1
|
Wang C, Guo L, Zhu J, Zhu L, Li C, Zhu H, Song A, Lu L, Teng GJ, Navab N, Jiang Z. Review of robotic systems for thoracoabdominal puncture interventional surgery. APL Bioeng 2024; 8:021501. [PMID: 38572313 PMCID: PMC10987197 DOI: 10.1063/5.0180494] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2023] [Accepted: 03/11/2024] [Indexed: 04/05/2024] Open
Abstract
Cancer, with high morbidity and high mortality, is one of the major burdens threatening human health globally. Intervention procedures via percutaneous puncture have been widely used by physicians due to its minimally invasive surgical approach. However, traditional manual puncture intervention depends on personal experience and faces challenges in terms of precisely puncture, learning-curve, safety and efficacy. The development of puncture interventional surgery robotic (PISR) systems could alleviate the aforementioned problems to a certain extent. This paper attempts to review the current status and prospective of PISR systems for thoracic and abdominal application. In this review, the key technologies related to the robotics, including spatial registration, positioning navigation, puncture guidance feedback, respiratory motion compensation, and motion control, are discussed in detail.
Collapse
Affiliation(s)
- Cheng Wang
- Hanglok-Tech Co. Ltd., Hengqin 519000, People's Republic of China
| | - Li Guo
- Hanglok-Tech Co. Ltd., Hengqin 519000, People's Republic of China
| | | | - Lifeng Zhu
- State Key Laboratory of Digital Medical Engineering, Jiangsu Key Lab of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, People's Republic of China
| | - Chichi Li
- School of Computer Science and Engineering, Macau University of Science and Technology, Macau, 999078, People's Republic of China
| | - Haidong Zhu
- Center of Interventional Radiology and Vascular Surgery, Department of Radiology, Zhongda Hospital, Medical School, Southeast University, Nanjing 210009, People's Republic of China
| | - Aiguo Song
- State Key Laboratory of Digital Medical Engineering, Jiangsu Key Lab of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, People's Republic of China
| | | | - Gao-Jun Teng
- Center of Interventional Radiology and Vascular Surgery, Department of Radiology, Zhongda Hospital, Medical School, Southeast University, Nanjing 210009, People's Republic of China
| | | | - Zhongliang Jiang
- Computer Aided Medical Procedures, Technical University of Munich, Munich 80333, Germany
| |
Collapse
|
2
|
Su K, Liu J, Ren X, Huo Y, Du G, Zhao W, Wang X, Liang B, Li D, Liu PX. A fully autonomous robotic ultrasound system for thyroid scanning. Nat Commun 2024; 15:4004. [PMID: 38734697 DOI: 10.1038/s41467-024-48421-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/08/2023] [Accepted: 04/23/2024] [Indexed: 05/13/2024] Open
Abstract
The current thyroid ultrasound relies heavily on the experience and skills of the sonographer and the expertise of the radiologist, and the process is physically and cognitively exhausting. In this paper, we report a fully autonomous robotic ultrasound system, which is able to scan thyroid regions without human assistance and identify malignant nod- ules. In this system, human skeleton point recognition, reinforcement learning, and force feedback are used to deal with the difficulties in locating thyroid targets. The orientation of the ultrasound probe is adjusted dynamically via Bayesian optimization. Experimental results on human participants demonstrated that this system can perform high-quality ultrasound scans, close to manual scans obtained by clinicians. Additionally, it has the potential to detect thyroid nodules and provide data on nodule characteristics for American College of Radiology Thyroid Imaging Reporting and Data System (ACR TI-RADS) calculation.
Collapse
Affiliation(s)
- Kang Su
- School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510006, China
| | - Jingwei Liu
- School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510006, China
| | - Xiaoqi Ren
- School of Future Technology, South China University of Technology, Guangzhou, 511442, China
- Peng Cheng Laboratory, Shenzhen, 518000, China
| | - Yingxiang Huo
- School of Future Technology, South China University of Technology, Guangzhou, 511442, China
- Peng Cheng Laboratory, Shenzhen, 518000, China
| | - Guanglong Du
- School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510006, China.
| | - Wei Zhao
- Division of Vascular and Interventional Radiology, Nanfang Hospital Southern Medical University, Guangzhou, 510515, China
| | - Xueqian Wang
- Tsinghua Shenzhen International Graduate School, Tsinghua University, Shenzhen, 518055, China.
| | - Bin Liang
- Department of Automation, Tsinghua University, 100854, Beijing, China.
| | - Di Li
- School of Mechanical and Automotive Engineering, South China University of Technology, Guangzhou, 510641, China
| | - Peter Xiaoping Liu
- Department of Systems and Computer Engineering, Carleton University, Ottawa, ON, K1S 5B6, Canada.
| |
Collapse
|
3
|
Tang X, Wang H, Luo J, Jiang J, Nian F, Qi L, Sang L, Gan Z. Autonomous ultrasound scanning robotic system based on human posture recognition and image servo control: an application for cardiac imaging. Front Robot AI 2024; 11:1383732. [PMID: 38774468 PMCID: PMC11106497 DOI: 10.3389/frobt.2024.1383732] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2024] [Accepted: 04/12/2024] [Indexed: 05/24/2024] Open
Abstract
In traditional cardiac ultrasound diagnostics, the process of planning scanning paths and adjusting the ultrasound window relies solely on the experience and intuition of the physician, a method that not only affects the efficiency and quality of cardiac imaging but also increases the workload for physicians. To overcome these challenges, this study introduces a robotic system designed for autonomous cardiac ultrasound scanning, with the goal of advancing both the degree of automation and the quality of imaging in cardiac ultrasound examinations. The system achieves autonomous functionality through two key stages: initially, in the autonomous path planning stage, it utilizes a camera posture adjustment method based on the human body's central region and its planar normal vectors to achieve automatic adjustment of the camera's positioning angle; precise segmentation of the human body point cloud is accomplished through efficient point cloud processing techniques, and precise localization of the region of interest (ROI) based on keypoints of the human body. Furthermore, by applying isometric path slicing and B-spline curve fitting techniques, it independently plans the scanning path and the initial position of the probe. Subsequently, in the autonomous scanning stage, an innovative servo control strategy based on cardiac image edge correction is introduced to optimize the quality of the cardiac ultrasound window, integrating position compensation through admittance control to enhance the stability of autonomous cardiac ultrasound imaging, thereby obtaining a detailed view of the heart's structure and function. A series of experimental validations on human and cardiac models have assessed the system's effectiveness and precision in the correction of camera pose, planning of scanning paths, and control of cardiac ultrasound imaging quality, demonstrating its significant potential for clinical ultrasound scanning applications.
Collapse
Affiliation(s)
- Xiuhong Tang
- Academy for Engineering and Technology, Fudan University, Shanghai, China
| | - Hongbo Wang
- Academy for Engineering and Technology, Fudan University, Shanghai, China
| | - Jingjing Luo
- Academy for Engineering and Technology, Fudan University, Shanghai, China
| | - Jinlei Jiang
- Intelligent Robot Engineering Research Center of Ministry of Education, Shanghai, China
| | - Fan Nian
- Intelligent Robot Engineering Research Center of Ministry of Education, Shanghai, China
| | - Lizhe Qi
- Academy for Engineering and Technology, Fudan University, Shanghai, China
| | - Lingfeng Sang
- Institute of Intelligent Medical Care Technology, Ningbo, China
| | - Zhongxue Gan
- Academy for Engineering and Technology, Fudan University, Shanghai, China
| |
Collapse
|
4
|
Vinter-Hviid F, Sloth C, Savarimuthu TR, Iturrate I. Safe contact-based robot active search using Bayesian optimization and control barrier functions. Front Robot AI 2024; 11:1344367. [PMID: 38741717 PMCID: PMC11089096 DOI: 10.3389/frobt.2024.1344367] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2023] [Accepted: 03/07/2024] [Indexed: 05/16/2024] Open
Abstract
In robotics, active exploration and learning in uncertain environments must take into account safety, as the robot may otherwise damage itself or its surroundings. This paper presents a method for safe active search using Bayesian optimization and control barrier functions. As robot paths undertaken during sampling are continuous, we consider an informative continuous expected improvement acquisition function. To safely bound the contact forces between the robot and its surroundings, we leverage exponential control barrier functions, utilizing the derivative of the force in the contact model to increase robustness to uncertainty in the contact boundary. Our approach is demonstrated on a fully autonomous robot for ultrasound scanning of rheumatoid arthritis (RA). Here, active search is a critical component of ensuring high image quality. Furthermore, bounded contact forces between the ultrasound probe and the patient ensure patient safety and better scan quality. To the best of our knowledge, our results are both the first demonstration of safe active search on a fully autonomous robot for ultrasound scanning of rheumatoid arthritis and the first experimental evaluation of bounding contact forces in the context of medical robotics using control barrier functions. The results show that when search time is limited to less than 60 s, informative continuous expected improvement leads to a 92% success, a 13% improvement compared to expected improvement. Meanwhile, exponential control barrier functions can limit the force applied by the robot to under 5 N, even in cases where the contact boundary is specified incorrectly by -1 or +4 mm.
Collapse
Affiliation(s)
| | - Christoffer Sloth
- SDU Robotics, Maersk McKinney Moller Institute, University of Southern Denmark, Odense, Denmark
| | - Thiusius Rajeeth Savarimuthu
- ROPCA ApS, Odense, Denmark
- SDU Robotics, Maersk McKinney Moller Institute, University of Southern Denmark, Odense, Denmark
| | - Iñigo Iturrate
- SDU Robotics, Maersk McKinney Moller Institute, University of Southern Denmark, Odense, Denmark
| |
Collapse
|
5
|
Jiang Z, Zhou Y, Cao D, Navab N. DefCor-Net: Physics-aware ultrasound deformation correction. Med Image Anal 2023; 90:102923. [PMID: 37688982 DOI: 10.1016/j.media.2023.102923] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/03/2022] [Revised: 05/22/2023] [Accepted: 08/01/2023] [Indexed: 09/11/2023]
Abstract
The recovery of morphologically accurate anatomical images from deformed ones is challenging in ultrasound (US) image acquisition, but crucial to accurate and consistent diagnosis, particularly in the emerging field of computer-assisted diagnosis. This article presents a novel physics-aware deformation correction approach based on a coarse-to-fine, multi-scale deep neural network (DefCor-Net). To achieve pixel-wise performance, DefCor-Net incorporates biomedical knowledge by estimating pixel-wise stiffness online using a U-shaped feature extractor. The deformation field is then computed using polynomial regression by integrating the measured force applied by the US probe. Based on real-time estimation of pixel-by-pixel tissue properties, the learning-based approach enables the potential for anatomy-aware deformation correction. To demonstrate the effectiveness of the proposed DefCor-Net, images recorded at multiple locations on forearms and upper arms of six volunteers are used to train and validate DefCor-Net. The results demonstrate that DefCor-Net can significantly improve the accuracy of deformation correction to recover the original geometry (Dice Coefficient: from 14.3±20.9 to 82.6±12.1 when the force is 6N). Code:https://github.com/KarolineZhy/DefCorNet.
Collapse
Affiliation(s)
- Zhongliang Jiang
- Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany.
| | - Yue Zhou
- Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany
| | | | - Nassir Navab
- Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany; Computer Aided Medical Procedures, Johns Hopkins University, Baltimore, MD, USA
| |
Collapse
|
6
|
Bao X, Wang S, Zheng L. SAPM: Self-Adaptive Parallel Manipulator with Pose and Force Adjustment for Robotic Ultrasonography. IEEE TRANSACTIONS ON INDUSTRIAL ELECTRONICS (1982) 2023; 70:10333-10343. [PMID: 37323755 PMCID: PMC7614654 DOI: 10.1109/tie.2022.3220864] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/17/2023]
Abstract
Robotic ultrasonography potentially acts as an essential aid to medical diagnosis. To overcome the limitations in robotic ultrasonography, in this paper, we proposed a novel self-adaptive parallel manipulator (SAPM) that can automatically adjust the ultrasound (US) probe pose to adapt to various contours of scanned areas, provide approximate constant operating forces/torques, achieve mechanical measurement, and cushion undesired produced forces. A novel parallel adjustment mechanism is proposed to attain automatic pose adjustment with 3 degrees of freedom (DOFs). This mechanism enables the US probe to adapt to different scanned areas and to perform the scanning with approximate constant forces and torques. Besides, we present a mechanical measurement and safety protection method that can be integrated into the SAPM and used as operation status monitoring and early warning during scanning procedures by capturing operating forces and torques. Experiments were carried out to calibrate the measurement and buffer units and evaluate the performance of the SAPM. Experimental results show the ability of the SAPM to provide 3-DoFs motion and operating force/torque measurement and automatically adjust the US probe pose to capture US images of equally good quality compared to a manual sonographer scan. Moreover, it has characteristics similar to soft robots that could significantly improve operation safety, and could be extended to some other engineering or medical applications.
Collapse
Affiliation(s)
- Xianqiang Bao
- School of Biomedical Engineering & Imaging Sciences, King’s College London, SE1 7EH, United Kingdom
| | - Shuangyi Wang
- State Key Laboratory of Management and Control for Complex Systems, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China
| | - Lingling Zheng
- Faculty of Engineering and Design, Kagawa University, Takamatsu 761-0396, Japan
| |
Collapse
|
7
|
Jiang Z, Salcudean SE, Navab N. Robotic ultrasound imaging: State-of-the-art and future perspectives. Med Image Anal 2023; 89:102878. [PMID: 37541100 DOI: 10.1016/j.media.2023.102878] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/28/2022] [Revised: 04/27/2023] [Accepted: 06/22/2023] [Indexed: 08/06/2023]
Abstract
Ultrasound (US) is one of the most widely used modalities for clinical intervention and diagnosis due to the merits of providing non-invasive, radiation-free, and real-time images. However, free-hand US examinations are highly operator-dependent. Robotic US System (RUSS) aims at overcoming this shortcoming by offering reproducibility, while also aiming at improving dexterity, and intelligent anatomy and disease-aware imaging. In addition to enhancing diagnostic outcomes, RUSS also holds the potential to provide medical interventions for populations suffering from the shortage of experienced sonographers. In this paper, we categorize RUSS as teleoperated or autonomous. Regarding teleoperated RUSS, we summarize their technical developments, and clinical evaluations, respectively. This survey then focuses on the review of recent work on autonomous robotic US imaging. We demonstrate that machine learning and artificial intelligence present the key techniques, which enable intelligent patient and process-specific, motion and deformation-aware robotic image acquisition. We also show that the research on artificial intelligence for autonomous RUSS has directed the research community toward understanding and modeling expert sonographers' semantic reasoning and action. Here, we call this process, the recovery of the "language of sonography". This side result of research on autonomous robotic US acquisitions could be considered as valuable and essential as the progress made in the robotic US examination itself. This article will provide both engineers and clinicians with a comprehensive understanding of RUSS by surveying underlying techniques. Additionally, we present the challenges that the scientific community needs to face in the coming years in order to achieve its ultimate goal of developing intelligent robotic sonographer colleagues. These colleagues are expected to be capable of collaborating with human sonographers in dynamic environments to enhance both diagnostic and intraoperative imaging.
Collapse
Affiliation(s)
- Zhongliang Jiang
- Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany.
| | - Septimiu E Salcudean
- Department of Electrical and Computer Engineering, University of British Columbia, Vancouver, BC V6T 1Z4, Canada
| | - Nassir Navab
- Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany; Computer Aided Medical Procedures, Johns Hopkins University, Baltimore, MD, USA
| |
Collapse
|
8
|
Tan J, Li B, Leng Y, Li Y, Peng J, Wu J, Luo B, Chen X, Rong Y, Fu C. Fully Automatic Dual-Probe Lung Ultrasound Scanning Robot for Screening Triage. IEEE TRANSACTIONS ON ULTRASONICS, FERROELECTRICS, AND FREQUENCY CONTROL 2023; 70:975-988. [PMID: 36191095 DOI: 10.1109/tuffc.2022.3211532] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
Two-dimensional lung ultrasound (LUS) has widely emerged as a rapid and noninvasive imaging tool for the detection and diagnosis of coronavirus disease 2019 (COVID-19). However, image differences will be magnified due to changes in ultrasound (US) imaging experience, such as US probe attitude control and force control, which will directly affect the diagnosis results. In addition, the risk of virus transmission between sonographer and patients is increased due to frequent physical contact. In this study, a fully automatic dual-probe US scanning robot for the acquisition of LUS images is proposed and developed. Furthermore, the trajectory was optimized based on the velocity look-ahead strategy, the stability of contact force of the system and the scanning efficiency were improved by 24.13% and 29.46%, respectively. Also, the control ability of the contact force of robotic automatic scanning was 34.14 times higher than that of traditional manual scanning, which significantly improves the smoothness of scanning. Importantly, there was no significant difference in image quality obtained by robotic automatic scanning and manual scanning. Furthermore, the scanning time for a single person is less than 4 min, which greatly improves the efficiency of screening triage of group COVID-19 diagnosis and suspected patients and reduces the risk of virus exposure and spread.
Collapse
|
9
|
Cai Q, Hu J, Chen M, Prieto J, Rosenbaum AJ, Stringer JSA, Jiang X. Inertial Measurement Unit-Assisted Ultrasonic Tracking System for Ultrasound Probe Localization. IEEE TRANSACTIONS ON ULTRASONICS, FERROELECTRICS, AND FREQUENCY CONTROL 2023; 70:920-929. [PMID: 36150002 DOI: 10.1109/tuffc.2022.3207185] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
Ultrasonic tracking is a promising technique in indoor object localization. However, limited success has been reported in dynamic orientational and positional ultrasonic tracking for ultrasound (US) probes due to its instability and relatively low accuracy. This article aims at developing an inertial measurement unit (IMU)-assisted ultrasonic tracking system that enables a high accuracy positional and orientational localization. The system was designed with the acoustic pressure field simulation of the transmitter, receiver configuration, position-variant error simulation, and sensor fusion. The prototype was tested in a tracking volume required in typical obstetric sonography within the typical operation speed ranges (slow mode and fast mode) of US probe movement. The performance in two different speed ranges was evaluated against a commercial optical tracking device. The results show that the proposed IMU-assisted US tracking system achieved centimeter-level positional tracking accuracy with the mean absolute error (MAE) of 12 mm and the MAE of orientational tracking was less than 1°. The results indicate the possibility of implementing the IMU-assisted ultrasonic tracking system in US probe localization.
Collapse
|
10
|
Pornpipatsakul K, Chenviteesook A, Chaichaowarat R. Ultrasound Probe Movement Analysis Using Depth Camera with Compact Handle Design for Probe Contact Force Measurement. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2023; 2023:1-4. [PMID: 38082920 DOI: 10.1109/embc40787.2023.10340101] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/18/2023]
Abstract
The real-time information of the ultrasound probe movement and contact force during scanning is helpful for improving skill of medical professionals. This paper presents an affordable technique using an RGB-depth camera with the MediaPipe Hands framework for capturing the hand gesture to estimate the position and orientation of the ultrasound probe. The method does not require any additional marker which can interfere the motion and the feeling of handheld operation. The tracking accuracies of position and orientation were evaluated experimentally at different camera angles. The 3D printed handle was inserted into the grids of the XYZ plate and the tilt plate. Although the camera angle and the spatial position affect the accuracies, the maximum errors are always less than 7.5 mm and 10 degrees. The custom designed handle consisting of the inner and outer shells allows installation of the small three-axis force sensor for probe contact force measurement while scanning. The design is easy to assemble with an ultrasound probe without requiring any modification. The results of this work can be applied as a guideline for monitoring ultrasound training.
Collapse
|
11
|
Zakaria UMU, Mustaza SM, Zaman MHM, Rahni AAA. Development of Real-Time Contact Force Control of a Collaborative Robot for Automated Ultrasound Scanning. 2022 IEEE-EMBS CONFERENCE ON BIOMEDICAL ENGINEERING AND SCIENCES (IECBES) 2022. [DOI: 10.1109/iecbes54088.2022.10079599] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/02/2023]
Affiliation(s)
- Ungku M.Z. Ungku Zakaria
- Universiti Kebangsaan Malaysia,Faculty of Engineering & Built Environment,Bangi,Selangor,Malaysia,43600
| | - Seri M. Mustaza
- Universiti Kebangsaan Malaysia,Faculty of Engineering & Built Environment,Bangi,Selangor,Malaysia,43600
| | - Mohd H. Mohd Zaman
- Universiti Kebangsaan Malaysia,Faculty of Engineering & Built Environment,Bangi,Selangor,Malaysia,43600
| | - Ashrani A. Abd. Rahni
- Universiti Kebangsaan Malaysia,Faculty of Engineering & Built Environment,Bangi,Selangor,Malaysia,43600
| |
Collapse
|
12
|
Ma X, Kuo WY, Yang K, Rahaman A, Zhang HK. A-SEE: Active-Sensing End-effector Enabled Probe Self-Normal-Positioning for Robotic Ultrasound Imaging Applications. IEEE Robot Autom Lett 2022; 7:12475-12482. [PMID: 37325198 PMCID: PMC10266708 DOI: 10.1109/lra.2022.3218183] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/06/2023]
Abstract
Conventional manual ultrasound (US) imaging is a physically demanding procedure for sonographers. A robotic US system (RUSS) has the potential to overcome this limitation by automating and standardizing the imaging procedure. It also extends ultrasound accessibility in resource-limited environments with the shortage of human operators by enabling remote diagnosis. During imaging, keeping the US probe normal to the skin surface largely benefits the US image quality. However, an autonomous, real-time, low-cost method to align the probe towards the direction orthogonal to the skin surface without pre-operative information is absent in RUSS. We propose a novel end-effector design to achieve self-normal-positioning of the US probe. The end-effector embeds four laser distance sensors to estimate the desired rotation towards the normal direction. We then integrate the proposed end-effector with a RUSS system which allows the probe to be automatically and dynamically kept to normal direction during US imaging. We evaluated the normal positioning accuracy and the US image quality using a flat surface phantom, an upper torso mannequin, and a lung ultrasound phantom. Results show that the normal positioning accuracy is 4.17 ± 2.24 degrees on the flat surface and 14.67 ± 8.46 degrees on the mannequin. The quality of the RUSS collected US images from the lung ultrasound phantom was equivalent to that of the manually collected ones.
Collapse
Affiliation(s)
- Xihan Ma
- Department of Robotics Engineering, Worcester Polytechnic Institute, Worcester, MA, 01609, USA
| | - Wen-Yi Kuo
- Department of Robotics Engineering, Worcester Polytechnic Institute, Worcester, MA, 01609, USA
| | - Kehan Yang
- Department of Robotics Engineering, Worcester Polytechnic Institute, Worcester, MA, 01609, USA
| | - Ashiqur Rahaman
- Department of Robotics Engineering, Worcester Polytechnic Institute, Worcester, MA, 01609, USA
- Department of Biomedical Engineering, Worcester Polytechnic Institute, Worcester, MA, 01609, USA
| | - Haichong K Zhang
- Department of Robotics Engineering, Worcester Polytechnic Institute, Worcester, MA, 01609, USA
- Department of Biomedical Engineering, Worcester Polytechnic Institute, Worcester, MA, 01609, USA
| |
Collapse
|
13
|
Jiang Z, Gao Y, Xie L, Navab N. Towards Autonomous Atlas-Based Ultrasound Acquisitions in Presence of Articulated Motion. IEEE Robot Autom Lett 2022. [DOI: 10.1109/lra.2022.3180440] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Affiliation(s)
- Zhongliang Jiang
- Chair for Computer Aided Medical Procedures and Augmented Reality (CAMP), Technical University of Munich, Garching, Germany
| | - Yuan Gao
- Chair for Computer Aided Medical Procedures and Augmented Reality (CAMP), Technical University of Munich, Garching, Germany
| | - Le Xie
- Institute of Forming Technology and Equipment and Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, China
| | - Nassir Navab
- Chair for Computer Aided Medical Procedures and Augmented Reality (CAMP), Technical University of Munich, Garching, Germany
| |
Collapse
|
14
|
Bi Y, Jiang Z, Gao Y, Wendler T, Karlas A, Navab N. VesNet-RL: Simulation-Based Reinforcement Learning for Real-World US Probe Navigation. IEEE Robot Autom Lett 2022. [DOI: 10.1109/lra.2022.3176112] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Affiliation(s)
- Yuan Bi
- Chair for Computer-Aided Medical Procedures and Augmented Reality, Technical University of Munich, Garching bei München, Germany
| | - Zhongliang Jiang
- Chair for Computer-Aided Medical Procedures and Augmented Reality, Technical University of Munich, Garching bei München, Germany
| | - Yuan Gao
- Chair for Computer-Aided Medical Procedures and Augmented Reality, Technical University of Munich, Garching bei München, Germany
| | - Thomas Wendler
- Chair for Computer-Aided Medical Procedures and Augmented Reality, Technical University of Munich, Garching bei München, Germany
| | - Angelos Karlas
- Institute of Biological and Medical Imaging, Helmholtz Zentrum München, München, Germany
| | - Nassir Navab
- Chair for Computer-Aided Medical Procedures and Augmented Reality, Technical University of Munich, Garching bei München, Germany
| |
Collapse
|
15
|
Jiang Z, Zhou Y, Bi Y, Zhou M, Wendler T, Navab N. Deformation-Aware Robotic 3D Ultrasound. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2021.3099080] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
16
|
Ma X, Zhang Z, Zhang HK. Autonomous Scanning Target Localization for Robotic Lung Ultrasound Imaging. PROCEEDINGS OF THE ... IEEE/RSJ INTERNATIONAL CONFERENCE ON INTELLIGENT ROBOTS AND SYSTEMS. IEEE/RSJ INTERNATIONAL CONFERENCE ON INTELLIGENT ROBOTS AND SYSTEMS 2021; 2021:9467-9474. [PMID: 35965637 PMCID: PMC9373068 DOI: 10.1109/iros51168.2021.9635902] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Under the ceaseless global COVID-19 pandemic, lung ultrasound (LUS) is the emerging way for effective diagnosis and severeness evaluation of respiratory diseases. However, close physical contact is unavoidable in conventional clinical ultrasound, increasing the infection risk for health-care workers. Hence, a scanning approach involving minimal physical contact between an operator and a patient is vital to maximize the safety of clinical ultrasound procedures. A robotic ultrasound platform can satisfy this need by remotely manipulating the ultrasound probe with a robotic arm. This paper proposes a robotic LUS system that incorporates the automatic identification and execution of the ultrasound probe placement pose without manual input. An RGB-D camera is utilized to recognize the scanning targets on the patient through a learning-based human pose estimation algorithm and solve for the landing pose to attach the probe vertically to the tissue surface; A position/force controller is designed to handle intraoperative probe pose adjustment for maintaining the contact force. We evaluated the scanning area localization accuracy, motion execution accuracy, and ultrasound image acquisition capability using an upper torso mannequin and a realistic lung ultrasound phantom with healthy and COVID-19-infected lung anatomy. Results demonstrated the overall scanning target localization accuracy of 19.67 ± 4.92 mm and the probe landing pose estimation accuracy of 6.92 ± 2.75 mm in translation, 10.35 ± 2.97 deg in rotation. The contact force-controlled robotic scanning allowed the successful ultrasound image collection, capturing pathological landmarks.
Collapse
Affiliation(s)
- Xihan Ma
- Department of Robotics Engineering, Worcester Polytechnic Institute, Worcester, MA, 01609 USA
| | - Ziming Zhang
- Department of Electrical and Computer Engineering, Worcester Polytechnic Institute, Worcester, MA, 01609 USA
| | - Haichong K Zhang
- Department of Robotics Engineering, Worcester Polytechnic Institute, Worcester, MA, 01609 USA
- Department of Biomedical Engineering, Worcester Polytechnic Institute, Worcester, MA, 01609 USA
| |
Collapse
|
17
|
Chen S, Li Z, Lin Y, Wang F, Cao Q. Automatic ultrasound scanning robotic system with optical waveguide-based force measurement. Int J Comput Assist Radiol Surg 2021; 16:1015-1025. [PMID: 33939078 DOI: 10.1007/s11548-021-02385-2] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/25/2020] [Accepted: 04/19/2021] [Indexed: 12/18/2022]
Abstract
PURPOSE The three-dimensional (3D) ultrasound (US) imaging realized by continuous scanning of a region is of great value for medical diagnosis and robot-assisted needle insertion. During scanning, the contact force and posture between the probe and skin of the patient are crucial factors that determine the quality of US imaging. We propose a robotic system for automatic scanning of curved surfaces with a stable contact force and vertical contact posture (the probe is parallel to the normal of the surface at the contact point). METHODS A 6-DOF robotic arm is used to hold and drive a two-dimensional (2D) US probe to complete automatic scanning. Further, a path-planning strategy is proposed to generate the scan path covering the selected area automatically. We also developed a novel force-measuring device based on optical waveguides to measure the distributed contact force and contact posture. Based on the measured force and posture, the robotic arm automatically adjusts the position and orientation of the probe and maintains a stable contact force and vertical contact posture at each scan point. RESULTS The novel force-measuring device is easy to fabricate, integrates with the probe and has the capacity of measuring the force distributed on the contact surface and estimating the contact posture. The experimental results of automatic scanning of a US phantom and parts of the human body demonstrate that the proposed system performs well in automatically scanning curved surfaces, maintaining a stable contact force and vertical contact posture and producing a good quality 3D US volume. CONCLUSION An automatic US scanning robotic system with an optical waveguide-based force-measuring device was developed and tested successfully. Experimental results demonstrated the feasibility of the proposed system to scan the human body.
Collapse
Affiliation(s)
- Shihang Chen
- School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Zhaojun Li
- Shanghai General Hospital, School of Medicine, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Yanping Lin
- School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of China. .,Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China.
| | - Fang Wang
- Shanghai General Hospital, School of Medicine, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Qixin Cao
- School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of China.,Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| |
Collapse
|
18
|
Huang Y, Xiao W, Wang C, Liu H, Huang R, Sun Z. Towards Fully Autonomous Ultrasound Scanning Robot With Imitation Learning Based on Clinical Protocols. IEEE Robot Autom Lett 2021; 6:3671-3678. [DOI: 10.1109/lra.2021.3064283] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/29/2023]
|
19
|
von Haxthausen F, Böttger S, Wulff D, Hagenah J, García-Vázquez V, Ipsen S. Medical Robotics for Ultrasound Imaging: Current Systems and Future Trends. ACTA ACUST UNITED AC 2021; 2:55-71. [PMID: 34977593 PMCID: PMC7898497 DOI: 10.1007/s43154-020-00037-y] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 12/21/2020] [Indexed: 12/17/2022]
Abstract
Abstract
Purpose of Review
This review provides an overview of the most recent robotic ultrasound systems that have contemporary emerged over the past five years, highlighting their status and future directions. The systems are categorized based on their level of robot autonomy (LORA).
Recent Findings
Teleoperating systems show the highest level of technical maturity. Collaborative assisting and autonomous systems are still in the research phase, with a focus on ultrasound image processing and force adaptation strategies. However, missing key factors are clinical studies and appropriate safety strategies. Future research will likely focus on artificial intelligence and virtual/augmented reality to improve image understanding and ergonomics.
Summary
A review on robotic ultrasound systems is presented in which first technical specifications are outlined. Hereafter, the literature of the past five years is subdivided into teleoperation, collaborative assistance, or autonomous systems based on LORA. Finally, future trends for robotic ultrasound systems are reviewed with a focus on artificial intelligence and virtual/augmented reality.
Collapse
Affiliation(s)
- Felix von Haxthausen
- Institute for Robotics and Cognitive Systems, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
| | - Sven Böttger
- Institute for Robotics and Cognitive Systems, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
| | - Daniel Wulff
- Institute for Robotics and Cognitive Systems, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
| | - Jannis Hagenah
- Institute for Robotics and Cognitive Systems, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
| | - Verónica García-Vázquez
- Institute for Robotics and Cognitive Systems, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
| | - Svenja Ipsen
- Institute for Robotics and Cognitive Systems, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
| |
Collapse
|
20
|
Real-Time Multi-Modal Sensing and Feedback for Catheterization in Porcine Tissue. SENSORS 2021; 21:s21010273. [PMID: 33401617 PMCID: PMC7795440 DOI: 10.3390/s21010273] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/25/2020] [Revised: 12/22/2020] [Accepted: 12/30/2020] [Indexed: 12/23/2022]
Abstract
Objective: In this study, we introduce a multi-modal sensing and feedback framework aimed at assisting clinicians during endovascular surgeries and catheterization procedures. This framework utilizes state-of-the-art imaging and sensing sub-systems to produce a 3D visualization of an endovascular catheter and surrounding vasculature without the need for intra-operative X-rays. Methods: The catheterization experiments within this study are conducted inside a porcine limb undergoing motions. A hybrid position-force controller of a robotically-actuated ultrasound (US) transducer for uneven porcine tissue surfaces is introduced. The tissue, vasculature, and catheter are visualized by integrated real-time US images, 3D surface imaging, and Fiber Bragg Grating (FBG) sensors. Results: During externally-induced limb motions, the vasculature and catheter can be reliably reconstructed at mean accuracies of 1.9±0.3 mm and 0.82±0.21 mm, respectively. Conclusions: The conventional use of intra-operative X-ray imaging to visualize instruments and vasculature in the human body can be reduced by employing improved diagnostic technologies that do not operate via ionizing radiation or nephrotoxic contrast agents. Significance: The presented multi-modal framework enables the radiation-free and accurate reconstruction of significant tissues and instruments involved in catheterization procedures.
Collapse
|