1
|
Dahmani J, Petit Y, Laporte C. Quantitative validation of two model-based methods for the correction of probe pressure deformation in ultrasound. Int J Comput Assist Radiol Surg 2024; 19:309-320. [PMID: 37596378 DOI: 10.1007/s11548-023-03006-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2023] [Accepted: 07/17/2023] [Indexed: 08/20/2023]
Abstract
PURPOSE The acquisition of good quality ultrasound (US) images requires good acoustic coupling between the ultrasound probe and the patient's skin. In practice, this good coupling is achieved by the operator applying a force to the skin through the probe. This force induces a deformation of the tissues underlying the probe. The distorted images deteriorate the quality of the reconstructed 3D US image. METHODS In this work, we propose two methods to correct these deformations. These methods are based on the construction of a biomechanical model to predict the mechanical behavior of the imaged soft tissues. The originality of the methods is that they do not use external information (force or position value from sensors, or elasticity value from the literature). The model is parameterized thanks to the information contained in the image. This is allowed thanks to the optimization of two key parameters for the model which are the indentation d and the elasticity ratio α. RESULTS The validation is performed on real images acquired on a gelatin-based phantom using an ultrasound probe inducing an increasing vertical indentation using a step motor. The results showed a good correction of the two methods for indentations less than 4 mm. For larger indentations, one of the two methods (guided by the similarity score) provides a better quality of correction, presenting a Euclidean distance between the contours of the reference image and the corrected image of 0.71 mm. CONCLUSION The proposed methods ensured the correction of the deformed images induced by a linear probe pressure without using any information coming from sensors (force or position), or generic information about the mechanical parameters. The corrected images can be used to obtain a corrected 3D US image.
Collapse
Affiliation(s)
- Jawad Dahmani
- École de Technologie Supérieure, 1100 Notre-Dame Street West, Montreal, QC, Canada.
| | - Yvan Petit
- École de Technologie Supérieure, 1100 Notre-Dame Street West, Montreal, QC, Canada
| | - Catherine Laporte
- École de Technologie Supérieure, 1100 Notre-Dame Street West, Montreal, QC, Canada
| |
Collapse
|
2
|
Jiang Z, Zhou Y, Cao D, Navab N. DefCor-Net: Physics-aware ultrasound deformation correction. Med Image Anal 2023; 90:102923. [PMID: 37688982 DOI: 10.1016/j.media.2023.102923] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/03/2022] [Revised: 05/22/2023] [Accepted: 08/01/2023] [Indexed: 09/11/2023]
Abstract
The recovery of morphologically accurate anatomical images from deformed ones is challenging in ultrasound (US) image acquisition, but crucial to accurate and consistent diagnosis, particularly in the emerging field of computer-assisted diagnosis. This article presents a novel physics-aware deformation correction approach based on a coarse-to-fine, multi-scale deep neural network (DefCor-Net). To achieve pixel-wise performance, DefCor-Net incorporates biomedical knowledge by estimating pixel-wise stiffness online using a U-shaped feature extractor. The deformation field is then computed using polynomial regression by integrating the measured force applied by the US probe. Based on real-time estimation of pixel-by-pixel tissue properties, the learning-based approach enables the potential for anatomy-aware deformation correction. To demonstrate the effectiveness of the proposed DefCor-Net, images recorded at multiple locations on forearms and upper arms of six volunteers are used to train and validate DefCor-Net. The results demonstrate that DefCor-Net can significantly improve the accuracy of deformation correction to recover the original geometry (Dice Coefficient: from 14.3±20.9 to 82.6±12.1 when the force is 6N). Code:https://github.com/KarolineZhy/DefCorNet.
Collapse
Affiliation(s)
- Zhongliang Jiang
- Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany.
| | - Yue Zhou
- Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany
| | | | - Nassir Navab
- Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany; Computer Aided Medical Procedures, Johns Hopkins University, Baltimore, MD, USA
| |
Collapse
|
3
|
Ning G, Liang H, Zhang X, Liao H. Autonomous Robotic Ultrasound Vascular Imaging System With Decoupled Control Strategy for External-Vision-Free Environments. IEEE Trans Biomed Eng 2023; 70:3166-3177. [PMID: 37227912 DOI: 10.1109/tbme.2023.3279114] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/27/2023]
Abstract
OBJECTIVE Ultrasound (US) probes scan over the surface of the human body to acquire US images in clinical vascular US diagnosis. However, due to the deformation and specificity of different human surfaces, the relationship between the scan trajectory of the skin and the internal tissues is not fully correlated, which poses a challenge for autonomous robotic US imaging in a dynamic and external-vision-free environment. Here, we propose a decoupled control strategy for autonomous robotic vascular US imaging in an environment without external vision. METHODS The proposed system is divided into outer-loop posture control and inner-loop orientation control, which are separately determined by a deep learning (DL) agent and a reinforcement learning (RL) agent. First, we use a weakly supervised US vessel segmentation network to estimate the probe orientation. In the outer loop control, we use a force-guided reinforcement learning agent to maintain a specific angle between the US probe and the skin in the dynamic imaging processes. Finally, the orientation and the posture are integrated to complete the imaging process. RESULTS Evaluation experiments on several volunteers showed that our RUS could autonomously perform vascular imaging in arms with different stiffness, curvature, and size without additional system adjustments. Furthermore, our system achieved reproducible imaging and reconstruction of dynamic targets without relying on vision-based surface information. CONCLUSION AND SIGNIFICANCE Our system and control strategy provides a novel framework for the application of US robots in complex and external-vision-free environments.
Collapse
|
4
|
Jiang Z, Salcudean SE, Navab N. Robotic ultrasound imaging: State-of-the-art and future perspectives. Med Image Anal 2023; 89:102878. [PMID: 37541100 DOI: 10.1016/j.media.2023.102878] [Citation(s) in RCA: 11] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/28/2022] [Revised: 04/27/2023] [Accepted: 06/22/2023] [Indexed: 08/06/2023]
Abstract
Ultrasound (US) is one of the most widely used modalities for clinical intervention and diagnosis due to the merits of providing non-invasive, radiation-free, and real-time images. However, free-hand US examinations are highly operator-dependent. Robotic US System (RUSS) aims at overcoming this shortcoming by offering reproducibility, while also aiming at improving dexterity, and intelligent anatomy and disease-aware imaging. In addition to enhancing diagnostic outcomes, RUSS also holds the potential to provide medical interventions for populations suffering from the shortage of experienced sonographers. In this paper, we categorize RUSS as teleoperated or autonomous. Regarding teleoperated RUSS, we summarize their technical developments, and clinical evaluations, respectively. This survey then focuses on the review of recent work on autonomous robotic US imaging. We demonstrate that machine learning and artificial intelligence present the key techniques, which enable intelligent patient and process-specific, motion and deformation-aware robotic image acquisition. We also show that the research on artificial intelligence for autonomous RUSS has directed the research community toward understanding and modeling expert sonographers' semantic reasoning and action. Here, we call this process, the recovery of the "language of sonography". This side result of research on autonomous robotic US acquisitions could be considered as valuable and essential as the progress made in the robotic US examination itself. This article will provide both engineers and clinicians with a comprehensive understanding of RUSS by surveying underlying techniques. Additionally, we present the challenges that the scientific community needs to face in the coming years in order to achieve its ultimate goal of developing intelligent robotic sonographer colleagues. These colleagues are expected to be capable of collaborating with human sonographers in dynamic environments to enhance both diagnostic and intraoperative imaging.
Collapse
Affiliation(s)
- Zhongliang Jiang
- Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany.
| | - Septimiu E Salcudean
- Department of Electrical and Computer Engineering, University of British Columbia, Vancouver, BC V6T 1Z4, Canada
| | - Nassir Navab
- Computer Aided Medical Procedures, Technical University of Munich, Munich, Germany; Computer Aided Medical Procedures, Johns Hopkins University, Baltimore, MD, USA
| |
Collapse
|
5
|
Li Q, Zhang F, Xi Q, Jiao Z, Ni X. Nondeformed Ultrasound Image Production Method for Ultrasound-Guided Radiotherapy. Technol Cancer Res Treat 2023; 22:15330338231194546. [PMID: 37700675 PMCID: PMC10501062 DOI: 10.1177/15330338231194546] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2023] [Revised: 07/23/2023] [Accepted: 07/26/2023] [Indexed: 09/14/2023] Open
Abstract
Purpose: During ultrasound (US)-guided radiotherapy, the tissue is deformed by probe pressure, and the US image is limited by changes in tissue and organ position and geometry when the US image is aligned with computed tomography (CT) image, leading to poor alignment. Accordingly, a pixel displacement-based nondeformed US image production method is proposed. Methods: The correction of US image deformation is achieved by calculating the pixel displacement of an image. The positioning CT image (CTstd) is used as the gold standard. The deformed US image (USdef) is inputted into the Harris algorithm to extract corner points for selecting feature points, and the displacement of adjacent pixels of feature points in the US video stream is calculated using the Lucas-Kanade optical flow algorithm. The moving least squares algorithm is used to correct USdef globally and locally in accordance with image pixel displacement to generate a nondeformed US image (USrev). In addition, USdef and USrev were separately aligned with CTstd to evaluate the improvement of alignment accuracy through deformation correction. Results: In the phantom experiment, the overall and local average correction errors of the US image under the optimal probe pressure were 1.0944 and 0.7388 mm, respectively, and the registration accuracy of USdef and USrev with CTstd was 0.6764 and 0.9016, respectively. During the volunteer experiment, the correction error of all 12 patients' data ranged from -1.7525 to 1.5685 mm, with a mean absolute error of 0.8612 mm. The improvement range of US and CT registration accuracy, before and after image deformation correction in the 12 patients evaluated by a normalized correlation coefficient, was 0.1232 to 0.2476. Conclusion: The pixel displacement-based deformation correction method can solve the limitation imposed by image deformation on image alignment in US-guided radiotherapy. Compared with USdef, the alignment results of USrev with CT were better.
Collapse
Affiliation(s)
- Qixuan Li
- School of Microelectronics and Control Engineering, Changzhou University, Changzhou, China
- Department of Radiotherapy, The Affiliated Changzhou No.2 People's Hospital of Nanjing Medical University, Changzhou, China
- Jiangsu Province Engineering Research Center of Medical Physics, Changzhou, China
- Center for Medical Physics, Nanjing Medical University, Changzhou, China
| | - Fan Zhang
- School of Microelectronics and Control Engineering, Changzhou University, Changzhou, China
- Department of Radiotherapy, The Affiliated Changzhou No.2 People's Hospital of Nanjing Medical University, Changzhou, China
- Jiangsu Province Engineering Research Center of Medical Physics, Changzhou, China
- Center for Medical Physics, Nanjing Medical University, Changzhou, China
| | - Qianyi Xi
- School of Microelectronics and Control Engineering, Changzhou University, Changzhou, China
- Department of Radiotherapy, The Affiliated Changzhou No.2 People's Hospital of Nanjing Medical University, Changzhou, China
- Jiangsu Province Engineering Research Center of Medical Physics, Changzhou, China
- Center for Medical Physics, Nanjing Medical University, Changzhou, China
| | - Zhuqing Jiao
- School of Microelectronics and Control Engineering, Changzhou University, Changzhou, China
- School of Computer and Artificial Intelligence, Changzhou University, Changzhou, China
| | - Xinye Ni
- Department of Radiotherapy, The Affiliated Changzhou No.2 People's Hospital of Nanjing Medical University, Changzhou, China
- Jiangsu Province Engineering Research Center of Medical Physics, Changzhou, China
- Center for Medical Physics, Nanjing Medical University, Changzhou, China
| |
Collapse
|
6
|
Jiang Z, Zhou Y, Bi Y, Zhou M, Wendler T, Navab N. Deformation-Aware Robotic 3D Ultrasound. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2021.3099080] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
|
7
|
Good and bad boundaries in ultrasound compounding: preserving anatomic boundaries while suppressing artifacts. Int J Comput Assist Radiol Surg 2021; 16:1957-1968. [PMID: 34357525 PMCID: PMC8589734 DOI: 10.1007/s11548-021-02464-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/13/2021] [Accepted: 07/15/2021] [Indexed: 11/23/2022]
Abstract
Purpose Ultrasound compounding is to combine sonographic information captured from different angles and produce a single image. It is important for multi-view reconstruction, but as of yet there is no consensus on best practices for compounding. Current popular methods inevitably suppress or altogether leave out bright or dark regions that are useful and potentially introduce new artifacts. In this work, we establish a new algorithm to compound the overlapping pixels from different viewpoints in ultrasound. Methods Inspired by image fusion algorithms and ultrasound confidence, we uniquely leverage Laplacian and Gaussian pyramids to preserve the maximum boundary contrast without overemphasizing noise, speckles, and other artifacts in the compounded image, while taking the direction of the ultrasound probe into account. Besides, we designed an algorithm that detects the useful boundaries in ultrasound images to further improve the boundary contrast. Results We evaluate our algorithm by comparing it with previous algorithms both qualitatively and quantitatively, and we show that our approach not only preserves both light and dark details, but also somewhat suppresses noise and artifacts, rather than amplifying them. We also show that our algorithm can improve the performance of downstream tasks like segmentation. Conclusion Our proposed method that is based on confidence, contrast, and both Gaussian and Laplacian pyramids appears to be better at preserving contrast at anatomic boundaries while suppressing artifacts than any of the other approaches we tested. This algorithm may have future utility with downstream tasks such as 3D ultrasound volume reconstruction and segmentation.
Collapse
|
8
|
Chen S, Li Z, Lin Y, Wang F, Cao Q. Automatic ultrasound scanning robotic system with optical waveguide-based force measurement. Int J Comput Assist Radiol Surg 2021; 16:1015-1025. [PMID: 33939078 DOI: 10.1007/s11548-021-02385-2] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/25/2020] [Accepted: 04/19/2021] [Indexed: 12/18/2022]
Abstract
PURPOSE The three-dimensional (3D) ultrasound (US) imaging realized by continuous scanning of a region is of great value for medical diagnosis and robot-assisted needle insertion. During scanning, the contact force and posture between the probe and skin of the patient are crucial factors that determine the quality of US imaging. We propose a robotic system for automatic scanning of curved surfaces with a stable contact force and vertical contact posture (the probe is parallel to the normal of the surface at the contact point). METHODS A 6-DOF robotic arm is used to hold and drive a two-dimensional (2D) US probe to complete automatic scanning. Further, a path-planning strategy is proposed to generate the scan path covering the selected area automatically. We also developed a novel force-measuring device based on optical waveguides to measure the distributed contact force and contact posture. Based on the measured force and posture, the robotic arm automatically adjusts the position and orientation of the probe and maintains a stable contact force and vertical contact posture at each scan point. RESULTS The novel force-measuring device is easy to fabricate, integrates with the probe and has the capacity of measuring the force distributed on the contact surface and estimating the contact posture. The experimental results of automatic scanning of a US phantom and parts of the human body demonstrate that the proposed system performs well in automatically scanning curved surfaces, maintaining a stable contact force and vertical contact posture and producing a good quality 3D US volume. CONCLUSION An automatic US scanning robotic system with an optical waveguide-based force-measuring device was developed and tested successfully. Experimental results demonstrated the feasibility of the proposed system to scan the human body.
Collapse
Affiliation(s)
- Shihang Chen
- School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Zhaojun Li
- Shanghai General Hospital, School of Medicine, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Yanping Lin
- School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of China. .,Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China.
| | - Fang Wang
- Shanghai General Hospital, School of Medicine, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Qixin Cao
- School of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, People's Republic of China.,Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| |
Collapse
|
9
|
Ipsen S, Wulff D, Kuhlemann I, Schweikard A, Ernst F. Towards automated ultrasound imaging-robotic image acquisition in liver and prostate for long-term motion monitoring. Phys Med Biol 2021; 66. [PMID: 33770768 DOI: 10.1088/1361-6560/abf277] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/06/2021] [Accepted: 03/26/2021] [Indexed: 11/12/2022]
Abstract
Real-time volumetric (4D) ultrasound has shown high potential for diagnostic and therapy guidance tasks. One of the main drawbacks of ultrasound imaging to date is the reliance on manual probe positioning and the resulting user dependence. Robotic assistance could help overcome this issue and facilitate the acquisition of long-term image data to observe dynamic processesin vivoover time. The aim of this study is to assess the feasibility of robotic probe manipulation and organ motion quantification during extended imaging sessions. The system consists of a collaborative robot and a 4D ultrasound system providing real-time data access. Five healthy volunteers received liver and prostate scans during free breathing over 30 min. Initial probe placement was performed with real-time remote control with a predefined contact force of 10 N. During scan acquisition, the probe position was continuously adjusted to the body surface motion using impedance control. Ultrasound volumes, the pose of the end-effector and the estimated contact forces were recorded. For motion analysis, one anatomical landmark was manually annotated in a subset of ultrasound frames for each experiment. Probe contact was uninterrupted over the entire scan duration in all ten sessions. Organ drift and imaging artefacts were successfully compensated using remote control. The median contact force along the probe's longitudinal axis was 10.0 N with maximum values of 13.2 and 21.3 N for liver and prostate, respectively. Forces exceeding 11 N only occurred in 0.3% of the time. Probe and landmark motion were more pronounced in the liver, with median interquartile ranges of 1.5 and 9.6 mm, compared to 0.6 and 2.7 mm in the prostate. The results show that robotic ultrasound imaging with dynamic force control can be used for stable, long-term imaging of anatomical regions affected by motion. The system facilitates the acquisition of 4D image datain vivoover extended scanning periods for the first time and holds the potential to be used for motion monitoring for therapy guidance as well as diagnostic tasks.
Collapse
Affiliation(s)
- Svenja Ipsen
- Institute for Robotics and Cognitive Systems, University of Luebeck, Luebeck, Germany.,Fraunhofer Research Institution for Individualized and Cell-Based Medical Engineering IMTE, Luebeck, Germany
| | - Daniel Wulff
- Institute for Robotics and Cognitive Systems, University of Luebeck, Luebeck, Germany
| | - Ivo Kuhlemann
- Institute for Robotics and Cognitive Systems, University of Luebeck, Luebeck, Germany
| | - Achim Schweikard
- Institute for Robotics and Cognitive Systems, University of Luebeck, Luebeck, Germany
| | - Floris Ernst
- Institute for Robotics and Cognitive Systems, University of Luebeck, Luebeck, Germany
| |
Collapse
|
10
|
von Haxthausen F, Böttger S, Wulff D, Hagenah J, García-Vázquez V, Ipsen S. Medical Robotics for Ultrasound Imaging: Current Systems and Future Trends. ACTA ACUST UNITED AC 2021; 2:55-71. [PMID: 34977593 PMCID: PMC7898497 DOI: 10.1007/s43154-020-00037-y] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 12/21/2020] [Indexed: 12/17/2022]
Abstract
Abstract
Purpose of Review
This review provides an overview of the most recent robotic ultrasound systems that have contemporary emerged over the past five years, highlighting their status and future directions. The systems are categorized based on their level of robot autonomy (LORA).
Recent Findings
Teleoperating systems show the highest level of technical maturity. Collaborative assisting and autonomous systems are still in the research phase, with a focus on ultrasound image processing and force adaptation strategies. However, missing key factors are clinical studies and appropriate safety strategies. Future research will likely focus on artificial intelligence and virtual/augmented reality to improve image understanding and ergonomics.
Summary
A review on robotic ultrasound systems is presented in which first technical specifications are outlined. Hereafter, the literature of the past five years is subdivided into teleoperation, collaborative assistance, or autonomous systems based on LORA. Finally, future trends for robotic ultrasound systems are reviewed with a focus on artificial intelligence and virtual/augmented reality.
Collapse
Affiliation(s)
- Felix von Haxthausen
- Institute for Robotics and Cognitive Systems, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
| | - Sven Böttger
- Institute for Robotics and Cognitive Systems, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
| | - Daniel Wulff
- Institute for Robotics and Cognitive Systems, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
| | - Jannis Hagenah
- Institute for Robotics and Cognitive Systems, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
| | - Verónica García-Vázquez
- Institute for Robotics and Cognitive Systems, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
| | - Svenja Ipsen
- Institute for Robotics and Cognitive Systems, University of Lübeck, Ratzeburger Allee 160, 23562 Lübeck, Germany
| |
Collapse
|
11
|
Seitz PK, Baumann B, Johnen W, Lissek C, Seidel J, Bendl R. Development of a robot-assisted ultrasound-guided radiation therapy (USgRT). Int J Comput Assist Radiol Surg 2019; 15:491-501. [DOI: 10.1007/s11548-019-02104-y] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/13/2019] [Accepted: 12/04/2019] [Indexed: 11/30/2022]
|
12
|
Ultrasound simulation with deformable and patient-specific scatterer maps. Int J Comput Assist Radiol Surg 2019; 14:1589-1599. [PMID: 31435812 DOI: 10.1007/s11548-019-02054-5] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2019] [Accepted: 08/07/2019] [Indexed: 10/26/2022]
Abstract
PURPOSE Ray-tracing-based simulations model ultrasound (US) interactions with a custom geometric anatomical model, where US texture can be emulated via real-time point-spread function convolutions of a tissue scatterer representation. Such scatterer representations for realistic appearance are difficult to parameterize or model manually and do not respond to volumetric deformations such as those caused with tissue compression by the probe. Herein we utilize brightness mode (B-mode) estimated scatterer maps for ray tracing and propose to enhance the realism of ray-tracing-based simulations by incorporating dynamic speckle patterns that change compliant with tissue deformation. METHODS In this work, we realistically simulate US texture deformations in the scatterer domain via back-projection of ray segments into a nominal state before sampling during simulation runtime. We estimate scatterer maps from background in vivo images using a pretrained generative adversarial network. RESULTS We demonstrated our proposed scatterer estimation and runtime background fusion method on simulated transvaginal US scans of detailed surface-based foetal models. We show the viability of modelling deformations in the scatterer domain at interactive frame rates of 28 frames per second. A quantitative and a qualitative evaluations indicated improved realism in comparison to the state of the art. CONCLUSIONS Transferring a background image in a scatterer representation enables us to capture anatomical content in a physical space, in which deformations can be incorporated physically consistently before convolving with a US point-spread function during simulation runtime. This then uses the same imaging model on both the background and the hand-crafted models leading to a consistent and seamless compounding of contents in the scatterer space.
Collapse
|
13
|
Norton JC, Slawinski PR, Lay HS, Martin JW, Cox BF, Cummins G, Desmulliez MP, Clutton RE, Obstein KL, Cochran S, Valdastri P. Intelligent magnetic manipulation for gastrointestinal ultrasound. Sci Robot 2019; 4:eaav7725. [PMID: 31380501 PMCID: PMC6677276 DOI: 10.1126/scirobotics.aav7725] [Citation(s) in RCA: 33] [Impact Index Per Article: 6.6] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/19/2022]
Abstract
Diagnostic endoscopy in the gastrointestinal tract has remained largely unchanged for decades and is limited to the visualization of the tissue surface, the collection of biopsy samples for diagnoses, and minor interventions such as clipping or tissue removal. In this work, we present the autonomous servoing of a magnetic capsule robot for in-situ, subsurface diagnostics of microanatomy. We investigated and showed the feasibility of closed-loop magnetic control using digitized microultrasound (μUS) feedback; this is crucial for obtaining robust imaging in an unknown and unconstrained environment. We demonstrated the functionality of an autonomous servoing algorithm that uses μUS feedback, both on benchtop trials as well as in-vivo in a porcine model. We have validated this magnetic-μUS servoing in instances of autonomous linear probe motion and were able to locate markers in an agar phantom with 1.0 ± 0.9 mm position accuracy using a fusion of robot localization and μUS image information. This work demonstrates the feasibility of closed-loop robotic μUS imaging in the bowel without the need for either a rigid physical link between the transducer and extracorporeal tools or complex manual manipulation.
Collapse
Affiliation(s)
| | | | | | | | | | | | | | | | - Keith L. Obstein
- STORM Lab USA, Vanderbilt University, Nashville, USA
- Vanderbilt University Medical Center, Nashville, USA
| | - Sandy Cochran
- University of Glasgow, School of Mechanical Engineering, Glasgow, UK
| | | |
Collapse
|