Use the force: deformation correction in robotic 3D ultrasound

Quantitative validation of two model-based methods for the correction of probe pressure deformation in ultrasound

PURPOSE

The acquisition of good quality ultrasound (US) images requires good acoustic coupling between the ultrasound probe and the patient's skin. In practice, this good coupling is achieved by the operator applying a force to the skin through the probe. This force induces a deformation of the tissues underlying the probe. The distorted images deteriorate the quality of the reconstructed 3D US image.

METHODS

In this work, we propose two methods to correct these deformations. These methods are based on the construction of a biomechanical model to predict the mechanical behavior of the imaged soft tissues. The originality of the methods is that they do not use external information (force or position value from sensors, or elasticity value from the literature). The model is parameterized thanks to the information contained in the image. This is allowed thanks to the optimization of two key parameters for the model which are the indentation d and the elasticity ratio α.

RESULTS

The validation is performed on real images acquired on a gelatin-based phantom using an ultrasound probe inducing an increasing vertical indentation using a step motor. The results showed a good correction of the two methods for indentations less than 4 mm. For larger indentations, one of the two methods (guided by the similarity score) provides a better quality of correction, presenting a Euclidean distance between the contours of the reference image and the corrected image of 0.71 mm.

CONCLUSION

The proposed methods ensured the correction of the deformed images induced by a linear probe pressure without using any information coming from sensors (force or position), or generic information about the mechanical parameters. The corrected images can be used to obtain a corrected 3D US image.

DefCor-Net: Physics-aware ultrasound deformation correction

The recovery of morphologically accurate anatomical images from deformed ones is challenging in ultrasound (US) image acquisition, but crucial to accurate and consistent diagnosis, particularly in the emerging field of computer-assisted diagnosis. This article presents a novel physics-aware deformation correction approach based on a coarse-to-fine, multi-scale deep neural network (DefCor-Net). To achieve pixel-wise performance, DefCor-Net incorporates biomedical knowledge by estimating pixel-wise stiffness online using a U-shaped feature extractor. The deformation field is then computed using polynomial regression by integrating the measured force applied by the US probe. Based on real-time estimation of pixel-by-pixel tissue properties, the learning-based approach enables the potential for anatomy-aware deformation correction. To demonstrate the effectiveness of the proposed DefCor-Net, images recorded at multiple locations on forearms and upper arms of six volunteers are used to train and validate DefCor-Net. The results demonstrate that DefCor-Net can significantly improve the accuracy of deformation correction to recover the original geometry (Dice Coefficient: from 14.3±20.9 to 82.6±12.1 when the force is 6N). Code:https://github.com/KarolineZhy/DefCorNet.

Autonomous Robotic Ultrasound Vascular Imaging System With Decoupled Control Strategy for External-Vision-Free Environments

OBJECTIVE

Ultrasound (US) probes scan over the surface of the human body to acquire US images in clinical vascular US diagnosis. However, due to the deformation and specificity of different human surfaces, the relationship between the scan trajectory of the skin and the internal tissues is not fully correlated, which poses a challenge for autonomous robotic US imaging in a dynamic and external-vision-free environment. Here, we propose a decoupled control strategy for autonomous robotic vascular US imaging in an environment without external vision.

METHODS

The proposed system is divided into outer-loop posture control and inner-loop orientation control, which are separately determined by a deep learning (DL) agent and a reinforcement learning (RL) agent. First, we use a weakly supervised US vessel segmentation network to estimate the probe orientation. In the outer loop control, we use a force-guided reinforcement learning agent to maintain a specific angle between the US probe and the skin in the dynamic imaging processes. Finally, the orientation and the posture are integrated to complete the imaging process.

RESULTS

Evaluation experiments on several volunteers showed that our RUS could autonomously perform vascular imaging in arms with different stiffness, curvature, and size without additional system adjustments. Furthermore, our system achieved reproducible imaging and reconstruction of dynamic targets without relying on vision-based surface information.

CONCLUSION AND SIGNIFICANCE

Our system and control strategy provides a novel framework for the application of US robots in complex and external-vision-free environments.

Robotic ultrasound imaging: State-of-the-art and future perspectives

Ultrasound (US) is one of the most widely used modalities for clinical intervention and diagnosis due to the merits of providing non-invasive, radiation-free, and real-time images. However, free-hand US examinations are highly operator-dependent. Robotic US System (RUSS) aims at overcoming this shortcoming by offering reproducibility, while also aiming at improving dexterity, and intelligent anatomy and disease-aware imaging. In addition to enhancing diagnostic outcomes, RUSS also holds the potential to provide medical interventions for populations suffering from the shortage of experienced sonographers. In this paper, we categorize RUSS as teleoperated or autonomous. Regarding teleoperated RUSS, we summarize their technical developments, and clinical evaluations, respectively. This survey then focuses on the review of recent work on autonomous robotic US imaging. We demonstrate that machine learning and artificial intelligence present the key techniques, which enable intelligent patient and process-specific, motion and deformation-aware robotic image acquisition. We also show that the research on artificial intelligence for autonomous RUSS has directed the research community toward understanding and modeling expert sonographers' semantic reasoning and action. Here, we call this process, the recovery of the "language of sonography". This side result of research on autonomous robotic US acquisitions could be considered as valuable and essential as the progress made in the robotic US examination itself. This article will provide both engineers and clinicians with a comprehensive understanding of RUSS by surveying underlying techniques. Additionally, we present the challenges that the scientific community needs to face in the coming years in order to achieve its ultimate goal of developing intelligent robotic sonographer colleagues. These colleagues are expected to be capable of collaborating with human sonographers in dynamic environments to enhance both diagnostic and intraoperative imaging.

Nondeformed Ultrasound Image Production Method for Ultrasound-Guided Radiotherapy

Purpose: During ultrasound (US)-guided radiotherapy, the tissue is deformed by probe pressure, and the US image is limited by changes in tissue and organ position and geometry when the US image is aligned with computed tomography (CT) image, leading to poor alignment. Accordingly, a pixel displacement-based nondeformed US image production method is proposed. Methods: The correction of US image deformation is achieved by calculating the pixel displacement of an image. The positioning CT image (CTstd) is used as the gold standard. The deformed US image (USdef) is inputted into the Harris algorithm to extract corner points for selecting feature points, and the displacement of adjacent pixels of feature points in the US video stream is calculated using the Lucas-Kanade optical flow algorithm. The moving least squares algorithm is used to correct USdef globally and locally in accordance with image pixel displacement to generate a nondeformed US image (USrev). In addition, USdef and USrev were separately aligned with CTstd to evaluate the improvement of alignment accuracy through deformation correction. Results: In the phantom experiment, the overall and local average correction errors of the US image under the optimal probe pressure were 1.0944 and 0.7388 mm, respectively, and the registration accuracy of USdef and USrev with CTstd was 0.6764 and 0.9016, respectively. During the volunteer experiment, the correction error of all 12 patients' data ranged from -1.7525 to 1.5685 mm, with a mean absolute error of 0.8612 mm. The improvement range of US and CT registration accuracy, before and after image deformation correction in the 12 patients evaluated by a normalized correlation coefficient, was 0.1232 to 0.2476. Conclusion: The pixel displacement-based deformation correction method can solve the limitation imposed by image deformation on image alignment in US-guided radiotherapy. Compared with USdef, the alignment results of USrev with CT were better.

Good and bad boundaries in ultrasound compounding: preserving anatomic boundaries while suppressing artifacts

Purpose

Ultrasound compounding is to combine sonographic information captured from different angles and produce a single image. It is important for multi-view reconstruction, but as of yet there is no consensus on best practices for compounding. Current popular methods inevitably suppress or altogether leave out bright or dark regions that are useful and potentially introduce new artifacts. In this work, we establish a new algorithm to compound the overlapping pixels from different viewpoints in ultrasound.

Methods

Inspired by image fusion algorithms and ultrasound confidence, we uniquely leverage Laplacian and Gaussian pyramids to preserve the maximum boundary contrast without overemphasizing noise, speckles, and other artifacts in the compounded image, while taking the direction of the ultrasound probe into account. Besides, we designed an algorithm that detects the useful boundaries in ultrasound images to further improve the boundary contrast.

Results

We evaluate our algorithm by comparing it with previous algorithms both qualitatively and quantitatively, and we show that our approach not only preserves both light and dark details, but also somewhat suppresses noise and artifacts, rather than amplifying them. We also show that our algorithm can improve the performance of downstream tasks like segmentation.

Conclusion

Our proposed method that is based on confidence, contrast, and both Gaussian and Laplacian pyramids appears to be better at preserving contrast at anatomic boundaries while suppressing artifacts than any of the other approaches we tested. This algorithm may have future utility with downstream tasks such as 3D ultrasound volume reconstruction and segmentation.

Automatic ultrasound scanning robotic system with optical waveguide-based force measurement

PURPOSE

The three-dimensional (3D) ultrasound (US) imaging realized by continuous scanning of a region is of great value for medical diagnosis and robot-assisted needle insertion. During scanning, the contact force and posture between the probe and skin of the patient are crucial factors that determine the quality of US imaging. We propose a robotic system for automatic scanning of curved surfaces with a stable contact force and vertical contact posture (the probe is parallel to the normal of the surface at the contact point).

METHODS

A 6-DOF robotic arm is used to hold and drive a two-dimensional (2D) US probe to complete automatic scanning. Further, a path-planning strategy is proposed to generate the scan path covering the selected area automatically. We also developed a novel force-measuring device based on optical waveguides to measure the distributed contact force and contact posture. Based on the measured force and posture, the robotic arm automatically adjusts the position and orientation of the probe and maintains a stable contact force and vertical contact posture at each scan point.

RESULTS

The novel force-measuring device is easy to fabricate, integrates with the probe and has the capacity of measuring the force distributed on the contact surface and estimating the contact posture. The experimental results of automatic scanning of a US phantom and parts of the human body demonstrate that the proposed system performs well in automatically scanning curved surfaces, maintaining a stable contact force and vertical contact posture and producing a good quality 3D US volume.

CONCLUSION

An automatic US scanning robotic system with an optical waveguide-based force-measuring device was developed and tested successfully. Experimental results demonstrated the feasibility of the proposed system to scan the human body.

Towards automated ultrasound imaging-robotic image acquisition in liver and prostate for long-term motion monitoring

Real-time volumetric (4D) ultrasound has shown high potential for diagnostic and therapy guidance tasks. One of the main drawbacks of ultrasound imaging to date is the reliance on manual probe positioning and the resulting user dependence. Robotic assistance could help overcome this issue and facilitate the acquisition of long-term image data to observe dynamic processesin vivoover time. The aim of this study is to assess the feasibility of robotic probe manipulation and organ motion quantification during extended imaging sessions. The system consists of a collaborative robot and a 4D ultrasound system providing real-time data access. Five healthy volunteers received liver and prostate scans during free breathing over 30 min. Initial probe placement was performed with real-time remote control with a predefined contact force of 10 N. During scan acquisition, the probe position was continuously adjusted to the body surface motion using impedance control. Ultrasound volumes, the pose of the end-effector and the estimated contact forces were recorded. For motion analysis, one anatomical landmark was manually annotated in a subset of ultrasound frames for each experiment. Probe contact was uninterrupted over the entire scan duration in all ten sessions. Organ drift and imaging artefacts were successfully compensated using remote control. The median contact force along the probe's longitudinal axis was 10.0 N with maximum values of 13.2 and 21.3 N for liver and prostate, respectively. Forces exceeding 11 N only occurred in 0.3% of the time. Probe and landmark motion were more pronounced in the liver, with median interquartile ranges of 1.5 and 9.6 mm, compared to 0.6 and 2.7 mm in the prostate. The results show that robotic ultrasound imaging with dynamic force control can be used for stable, long-term imaging of anatomical regions affected by motion. The system facilitates the acquisition of 4D image datain vivoover extended scanning periods for the first time and holds the potential to be used for motion monitoring for therapy guidance as well as diagnostic tasks.

Medical Robotics for Ultrasound Imaging: Current Systems and Future Trends

Purpose of Review

This review provides an overview of the most recent robotic ultrasound systems that have contemporary emerged over the past five years, highlighting their status and future directions. The systems are categorized based on their level of robot autonomy (LORA).

Recent Findings

Teleoperating systems show the highest level of technical maturity. Collaborative assisting and autonomous systems are still in the research phase, with a focus on ultrasound image processing and force adaptation strategies. However, missing key factors are clinical studies and appropriate safety strategies. Future research will likely focus on artificial intelligence and virtual/augmented reality to improve image understanding and ergonomics.

Summary

A review on robotic ultrasound systems is presented in which first technical specifications are outlined. Hereafter, the literature of the past five years is subdivided into teleoperation, collaborative assistance, or autonomous systems based on LORA. Finally, future trends for robotic ultrasound systems are reviewed with a focus on artificial intelligence and virtual/augmented reality.

Ultrasound simulation with deformable and patient-specific scatterer maps

PURPOSE

Ray-tracing-based simulations model ultrasound (US) interactions with a custom geometric anatomical model, where US texture can be emulated via real-time point-spread function convolutions of a tissue scatterer representation. Such scatterer representations for realistic appearance are difficult to parameterize or model manually and do not respond to volumetric deformations such as those caused with tissue compression by the probe. Herein we utilize brightness mode (B-mode) estimated scatterer maps for ray tracing and propose to enhance the realism of ray-tracing-based simulations by incorporating dynamic speckle patterns that change compliant with tissue deformation.

METHODS

In this work, we realistically simulate US texture deformations in the scatterer domain via back-projection of ray segments into a nominal state before sampling during simulation runtime. We estimate scatterer maps from background in vivo images using a pretrained generative adversarial network.

RESULTS

We demonstrated our proposed scatterer estimation and runtime background fusion method on simulated transvaginal US scans of detailed surface-based foetal models. We show the viability of modelling deformations in the scatterer domain at interactive frame rates of 28 frames per second. A quantitative and a qualitative evaluations indicated improved realism in comparison to the state of the art.

CONCLUSIONS

Transferring a background image in a scatterer representation enables us to capture anatomical content in a physical space, in which deformations can be incorporated physically consistently before convolving with a US point-spread function during simulation runtime. This then uses the same imaging model on both the background and the hand-crafted models leading to a consistent and seamless compounding of contents in the scatterer space.

Intelligent magnetic manipulation for gastrointestinal ultrasound

Diagnostic endoscopy in the gastrointestinal tract has remained largely unchanged for decades and is limited to the visualization of the tissue surface, the collection of biopsy samples for diagnoses, and minor interventions such as clipping or tissue removal. In this work, we present the autonomous servoing of a magnetic capsule robot for in-situ, subsurface diagnostics of microanatomy. We investigated and showed the feasibility of closed-loop magnetic control using digitized microultrasound (μUS) feedback; this is crucial for obtaining robust imaging in an unknown and unconstrained environment. We demonstrated the functionality of an autonomous servoing algorithm that uses μUS feedback, both on benchtop trials as well as in-vivo in a porcine model. We have validated this magnetic-μUS servoing in instances of autonomous linear probe motion and were able to locate markers in an agar phantom with 1.0 ± 0.9 mm position accuracy using a fusion of robot localization and μUS image information. This work demonstrates the feasibility of closed-loop robotic μUS imaging in the bowel without the need for either a rigid physical link between the transducer and extracorporeal tools or complex manual manipulation.