An unsupervised learning approach to ultrasound strain elastography with spatio-temporal consistency

Alternating direction method of multipliers for displacement estimation in ultrasound strain elastography

BACKGROUND

Ultrasound strain imaging, which delineates mechanical properties to detect tissue abnormalities, involves estimating the time delay between two radio-frequency (RF) frames collected before and after tissue deformation. The existing regularized optimization-based time-delay estimation (TDE) techniques suffer from at least one of the following drawbacks: (1) The regularizer is not aligned with the tissue deformation physics due to taking only the first-order displacement derivative into account; (2) TheL 2 $L2$ -norm of the displacement derivatives, which oversmooths the estimated time-delay, is utilized as the regularizer; (3) The modulus function defined mathematically should be approximated by a smooth function to facilitate the optimization ofL 1 $L1$ -norm.

PURPOSE

Our purpose is to develop a novel TDE technique that resolves the aforementioned shortcomings of the existing algorithms.

METHODS

Herein, we propose employing the alternating direction method of multipliers (ADMM) for optimizing a novel cost function consisting ofL 2 $L2$ -norm data fidelity term andL 1 $L1$ -norm first- and second-order spatial continuity terms. ADMM empowers the proposed algorithm to use different techniques for optimizing different parts of the cost function and obtain high-contrast strain images with smooth backgrounds and sharp boundaries. We name our technique ADMM for totaL variaTion RegUlarIzation in ultrasound STrain imaging (ALTRUIST). ALTRUIST's efficacy is quantified using absolute error (AE), Structural SIMilarity (SSIM), signal-to-noise ratio (SNR), contrast-to-noise ratio (CNR), and strain ratio (SR) with respect to GLUE, OVERWIND, andL 1 $L1$ -SOUL, three recently published energy-based techniques, and UMEN-Net, a state-of-the-art deep learning-based algorithm. Analysis of variance (ANOVA)-led multiple comparison tests and paired t $t$ -tests at5 % $5\%$ overall significance level were conducted to assess the statistical significance of our findings. The Bonferroni correction was taken into account in all statistical tests. Two simulated layer phantoms, three simulated resolution phantoms, one hard-inclusion simulated phantom, one multi-inclusion simulated phantom, one experimental breast phantom, and three in vivo liver cancer datasets have been used for validation experiments. We have published the ALTRUIST code at http://code.sonography.ai.

RESULTS

ALTRUIST substantially outperforms the four state-of-the-art benchmarks in all validation experiments, both qualitatively and quantitatively. ALTRUIST yields up to573 % ∗ ${573\%}^{*}$ ,41 % ∗ ${41\%}^{*}$ , and51 % ∗ ${51\%}^{*}$ SNR improvements and443 % ∗ ${443\%}^{*}$ ,53 % ∗ ${53\%}^{*}$ , and15 % ∗ ${15\%}^{*}$ CNR improvements overL 1 $L1$ -SOUL, its closest competitor, for simulated, phantom, and in vivo liver cancer datasets, respectively, where the asterisk (*) indicates statistical significance. In addition, ANOVA-led multiple comparison tests and paired t $t$ -tests indicate that ALTRUIST generally achieves statistically significant improvements over GLUE, UMEN-Net, OVERWIND, andL 1 $L1$ -SOUL in terms of AE, SSIM map, SNR, and CNR.

CONCLUSIONS

A novel ultrasonic displacement tracking algorithm named ALTRUIST has been developed. The principal novelty of ALTRUIST is incorporating ADMM for optimizing anL 1 $L1$ -norm regularization-based cost function. ALTRUIST exhibits promising performance in simulation, phantom, and in vivo experiments.

Passive Elastography for Clinical HIFU Lesion Detection

High-intensity Focused Ultrasound (HIFU) is a promising treatment modality for a wide range of pathologies including prostate cancer. However, the lack of a reliable ultrasound-based monitoring technique limits its clinical use. Ultrasound currently provides real-time HIFU planning, but its use for monitoring is usually limited to detecting the backscatter increase resulting from chaotic bubble appearance. HIFU has been shown to generate stiffening in various tissues, so elastography is an interesting lead for ablation monitoring. However, the standard techniques usually require the generation of a controlled push which can be problematic in deeper organs. Passive elastography offers a potential alternative as it uses the physiological wave field to estimate the elasticity in tissues and not an external perturbation. This technique was adapted to process B-mode images acquired with a clinical system. It was first shown to faithfully assess elasticity in calibrated phantoms. The technique was then implemented on the Focal One® clinical system to evaluate its capacity to detect HIFU lesions in vitro (CNR = 9.2 dB) showing its independence regarding the bubbles resulting from HIFU and in vivo where the physiological wave field was successfully used to detect and delineate lesions of different sizes in porcine liver. Finally, the technique was performed for the very first time in four prostate cancer patients showing strong variation in elasticity before and after HIFU treatment (average variation of 33.0 ± 16.0 % ). Passive elastography has shown evidence of its potential to monitor HIFU treatment and thus help spread its use.

Ultrafast Cardiac Imaging Using Deep Learning for Speckle-Tracking Echocardiography

High-quality ultrafast ultrasound imaging is based on coherent compounding from multiple transmissions of plane waves (PW) or diverging waves (DW). However, compounding results in reduced frame rate, as well as destructive interferences from high-velocity tissue motion if motion compensation (MoCo) is not considered. While many studies have recently shown the interest of deep learning for the reconstruction of high-quality static images from PW or DW, its ability to achieve such performance while maintaining the capability of tracking cardiac motion has yet to be assessed. In this article, we addressed such issue by deploying a complex-weighted convolutional neural network (CNN) for image reconstruction and a state-of-the-art speckle-tracking method. The evaluation of this approach was first performed by designing an adapted simulation framework, which provides specific reference data, i.e., high-quality, motion artifact-free cardiac images. The obtained results showed that, while using only three DWs as input, the CNN-based approach yielded an image quality and a motion accuracy equivalent to those obtained by compounding 31 DWs free of motion artifacts. The performance was then further evaluated on nonsimulated, experimental in vitro data, using a spinning disk phantom. This experiment demonstrated that our approach yielded high-quality image reconstruction and motion estimation, under a large range of velocities and outperforms a state-of-the-art MoCo-based approach at high velocities. Our method was finally assessed on in vivo datasets and showed consistent improvement in image quality and motion estimation compared to standard compounding. This demonstrates the feasibility and effectiveness of deep learning reconstruction for ultrafast speckle-tracking echocardiography.

Unsupervised deep learning-based displacement estimation for vascular elasticity imaging applications

Objective. Arterial wall stiffness can provide valuable information on the proper function of the cardiovascular system. Ultrasound elasticity imaging techniques have shown great promise as a low-cost and non-invasive tool to enable localized maps of arterial wall stiffness. Such techniques rely upon motion detection algorithms that provide arterial wall displacement estimation.Approach. In this study, we propose an unsupervised deep learning-based approach, originally proposed for image registration, in order to enable improved quality arterial wall displacement estimation at high temporal and spatial resolutions. The performance of the proposed network was assessed through phantom experiments, where various models were trained by using ultrasound RF signals, or B-mode images, as well as different loss functions.Main results. Using the mean square error (MSE) for the training process provided the highest signal-to-noise ratio when training on the B-modes images (30.36 ± 1.14 dB) and highest contrast-to-noise ratio when training on the RF signals (32.84 ± 1.89 dB). In addition, training the model on RF signals demonstrated the capability of providing accurate localized pulse wave velocity (PWV) maps, with a mean relative error (MREPWV) of 3.32 ± 1.80% and anR2 of 0.97 ± 0.03. Finally, the developed model was tested in human common carotid arteriesin vivo, providing accurate tracking of the distension pulse wave propagation, with an MREPWV= 3.86 ± 2.69% andR2 = 0.95 ± 0.03.Significance. In conclusion, a novel displacement estimation approach was presented, showing promise in improving vascular elasticity imaging techniques.

A Data-Efficient Deep Learning Strategy for Tissue Characterization via Quantitative Ultrasound: Zone Training

Deep learning (DL) powered biomedical ultrasound imaging is an emerging research field where researchers adapt the image analysis capabilities of DL algorithms to biomedical ultrasound imaging settings. A major roadblock to wider adoption of DL powered biomedical ultrasound imaging is that acquisition of large and diverse datasets is expensive in clinical settings, which is a requirement for successful DL implementation. Hence, there is a constant need for developing data-efficient DL techniques to turn DL powered biomedical ultrasound imaging into reality. In this work, we develop a data-efficient DL training strategy for classifying tissues based on the ultrasonic backscattered RF data, i.e., quantitative ultrasound (QUS), which we named zone training. In zone training, we propose to divide the complete field of view of an ultrasound image into multiple zones associated with different regions of a diffraction pattern and then, train separate DL networks for each zone. The main advantage of zone training is that it requires less training data to achieve high accuracy. In this work, three different tissue-mimicking phantoms were classified by a DL network. The results demonstrated that zone training can require a factor of 2-3 less training data in low data regime to achieve similar classification accuracies compared to a conventional training strategy.

Lateral Strain Imaging Using Self-Supervised and Physically Inspired Constraints in Unsupervised Regularized Elastography

Convolutional Neural Networks (CNN) have shown promising results for displacement estimation in UltraSound Elastography (USE). Many modifications have been proposed to improve the displacement estimation of CNNs for USE in the axial direction. However, the lateral strain, which is essential in several downstream tasks such as the inverse problem of elasticity imaging, remains a challenge. The lateral strain estimation is complicated since the motion and the sampling frequency in this direction are substantially lower than the axial one, and a lack of carrier signal in this direction. In computer vision applications, the axial and the lateral motions are independent. In contrast, the tissue motion pattern in USE is governed by laws of physics which link the axial and lateral displacements. In this paper, inspired by Hooke's law, we, first propose Physically Inspired ConsTraint for Unsupervised Regularized Elastography (PICTURE), where we impose a constraint on the Effective Poisson's ratio (EPR) to improve the lateral strain estimation. In the next step, we propose self-supervised PICTURE (sPICTURE) to further enhance the strain image estimation. Extensive experiments on simulation, experimental phantom and in vivo data demonstrate that the proposed methods estimate accurate axial and lateral strain maps.

Convolutional Neural Network-Based Speckle Tracking for Ultrasound Strain Elastography: An Unsupervised Learning Approach

Accurate and computationally efficient motion estimation is a critical component of real-time ultrasound strain elastography (USE). With the advent of deep-learning neural network models, a growing body of work has explored supervised convolutional neural network (CNN)-based optical flow in the framework of USE. However, the above-said supervised learning was often done using simulated ultrasound data. The research community has questioned whether simulated ultrasound data containing simple motion can train deep-learning CNN models that can reliably track complex in vivo speckle motion. In parallel with other research groups' efforts, this study developed an unsupervised motion estimation neural network (UMEN-Net) for USE by adapting a well-established CNN model named PWC-Net. Our network's input is a pair of predeformation and postdeformation radio frequency (RF) echo signals. The proposed network outputs both axial and lateral displacement fields. The loss function consists of a correlation between the predeformation signal and the motion-compensated postcompression signal, smoothness of the displacement fields, and tissue incompressibility. Notably, an innovative correlation method known as the globally optimized correspondence (GOCor) volumes module developed by Truong et al. was used to replace the original Corr module to enhance our evaluation of signal correlation. The proposed CNN model was tested using simulated, phantom, and in vivo ultrasound data containing biologically confirmed breast lesions. Its performance was compared against other state-of-the-art methods, including two deep-learning-based tracking methods (MPWC-Net++ and ReUSENet) and two conventional tracking methods (GLUE and BRGMT-LPF). In summary, compared with the four known methods mentioned above, our unsupervised CNN model not only obtained higher signal-to-noise ratios (SNRs) and contrast-to-noise ratios (CNRs) for axial strain estimates but also improved the quality of the lateral strain estimates.

Unsupervised Convolutional Neural Network for Motion Estimation in Ultrasound Elastography

High-quality motion estimation is essential for ultrasound elastography (USE). Traditional motion estimation algorithms based on speckle tracking such as normalized cross correlation (NCC) or regularization such as global ultrasound elastography (GLUE) are time-consuming. In order to reduce the computational cost and ensure the accuracy of motion estimation, many convolutional neural networks have been introduced into USE. Most of these networks such as radio-frequency modified pyramid, warping and cost volume network (RFMPWC-Net) are supervised and need many ground truths as labels in network training. However, the ground truths are laborious to collect for USE. In this study, we proposed a MaskFlownet-based unsupervised convolutional neural network (MF-UCNN) for fast and high-quality motion estimation in USE. The inputs to MF-UCNN are the concatenation of RF, envelope, and B-mode images before and after deformation, while the outputs are the axial and lateral displacement fields. The similarity between the predeformed image and the warped image (i.e., the postdeformed image compensated by the estimated displacement fields) and the smoothness of the estimated displacement fields were incorporated in the loss function. The network was compared with modified pyramid, warping and cost volume network (MPWC-Net)++, RFMPWC-Net, GLUE, and NCC. Results of simulations, breast phantom, and in vivo experiments show that MF-UCNN obtains higher signal-to-noise ratio (SNR) and higher contrast-to-noise ratio (CNR). MF-UCNN achieves high-quality motion estimation with significantly reduced computation time. It is unsupervised and does not need any ground truths as labels in the training, and, thus, has great potential for motion estimation in USE.

Bi-Directional Semi-Supervised Training of Convolutional Neural Networks for Ultrasound Elastography Displacement Estimation

The performance of ultrasound elastography (USE) heavily depends on the accuracy of displacement estimation. Recently, convolutional neural networks (CNNs) have shown promising performance in optical flow estimation and have been adopted for USE displacement estimation. Networks trained on computer vision images are not optimized for USE displacement estimation since there is a large gap between the computer vision images and the high-frequency radio frequency (RF) ultrasound data. Many researchers tried to adopt the optical flow CNNs to USE by applying transfer learning to improve the performance of CNNs for USE. However, the ground-truth displacement in real ultrasound data is unknown, and simulated data exhibit a domain shift compared to the real data and are also computationally expensive to generate. To resolve this issue, semisupervised methods have been proposed in which the networks pretrained on computer vision images are fine-tuned using real ultrasound data. In this article, we employ a semisupervised method by exploiting the first- and second-order derivatives of the displacement field for regularization. We also modify the network structure to estimate both forward and backward displacements and propose to use consistency between the forward and backward strains as an additional regularizer to further enhance the performance. We validate our method using several experimental phantom and in vivo data. We also show that the network fine-tuned by our proposed method using experimental phantom data performs well on in vivo data similar to the network fine-tuned on in vivo data. Our results also show that the proposed method outperforms current deep learning methods and is comparable to computationally expensive optimization-based algorithms.

Robust Scatterer Number Density Segmentation of Ultrasound Images

Quantitative ultrasound (QUS) aims to reveal information about the tissue microstructure using backscattered echo signals from clinical scanners. Among different QUS parameters, scatterer number density is an important property that can affect the estimation of other QUS parameters. Scatterer number density can be classified into high or low scatterer densities. If there are more than ten scatterers inside the resolution cell, the envelope data are considered as fully developed speckle (FDS) and, otherwise, as underdeveloped speckle (UDS). In conventional methods, the envelope data are divided into small overlapping windows (a strategy here we refer to as patching), and statistical parameters, such as SNR and skewness, are employed to classify each patch of envelope data. However, these parameters are system-dependent, meaning that their distribution can change by the imaging settings and patch size. Therefore, reference phantoms that have known scatterer number density are imaged with the same imaging settings to mitigate system dependency. In this article, we aim to segment regions of ultrasound data without any patching. A large dataset is generated, which has different shapes of scatterer number density and mean scatterer amplitude using a fast simulation method. We employ a convolutional neural network (CNN) for the segmentation task and investigate the effect of domain shift when the network is tested on different datasets with different imaging settings. Nakagami parametric image is employed for multitask learning to improve performance. Furthermore, inspired by the reference phantom methods in QUS, a domain adaptation stage is proposed, which requires only two frames of data from FDS and UDS classes. We evaluate our method for different experimental phantoms and in vivo data.

Spatiotemporal Bayesian Regularization for Cardiac Strain Imaging: Simulation and In Vivo Results

Cardiac strain imaging (CSI) plays a critical role in the detection of myocardial motion abnormalities. Displacement estimation is an important processing step to ensure the accuracy and precision of derived strain tensors. In this paper, we propose and implement Spatiotemporal Bayesian regularization (STBR) algorithms for two-dimensional (2-D) normalized cross-correlation (NCC) based multi-level block matching along with incorporation into a Lagrangian cardiac strain estimation framework. Assuming smooth temporal variation over a short span of time, the proposed STBR algorithm performs displacement estimation using at least four consecutive ultrasound radio-frequency (RF) frames by iteratively regularizing 2-D NCC matrices using information from a local spatiotemporal neighborhood in a Bayesian sense. Two STBR schemes are proposed to construct Bayesian likelihood functions termed as Spatial then Temporal Bayesian (STBR-1) and simultaneous Spatiotemporal Bayesian (STBR-2). Radial and longitudinal strain estimated from a finite-element-analysis (FEA) model of realistic canine myocardial deformation were utilized to quantify strain bias, normalized strain error and total temporal relative error (TTR). Statistical analysis with one-way analysis of variance (ANOVA) showed that all Bayesian regularization methods significantly outperform NCC with lower bias and errors (p < 0.001). However, there was no significant difference among Bayesian methods. For example, mean longitudinal TTR for NCC, SBR, STBR-1 and STBR-2 were 25.41%, 9.27%, 10.38% and 10.13% respectively An in vivo feasibility study using RF data from ten healthy mice hearts were used to compare the elastographic signal-to-noise ratio (SNR_e) calculated using stochastic analysis. STBR-2 had the highest expected SNR_e both for radial and longitudinal strain. The mean expected SNR_e values for accumulated radial strain for NCC, SBR, STBR-1 and STBR-2 were 5.03, 9.43, 9.42 and 10.58, respectively. Overall results suggest that STBR improves CSI in vivo.