Unsupervised Convolutional Neural Network for Motion Estimation in Ultrasound Elastography

Knowledge fused latent representation from lung ultrasound examination for COVID-19 pneumonia severity assessment

COVID-19 pneumonia severity assessment is of great clinical importance, and lung ultrasound (LUS) plays a crucial role in aiding the severity assessment of COVID-19 pneumonia due to its safety and portability. However, its reliance on qualitative and subjective observations by clinicians is a limitation. Moreover, LUS images often exhibit significant heterogeneity, emphasizing the need for more quantitative assessment methods. In this paper, we propose a knowledge fused latent representation framework tailored for COVID-19 pneumonia severity assessment using LUS examinations. The framework transforms the LUS examination into latent representation and extracts knowledge from regions labeled by clinicians to improve accuracy. To fuse the knowledge into the latent representation, we employ a knowledge fusion with latent representation (KFLR) model. This model significantly reduces errors compared to approaches that lack prior knowledge integration. Experimental results demonstrate the effectiveness of our method, achieving high accuracy of 96.4 % and 87.4 % for binary-level and four-level COVID-19 pneumonia severity assessments, respectively. It is worth noting that only a limited number of studies have reported accuracy for clinically valuable exam level assessments, and our method surpass existing methods in this context. These findings highlight the potential of the proposed framework for monitoring disease progression and patient stratification in COVID-19 pneumonia cases.

Surgical-DINO: adapter learning of foundation models for depth estimation in endoscopic surgery

PURPOSE

Depth estimation in robotic surgery is vital in 3D reconstruction, surgical navigation and augmented reality visualization. Although the foundation model exhibits outstanding performance in many vision tasks, including depth estimation (e.g., DINOv2), recent works observed its limitations in medical and surgical domain-specific applications. This work presents a low-ranked adaptation (LoRA) of the foundation model for surgical depth estimation.

METHODS

We design a foundation model-based depth estimation method, referred to as Surgical-DINO, a low-rank adaptation of the DINOv2 for depth estimation in endoscopic surgery. We build LoRA layers and integrate them into DINO to adapt with surgery-specific domain knowledge instead of conventional fine-tuning. During training, we freeze the DINO image encoder, which shows excellent visual representation capacity, and only optimize the LoRA layers and depth decoder to integrate features from the surgical scene.

RESULTS

Our model is extensively validated on a MICCAI challenge dataset of SCARED, which is collected from da Vinci Xi endoscope surgery. We empirically show that Surgical-DINO significantly outperforms all the state-of-the-art models in endoscopic depth estimation tasks. The analysis with ablation studies has shown evidence of the remarkable effect of our LoRA layers and adaptation.

CONCLUSION

Surgical-DINO shed some light on the successful adaptation of the foundation models into the surgical domain for depth estimation. There is clear evidence in the results that zero-shot prediction on pre-trained weights in computer vision datasets or naive fine-tuning is not sufficient to use the foundation model in the surgical domain directly.

Alternating direction method of multipliers for displacement estimation in ultrasound strain elastography

BACKGROUND

Ultrasound strain imaging, which delineates mechanical properties to detect tissue abnormalities, involves estimating the time delay between two radio-frequency (RF) frames collected before and after tissue deformation. The existing regularized optimization-based time-delay estimation (TDE) techniques suffer from at least one of the following drawbacks: (1) The regularizer is not aligned with the tissue deformation physics due to taking only the first-order displacement derivative into account; (2) TheL 2 $L2$ -norm of the displacement derivatives, which oversmooths the estimated time-delay, is utilized as the regularizer; (3) The modulus function defined mathematically should be approximated by a smooth function to facilitate the optimization ofL 1 $L1$ -norm.

PURPOSE

Our purpose is to develop a novel TDE technique that resolves the aforementioned shortcomings of the existing algorithms.

METHODS

Herein, we propose employing the alternating direction method of multipliers (ADMM) for optimizing a novel cost function consisting ofL 2 $L2$ -norm data fidelity term andL 1 $L1$ -norm first- and second-order spatial continuity terms. ADMM empowers the proposed algorithm to use different techniques for optimizing different parts of the cost function and obtain high-contrast strain images with smooth backgrounds and sharp boundaries. We name our technique ADMM for totaL variaTion RegUlarIzation in ultrasound STrain imaging (ALTRUIST). ALTRUIST's efficacy is quantified using absolute error (AE), Structural SIMilarity (SSIM), signal-to-noise ratio (SNR), contrast-to-noise ratio (CNR), and strain ratio (SR) with respect to GLUE, OVERWIND, andL 1 $L1$ -SOUL, three recently published energy-based techniques, and UMEN-Net, a state-of-the-art deep learning-based algorithm. Analysis of variance (ANOVA)-led multiple comparison tests and paired t $t$ -tests at5 % $5\%$ overall significance level were conducted to assess the statistical significance of our findings. The Bonferroni correction was taken into account in all statistical tests. Two simulated layer phantoms, three simulated resolution phantoms, one hard-inclusion simulated phantom, one multi-inclusion simulated phantom, one experimental breast phantom, and three in vivo liver cancer datasets have been used for validation experiments. We have published the ALTRUIST code at http://code.sonography.ai.

RESULTS

ALTRUIST substantially outperforms the four state-of-the-art benchmarks in all validation experiments, both qualitatively and quantitatively. ALTRUIST yields up to573 % ∗ ${573\%}^{*}$ ,41 % ∗ ${41\%}^{*}$ , and51 % ∗ ${51\%}^{*}$ SNR improvements and443 % ∗ ${443\%}^{*}$ ,53 % ∗ ${53\%}^{*}$ , and15 % ∗ ${15\%}^{*}$ CNR improvements overL 1 $L1$ -SOUL, its closest competitor, for simulated, phantom, and in vivo liver cancer datasets, respectively, where the asterisk (*) indicates statistical significance. In addition, ANOVA-led multiple comparison tests and paired t $t$ -tests indicate that ALTRUIST generally achieves statistically significant improvements over GLUE, UMEN-Net, OVERWIND, andL 1 $L1$ -SOUL in terms of AE, SSIM map, SNR, and CNR.

CONCLUSIONS

A novel ultrasonic displacement tracking algorithm named ALTRUIST has been developed. The principal novelty of ALTRUIST is incorporating ADMM for optimizing anL 1 $L1$ -norm regularization-based cost function. ALTRUIST exhibits promising performance in simulation, phantom, and in vivo experiments.

Ultrafast Cardiac Imaging Using Deep Learning for Speckle-Tracking Echocardiography

High-quality ultrafast ultrasound imaging is based on coherent compounding from multiple transmissions of plane waves (PW) or diverging waves (DW). However, compounding results in reduced frame rate, as well as destructive interferences from high-velocity tissue motion if motion compensation (MoCo) is not considered. While many studies have recently shown the interest of deep learning for the reconstruction of high-quality static images from PW or DW, its ability to achieve such performance while maintaining the capability of tracking cardiac motion has yet to be assessed. In this article, we addressed such issue by deploying a complex-weighted convolutional neural network (CNN) for image reconstruction and a state-of-the-art speckle-tracking method. The evaluation of this approach was first performed by designing an adapted simulation framework, which provides specific reference data, i.e., high-quality, motion artifact-free cardiac images. The obtained results showed that, while using only three DWs as input, the CNN-based approach yielded an image quality and a motion accuracy equivalent to those obtained by compounding 31 DWs free of motion artifacts. The performance was then further evaluated on nonsimulated, experimental in vitro data, using a spinning disk phantom. This experiment demonstrated that our approach yielded high-quality image reconstruction and motion estimation, under a large range of velocities and outperforms a state-of-the-art MoCo-based approach at high velocities. Our method was finally assessed on in vivo datasets and showed consistent improvement in image quality and motion estimation compared to standard compounding. This demonstrates the feasibility and effectiveness of deep learning reconstruction for ultrafast speckle-tracking echocardiography.

Exploiting Mechanics-Based Priors for Lateral Displacement Estimation in Ultrasound Elastography

Tracking the displacement between the pre- and post-deformed radio-frequency (RF) frames is a pivotal step of ultrasound elastography, which depicts tissue mechanical properties to identify pathologies. Due to ultrasound's poor ability to capture information pertaining to the lateral direction, the existing displacement estimation techniques fail to generate an accurate lateral displacement or strain map. The attempts made in the literature to mitigate this well-known issue suffer from one of the following limitations: 1) Sampling size is substantially increased, rendering the method computationally and memory expensive. 2) The lateral displacement estimation entirely depends on the axial one, ignoring data fidelity and creating large errors. This paper proposes exploiting the effective Poisson's ratio (EPR)-based mechanical correspondence between the axial and lateral strains along with the RF data fidelity and displacement continuity to improve the lateral displacement and strain estimation accuracies. We call our techniques MechSOUL (Mechanically-constrained Second-Order Ultrasound eLastography) and L1 -MechSOUL ( L1 -norm-based MechSOUL), which optimize L2 - and L1 -norm-based penalty functions, respectively. Extensive validation experiments with simulated, phantom, and in vivo datasets demonstrate that MechSOUL and L1 -MechSOUL's lateral strain and EPR estimation abilities are substantially superior to those of the recently-published elastography techniques. We have published the MATLAB codes of MechSOUL and L1 -MechSOUL at https://code.sonography.ai.

A Data-Efficient Deep Learning Strategy for Tissue Characterization via Quantitative Ultrasound: Zone Training

Deep learning (DL) powered biomedical ultrasound imaging is an emerging research field where researchers adapt the image analysis capabilities of DL algorithms to biomedical ultrasound imaging settings. A major roadblock to wider adoption of DL powered biomedical ultrasound imaging is that acquisition of large and diverse datasets is expensive in clinical settings, which is a requirement for successful DL implementation. Hence, there is a constant need for developing data-efficient DL techniques to turn DL powered biomedical ultrasound imaging into reality. In this work, we develop a data-efficient DL training strategy for classifying tissues based on the ultrasonic backscattered RF data, i.e., quantitative ultrasound (QUS), which we named zone training. In zone training, we propose to divide the complete field of view of an ultrasound image into multiple zones associated with different regions of a diffraction pattern and then, train separate DL networks for each zone. The main advantage of zone training is that it requires less training data to achieve high accuracy. In this work, three different tissue-mimicking phantoms were classified by a DL network. The results demonstrated that zone training can require a factor of 2-3 less training data in low data regime to achieve similar classification accuracies compared to a conventional training strategy.

Lateral Strain Imaging Using Self-Supervised and Physically Inspired Constraints in Unsupervised Regularized Elastography

Convolutional Neural Networks (CNN) have shown promising results for displacement estimation in UltraSound Elastography (USE). Many modifications have been proposed to improve the displacement estimation of CNNs for USE in the axial direction. However, the lateral strain, which is essential in several downstream tasks such as the inverse problem of elasticity imaging, remains a challenge. The lateral strain estimation is complicated since the motion and the sampling frequency in this direction are substantially lower than the axial one, and a lack of carrier signal in this direction. In computer vision applications, the axial and the lateral motions are independent. In contrast, the tissue motion pattern in USE is governed by laws of physics which link the axial and lateral displacements. In this paper, inspired by Hooke's law, we, first propose Physically Inspired ConsTraint for Unsupervised Regularized Elastography (PICTURE), where we impose a constraint on the Effective Poisson's ratio (EPR) to improve the lateral strain estimation. In the next step, we propose self-supervised PICTURE (sPICTURE) to further enhance the strain image estimation. Extensive experiments on simulation, experimental phantom and in vivo data demonstrate that the proposed methods estimate accurate axial and lateral strain maps.

Convolutional Neural Network-Based Speckle Tracking for Ultrasound Strain Elastography: An Unsupervised Learning Approach

Accurate and computationally efficient motion estimation is a critical component of real-time ultrasound strain elastography (USE). With the advent of deep-learning neural network models, a growing body of work has explored supervised convolutional neural network (CNN)-based optical flow in the framework of USE. However, the above-said supervised learning was often done using simulated ultrasound data. The research community has questioned whether simulated ultrasound data containing simple motion can train deep-learning CNN models that can reliably track complex in vivo speckle motion. In parallel with other research groups' efforts, this study developed an unsupervised motion estimation neural network (UMEN-Net) for USE by adapting a well-established CNN model named PWC-Net. Our network's input is a pair of predeformation and postdeformation radio frequency (RF) echo signals. The proposed network outputs both axial and lateral displacement fields. The loss function consists of a correlation between the predeformation signal and the motion-compensated postcompression signal, smoothness of the displacement fields, and tissue incompressibility. Notably, an innovative correlation method known as the globally optimized correspondence (GOCor) volumes module developed by Truong et al. was used to replace the original Corr module to enhance our evaluation of signal correlation. The proposed CNN model was tested using simulated, phantom, and in vivo ultrasound data containing biologically confirmed breast lesions. Its performance was compared against other state-of-the-art methods, including two deep-learning-based tracking methods (MPWC-Net++ and ReUSENet) and two conventional tracking methods (GLUE and BRGMT-LPF). In summary, compared with the four known methods mentioned above, our unsupervised CNN model not only obtained higher signal-to-noise ratios (SNRs) and contrast-to-noise ratios (CNRs) for axial strain estimates but also improved the quality of the lateral strain estimates.

Modified UNet++ with atrous spatial pyramid pooling for blood cell image segmentation

Blood cell image segmentation is an important part of the field of computer-aided diagnosis. However, due to the low contrast, large differences in cell morphology and the scarcity of labeled images, the segmentation performance of cells cannot meet the requirements of an actual diagnosis. To address the above limitations, we present a deep learning-based approach to study cell segmentation on pathological images. Specifically, the algorithm selects UNet++ as the backbone network to extract multi-scale features. Then, the skip connection is redesigned to improve the degradation problem and reduce the computational complexity. In addition, the atrous spatial pyramid pooling (ASSP) is introduced to obtain cell image information features from each layer through different receptive domains. Finally, the multi-sided output fusion (MSOF) strategy is utilized to fuse the features of different semantic levels, so as to improve the accuracy of target segmentation. Experimental results on blood cell images for segmentation and classification (BCISC) dataset show that the proposed method has significant improvement in Matthew's correlation coefficient (Mcc), Dice and Jaccard values, which are better than the classical semantic segmentation network.