Deep Learning for Ultrasound Localization Microscopy

Context-aware deep learning enables high-efficacy localization of high concentration microbubbles for super-resolution ultrasound localization microscopy

Ultrasound localization microscopy (ULM) enables deep tissue microvascular imaging by localizing and tracking intravenously injected microbubbles circulating in the bloodstream. However, conventional localization techniques require spatially isolated microbubbles, resulting in prolonged imaging time to obtain detailed microvascular maps. Here, we introduce LOcalization with Context Awareness (LOCA)-ULM, a deep learning-based microbubble simulation and localization pipeline designed to enhance localization performance in high microbubble concentrations. In silico, LOCA-ULM enhanced microbubble detection accuracy to 97.8% and reduced the missing rate to 23.8%, outperforming conventional and deep learning-based localization methods up to 17.4% in accuracy and 37.6% in missing rate reduction. In in vivo rat brain imaging, LOCA-ULM revealed dense cerebrovascular networks and spatially adjacent microvessels undetected by conventional ULM. We further demonstrate the superior localization performance of LOCA-ULM in functional ULM (fULM) where LOCA-ULM significantly increased the functional imaging sensitivity of fULM to hemodynamic responses invoked by whisker stimulations in the rat brain.

Improving sub-pixel accuracy in ultrasound localization microscopy using supervised and self-supervised deep learning

With a spatial resolution of tens of microns, ultrasound localization microscopy (ULM) reconstructs microvascular structures and measures intravascular flows by tracking microbubbles (1-5 μm) in contrast enhanced ultrasound (CEUS) images. Since the size of CEUS bubble traces, e.g. 0.5-1 mm for ultrasound with a wavelength λ = 280 μm, is typically two orders of magnitude larger than the bubble diameter, accurately localizing microbubbles in noisy CEUS data is vital to the fidelity of the ULM results. In this paper, we introduce a residual learning based supervised super-resolution blind deconvolution network (SupBD-net), and a new loss function for a self-supervised blind deconvolution network (SelfBD-net), for detecting bubble centers at a spatial resolution finer than λ/10. Our ultimate purpose is to improve the ability to distinguish closely located microvessels and the accuracy of the velocity profile measurements in macrovessels. Using realistic synthetic data, the performance of these methods is calibrated and compared against several recently introduced deep learning and blind deconvolution techniques. For bubble detection, errors in bubble center location increase with the trace size, noise level, and bubble concentration. For all cases, SupBD-net yields the least error, keeping it below 0.1 λ. For unknown bubble trace morphology, where all the supervised learning methods fail, SelfBD-net can still maintain an error of less than 0.15 λ. SupBD-net also outperforms the other methods in separating closely located bubbles and parallel microvessels. In macrovessels, SupBD-net maintains the least errors in the vessel radius and velocity profile after introducing a procedure that corrects for terminated tracks caused by overlapping traces. Application of these methods is demonstrated by mapping the cerebral microvasculature of a neonatal pig, where neighboring microvessels separated by 0.15 λ can be readily distinguished by SupBD-net and SelfBD-net, but not by the other techniques. Hence, the newly proposed residual learning based methods improve the spatial resolution and accuracy of ULM in micro- and macro-vessels.

Deep learning for fast super-resolution ultrasound microvessel imaging

Objective. Ultrasound localization microscopy (ULM) enables microvascular reconstruction by localizing microbubbles (MBs). Although ULM can obtain microvascular images that are beyond the ultimate resolution of the ultrasound (US) diffraction limit, it requires long data processing time, and the imaging accuracy is susceptible to the density of MBs. Deep learning (DL)-based ULM is proposed to alleviate these limitations, which simulated MBs at low-resolution and mapped them to coordinates at high-resolution by centroid localization. However, traditional DL-based ULMs are imprecise and computationally complex. Also, the performance of DL is highly dependent on the training datasets, which are difficult to realistically simulate.Approach. A novel architecture called adaptive matching network (AM-Net) and a dataset generation method named multi-mapping (MMP) was proposed to overcome the above challenges. The imaging performance and processing time of the AM-Net have been assessed by simulation andin vivoexperiments.Main results. Simulation results show that at high density (20 MBs/frame), when compared to other DL-based ULM, AM-Net achieves higher localization accuracy in the lateral/axial direction.In vivoexperiment results show that the AM-Net can reconstruct ∼24.3μm diameter micro-vessels and separate two ∼28.3μm diameter micro-vessels. Furthermore, when processing a 128 × 128 pixels image in simulation experiments and an 896 × 1280 pixels imagein vivoexperiment, the processing time of AM-Net is ∼13 s and ∼33 s, respectively, which are 0.3-0.4 orders of magnitude faster than other DL-based ULM.Significance. We proposes a promising solution for ULM with low computing costs and high imaging performance.

Three-dimensional super resolution ultrasound imaging with a multi-frequency hemispherical phased array

BACKGROUND

High resolution imaging of the microvasculature plays an important role in both diagnostic and therapeutic applications in the brain. However, ultrasound pulse-echo sonography imaging the brain vasculatures has been limited to narrow acoustic windows and low frequencies due to the distortion of the skull bone, which sacrifices axial resolution since it is pulse length dependent.

PURPOSE

To overcome the detect limit, a large aperture 256-module sparse hemispherical transmit/receive array was used to visualize the acoustic emissions of ultrasound-vaporized lipid-coated decafluorobutane nanodroplets flowing through tube phantoms and within rabbit cerebral vasculature in vivo via passive acoustic mapping and super resolution techniques.

METHODS

Nanodroplets were vaporized with 55 kHz burst-mode ultrasound (burst length = 145 μs, burst repetition frequency = 9-45 Hz, peak negative acoustic pressure = 0.10-0.22 MPa), which propagates through overlying tissues well without suffering from severe distortions. The resulting emissions were received at a higher frequency (612 or 1224 kHz subarray) to improve the resulting spatial resolution during passive beamforming. Normal resolution three-dimensional images were formed using a delay, sum, and integrate beamforming algorithm, and super-resolved images were extracted via Gaussian fitting of the estimated point-spread-function to the normal resolution data.

RESULTS

With super resolution techniques, the mean lateral (axial) full-width-at-half-maximum image intensity was 16 ± 3 (32 ± 6) μm, and 7 ± 1 (15 ± 2) μm corresponding to ∼1/67 of the normal resolution at 612 and 1224 kHz, respectively. The mean positional uncertainties were ∼1/350 (lateral) and ∼1/180 (axial) of the receive wavelength in water. In addition, a temporal correlation between nanodroplet vaporization and the transmit waveform shape was observed, which may provide the opportunity to enhance the signal-to-noise ratio in future studies.

CONCLUSIONS

Here, we demonstrate the feasibility of vaporizing nanodroplets via low frequency ultrasound and simultaneously performing spatial mapping via passive beamforming at higher frequencies to improve the resulting spatial resolution of super resolution imaging techniques. This method may enable complete four-dimensional vascular mapping in organs where a hemispherical array could be positioned to surround the target, such as the brain, breast, or testicles.

Deep learning for fast denoising filtering in ultrasound localization microscopy

Objective.Addition of a denoising filter step in ultrasound localization microscopy (ULM) has been shown to effectively reduce the error localizations of microbubbles (MBs) and achieve resolution improvement for super-resolution ultrasound (SR-US) imaging. However, previous image-denoising methods (e.g. block-matching 3D, BM3D) requires long data processing times, making ULM only able to be processed offline. This work introduces a new way to reduce data processing time through deep learning.Approach.In this study, we propose deep learning (DL) denoising based on contrastive semi-supervised network (CS-Net). The neural network is mainly trained with simulated MBs data to extract MB signals from noise. And the performances of CS-Net denoising are evaluated in bothin vitroflow phantom experiment andin vivoexperiment of New Zealand rabbit tumor.Main results.Forin vitroflow phantom experiment, the signal-to-noise ratio (SNR) and contrast-to-noise ratio (CNR) of single microbubble image are 26.91 dB and 4.01 dB, repectively. Forin vivoanimal experiment , the SNR and CNR were 12.29 dB and 6.06 dB. In addition, single microvessel of 24μm and two microvessels separated by 46μm could be clearly displayed. Most importantly,, the CS-Net denoising speeds forin vitroandin vivoexperiments were 0.041 s frame-1and 0.062 s frame-1, respectively.Significance.DL denoising based on CS-Net can improve the resolution of SR-US as well as reducing denoising time, thereby making further contributions to the clinical real-time imaging of ULM.

Nanodroplet-Based Super-Resolution Ultrasound Localization Microscopy

Over the past decade, super-resolution ultrasound localization microscopy (SR-ULM) has revolutionized ultrasound imaging with its capability to resolve the microvascular structures below the ultrasound diffraction limit. The introduction of this imaging technique enables the visualization, quantification, and characterization of tissue microvasculature. The early implementations of SR-ULM utilize microbubbles (MBs) that require a long image acquisition time due to the requirement of capturing sparsely isolated microbubble signals. The next-generation SR-ULM employs nanodroplets that have the potential to significantly reduce the image acquisition time without sacrificing the resolution. This review discusses various nanodroplet-based ultrasound localization microscopy techniques and their corresponding imaging mechanisms. A summary is given on the preclinical applications of SR-ULM with nanodroplets, and the challenges in the clinical translation of nanodroplet-based SR-ULM are presented while discussing the future perspectives. In conclusion, ultrasound localization microscopy is a promising microvasculature imaging technology that can provide new diagnostic and prognostic information for a wide range of pathologies, such as cancer, heart conditions, and autoimmune diseases, and enable personalized treatment monitoring at a microlevel.

Localization Free Super-Resolution Microbubble Velocimetry Using a Long Short-Term Memory Neural Network

Ultrasound localization microscopy is a super-resolution imaging technique that exploits the unique characteristics of contrast microbubbles to side-step the fundamental trade-off between imaging resolution and penetration depth. However, the conventional reconstruction technique is confined to low microbubble concentrations to avoid localization and tracking errors. Several research groups have introduced sparsity- and deep learning-based approaches to overcome this constraint to extract useful vascular structural information from overlapping microbubble signals, but these solutions have not been demonstrated to produce blood flow velocity maps of the microcirculation. Here, we introduce Deep-SMV, a localization free super-resolution microbubble velocimetry technique, based on a long short-term memory neural network, that provides high imaging speed and robustness to high microbubble concentrations, and directly outputs blood velocity measurements at a super-resolution. Deep-SMV is trained efficiently using microbubble flow simulation on real in vivo vascular data and demonstrates real-time velocity map reconstruction suitable for functional vascular imaging and pulsatility mapping at super-resolution. The technique is successfully applied to a wide variety of imaging scenarios, include flow channel phantoms, chicken embryo chorioallantoic membranes, and mouse brain imaging. An implementation of Deep-SMV is openly available at https://github.com/chenxiptz/SR_microvessel_velocimetry, with two pre-trained models available at https://doi.org/10.7910/DVN/SECUFD.

Super-resolution ultrasound microvascular imaging: Is it ready for clinical use?

The field of super-resolution ultrasound microvascular imaging has been rapidly growing over the past decade. By leveraging contrast microbubbles as point targets for localization and tracking, super-resolution ultrasound pinpoints the location of microvessels and measures their blood flow velocity. Super-resolution ultrasound is the first in vivo imaging modality that can image micron-scale vessels at a clinically relevant imaging depth without tissue destruction. These unique capabilities of super-resolution ultrasound provide structural (vessel morphology) and functional (vessel blood flow) assessments of tissue microvasculature on a global and local scale, which opens new doors for many enticing preclinical and clinical applications that benefit from microvascular biomarkers. The goal of this short review is to provide an update on recent advancements in super-resolution ultrasound imaging, with a focus on summarizing existing applications and discussing the prospects of translating super-resolution imaging to clinical practice and research. In this review, we also provide brief introductions of how super-resolution ultrasound works, how does it compare with other imaging modalities, and what are the tradeoffs and limitations for an audience who is not familiar with the technology.

Deep learning algorithm for predicting subacromial motion trajectory: Dynamic shoulder ultrasound analysis

Subacromial motion metrics can be extracted from dynamic shoulder ultrasonography, which is useful for identifying abnormal motion patterns in painful shoulders. However, frame-by-frame manual labeling of anatomical landmarks in ultrasound images is time consuming. The present study aims to investigate the feasibility of a deep learning algorithm for extracting subacromial motion metrics from dynamic ultrasonography. Dynamic ultrasound imaging was retrieved by asking 17 participants to perform cyclic shoulder abduction and adduction along the scapular plane, whereby the trajectory of the humeral greater tubercle (in relation to the lateral acromion) was depicted by the deep learning algorithm. Extraction of the subacromial motion metrics was conducted using a convolutional neural network (CNN) or a self-transfer learning-based (STL)-CNN with or without an autoencoder (AE). The mean absolute error (MAE) compared with the manually-labeled data (ground truth) served as the main outcome variable. Using eight-fold cross-validation, the average MAE was proven to be significantly higher in the group using CNN than in those using STL-CNN or STL-CNN+AE for the relative difference between the greater tubercle and lateral acromion on the horizontal axis. The MAE for the localization of the two aforementioned landmarks on the vertical axis also seemed to be enlarged in those using CNN compared with those using STL-CNN. In the testing dataset, the errors in relation to the ground truth for the minimal vertical acromiohumeral distance were 0.081-0.333 cm using CNN, compared with 0.002-0.007 cm using STL-CNN. We successfully demonstrated the feasibility of a deep learning algorithm for automatic detection of the greater tubercle and lateral acromion during dynamic shoulder ultrasonography. Our framework also demonstrated the capability of capturing the minimal vertical acromiohumeral distance, which is the most important indicator of subacromial motion metrics in daily clinical practice.

In vivo ocular microvasculature imaging in rabbits with 3D ultrasound localization microscopy

Morphological and hemodynamic changes in the ocular vasculature are important signs of various ocular diseases. The evaluation of the ocular microvasculature with high resolution is valuable in comprehensive diagnoses. However, it is difficult for current optical imaging techniques to visualize the posterior segment and retrobulbar microvasculature due to the limited penetration depth of light, particularly when the refractive medium is opaque. Thus, we have developed a 3D ultrasound localization microscopy (ULM) imaging method to visualize the ocular microvasculature in rabbits with micron-scale resolution. We used a 32 × 32 matrix array transducer (center frequency: 8 MHz) with a compounding plane wave sequence and microbubbles. Block-wise singular value decomposition spatiotemporal clutter filtering and block-matching 3D denoising were implemented to extract the flowing microbubble signals at different imaging depths with high signal-to-noise ratios. The center points of microbubbles were localized and tracked in 3D space to achieve the micro-angiography. The in vivo results demonstrate the ability of 3D ULM to visualize the microvasculature of the eye in rabbits, where vessels down to 54 μm were successfully revealed. Moreover, the microvascular maps indicated the morphological abnormalities in the eye with retinal detachment. This efficient modality shows potential for use in the diagnosis of ocular diseases.

A Data-Efficient Deep Learning Strategy for Tissue Characterization via Quantitative Ultrasound: Zone Training

Deep learning (DL) powered biomedical ultrasound imaging is an emerging research field where researchers adapt the image analysis capabilities of DL algorithms to biomedical ultrasound imaging settings. A major roadblock to wider adoption of DL powered biomedical ultrasound imaging is that acquisition of large and diverse datasets is expensive in clinical settings, which is a requirement for successful DL implementation. Hence, there is a constant need for developing data-efficient DL techniques to turn DL powered biomedical ultrasound imaging into reality. In this work, we develop a data-efficient DL training strategy for classifying tissues based on the ultrasonic backscattered RF data, i.e., quantitative ultrasound (QUS), which we named zone training. In zone training, we propose to divide the complete field of view of an ultrasound image into multiple zones associated with different regions of a diffraction pattern and then, train separate DL networks for each zone. The main advantage of zone training is that it requires less training data to achieve high accuracy. In this work, three different tissue-mimicking phantoms were classified by a DL network. The results demonstrated that zone training can require a factor of 2-3 less training data in low data regime to achieve similar classification accuracies compared to a conventional training strategy.

Laser diode photoacoustic point source detection: machine learning-based denoising and reconstruction

A new development in photoacoustic (PA) imaging has been the use of compact, portable and low-cost laser diodes (LDs), but LD-based PA imaging suffers from low signal intensity recorded by the conventional transducers. A common method to improve signal strength is temporal averaging, which reduces frame rate and increases laser exposure to patients. To tackle this problem, we propose a deep learning method that will denoise point source PA radio-frequency (RF) data before beamforming with a very few frames, even one. We also present a deep learning method to automatically reconstruct point sources from noisy pre-beamformed data. Finally, we employ a strategy of combined denoising and reconstruction, which can supplement the reconstruction algorithm for very low signal-to-noise ratio inputs.

Deep learning using a residual deconvolutional network enables real-time high-density single-molecule localization microscopy

High-density localization based on deep learning is a very effective method to accelerate single molecule localization microscopy (SMLM). Compared with traditional high-density localization methods, deep learning-based methods enable a faster data processing speed and a higher localization accuracy. However, the reported high-density localization methods based on deep learning are still not fast enough to enable real time data processing for large batches of raw images, which is probably due to the heavy computational burden and computation complexity in the U-shape architecture used in these models. Here we propose a high-density localization method called FID-STORM, which is based on an improved residual deconvolutional network for the real-time processing of raw images. In FID-STORM, we use a residual network to extract the features directly from low-resolution raw images rather than the U-shape network from interpolated images. We also use a model fusion from TensorRT to further accelerate the inference of the model. In addition, we process the sum of the localization images directly on GPU to obtain an additional speed gain. Using simulated and experimental data, we verified that the FID-STORM method achieves a processing speed of 7.31 ms/frame at 256 × 256 pixels @ Nvidia RTX 2080 Ti graphic card, which is shorter than the typical exposure time of 10∼30 ms, thus enabling real-time data processing in high-density SMLM. Moreover, compared with a popular interpolated image-based method called Deep-STORM, FID-STORM enables a speed gain of ∼26 times, without loss of reconstruction accuracy. We also provided an ImageJ plugin for our new method.

Ultrasound Signal Processing: From Models to Deep Learning

Medical ultrasound imaging relies heavily on high-quality signal processing to provide reliable and interpretable image reconstructions. Conventionally, reconstruction algorithms have been derived from physical principles. These algorithms rely on assumptions and approximations of the underlying measurement model, limiting image quality in settings where these assumptions break down. Conversely, more sophisticated solutions based on statistical modeling or careful parameter tuning or derived from increased model complexity can be sensitive to different environments. Recently, deep learning-based methods, which are optimized in a data-driven fashion, have gained popularity. These model-agnostic techniques often rely on generic model structures and require vast training data to converge to a robust solution. A relatively new paradigm combines the power of the two: leveraging data-driven deep learning and exploiting domain knowledge. These model-based solutions yield high robustness and require fewer parameters and training data than conventional neural networks. In this work we provide an overview of these techniques from the recent literature and discuss a wide variety of ultrasound applications. We aim to inspire the reader to perform further research in this area and to address the opportunities within the field of ultrasound signal processing. We conclude with a future perspective on model-based deep learning techniques for medical ultrasound.

Ultrafast Ultrasound Localization Microscopy by Conditional Generative Adversarial Network

Ultrasound localization microscopy (ULM) overcomes the acoustic diffraction limit and enables the visualization of microvasculature at subwavelength resolution. However, challenges remain in ultrafast ULM implementation, where short data acquisition time, efficient data processing speed, and high imaging resolution need to be considered simultaneously. Recently, deep learning (DL)-based methods have exhibited potential in speeding up ULM imaging. Nevertheless, a certain number of ultrasound (US) data ( L frames) are still required to accumulate enough localized microbubble (MB) events, leading to an acquisition time within a time span of tens of seconds. To further speed up ULM imaging, in this article, we present a new DL-based method, termed as ULM-GAN. By using a modified conditional generative adversarial network (cGAN) framework, ULM-GAN is able to reconstruct a superresolution image directly from a temporal mean low-resolution (LR) image generated by averaging l -frame raw US images with l being significantly smaller than L . To evaluate the performance of ULM-GAN, a series of numerical simulations and phantom experiments are both implemented. The results of the numerical simulations demonstrate that when performing ULM imaging, ULM-GAN allows  ∼ 40 -fold reduction in data acquisition time and  ∼ 61 -fold reduction in computational time compared with the conventional Gaussian fitting method, without compromising spatial resolution according to the resolution scaled error (RSE). For the phantom experiments, ULM-GAN offers an implementation of ULM with ultrafast data acquisition time (  ∼ 0.33 s) and ultrafast data processing speed (  ∼ 0.60 s) that makes it promising to observe rapid biological activities in vivo.

Three-Dimensional Ultrasound Localization Microscopy with Bipartite Graph-Based Microbubble Pairing and Kalman-Filtering-Based Tracking on a 256-Channel Verasonics Ultrasound System with a 32 × 32 Matrix Array

Three-dimensional (3D) ultrasound localization microscopy (ULM) using a 2-D matrix probe and microbubbles (MBs) has been recently proposed to visualize microvasculature beyond the ultrasound diffraction limit in three spatial dimensions. However, 3D ULM suffers from several limitations: (1) high system complexity due to numerous channel counts, (2) complex MB flow dynamics in 3D, and (3) extremely long acquisition time. To reduce the system complexity while maintaining high image quality, we used a sub-aperture process to reduce received channel counts. To address the second issue, a 3D bipartite graph-based method with Kalman filtering-based tracking was used in this study for MB tracking. An MB separation approach was incorporated to separate high concentration MB data into multiple, sparser MB datasets, allowing better MB localization and tracking for a limited acquisition time. The proposed method was first validated in a flow channel phantom, showing improved spatial resolutions compared with the contrasted enhanced power Doppler image. Then the proposed method was evaluated with an in vivo chicken embryo brain dataset. Results showed that the reconstructed 3D super-resolution image achieved a spatial resolution of around 52 μm (smaller than the wavelength of around 200 μm). Microvessels that cannot be resolved clearly using localization only, can be well identified with the tailored 3D pairing and tracking algorithms. To sum up, the feasibility of the 3D ULM is shown, indicating the great possibility in clinical applications.

Curvelet Transform-Based Sparsity Promoting Algorithm for Fast Ultrasound Localization Microscopy

Ultrasound localization microscopy (ULM) based on microbubble (MB) localization was recently introduced to overcome the resolution limit of conventional ultrasound. However, ULM is currently challenged by the requirement for long data acquisition times to accumulate adequate MB events to fully reconstruct vasculature. In this study, we present a curvelet transform-based sparsity promoting (CTSP) algorithm that improves ULM imaging speed by recovering missing MB localization signal from data with very short acquisition times. CTSP was first validated in a simulated microvessel model, followed by the chicken embryo chorioallantoic membrane (CAM), and finally, in the mouse brain. In the simulated microvessel study, CTSP robustly recovered the vessel model to achieve an 86.94% vessel filling percentage from a corrupted image with only 4.78% of the true vessel pixels. In the chicken embryo CAM study, CTSP effectively recovered the missing MB signal within the vasculature, leading to marked improvement in ULM imaging quality with a very short data acquisition. Taking the optical image as reference, the vessel filling percentage increased from 2.7% to 42.2% using 50ms of data acquisition after applying CTSP. CTSP used 80% less time to achieve the same 90% maximum saturation level as compared with conventional MB localization. We also applied CTSP on the microvessel flow speed maps and found that CTSP was able to use only 1.6s of microbubble data to recover flow speed images that have similar qualities as those constructed using 33.6s of data. In the mouse brain study, CTSP was able to reconstruct the majority of the cerebral vasculature using 1-2s of data acquisition. Additionally, CTSP only needed 3.2s of microbubble data to generate flow velocity maps that are comparable to those using 129.6s of data. These results suggest that CTSP can facilitate fast and robust ULM imaging especially under the circumstances of inadequate microbubble localizations.

Super-Resolution Ultrasound Through Sparsity-Based Deconvolution and Multi-Feature Tracking

Ultrasound super-resolution imaging through localisation and tracking of microbubbles can achieve sub-wave-diffraction resolution in mapping both micro-vascular structure and flow dynamics in deep tissue in vivo. Currently, it is still challenging to achieve high accuracy in localisation and tracking particularly with limited imaging frame rates and in the presence of high bubble concentrations. This study introduces microbubble image features into a Kalman tracking framework, and makes the framework compatible with sparsity-based deconvolution to address these key challenges. The performance of the method is evaluated on both simulations using individual bubble signals segmented from in vivo data and experiments on a mouse brain and a human lymph node. The simulation results show that the deconvolution not only significantly improves the accuracy of isolating overlapping bubbles, but also preserves some image features of the bubbles. The combination of such features with Kalman motion model can achieve a significant improvement in tracking precision at a low frame rate over that using the distance measure, while the improvement is not significant at the highest frame rate. The in vivo results show that the proposed framework generates SR images that are significantly different from the current methods with visual improvement, and is more robust to high bubble concentrations and low frame rates.

Deep learning for dose assessment in radiotherapy by the super-localization of vaporized nanodroplets in high frame rate ultrasound imaging

Objective. External beam radiotherapy is aimed to precisely deliver a high radiation dose to malignancies, while optimally sparing surrounding healthy tissues. With the advent of increasingly complex treatment plans, the delivery should preferably be verified by quality assurance methods. Recently, online ultrasound imaging of vaporized radiosensitive nanodroplets was proposed as a promising tool for in vivo dosimetry in radiotherapy. Previously, the detection of sparse vaporization events was achieved by applying differential ultrasound (US) imaging followed by intensity thresholding using subjective parameter tuning, which is sensitive to image artifacts. Approach. A generalized deep learning solution (i.e. BubbleNet) is proposed to localize vaporized nanodroplets on differential US frames, while overcoming the aforementioned limitation. A 5-fold cross-validation was performed on a diversely composed 5747-frame training/validation dataset by manual segmentation. BubbleNet was then applied on a test dataset of 1536 differential US frames to evaluate dosimetric features. The intra-observer variability was determined by scoring the Dice similarity coefficient (DSC) on 150 frames segmented twice. Additionally, the BubbleNet generalization capability was tested on an external test dataset of 432 frames acquired by a phased array transducer at a much lower ultrasound frequency and reconstructed with unconventional pixel dimensions with respect to the training dataset. Main results. The median DSC in the 5-fold cross validation was equal to ∼0.88, which was in line with the intra-observer variability (=0.86). Next, BubbleNet was employed to detect vaporizations in differential US frames obtained during the irradiation of phantoms with a 154 MeV proton beam or a 6 MV photon beam. BubbleNet improved the bubble-count statistics by ∼30% compared to the earlier established intensity-weighted thresholding. The proton range was verified with a −0.8 mm accuracy. Significance. BubbleNet is a flexible tool to localize individual vaporized nanodroplets on experimentally acquired US images, which improves the sensitivity compared to former thresholding-weighted methods.

Deep Learning-Based Microbubble Localization for Ultrasound Localization Microscopy

Ultrasound localization microscopy (ULM) is an emerging vascular imaging technique that overcomes the resolution-penetration compromise of ultrasound imaging. Accurate and robust microbubble (MB) localization is essential for successful ULM. In this study, we present a deep learning (DL)-based localization technique that uses both Field-II simulation and in vivo chicken embryo chorioallantoic membrane (CAM) data for training. Both radio frequency (RF) and in-phase and quadrature (IQ) data were tested in this study. The simulation experiment shows that the proposed DL-based localization was able to reduce both missing MB localization rate and MB localization error. In general, RF data showed better performance than IQ. For the in vivo CAM study with high MB concentration, DL-based localization was able to reduce the vessel MB saturation time by more than 50% compared to conventional localization. In addition, we propose a DL-based framework for real-time visualization of the high-resolution microvasculature. The findings of this article support the use of DL for more robust and faster MB localization, especially under high MB concentrations. The results indicate that further improvement could be achieved by incorporating temporal information of the MB data.

Subwavelength ultrasonic imaging using a deep convolutional neural network trained on structural noise

Subwavelength ultrasonic imaging (SUI) can detect subwavelength flaws beyond the diffraction limit, however, SUI sometimes fails to clearly reveal flaws in C-scans when the signal-to-noise ratio (SNR) is low. In this work, a convolutional neural network (CNN) that takes structural noise into account is developed for SUI to distinguish flaw echoes from structural noise. The network contains a regression CNN for learning features from the structural noise and a learnable soft thresholding layer for classification. Experiments show that the proposed method performs well for imaging subwavelength flaws at different depths and of different sizes. It achieved an F₁ score of 97.69 ± 1.56% in detecting flaws as compared to the enhanced ultrasonic flaw detection method with time-dependent threshold. As an example of general application of the method, we also performed SUI on natural flaws in a spheroidal graphite cast iron specimen. The results show that the method can achieve SUI without a theoretical backscattering model and is not limited by noise distribution, multiple scattering, or complex microstructures. Furthermore, the network does not need to prepare flaw echoes for training.

ApodNet: Learning for High Frame Rate Synthetic Transmit Aperture Ultrasound Imaging

Two-way dynamic focusing in synthetic transmit aperture (STA) beamforming can benefit high-quality ultrasound imaging with higher lateral spatial resolution and contrast resolution. However, STA requires the complete dataset for beamforming in a relatively low frame rate and transmit power. This paper proposes a deep-learning architecture to achieve high frame rate STA imaging with two-way dynamic focusing. The network consists of an encoder and a joint decoder. The encoder trains a set of binary weights as the apodizations of the high-frame-rate plane wave transmissions. In this respect, we term our network ApodNet. The decoder can recover the complete dataset from the acquired channel data to achieve dynamic transmit focusing. We evaluate the proposed method by simulations at different levels of noise and in-vivo experiments on the human biceps brachii and common carotid artery. The experimental results demonstrate that ApodNet provides a promising strategy for high frame rate STA imaging, obtaining comparable lateral resolution and contrast resolution with four-times higher frame rate than conventional STA imaging in the in-vivo experiments. Particularly, ApodNet improves contrast resolution of the hypoechoic targets with much shorter computational time when compared with other high-frame-rate methods in both simulations and in-vivo experiments.

Ultrasound microvasculature imaging with entropy-based radiality super-resolution (ERSR)

Objective:Microvasculature is highly relevant to the occurrence and development of pathologies such as cancer and diabetes. Ultrasound localization microscopy (ULM) has bypassed the diffraction limit and demonstrated its great potential to provide new imaging modality and establish new diagnostic criteria in clinical application. However, sparse microbubble distribution can be a significant bottleneck for improving temporal resolution, even for further clinical translation. Other important challenges for ULM to tackle in clinic also include high microbubble concentration and low frame rate.Approach:As part of the efforts to facilitate clinical translation, this paper focused on the low frame rate and the high microbubble distribution issue and proposed a new super-resolution imaging strategy called entropy-based radiality super-resolution (ERSR). The feasibility of ERSR is validated with simulations, phantom experiment and contrast-enhanced ultrasound scan of rabbit sciatic nerve with clinical accessible ultrasound system.Main results:ERSR can achieve 10 times improvement in spatial resolution compared to conventional ultrasound imaging, higher temporal resolution (∼10 times higher) and contrast-to-noise ratio under high-density microbubbles, compared with ULM under low-density microbubbles.Significance:We conclude ERSR could be a valuable imaging tool with high spatio-temporal resolution for clinical diagnosis and assessment of diseases potentially.

Faster super-resolution ultrasound imaging with a deep learning model for tissue decluttering and contrast agent localization

Super-resolution ultrasound (SR-US) imaging allows visualization of microvascular structures as small as tens of micrometers in diameter. However, use in the clinical setting has been impeded in part by ultrasound (US) acquisition times exceeding a breath-hold and by the need for extensive offline computation. Deep learning techniques have been shown to be effective in modeling the two more computationally intensive steps of microbubble (MB) contrast agent detection and localization. Performance gains by deep networks over conventional methods are more than two orders of magnitude and in addition the networks can localize overlapping MBs. The ability to separate overlapping MBs allows use of higher contrast agent concentrations and reduces US image acquisition time. Herein we propose a fully convolutional neural network (CNN) architecture to perform the operations of MB detection as well as localization in a single model. Termed SRUSnet, the network is based on the MobileNetV3 architecture modified for 3-D input data, minimal convergence time, and high-resolution data output using a flexible regression head. Also, we propose to combine linear B-mode US imaging and nonlinear contrast pulse sequencing (CPS) which has been shown to increase MB detection and further reduce the US image acquisition time. The network was trained within silicodata and tested onin vitrodata from a tissue-mimicking flow phantom, and onin vivodata from the rat hind limb (N = 3). Images were collected with a programmable US system (Vantage 256, Verasonics Inc., Kirkland, WA) using an L11-4v linear array transducer. The network exceeded 99.9% detection accuracy onin silicodata. The average localization accuracy was smaller than the resolution of a pixel (i.e.λ/8). The average processing time on a Nvidia GeForce 2080Ti GPU was 64.5 ms for a 128 × 128-pixel image.

In Vivo Motion Correction in Super-Resolution Imaging of Rat Kidneys

Super-resolution (SR) imaging has the potential of visualizing the microvasculature down to the 10- [Formula: see text] level, but motion induced by breathing, heartbeats, and muscle contractions are often significantly above this level. This article, therefore, introduces a method for estimating tissue motion and compensating for this. The processing pipeline is described and validated using Field II simulations of an artificial kidney. In vivo measurements were conducted using a modified bk5000 research scanner (BK Medical, Herlev, Denmark) with a BK 9009 linear array probe employing a pulse amplitude modulation scheme. The left kidney of ten Sprague-Dawley rats was scanned during open laparotomy. A 1:10 diluted SonoVue contrast agent (Bracco, Milan, Italy) was injected through a jugular vein catheter at 100 [Formula: see text]/min. Motion was estimated using speckle tracking and decomposed into contributions from the heartbeats, breathing, and residual motion. The estimated peak motions and their precisions were: heart: axial- [Formula: see text] and lateral- [Formula: see text], breathing: axial- [Formula: see text] and lateral- [Formula: see text], and residual: axial-30 [Formula: see text] and lateral-90 [Formula: see text]. The motion corrected microbubble tracks yielded SR images of both bubble density and blood vector velocity. The estimation was, thus, sufficiently precise to correct shifts down to the 10- [Formula: see text] capillary level. Similar results were found in the other kidney measurements with a restoration of resolution for the small vessels demonstrating that motion correction in 2-D can enhance SR imaging quality.

A Deep Learning Framework for Spatiotemporal Ultrasound Localization Microscopy

Ultrasound Localization Microscopy (ULM) can resolve the microvascular bed down to a few micrometers. To achieve such performance, microbubble contrast agents must perfuse the entire microvascular network. Microbubbles are then located individually and tracked over time to sample individual vessels, typically over hundreds of thousands of images. To overcome the fundamental limit of diffraction and achieve a dense reconstruction of the network, low microbubble concentrations must be used, which leads to acquisitions lasting several minutes. Conventional processing pipelines are currently unable to deal with interference from multiple nearby microbubbles, further reducing achievable concentrations. This work overcomes this problem by proposing a Deep Learning approach to recover dense vascular networks from ultrasound acquisitions with high microbubble concentrations. A realistic mouse brain microvascular network, segmented from 2-photon microscopy, was used to train a three-dimensional convolutional neural network (CNN) based on a V-net architecture. Ultrasound data sets from multiple microbubbles flowing through the microvascular network were simulated and used as ground truth to train the 3D CNN to track microbubbles. The 3D-CNN approach was validated in silico using a subset of the data and in vivo in a rat brain. In silico, the CNN reconstructed vascular networks with higher precision (81%) than a conventional ULM framework (70%). In vivo, the CNN could resolve micro vessels as small as 10 μ m with an improvement in resolution when compared against a conventional approach.

Super-resolution ultrasound localization microscopy based on a high frame-rate clinical ultrasound scanner: an in-human feasibility study

Non-invasive detection of microvascular alterations in deep tissuesin vivoprovides critical information for clinical diagnosis and evaluation of a broad-spectrum of pathologies. Recently, the emergence of super-resolution ultrasound localization microscopy (ULM) offers new possibilities for clinical imaging of microvasculature at capillary level. Currently, the clinical utility of ULM on clinical ultrasound scanners is hindered by the technical limitations, such as long data acquisition time, high microbubble (MB) concentration, and compromised tracking performance associated with low imaging frame-rate. Here we present a robust in-human ULM on a high frame-rate (HFR) clinical ultrasound scanner to achieve super-resolution microvessel imaging using a short acquisition time (<10 s). Ultrasound MB data were acquired from different human tissues, including a healthy liver and a diseased liver with acute-on-chronic liver failure, a kidney, a pancreatic tumor, and a breast mass using an HFR clinical scanner. By leveraging the HFR and advanced processing techniques including sub-pixel motion registration, MB signal separation, and Kalman filter-based tracking, MBs can be robustly localized and tracked for ULM under the circumstances of relatively high MB concentration associated with standard clinical MB administration and limited data acquisition time in humans. Subtle morphological and hemodynamic information in microvasculature were shown based on data acquired with single breath-hold and free-hand scanning. Compared with contrast-enhanced power Doppler generated based on the same MB dataset, ULM showed a 5.7-fold resolution improvement in a vessel based on a linear transducer, and provided a wide-range blood flow speed measurement that is Doppler angle-independent. Microvasculatures with complex hemodynamics can be well-differentiated at super-resolution in both normal and pathological tissues. This preliminary study implemented the ultrafast in-human ULM in various human tissues based on a clinical scanner that supports HFR imaging, indicating the potentials of the technique for various clinical applications. However, rigorous validation of the technique in imaging human microvasculature (especially for those tiny vessel structure), preferably with a gold standard, is still required.

Compressed Sensing-Based Super-Resolution Ultrasound Imaging for Faster Acquisition and High Quality Images

GOAL

Typical SRUS images are reconstructed by localizing ultrasound microbubbles (MBs) injected in a vessel using normalized 2-dimensional cross-correlation (2DCC) between MBs signals and the point spread function of the system. However, current techniques require isolated MBs in a confined area due to inaccurate localization of densely populated MBs. To overcome this limitation, we developed the ℓ1-homotopy based compressed sensing (L1H-CS) based SRUS imaging technique which localizes densely populated MBs to visualize microvasculature in vivo.

METHODS

To evaluate the performance of L1H-CS, we compared the performance of 2DCC, interior-point method based compressed sensing (CVX-CS), and L1H-CS algorithms. Localization efficiency was compared using axially and laterally aligned point targets (PTs) with known distances and randomly distributed PTs generated by simulation. We developed post-processing techniques including clutter reduction, noise equalization, motion compensation, and spatiotemporal noise filtering for in vivo imaging. We then validated the capabilities of L1H-CS based SRUS imaging technique with high-density MBs in a mouse tumor model, kidney, and zebrafish dorsal trunk, and brain.

RESULTS

Compared to 2DCC and CVX-CS algorithms, L1H-CS achieved faster data acquisition time and considerable improvement in SRUS image quality.

CONCLUSIONS AND SIGNIFICANCE

These results demonstrate that the L1H-CS based SRUS imaging technique has the potential to examine microvasculature with reduced acquisition and reconstruction time to acquire enhanced SRUS image quality, which may be necessary to translate it into clinics.

Current Development and Applications of Super-Resolution Ultrasound Imaging

Abnormal changes of the microvasculature are reported to be key evidence of the development of several critical diseases, including cancer, progressive kidney disease, and atherosclerotic plaque. Super-resolution ultrasound imaging is an emerging technology that can identify the microvasculature noninvasively, with unprecedented spatial resolution beyond the acoustic diffraction limit. Therefore, it is a promising approach for diagnosing and monitoring the development of diseases. In this review, we introduce current super-resolution ultrasound imaging approaches and their preclinical applications on different animals and disease models. Future directions and challenges to overcome for clinical translations are also discussed.

Fast super-resolution ultrasound microvessel imaging using spatiotemporal data with deep fully convolutional neural network

Ultrasound localization microscopy (ULM) has been proposed to image microvasculature beyond the ultrasound diffraction limit. Although ULM can attain microvascular images with a sub-diffraction resolution, long data acquisition time and processing time are the critical limitations. Deep learning-based ULM (deep-ULM) has been proposed to mitigate these limitations. However, microbubble (MB) localization used in deep-ULMs is currently based on spatial information without the use of temporal information. The highly spatiotemporally coherent MB signals provide a strong feature that can be used to differentiate MB signals from background artifacts. In this study, a deep neural network was employed and trained with spatiotemporal ultrasound datasets to better identify the MB signals by leveraging both the spatial and temporal information of the MB signals. Training, validation and testing datasets were acquired from MB suspension to mimic the realistic intensity-varying and moving MB signals. The performance of the proposed network was first demonstrated in the chicken embryo chorioallantoic membrane dataset with an optical microscopic image as the reference standard. Substantial improvement in spatial resolution was shown for the reconstructed super-resolved images compared with power Doppler images. The full-width-half-maximum (FWHM) of a microvessel was improved from 133μm to 35μm, which is smaller than the ultrasound wavelength (73μm). The proposed method was further tested in anin vivohuman liver data. Results showed the reconstructed super-resolved images could resolve a microvessel of nearly 170μm (FWHM). Adjacent microvessels with a distance of 670μm, which cannot be resolved with power Doppler imaging, can be well-separated with the proposed method. Improved contrast ratios using the proposed method were shown compared with that of the conventional deep-ULM method. Additionally, the processing time to reconstruct a high-resolution ultrasound frame with an image size of 1024 × 512 pixels was around 16 ms, comparable to state-of-the-art deep-ULMs.

Ultrasound image reconstruction from plane wave radio-frequency data by self-supervised deep neural network

Image reconstruction from radio-frequency (RF) data is crucial for ultrafast plane wave ultrasound (PWUS) imaging. Compared with the traditional delay-and-sum (DAS) method based on relatively imprecise assumptions, sparse regularization (SR) method directly solves the inverse problem of image reconstruction and has presented significant improvement in the image quality when the frame rate remains high. However, the computational complexity of SR is too high for practical implementation, which is inherently associated with its iterative process. In this work, a deep neural network (DNN), which is trained with an incorporated loss function including sparse regularization terms, is proposed to reconstruct PWUS images from RF data with significantly reduced computational time. It is remarkable that, a self-supervised learning scheme, in which the RF data are utilized as both the inputs and the labels during the training process, is employed to overcome the lack of the "ideal" ultrasound images as the labels for DNN. In addition, it has been also verified that the trained network can be used on the RF data obtained with steered plane waves (PWs), and thus the image quality can be further improved with coherent compounding. Using simulation data, the proposed method has significantly shorter reconstruction time (∼10 ms) than the conventional SR method (∼1-5 mins), with comparable spatial resolution and 1.5-dB higher contrast-to-noise ratio (CNR). Besides, the proposed method with single PW can achieve higher CNR than DAS with 75 PWs in reconstruction of in-vivo images of human carotid arteries.

Technology trends and applications of deep learning in ultrasonography: image quality enhancement, diagnostic support, and improving workflow efficiency

In this review of the most recent applications of deep learning to ultrasound imaging, the architectures of deep learning networks are briefly explained for the medical imaging applications of classification, detection, segmentation, and generation. Ultrasonography applications for image processing and diagnosis are then reviewed and summarized, along with some representative imaging studies of the breast, thyroid, heart, kidney, liver, and fetal head. Efforts towards workflow enhancement are also reviewed, with an emphasis on view recognition, scanning guide, image quality assessment, and quantification and measurement. Finally some future prospects are presented regarding image quality enhancement, diagnostic support, and improvements in workflow efficiency, along with remarks on hurdles, benefits, and necessary collaborations.