Super-Resolution Ultrasound Localization Microscopy Through Deep Learning

Inspiring a convergent engineering approach to measure and model the tissue microenvironment

Understanding the molecular and physical complexity of the tissue microenvironment (TiME) in the context of its spatiotemporal organization has remained an enduring challenge. Recent advances in engineering and data science are now promising the ability to study the structure, functions, and dynamics of the TiME in unprecedented detail; however, many advances still occur in silos that rarely integrate information to study the TiME in its full detail. This review provides an integrative overview of the engineering principles underlying chemical, optical, electrical, mechanical, and computational science to probe, sense, model, and fabricate the TiME. In individual sections, we first summarize the underlying principles, capabilities, and scope of emerging technologies, the breakthrough discoveries enabled by each technology and recent, promising innovations. We provide perspectives on the potential of these advances in answering critical questions about the TiME and its role in various disease and developmental processes. Finally, we present an integrative view that appreciates the major scientific and educational aspects in the study of the TiME.

Microbubble detection on ultrasound imaging by utilizing phase patterned waves

Objective.Super-resolution ultrasonography offers the advantage of visualization of intricate microvasculature, which is crucial for disease diagnosis. Mapping of microvessels is possible by localizing microbubbles (MBs) that act as contrast agents and tracking their location. However, there are limitations such as the low detectability of MBs and the utilization of a diluted concentration of MBs, leading to the extension of the acquisition time. We aim to enhance the detectability of MBs to reduce the acquisition time of acoustic data necessary for mapping the microvessels.Approach.We propose utilizing phase patterned waves (PPWs) characterized by spatially patterned phase distributions in the incident beam to achieve this. In contrast to conventional ultrasound irradiation methods, this irradiation method alters bubble interactions, enhancing the oscillation response of MBs and generating more significant scattered waves from specific MBs. This enhances the detectability of MBs, thereby enabling the detection of MBs that were undetectable by the conventional method. The objective is to maximize the overall detection of bubbles by utilizing ultrasound imaging with additional PPWs, including the conventional method. In this paper, we apply PPWs to ultrasound imaging simulations considering bubble-bubble interactions to elucidate the characteristics of PPWs and demonstrate their efficacy by employing PPWs on MBs fixed in a phantom by the experiment.Main results.By utilizing two types of PPWs in addition to the conventional ultrasound irradiation method, we confirmed the detection of up to 93.3% more MBs compared to those detected using the conventional method alone.Significance.Ultrasound imaging using additional PPWs made it possible to increase the number of detected MBs, which is expected to improve the efficiency of bubble detection.

Super-Resolution Ultrasound Imaging for Monitoring the Therapeutic Efficacy of a Vascular Disrupting Agent in an Animal Model of Breast Cancer

OBJECTIVE

Evaluate the use of super-resolution ultrasound (SRUS) imaging for the early detection of tumor response to treatment using a vascular-disrupting agent (VDA).

METHODS

A population of 28 female nude athymic mice (Charles River Laboratories) were implanted with human breast cancer cells (MDA-MB-231, ATCC) in the mammary fat pad and allowed to grow. Ultrasound imaging was performed using a Vevo 3100 scanner (FUJIFILM VisualSonics Inc) equipped with the MX250 linear array transducer immediately before and after receiving bolus injections of a microbubble (MB) contrast agent (Definity, Lantheus Medical Imaging) via the tail vein. Following baseline ultrasound imaging, VDA drug (combretastatin A4 phosphate, CA4P, Sigma Aldrich) or control saline was injected via the placed catheter. After 4 or 24 hours, repeat ultrasound imaging along the same tumor cross-section occurred. Direct intratumoral pressure measurements were obtained using a calibrated sensor. All raw ultrasound data were saved for offline processing and SRUS image reconstruction using custom MATLAB software (MathWorks Inc). From a region encompassing the tumor space and the entire postprocessed ultrasound image sequence, time MB count (TMC) curves were generated in addition to traditional SRUS maps reflecting MB enumeration at each pixel location. Peak enhancement (PE) and wash-in rate (WIR) were extracted from these TMC curves. At termination, intratumoral microvessel density (MVD) was quantified using tomato lectin labeling of patent blood vessels.

RESULTS

SRUS images exhibited a clear difference between control and treated tumors. While there was no difference in any group parameters at baseline (0 hour, P > .09), both SRUS-derived PE and WIR measurements in tumors treated with VDA exhibited significant decreases by 4 (P = .03 and P = .05, respectively) and 24 hours (P = .02 and P = .01, respectively), but not in control group tumors (P > .22). Similarly, SRUS derived microvascular maps were not different at baseline (P = .81), but measures of vessel density were lower in treated tumors at both 4 and 24 hours (P < .04). An inverse relationship between intratumoral pressure and both PE and WIR parameters were found in control tumors (R2 > .09, P < .03).

CONCLUSION

SRUS imaging is a new modality for assessing tumor response to treatment using a VDA.

A Residual Dense Attention Generative Adversarial Network for Microscopic Image Super-Resolution

With the development of deep learning, the Super-Resolution (SR) reconstruction of microscopic images has improved significantly. However, the scarcity of microscopic images for training, the underutilization of hierarchical features in original Low-Resolution (LR) images, and the high-frequency noise unrelated with the image structure generated during the reconstruction process are still challenges in the Single Image Super-Resolution (SISR) field. Faced with these issues, we first collected sufficient microscopic images through Motic, a company engaged in the design and production of optical and digital microscopes, to establish a dataset. Secondly, we proposed a Residual Dense Attention Generative Adversarial Network (RDAGAN). The network comprises a generator, an image discriminator, and a feature discriminator. The generator includes a Residual Dense Block (RDB) and a Convolutional Block Attention Module (CBAM), focusing on extracting the hierarchical features of the original LR image. Simultaneously, the added feature discriminator enables the network to generate high-frequency features pertinent to the image's structure. Finally, we conducted experimental analysis and compared our model with six classic models. Compared with the best model, our model improved PSNR and SSIM by about 1.5 dB and 0.2, respectively.

Super-resolution ultrasound and microvasculomics: a consensus statement

This is a summary of a consensus statement on the introduction of "Ultrasound microvasculomics" produced by The Chinese Artificial Intelligence Alliance for Thyroid and Breast Ultrasound. The evaluation of microvessels is a very important part for the assessment of diseases. Super-resolution ultrasound (SRUS) microvascular imaging surpasses traditional ultrasound imaging in the morphological and functional analysis of microcirculation. SRUS microvascular imaging relies on contrast microbubbles to gain sensitivity to microvessels and improves the spatial resolution of ultrasound blood flow imaging for a more detailed depiction of vascular structures and hemodynamics. This method has been applied in preclinical animal models and pilot clinical studies, involving areas including neurology, oncology, nephrology, and cardiology. However, the current quantitative parameters of SRUS images are not enough for precise evaluation of microvessels. Therefore, by employing omics methods, more quantification indicators can be obtained, enabling a more precise and personalized assessment of microvascular status. Ultrasound microvasculomics - a high-throughput extraction of image features from SRUS images - is one novel approach that holds great promise but needs further validation in both bench and clinical settings. CLINICAL RELEVANCE STATEMENT: Super-resolution Ultrasound (SRUS) blood flow imaging improves spatial resolution. Ultrasound microvasculomics is possible to acquire high-throughput information of features from SRUS images. It provides more precise and abundant micro-blood flow information in clinical medicine. KEY POINTS: This consensus statement reviews the development and application of super-resolution ultrasound (SRUS). The shortcomings of the current quantification indicators of SRUS and strengths of the omics methodology are addressed. "Ultrasound microvasculomics" is introduced for a high-throughput extraction of image features from SRUS images.

Understanding the effects of microbubble concentration on localization accuracy in super-resolution ultrasound imaging

Super-resolution ultrasound (SRUS) through localising and tracking of microbubbles (MBs) can achieve sub-wavelength resolution for imaging microvascular structure and flow dynamics in deep tissuein vivo. The technique assumes that signals from individual MBs can be isolated and localised accurately, but this assumption starts to break down when the MB concentration increases and the signals from neighbouring MBs start to interfere. The aim of this study is to gain understanding of the effect of MB-MB distance on ultrasound images and their localisation. Ultrasound images of two MBs approaching each other were synthesised by simulating both ultrasound field propagation and nonlinear MB dynamics. Besides the distance between MBs, a range of other influencing factors including MB size, ultrasound frequency, transmit pulse sequence, pulse amplitude and localisation methods were studied. The results show that as two MBs approach each other, the interference fringes can lead to significant and oscillating localisation errors, which are affected by both the MB and imaging parameters. When modelling a clinical linear array probe operating at 6 MHz, localisation errors between 20 and 30μm (∼1/10 wavelength) can be generated when MBs are ∼500μm (2 wavelengths or ∼1.7 times the point spread function (PSF)) away from each other. When modelling a cardiac probe operating at 1.5 MHz, the localisation errors were as high as 200μm (∼1/5 wavelength) even when the MBs were more than 10 wavelengths apart (2.9 times the PSF). For both frequencies, at smaller separation distances, the two MBs were misinterpreted as one MB located in between the two true positions. Cross-correlation or Gaussian fitting methods were found to generate slightly smaller localisation errors than centroiding. In conclusion, caution should be taken when generating and interpreting SRUS images obtained using high agent concentration with MBs separated by less than 1.7 to 3 times the PSF, as significant localisation errors can be generated due to interference between neighbouring MBs.

Time Efficient Ultrasound Localization Microscopy Based on A Novel Radial Basis Function 2D Interpolation

Ultrasound localization microscopy (ULM) allows for the generation of super-resolved (SR) images of the vasculature by precisely localizing intravenously injected microbubbles. Although SR images may be useful for diagnosing and treating patients, their use in the clinical context is limited by the need for prolonged acquisition times and high frame rates. The primary goal of our study is to relax the requirement of high frame rates to obtain SR images. To this end, we propose a new time-efficient ULM (TEULM) pipeline built on a cutting-edge interpolation method. More specifically, we suggest employing Radial Basis Functions (RBFs) as interpolators to estimate the missing values in the 2-dimensional (2D) spatio-temporal structures. To evaluate this strategy, we first mimic the data acquisition at a reduced frame rate by applying a down-sampling (DS = 2, 4, 8, and 10) factor to high frame rate ULM data. Then, we up-sample the data to the original frame rate using the suggested interpolation to reconstruct the missing frames. Finally, using both the original high frame rate data and the interpolated one, we reconstruct SR images using the ULM framework steps. We evaluate the proposed TEULM using four in vivo datasets, a Rat brain (dataset A), a Rat kidney (dataset B), a Rat tumor (dataset C) and a Rat brain bolus (dataset D), interpolating at the in-phase and quadrature (IQ) level. Results demonstrate the effectiveness of TEULM in recovering vascular structures, even at a DS rate of 10 (corresponding to a frame rate of sub-100Hz). In conclusion, the proposed technique is successful in reconstructing accurate SR images while requiring frame rates of one order of magnitude lower than standard ULM.

Super-resolution techniques for biomedical applications and challenges

Super-resolution (SR) techniques have revolutionized the field of biomedical applications by detailing the structures at resolutions beyond the limits of imaging or measuring tools. These techniques have been applied in various biomedical applications, including microscopy, magnetic resonance imaging (MRI), computed tomography (CT), X-ray, electroencephalogram (EEG), ultrasound, etc. SR methods are categorized into two main types: traditional non-learning-based methods and modern learning-based approaches. In both applications, SR methodologies have been effectively utilized on biomedical images, enhancing the visualization of complex biological structures. Additionally, these methods have been employed on biomedical data, leading to improvements in computational precision and efficiency for biomedical simulations. The use of SR techniques has resulted in more detailed and accurate analyses in diagnostics and research, essential for early disease detection and treatment planning. However, challenges such as computational demands, data interpretation complexities, and the lack of unified high-quality data persist. The article emphasizes these issues, underscoring the need for ongoing development in SR technologies to further improve biomedical research and patient care outcomes.

Context-aware deep learning enables high-efficacy localization of high concentration microbubbles for super-resolution ultrasound localization microscopy

Ultrasound localization microscopy (ULM) enables deep tissue microvascular imaging by localizing and tracking intravenously injected microbubbles circulating in the bloodstream. However, conventional localization techniques require spatially isolated microbubbles, resulting in prolonged imaging time to obtain detailed microvascular maps. Here, we introduce LOcalization with Context Awareness (LOCA)-ULM, a deep learning-based microbubble simulation and localization pipeline designed to enhance localization performance in high microbubble concentrations. In silico, LOCA-ULM enhanced microbubble detection accuracy to 97.8% and reduced the missing rate to 23.8%, outperforming conventional and deep learning-based localization methods up to 17.4% in accuracy and 37.6% in missing rate reduction. In in vivo rat brain imaging, LOCA-ULM revealed dense cerebrovascular networks and spatially adjacent microvessels undetected by conventional ULM. We further demonstrate the superior localization performance of LOCA-ULM in functional ULM (fULM) where LOCA-ULM significantly increased the functional imaging sensitivity of fULM to hemodynamic responses invoked by whisker stimulations in the rat brain.

Improving sub-pixel accuracy in ultrasound localization microscopy using supervised and self-supervised deep learning

With a spatial resolution of tens of microns, ultrasound localization microscopy (ULM) reconstructs microvascular structures and measures intravascular flows by tracking microbubbles (1-5 μm) in contrast enhanced ultrasound (CEUS) images. Since the size of CEUS bubble traces, e.g. 0.5-1 mm for ultrasound with a wavelength λ = 280 μm, is typically two orders of magnitude larger than the bubble diameter, accurately localizing microbubbles in noisy CEUS data is vital to the fidelity of the ULM results. In this paper, we introduce a residual learning based supervised super-resolution blind deconvolution network (SupBD-net), and a new loss function for a self-supervised blind deconvolution network (SelfBD-net), for detecting bubble centers at a spatial resolution finer than λ/10. Our ultimate purpose is to improve the ability to distinguish closely located microvessels and the accuracy of the velocity profile measurements in macrovessels. Using realistic synthetic data, the performance of these methods is calibrated and compared against several recently introduced deep learning and blind deconvolution techniques. For bubble detection, errors in bubble center location increase with the trace size, noise level, and bubble concentration. For all cases, SupBD-net yields the least error, keeping it below 0.1 λ. For unknown bubble trace morphology, where all the supervised learning methods fail, SelfBD-net can still maintain an error of less than 0.15 λ. SupBD-net also outperforms the other methods in separating closely located bubbles and parallel microvessels. In macrovessels, SupBD-net maintains the least errors in the vessel radius and velocity profile after introducing a procedure that corrects for terminated tracks caused by overlapping traces. Application of these methods is demonstrated by mapping the cerebral microvasculature of a neonatal pig, where neighboring microvessels separated by 0.15 λ can be readily distinguished by SupBD-net and SelfBD-net, but not by the other techniques. Hence, the newly proposed residual learning based methods improve the spatial resolution and accuracy of ULM in micro- and macro-vessels.

Motor Interaction Control Based on Muscle Force Model and Depth Reinforcement Strategy

The current motion interaction model has the problems of insufficient motion fidelity and lack of self-adaptation to complex environments. To address this problem, this study proposed to construct a human motion control model based on the muscle force model and stage particle swarm, and based on this, this study utilized the deep deterministic gradient strategy algorithm to construct a motion interaction control model based on the muscle force model and the deep reinforcement strategy. Empirical analysis of the human motion control model proposed in this study revealed that the joint trajectory correlation and muscle activity correlation of the model were higher than those of other comparative models, and its joint trajectory correlation was up to 0.90, and its muscle activity correlation was up to 0.84. In addition, this study validated the effectiveness of the motion interaction control model using the depth reinforcement strategy and found that in the mixed-obstacle environment, the model's desired results were obtained by training 1.1 × 103 times, and the walking distance was 423 m, which was better than other models. In summary, the proposed motor interaction control model using the muscle force model and deep reinforcement strategy has higher motion fidelity and can realize autonomous decision making and adaptive control in the face of complex environments. It can provide a theoretical reference for improving the effect of motion control and realizing intelligent motion interaction.

Artificial intelligence-based speckle featurization and localization for ultrasound speckle tracking velocimetry

Deep learning-based super-resolution ultrasound (DL-SRU) framework has been successful in improving spatial resolution and measuring the velocity field information of a blood flows by localizing and tracking speckle signals of red blood cells (RBCs) without using any contrast agents. However, DL-SRU can localize only a small part of the speckle signals of blood flow owing to ambiguity problems encountered in the classification of blood flow signals from ultrasound B-mode images and the building up of suitable datasets required for training artificial neural networks, as well as the structural limitations of the neural network itself. An artificial intelligence-based speckle featurization and localization (AI-SFL) framework is proposed in this study. It includes a machine learning-based algorithm for classifying blood flow signals from ultrasound B-mode images, dimensionality reduction for featurizing speckle patterns of the classified blood flow signals by approximating them with quantitative values. A novel and robust neural network (ResSU-net) is trained using the online data generation (ODG) method and the extracted speckle features. The super-resolution performance of the proposed AI-SFL and ODG method is evaluated and compared with the results of previous U-net and conventional data augmentation methods under in silico conditions. The predicted locations of RBCs by the AI-SFL and DL-SRU for speckle patterns of blood flow are applied to a PTV algorithm to measure quantitative velocity fields of the flow. Finally, the feasibility of the proposed AI-SFL framework for measuring real blood flows is verified under in vivo conditions.

An Automated Heart Shunt Recognition Pipeline Using Deep Neural Networks

Automated recognition of heart shunts using saline contrast transthoracic echocardiography (SC-TTE) has the potential to transform clinical practice, enabling non-experts to assess heart shunt lesions. This study aims to develop a fully automated and scalable analysis pipeline for distinguishing heart shunts, utilizing a deep neural network-based framework. The pipeline consists of three steps: (1) chamber segmentation, (2) ultrasound microbubble localization, and (3) disease classification model establishment. The study's normal control group included 91 patients with intracardiac shunts, 61 patients with extracardiac shunts, and 84 asymptomatic individuals. Participants' SC-TTE images were segmented using the U-Net model to obtain cardiac chambers. The segmentation results were combined with ultrasound microbubble localization to generate multivariate time series data on microbubble counts in each chamber. A classification model was then trained using this data to distinguish between intracardiac and extracardiac shunts. The proposed framework accurately segmented heart chambers (dice coefficient = 0.92 ± 0.1) and localized microbubbles. The disease classification model achieved high accuracy, sensitivity, specificity, F1 score, kappa value, and AUC value for both intracardiac and extracardiac shunts. For intracardiac shunts, accuracy was 0.875 ± 0.008, sensitivity was 0.891 ± 0.002, specificity was 0.865 ± 0.012, F1 score was 0.836 ± 0.011, kappa value was 0.735 ± 0.017, and AUC value was 0.942 ± 0.014. For extracardiac shunts, accuracy was 0.902 ± 0.007, sensitivity was 0.763 ± 0.014, specificity was 0.966 ± 0.008, F1 score was 0.830 ± 0.012, kappa value was 0.762 ± 0.017, and AUC value was 0.916 ± 0.006. The proposed framework utilizing deep neural networks offers a fast, convenient, and accurate method for identifying intracardiac and extracardiac shunts. It aids in shunt recognition and generates valuable quantitative indices, assisting clinicians in diagnosing these conditions.

Deep learning for fast super-resolution ultrasound microvessel imaging

Objective. Ultrasound localization microscopy (ULM) enables microvascular reconstruction by localizing microbubbles (MBs). Although ULM can obtain microvascular images that are beyond the ultimate resolution of the ultrasound (US) diffraction limit, it requires long data processing time, and the imaging accuracy is susceptible to the density of MBs. Deep learning (DL)-based ULM is proposed to alleviate these limitations, which simulated MBs at low-resolution and mapped them to coordinates at high-resolution by centroid localization. However, traditional DL-based ULMs are imprecise and computationally complex. Also, the performance of DL is highly dependent on the training datasets, which are difficult to realistically simulate.Approach. A novel architecture called adaptive matching network (AM-Net) and a dataset generation method named multi-mapping (MMP) was proposed to overcome the above challenges. The imaging performance and processing time of the AM-Net have been assessed by simulation andin vivoexperiments.Main results. Simulation results show that at high density (20 MBs/frame), when compared to other DL-based ULM, AM-Net achieves higher localization accuracy in the lateral/axial direction.In vivoexperiment results show that the AM-Net can reconstruct ∼24.3μm diameter micro-vessels and separate two ∼28.3μm diameter micro-vessels. Furthermore, when processing a 128 × 128 pixels image in simulation experiments and an 896 × 1280 pixels imagein vivoexperiment, the processing time of AM-Net is ∼13 s and ∼33 s, respectively, which are 0.3-0.4 orders of magnitude faster than other DL-based ULM.Significance. We proposes a promising solution for ULM with low computing costs and high imaging performance.

Three-dimensional super resolution ultrasound imaging with a multi-frequency hemispherical phased array

BACKGROUND

High resolution imaging of the microvasculature plays an important role in both diagnostic and therapeutic applications in the brain. However, ultrasound pulse-echo sonography imaging the brain vasculatures has been limited to narrow acoustic windows and low frequencies due to the distortion of the skull bone, which sacrifices axial resolution since it is pulse length dependent.

PURPOSE

To overcome the detect limit, a large aperture 256-module sparse hemispherical transmit/receive array was used to visualize the acoustic emissions of ultrasound-vaporized lipid-coated decafluorobutane nanodroplets flowing through tube phantoms and within rabbit cerebral vasculature in vivo via passive acoustic mapping and super resolution techniques.

METHODS

Nanodroplets were vaporized with 55 kHz burst-mode ultrasound (burst length = 145 μs, burst repetition frequency = 9-45 Hz, peak negative acoustic pressure = 0.10-0.22 MPa), which propagates through overlying tissues well without suffering from severe distortions. The resulting emissions were received at a higher frequency (612 or 1224 kHz subarray) to improve the resulting spatial resolution during passive beamforming. Normal resolution three-dimensional images were formed using a delay, sum, and integrate beamforming algorithm, and super-resolved images were extracted via Gaussian fitting of the estimated point-spread-function to the normal resolution data.

RESULTS

With super resolution techniques, the mean lateral (axial) full-width-at-half-maximum image intensity was 16 ± 3 (32 ± 6) μm, and 7 ± 1 (15 ± 2) μm corresponding to ∼1/67 of the normal resolution at 612 and 1224 kHz, respectively. The mean positional uncertainties were ∼1/350 (lateral) and ∼1/180 (axial) of the receive wavelength in water. In addition, a temporal correlation between nanodroplet vaporization and the transmit waveform shape was observed, which may provide the opportunity to enhance the signal-to-noise ratio in future studies.

CONCLUSIONS

Here, we demonstrate the feasibility of vaporizing nanodroplets via low frequency ultrasound and simultaneously performing spatial mapping via passive beamforming at higher frequencies to improve the resulting spatial resolution of super resolution imaging techniques. This method may enable complete four-dimensional vascular mapping in organs where a hemispherical array could be positioned to surround the target, such as the brain, breast, or testicles.

Contrast-Free Super-Resolution Power Doppler (CS-PD) Based on Deep Neural Networks

Super-resolution ultrasound microvessel imaging based on ultrasound localization microscopy (ULM) is an emerging imaging modality that is capable of resolving micrometer-scaled vessels deep into tissue. In practice, ULM is limited by the need for contrast injection, long data acquisition, and computationally expensive postprocessing times. In this study, we present a contrast-free super-resolution power Doppler (CS-PD) technique that uses deep networks to achieve super-resolution with short data acquisition. The training dataset is comprised of spatiotemporal ultrafast ultrasound signals acquired from in vivo mouse brains, while the testing dataset includes in vivo mouse brain, chicken embryo chorioallantoic membrane (CAM), and healthy human subjects. The in vivo mouse imaging studies demonstrate that CS-PD could achieve an approximate twofold improvement in spatial resolution when compared with conventional power Doppler. In addition, the microvascular images generated by CS-PD showed good agreement with the corresponding ULM images as indicated by a structural similarity index of 0.7837 and a peak signal-to-noise ratio (PSNR) of 25.52. Moreover, CS-PD was able to preserve the temporal profile of the blood flow (e.g., pulsatility) that is similar to conventional power Doppler. Finally, the generalizability of CS-PD was demonstrated on testing data of different tissues using different imaging settings. The fast inference time of the proposed deep neural network also allows CS-PD to be implemented for real-time imaging. These features of CS-PD offer a practical, fast, and robust microvascular imaging solution for many preclinical and clinical applications of Doppler ultrasound.

Nanodroplet-Based Super-Resolution Ultrasound Localization Microscopy

Over the past decade, super-resolution ultrasound localization microscopy (SR-ULM) has revolutionized ultrasound imaging with its capability to resolve the microvascular structures below the ultrasound diffraction limit. The introduction of this imaging technique enables the visualization, quantification, and characterization of tissue microvasculature. The early implementations of SR-ULM utilize microbubbles (MBs) that require a long image acquisition time due to the requirement of capturing sparsely isolated microbubble signals. The next-generation SR-ULM employs nanodroplets that have the potential to significantly reduce the image acquisition time without sacrificing the resolution. This review discusses various nanodroplet-based ultrasound localization microscopy techniques and their corresponding imaging mechanisms. A summary is given on the preclinical applications of SR-ULM with nanodroplets, and the challenges in the clinical translation of nanodroplet-based SR-ULM are presented while discussing the future perspectives. In conclusion, ultrasound localization microscopy is a promising microvasculature imaging technology that can provide new diagnostic and prognostic information for a wide range of pathologies, such as cancer, heart conditions, and autoimmune diseases, and enable personalized treatment monitoring at a microlevel.

Ultrasound localization microscopy

Ultrasound Localization Microscopy (ULM) is an emerging technique that provides impressive super-resolved images of microvasculature, i.e., images with much better resolution than the conventional diffraction-limited ultrasound techniques and is already taking its first steps from preclinical to clinical applications. In comparison to the established perfusion or flow measurement methods, namely contrast-enhanced ultrasound (CEUS) and Doppler techniques, ULM allows imaging and flow measurements even down to the capillary level. As ULM can be realized as a post-processing method, conventional ultrasound systems can be used for. ULM relies on the localization of single microbubbles (MB) of commercial, clinically approved contrast agents. In general, these very small and strong scatterers with typical radii of 1-3 µm are imaged much larger in ultrasound images than they actually are due to the point spread function of the imaging system. However, by applying appropriate methods, these MBs can be localized with sub-pixel precision. Then, by tracking MBs over successive frames of image sequences, not only the morphology of vascular trees but also functional information such as flow velocities or directions can be obtained and visualized. In addition, quantitative parameters can be derived to describe pathological and physiological changes in the microvasculature. In this review, the general concept of ULM and conditions for its applicability to microvessel imaging are explained. Based on this, various aspects of the different processing steps for a concrete implementation are discussed. The trade-off between complete reconstruction of the microvasculature and the necessary measurement time as well as the implementation in 3D are reviewed in more detail, as they are the focus of current research. Through an overview of potential or already realized preclinical and clinical applications - pathologic angiogenesis or degeneration of vessels, physiological angiogenesis, or the general understanding of organ or tissue function - the great potential of ULM is demonstrated.

Localization Free Super-Resolution Microbubble Velocimetry Using a Long Short-Term Memory Neural Network

Ultrasound localization microscopy is a super-resolution imaging technique that exploits the unique characteristics of contrast microbubbles to side-step the fundamental trade-off between imaging resolution and penetration depth. However, the conventional reconstruction technique is confined to low microbubble concentrations to avoid localization and tracking errors. Several research groups have introduced sparsity- and deep learning-based approaches to overcome this constraint to extract useful vascular structural information from overlapping microbubble signals, but these solutions have not been demonstrated to produce blood flow velocity maps of the microcirculation. Here, we introduce Deep-SMV, a localization free super-resolution microbubble velocimetry technique, based on a long short-term memory neural network, that provides high imaging speed and robustness to high microbubble concentrations, and directly outputs blood velocity measurements at a super-resolution. Deep-SMV is trained efficiently using microbubble flow simulation on real in vivo vascular data and demonstrates real-time velocity map reconstruction suitable for functional vascular imaging and pulsatility mapping at super-resolution. The technique is successfully applied to a wide variety of imaging scenarios, include flow channel phantoms, chicken embryo chorioallantoic membranes, and mouse brain imaging. An implementation of Deep-SMV is openly available at https://github.com/chenxiptz/SR_microvessel_velocimetry, with two pre-trained models available at https://doi.org/10.7910/DVN/SECUFD.

Super-resolution ultrasound microvascular imaging: Is it ready for clinical use?

The field of super-resolution ultrasound microvascular imaging has been rapidly growing over the past decade. By leveraging contrast microbubbles as point targets for localization and tracking, super-resolution ultrasound pinpoints the location of microvessels and measures their blood flow velocity. Super-resolution ultrasound is the first in vivo imaging modality that can image micron-scale vessels at a clinically relevant imaging depth without tissue destruction. These unique capabilities of super-resolution ultrasound provide structural (vessel morphology) and functional (vessel blood flow) assessments of tissue microvasculature on a global and local scale, which opens new doors for many enticing preclinical and clinical applications that benefit from microvascular biomarkers. The goal of this short review is to provide an update on recent advancements in super-resolution ultrasound imaging, with a focus on summarizing existing applications and discussing the prospects of translating super-resolution imaging to clinical practice and research. In this review, we also provide brief introductions of how super-resolution ultrasound works, how does it compare with other imaging modalities, and what are the tradeoffs and limitations for an audience who is not familiar with the technology.

Backscattering amplitude in ultrasound localization microscopy

In the last decade, Ultrafast ultrasound localisation microscopy has taken non-invasive deep vascular imaging down to the microscopic level. By imaging diluted suspensions of circulating microbubbles in the blood stream at kHz frame rate and localizing the center of their individual point spread function with a sub-resolution precision, it enabled to break the unvanquished trade-off between depth of imaging and resolution by microscopically mapping the microbubbles flux and velocities deep into tissue. However, ULM also suffers limitations. Many small vessels are not visible in the ULM images due to the noise level in areas dimly explored by the microbubbles. Moreover, as the vast majority of studies are performed using 2D imaging, quantification is limited to in-plane velocity or flux measurements which hinders the accurate velocity determination and quantification. Here we show that the backscattering amplitude of each individual microbubble can also be exploited to produce backscattering images of the vascularization with a higher sensitivity compared to conventional ULM images. By providing valuable information about the relative distance of the microbubble to the 2D imaging plane in the out-of-plane direction, backscattering ULM images introduces a physically relevant 3D rendering perception in the vascular maps. It also retrieves the missing information about the out-of-plane motion of microbubbles and provides a way to improve 3D flow and velocity quantification using 2D ULM. These results pave the way to improved visualization and quantification for 2D and 3D ULM.

A Data-Efficient Deep Learning Strategy for Tissue Characterization via Quantitative Ultrasound: Zone Training

Deep learning (DL) powered biomedical ultrasound imaging is an emerging research field where researchers adapt the image analysis capabilities of DL algorithms to biomedical ultrasound imaging settings. A major roadblock to wider adoption of DL powered biomedical ultrasound imaging is that acquisition of large and diverse datasets is expensive in clinical settings, which is a requirement for successful DL implementation. Hence, there is a constant need for developing data-efficient DL techniques to turn DL powered biomedical ultrasound imaging into reality. In this work, we develop a data-efficient DL training strategy for classifying tissues based on the ultrasonic backscattered RF data, i.e., quantitative ultrasound (QUS), which we named zone training. In zone training, we propose to divide the complete field of view of an ultrasound image into multiple zones associated with different regions of a diffraction pattern and then, train separate DL networks for each zone. The main advantage of zone training is that it requires less training data to achieve high accuracy. In this work, three different tissue-mimicking phantoms were classified by a DL network. The results demonstrated that zone training can require a factor of 2-3 less training data in low data regime to achieve similar classification accuracies compared to a conventional training strategy.

Assessment of Transarterial Chemoembolization Using Super-resolution Ultrasound Imaging and a Rat Model of Hepatocellular Carcinoma

OBJECTIVE

Hepatocellular carcinoma (HCC) is a highly prevalent form of liver cancer diagnosed annually in 600,000 people worldwide. A common treatment is transarterial chemoembolization (TACE), which interrupts the blood supply of oxygen and nutrients to the tumor mass. The need for repeat TACE treatments may be assessed in the weeks after therapy with contrast-enhanced ultrasound (CEUS) imaging. Although the spatial resolution of traditional CEUS has been restricted by the diffraction limit of ultrasound (US), this physical barrier has been overcome by a recent innovation known as super-resolution US (SRUS) imaging. In short, SRUS enhances the visible details of smaller microvascular structures on the 10 to 100 µm scale, which unlocks a host of new clinical opportunities for US.

METHODS

In this study, a rat model of orthotopic HCC is introduced and TACE treatment response (to a doxorubicin-lipiodol emulsion) is assessed using longitudinal SRUS and magnetic resonance imaging (MRI) performed at 0, 7 and 14 d. Animals were euthanized at 14 d for histological analysis of excised tumor tissue and determination of TACE response, that is, control, partial response or complete response. CEUS imaging was performed using a pre-clinical US system (Vevo 3100, FUJIFILM VisualSonics Inc.) equipped with an MX201 linear array transducer. After administration of a microbubble contrast agent (Definity, Lantheus Medical Imaging), a series of CEUS images were collected at each tissue cross-section as the transducer was mechanically stepped at 100 μm increments. SRUS images were formed at each spatial position, and a microvascular density metric was calculated. Microscale computed tomography (microCT, OI/CT, MILabs) was used to confirm TACE procedure success, and tumor size was monitored using a small animal MRI system (BioSpec 3T, Bruker Corp.).

RESULTS

Although there were no differences at baseline (p > 0.15), both microvascular density levels and tumor size measures from the complete responder cases at 14 d were considerably lower and smaller, respectively, than those in the partial responder or control group animals. Histological analysis revealed tumor-to-necrosis levels of 8.4%, 51.1% and 100%, for the control, partial responder and complete responder groups, respectively (p < 0.005).

CONCLUSION

SRUS imaging is a promising modality for assessing early changes in microvascular networks in response to tissue perfusion-altering interventions such as TACE treatment of HCC.

Laser diode photoacoustic point source detection: machine learning-based denoising and reconstruction

A new development in photoacoustic (PA) imaging has been the use of compact, portable and low-cost laser diodes (LDs), but LD-based PA imaging suffers from low signal intensity recorded by the conventional transducers. A common method to improve signal strength is temporal averaging, which reduces frame rate and increases laser exposure to patients. To tackle this problem, we propose a deep learning method that will denoise point source PA radio-frequency (RF) data before beamforming with a very few frames, even one. We also present a deep learning method to automatically reconstruct point sources from noisy pre-beamformed data. Finally, we employ a strategy of combined denoising and reconstruction, which can supplement the reconstruction algorithm for very low signal-to-noise ratio inputs.

Ultrasound Signal Processing: From Models to Deep Learning

Medical ultrasound imaging relies heavily on high-quality signal processing to provide reliable and interpretable image reconstructions. Conventionally, reconstruction algorithms have been derived from physical principles. These algorithms rely on assumptions and approximations of the underlying measurement model, limiting image quality in settings where these assumptions break down. Conversely, more sophisticated solutions based on statistical modeling or careful parameter tuning or derived from increased model complexity can be sensitive to different environments. Recently, deep learning-based methods, which are optimized in a data-driven fashion, have gained popularity. These model-agnostic techniques often rely on generic model structures and require vast training data to converge to a robust solution. A relatively new paradigm combines the power of the two: leveraging data-driven deep learning and exploiting domain knowledge. These model-based solutions yield high robustness and require fewer parameters and training data than conventional neural networks. In this work we provide an overview of these techniques from the recent literature and discuss a wide variety of ultrasound applications. We aim to inspire the reader to perform further research in this area and to address the opportunities within the field of ultrasound signal processing. We conclude with a future perspective on model-based deep learning techniques for medical ultrasound.

HD2A-Net: A novel dual gated attention network using comprehensive hybrid dilated convolutions for medical image segmentation

The convolutional neural networks (CNNs) have been widely proposed in the medical image analysis tasks, especially in the image segmentations. In recent years, the encoder-decoder structures, such as the U-Net, were rendered. However, the multi-scale information transmission and effective modeling for long-range feature dependencies in these structures were not sufficiently considered. To improve the performance of the existing methods, we propose a novel hybrid dual dilated attention network (HD²A-Net) to conduct the lesion region segmentations. In the proposed network, we innovatively present the comprehensive hybrid dilated convolution (CHDC) module, which facilitates the transmission of the multi-scale information. Based on the CHDC module and the attention mechanisms, we design a novel dual dilated gated attention (DDGA) block to enhance the saliency of related regions from the multi-scale aspect. Besides, a dilated dense (DD) block is designed to expand the receptive fields. The ablation studies were performed to verify our proposed blocks. Besides, the interpretability of the HD²A-Net was analyzed through the visualization of the attention weight maps from the key blocks. Compared to the state-of-the-art methods including CA-Net, DeepLabV3+, and Attention U-Net, the HD²A-Net outperforms significantly, with the metrics of Dice, Average Symmetric Surface Distance (ASSD), and mean Intersection-over-Union (mIoU) reaching 93.16%, 93.63%, and 94.72%, 0.36 pix, 0.69 pix, and 0.52 pix, and 88.03%, 88.67%, and 90.33% on three publicly available medical image datasets: MAEDE-MAFTOUNI (COVID-19 CT), ISIC-2018 (Melanoma Dermoscopy), and Kvasir-SEG (Gastrointestinal Disease Polyp), respectively.

Ultrafast Ultrasound Localization Microscopy by Conditional Generative Adversarial Network

Ultrasound localization microscopy (ULM) overcomes the acoustic diffraction limit and enables the visualization of microvasculature at subwavelength resolution. However, challenges remain in ultrafast ULM implementation, where short data acquisition time, efficient data processing speed, and high imaging resolution need to be considered simultaneously. Recently, deep learning (DL)-based methods have exhibited potential in speeding up ULM imaging. Nevertheless, a certain number of ultrasound (US) data ( L frames) are still required to accumulate enough localized microbubble (MB) events, leading to an acquisition time within a time span of tens of seconds. To further speed up ULM imaging, in this article, we present a new DL-based method, termed as ULM-GAN. By using a modified conditional generative adversarial network (cGAN) framework, ULM-GAN is able to reconstruct a superresolution image directly from a temporal mean low-resolution (LR) image generated by averaging l -frame raw US images with l being significantly smaller than L . To evaluate the performance of ULM-GAN, a series of numerical simulations and phantom experiments are both implemented. The results of the numerical simulations demonstrate that when performing ULM imaging, ULM-GAN allows  ∼ 40 -fold reduction in data acquisition time and  ∼ 61 -fold reduction in computational time compared with the conventional Gaussian fitting method, without compromising spatial resolution according to the resolution scaled error (RSE). For the phantom experiments, ULM-GAN offers an implementation of ULM with ultrafast data acquisition time (  ∼ 0.33 s) and ultrafast data processing speed (  ∼ 0.60 s) that makes it promising to observe rapid biological activities in vivo.

Quantitative tissue perfusion imaging using nonlinear ultrasound localization microscopy

Ultrasound localization microscopy (ULM) is a recent advancement in ultrasound imaging that uses microbubble contrast agents to yield vascular images that break the classical diffraction limit on spatial resolution. Current approaches cannot image blood flow at the tissue perfusion level since they rely solely on differences in velocity to separate tissue and microbubble signals; lower velocity microbubble echoes are removed during high pass wall filtering. To visualize blood flow in the entire vascular tree, we have developed nonlinear ULM, which combines nonlinear pulsing sequences with plane-wave imaging to segment microbubble signals independent of their velocity. Bubble localization and inter-frame tracking produces super-resolved images and, with parameters derived from the bubble tracks, a rich quantitative feature set that can describe the relative quality of microcirculatory flow. Using the rat spinal cord as a model system, we showed that nonlinear ULM better resolves some smaller branching vasculature compared to conventional ULM. Following contusion injury, both gold-standard histological techniques and nonlinear ULM depicted reduced in-plane vessel length between the penumbra and contralateral gray matter (-16.7% vs. -20.5%, respectively). Here, we demonstrate that nonlinear ULM uniquely enables investigation and potential quantification of tissue perfusion, arguably the most important component of blood flow.

Three-Dimensional Ultrasound Localization Microscopy with Bipartite Graph-Based Microbubble Pairing and Kalman-Filtering-Based Tracking on a 256-Channel Verasonics Ultrasound System with a 32 × 32 Matrix Array

Three-dimensional (3D) ultrasound localization microscopy (ULM) using a 2-D matrix probe and microbubbles (MBs) has been recently proposed to visualize microvasculature beyond the ultrasound diffraction limit in three spatial dimensions. However, 3D ULM suffers from several limitations: (1) high system complexity due to numerous channel counts, (2) complex MB flow dynamics in 3D, and (3) extremely long acquisition time. To reduce the system complexity while maintaining high image quality, we used a sub-aperture process to reduce received channel counts. To address the second issue, a 3D bipartite graph-based method with Kalman filtering-based tracking was used in this study for MB tracking. An MB separation approach was incorporated to separate high concentration MB data into multiple, sparser MB datasets, allowing better MB localization and tracking for a limited acquisition time. The proposed method was first validated in a flow channel phantom, showing improved spatial resolutions compared with the contrasted enhanced power Doppler image. Then the proposed method was evaluated with an in vivo chicken embryo brain dataset. Results showed that the reconstructed 3D super-resolution image achieved a spatial resolution of around 52 μm (smaller than the wavelength of around 200 μm). Microvessels that cannot be resolved clearly using localization only, can be well identified with the tailored 3D pairing and tracking algorithms. To sum up, the feasibility of the 3D ULM is shown, indicating the great possibility in clinical applications.

Automated segmentation of vertebral cortex with 3D U-Net-based deep convolutional neural network

Objectives: We developed a 3D U-Net-based deep convolutional neural network for the automatic segmentation of the vertebral cortex. The purpose of this study was to evaluate the accuracy of the 3D U-Net deep learning model.

Methods: In this study, a fully automated vertebral cortical segmentation method with 3D U-Net was developed, and ten-fold cross-validation was employed. Through data augmentation, we obtained 1,672 3D images of chest CT scans. Segmentation was performed using a conventional image processing method and manually corrected by a senior radiologist to create the gold standard. To compare the segmentation performance, 3D U-Net, Res U-Net, Ki U-Net, and Seg Net were used to segment the vertebral cortex in CT images. The segmentation performance of 3D U-Net and the other three deep learning algorithms was evaluated using DSC, mIoU, MPA, and FPS.

Results: The DSC, mIoU, and MPA of 3D U-Net are better than the other three strategies, reaching 0.71 ± 0.03, 0.74 ± 0.08, and 0.83 ± 0.02, respectively, indicating promising automated segmentation results. The FPS is slightly lower than that of Seg Net (23.09 ± 1.26 vs. 30.42 ± 3.57).

Conclusion: Cortical bone can be effectively segmented based on 3D U-net.

Acoustic Molecular Imaging Beyond the Diffraction Limit In Vivo

Ultrasound molecular imaging (USMI) is a technique used to noninvasively estimate the distribution of molecular markers in vivo by imaging microbubble contrast agents (MCAs) that have been modified to target receptors of interest on the vascular endothelium. USMI is especially relevant for preclinical and clinical cancer research and has been used to predict tumor malignancy and response to treatment. In the last decade, methods that improve the resolution of contrast-enhanced ultrasound by an order of magnitude and allow researchers to noninvasively image individual capillaries have emerged. However, these approaches do not translate directly to molecular imaging. In this work, we demonstrate super-resolution visualization of biomarker expression in vivo using superharmonic ultrasound imaging (SpHI) with dual-frequency transducers, targeted contrast agents, and localization microscopy processing. We validate and optimize the proposed method in vitro using concurrent optical and ultrasound microscopy and a microvessel phantom. With the same technique, we perform a proof-of-concept experiment in vivo in a rat fibrosarcoma model and create maps of biomarker expression co-registered with images of microvasculature. From these images, we measure a resolution of 23 μm, a nearly fivefold improvement in resolution compared to previous diffraction-limited molecular imaging studies.

Curvelet Transform-Based Sparsity Promoting Algorithm for Fast Ultrasound Localization Microscopy

Ultrasound localization microscopy (ULM) based on microbubble (MB) localization was recently introduced to overcome the resolution limit of conventional ultrasound. However, ULM is currently challenged by the requirement for long data acquisition times to accumulate adequate MB events to fully reconstruct vasculature. In this study, we present a curvelet transform-based sparsity promoting (CTSP) algorithm that improves ULM imaging speed by recovering missing MB localization signal from data with very short acquisition times. CTSP was first validated in a simulated microvessel model, followed by the chicken embryo chorioallantoic membrane (CAM), and finally, in the mouse brain. In the simulated microvessel study, CTSP robustly recovered the vessel model to achieve an 86.94% vessel filling percentage from a corrupted image with only 4.78% of the true vessel pixels. In the chicken embryo CAM study, CTSP effectively recovered the missing MB signal within the vasculature, leading to marked improvement in ULM imaging quality with a very short data acquisition. Taking the optical image as reference, the vessel filling percentage increased from 2.7% to 42.2% using 50ms of data acquisition after applying CTSP. CTSP used 80% less time to achieve the same 90% maximum saturation level as compared with conventional MB localization. We also applied CTSP on the microvessel flow speed maps and found that CTSP was able to use only 1.6s of microbubble data to recover flow speed images that have similar qualities as those constructed using 33.6s of data. In the mouse brain study, CTSP was able to reconstruct the majority of the cerebral vasculature using 1-2s of data acquisition. Additionally, CTSP only needed 3.2s of microbubble data to generate flow velocity maps that are comparable to those using 129.6s of data. These results suggest that CTSP can facilitate fast and robust ULM imaging especially under the circumstances of inadequate microbubble localizations.

Fast DNA-PAINT imaging using a deep neural network

DNA points accumulation for imaging in nanoscale topography (DNA-PAINT) is a super-resolution technique with relatively easy-to-implement multi-target imaging. However, image acquisition is slow as sufficient statistical data has to be generated from spatio-temporally isolated single emitters. Here, we train the neural network (NN) DeepSTORM to predict fluorophore positions from high emitter density DNA-PAINT data. This achieves image acquisition in one minute. We demonstrate multi-colour super-resolution imaging of structure-conserved semi-thin neuronal tissue and imaging of large samples. This improvement can be integrated into any single-molecule imaging modality to enable fast single-molecule super-resolution microscopy.

Super-Resolution Ultrasound Through Sparsity-Based Deconvolution and Multi-Feature Tracking

Ultrasound super-resolution imaging through localisation and tracking of microbubbles can achieve sub-wave-diffraction resolution in mapping both micro-vascular structure and flow dynamics in deep tissue in vivo. Currently, it is still challenging to achieve high accuracy in localisation and tracking particularly with limited imaging frame rates and in the presence of high bubble concentrations. This study introduces microbubble image features into a Kalman tracking framework, and makes the framework compatible with sparsity-based deconvolution to address these key challenges. The performance of the method is evaluated on both simulations using individual bubble signals segmented from in vivo data and experiments on a mouse brain and a human lymph node. The simulation results show that the deconvolution not only significantly improves the accuracy of isolating overlapping bubbles, but also preserves some image features of the bubbles. The combination of such features with Kalman motion model can achieve a significant improvement in tracking precision at a low frame rate over that using the distance measure, while the improvement is not significant at the highest frame rate. The in vivo results show that the proposed framework generates SR images that are significantly different from the current methods with visual improvement, and is more robust to high bubble concentrations and low frame rates.

Influences of Magnetic Resonance Imaging Superresolution Algorithm-Based Transition Care on Prognosis of Children with Severe Viral Encephalitis

Objective

Its goal was to see how convolutional neural network- (CNN-) based superresolution (SR) technology magnetic resonance imaging- (MRI-) assisted transition care (TC) affected the prognosis of children with severe viral encephalitis (SVE) and how effective it was.

Methods

90 SVE children were selected as the research objects and divided into control group (39 cases receiving conventional nursing intervention) and observation group (51 cases performed with conventional nursing intervention and TC intervention) according to their nursing purpose. Based on SR-CNN-optimized MRI images, diagnosis was implemented. Life treatment and sequelae in two groups were compared.

Results

After the processing by CNN algorithm-based SR, peak signal to noise ratio (PSNR) (40.08 dB) and structural similarity (SSIM) (0.98) of MRI images were both higher than those of fully connected neural network (FNN) (38.01 dB, 0.93) and recurrent neural network (RNN) (37.21 dB, 0.93) algorithms. Diagnostic sensitivity (95.34%), specificity (75%), and accuracy (94.44%) of MRI images were obviously superior to those of conventional MRI (81.40%, 50%, and 80%). PedsQLTM 4.0 scores of the observation group 1 to 3 months after discharge were all higher than those of the control group (54.55 ± 5.76 vs. 52.32 ± 5.12 and 66.32 ± 8.89 vs. 55.02 ± 5.87). Sequela incidence in the observation group (13.73%) was apparently lower than that in the control group (43.59%) (P < 0.05).

Conclusion

(1) SR-CNN algorithm could increase the definition and diagnostic ability of MRI images. (2) TC could reduce sequelae incidence among SVE children and improve their quality of life (QOL).

Deep learning for improving the spatial resolution of magnetic particle imaging

Objective.Magnetic particle imaging (MPI) is a new medical, non-destructive, imaging method for visualizing the spatial distribution of superparamagnetic iron oxide nanoparticles. In MPI, spatial resolution is an important indicator of efficiency; traditional techniques for improving the spatial resolution may result in higher costs, lower sensitivity, or reduced contrast.Approach.Therefore, we propose a deep-learning approach to improve the spatial resolution of MPI by fusing a dual-sampling convolutional neural network (FDS-MPI). An end-to-end model is established to generate high-spatial-resolution images from low-spatial-resolution images, avoiding the aforementioned shortcomings.Main results.We evaluate the performance of the proposed FDS-MPI model through simulation and phantom experiments. The results demonstrate that the FDS-MPI model can improve the spatial resolution by a factor of two.Significance.This significant improvement in MPI could facilitate the preclinical application of medical imaging modalities in the future.

Deep learning for dose assessment in radiotherapy by the super-localization of vaporized nanodroplets in high frame rate ultrasound imaging

Objective. External beam radiotherapy is aimed to precisely deliver a high radiation dose to malignancies, while optimally sparing surrounding healthy tissues. With the advent of increasingly complex treatment plans, the delivery should preferably be verified by quality assurance methods. Recently, online ultrasound imaging of vaporized radiosensitive nanodroplets was proposed as a promising tool for in vivo dosimetry in radiotherapy. Previously, the detection of sparse vaporization events was achieved by applying differential ultrasound (US) imaging followed by intensity thresholding using subjective parameter tuning, which is sensitive to image artifacts. Approach. A generalized deep learning solution (i.e. BubbleNet) is proposed to localize vaporized nanodroplets on differential US frames, while overcoming the aforementioned limitation. A 5-fold cross-validation was performed on a diversely composed 5747-frame training/validation dataset by manual segmentation. BubbleNet was then applied on a test dataset of 1536 differential US frames to evaluate dosimetric features. The intra-observer variability was determined by scoring the Dice similarity coefficient (DSC) on 150 frames segmented twice. Additionally, the BubbleNet generalization capability was tested on an external test dataset of 432 frames acquired by a phased array transducer at a much lower ultrasound frequency and reconstructed with unconventional pixel dimensions with respect to the training dataset. Main results. The median DSC in the 5-fold cross validation was equal to ∼0.88, which was in line with the intra-observer variability (=0.86). Next, BubbleNet was employed to detect vaporizations in differential US frames obtained during the irradiation of phantoms with a 154 MeV proton beam or a 6 MV photon beam. BubbleNet improved the bubble-count statistics by ∼30% compared to the earlier established intensity-weighted thresholding. The proton range was verified with a −0.8 mm accuracy. Significance. BubbleNet is a flexible tool to localize individual vaporized nanodroplets on experimentally acquired US images, which improves the sensitivity compared to former thresholding-weighted methods.

Subwavelength ultrasonic imaging using a deep convolutional neural network trained on structural noise

Subwavelength ultrasonic imaging (SUI) can detect subwavelength flaws beyond the diffraction limit, however, SUI sometimes fails to clearly reveal flaws in C-scans when the signal-to-noise ratio (SNR) is low. In this work, a convolutional neural network (CNN) that takes structural noise into account is developed for SUI to distinguish flaw echoes from structural noise. The network contains a regression CNN for learning features from the structural noise and a learnable soft thresholding layer for classification. Experiments show that the proposed method performs well for imaging subwavelength flaws at different depths and of different sizes. It achieved an F₁ score of 97.69 ± 1.56% in detecting flaws as compared to the enhanced ultrasonic flaw detection method with time-dependent threshold. As an example of general application of the method, we also performed SUI on natural flaws in a spheroidal graphite cast iron specimen. The results show that the method can achieve SUI without a theoretical backscattering model and is not limited by noise distribution, multiple scattering, or complex microstructures. Furthermore, the network does not need to prepare flaw echoes for training.

Upstream Machine Learning in Radiology

Machine learning (ML) and Artificial intelligence (AI) has the potential to dramatically improve radiology practice at multiple stages of the imaging pipeline. Most of the attention has been garnered by applications focused on improving the end of the pipeline: image interpretation. However, this article reviews how AI/ML can be applied to improve upstream components of the imaging pipeline, including exam modality selection, hardware design, exam protocol selection, data acquisition, image reconstruction, and image processing. A breadth of applications and their potential for impact is shown across multiple imaging modalities, including ultrasound, computed tomography, and MRI.

Ultrasound microvasculature imaging with entropy-based radiality super-resolution (ERSR)

Objective:Microvasculature is highly relevant to the occurrence and development of pathologies such as cancer and diabetes. Ultrasound localization microscopy (ULM) has bypassed the diffraction limit and demonstrated its great potential to provide new imaging modality and establish new diagnostic criteria in clinical application. However, sparse microbubble distribution can be a significant bottleneck for improving temporal resolution, even for further clinical translation. Other important challenges for ULM to tackle in clinic also include high microbubble concentration and low frame rate.Approach:As part of the efforts to facilitate clinical translation, this paper focused on the low frame rate and the high microbubble distribution issue and proposed a new super-resolution imaging strategy called entropy-based radiality super-resolution (ERSR). The feasibility of ERSR is validated with simulations, phantom experiment and contrast-enhanced ultrasound scan of rabbit sciatic nerve with clinical accessible ultrasound system.Main results:ERSR can achieve 10 times improvement in spatial resolution compared to conventional ultrasound imaging, higher temporal resolution (∼10 times higher) and contrast-to-noise ratio under high-density microbubbles, compared with ULM under low-density microbubbles.Significance:We conclude ERSR could be a valuable imaging tool with high spatio-temporal resolution for clinical diagnosis and assessment of diseases potentially.

Faster super-resolution ultrasound imaging with a deep learning model for tissue decluttering and contrast agent localization

Super-resolution ultrasound (SR-US) imaging allows visualization of microvascular structures as small as tens of micrometers in diameter. However, use in the clinical setting has been impeded in part by ultrasound (US) acquisition times exceeding a breath-hold and by the need for extensive offline computation. Deep learning techniques have been shown to be effective in modeling the two more computationally intensive steps of microbubble (MB) contrast agent detection and localization. Performance gains by deep networks over conventional methods are more than two orders of magnitude and in addition the networks can localize overlapping MBs. The ability to separate overlapping MBs allows use of higher contrast agent concentrations and reduces US image acquisition time. Herein we propose a fully convolutional neural network (CNN) architecture to perform the operations of MB detection as well as localization in a single model. Termed SRUSnet, the network is based on the MobileNetV3 architecture modified for 3-D input data, minimal convergence time, and high-resolution data output using a flexible regression head. Also, we propose to combine linear B-mode US imaging and nonlinear contrast pulse sequencing (CPS) which has been shown to increase MB detection and further reduce the US image acquisition time. The network was trained within silicodata and tested onin vitrodata from a tissue-mimicking flow phantom, and onin vivodata from the rat hind limb (N = 3). Images were collected with a programmable US system (Vantage 256, Verasonics Inc., Kirkland, WA) using an L11-4v linear array transducer. The network exceeded 99.9% detection accuracy onin silicodata. The average localization accuracy was smaller than the resolution of a pixel (i.e.λ/8). The average processing time on a Nvidia GeForce 2080Ti GPU was 64.5 ms for a 128 × 128-pixel image.

Energy Migration Control of Multimodal Emissions in an Er3+ -Doped Nanostructure for Information Encryption and Deep-Learning Decoding

Modulating the emission wavelengths of materials has always been a primary focus of fluorescence technology. Nanocrystals (NCs) doped with lanthanide ions with rich energy levels can produce a variety of emissions at different excitation wavelengths. However, the control of multimodal emissions of these ions has remained a challenge. Herein, we present a new composition of Er³⁺ -based lanthanide NCs with color-switchable output under irradiation with 980, 808, or 1535 nm light for information security. The variation of excitation wavelengths changes the intensity ratio of visible (Vis)/near-infrared (NIR-II) emissions. Taking advantage of the Vis/NIR-II multimodal emissions of NCs and deep learning, we successfully demonstrated the storage and decoding of visible light information in pork tissue.

A Deep Learning Framework for Spatiotemporal Ultrasound Localization Microscopy

Ultrasound Localization Microscopy (ULM) can resolve the microvascular bed down to a few micrometers. To achieve such performance, microbubble contrast agents must perfuse the entire microvascular network. Microbubbles are then located individually and tracked over time to sample individual vessels, typically over hundreds of thousands of images. To overcome the fundamental limit of diffraction and achieve a dense reconstruction of the network, low microbubble concentrations must be used, which leads to acquisitions lasting several minutes. Conventional processing pipelines are currently unable to deal with interference from multiple nearby microbubbles, further reducing achievable concentrations. This work overcomes this problem by proposing a Deep Learning approach to recover dense vascular networks from ultrasound acquisitions with high microbubble concentrations. A realistic mouse brain microvascular network, segmented from 2-photon microscopy, was used to train a three-dimensional convolutional neural network (CNN) based on a V-net architecture. Ultrasound data sets from multiple microbubbles flowing through the microvascular network were simulated and used as ground truth to train the 3D CNN to track microbubbles. The 3D-CNN approach was validated in silico using a subset of the data and in vivo in a rat brain. In silico, the CNN reconstructed vascular networks with higher precision (81%) than a conventional ULM framework (70%). In vivo, the CNN could resolve micro vessels as small as 10 μ m with an improvement in resolution when compared against a conventional approach.

Super-resolution ultrasound localization microscopy based on a high frame-rate clinical ultrasound scanner: an in-human feasibility study

Non-invasive detection of microvascular alterations in deep tissuesin vivoprovides critical information for clinical diagnosis and evaluation of a broad-spectrum of pathologies. Recently, the emergence of super-resolution ultrasound localization microscopy (ULM) offers new possibilities for clinical imaging of microvasculature at capillary level. Currently, the clinical utility of ULM on clinical ultrasound scanners is hindered by the technical limitations, such as long data acquisition time, high microbubble (MB) concentration, and compromised tracking performance associated with low imaging frame-rate. Here we present a robust in-human ULM on a high frame-rate (HFR) clinical ultrasound scanner to achieve super-resolution microvessel imaging using a short acquisition time (<10 s). Ultrasound MB data were acquired from different human tissues, including a healthy liver and a diseased liver with acute-on-chronic liver failure, a kidney, a pancreatic tumor, and a breast mass using an HFR clinical scanner. By leveraging the HFR and advanced processing techniques including sub-pixel motion registration, MB signal separation, and Kalman filter-based tracking, MBs can be robustly localized and tracked for ULM under the circumstances of relatively high MB concentration associated with standard clinical MB administration and limited data acquisition time in humans. Subtle morphological and hemodynamic information in microvasculature were shown based on data acquired with single breath-hold and free-hand scanning. Compared with contrast-enhanced power Doppler generated based on the same MB dataset, ULM showed a 5.7-fold resolution improvement in a vessel based on a linear transducer, and provided a wide-range blood flow speed measurement that is Doppler angle-independent. Microvasculatures with complex hemodynamics can be well-differentiated at super-resolution in both normal and pathological tissues. This preliminary study implemented the ultrafast in-human ULM in various human tissues based on a clinical scanner that supports HFR imaging, indicating the potentials of the technique for various clinical applications. However, rigorous validation of the technique in imaging human microvasculature (especially for those tiny vessel structure), preferably with a gold standard, is still required.

Current Development and Applications of Super-Resolution Ultrasound Imaging

Abnormal changes of the microvasculature are reported to be key evidence of the development of several critical diseases, including cancer, progressive kidney disease, and atherosclerotic plaque. Super-resolution ultrasound imaging is an emerging technology that can identify the microvasculature noninvasively, with unprecedented spatial resolution beyond the acoustic diffraction limit. Therefore, it is a promising approach for diagnosing and monitoring the development of diseases. In this review, we introduce current super-resolution ultrasound imaging approaches and their preclinical applications on different animals and disease models. Future directions and challenges to overcome for clinical translations are also discussed.

Fast super-resolution ultrasound microvessel imaging using spatiotemporal data with deep fully convolutional neural network

Ultrasound localization microscopy (ULM) has been proposed to image microvasculature beyond the ultrasound diffraction limit. Although ULM can attain microvascular images with a sub-diffraction resolution, long data acquisition time and processing time are the critical limitations. Deep learning-based ULM (deep-ULM) has been proposed to mitigate these limitations. However, microbubble (MB) localization used in deep-ULMs is currently based on spatial information without the use of temporal information. The highly spatiotemporally coherent MB signals provide a strong feature that can be used to differentiate MB signals from background artifacts. In this study, a deep neural network was employed and trained with spatiotemporal ultrasound datasets to better identify the MB signals by leveraging both the spatial and temporal information of the MB signals. Training, validation and testing datasets were acquired from MB suspension to mimic the realistic intensity-varying and moving MB signals. The performance of the proposed network was first demonstrated in the chicken embryo chorioallantoic membrane dataset with an optical microscopic image as the reference standard. Substantial improvement in spatial resolution was shown for the reconstructed super-resolved images compared with power Doppler images. The full-width-half-maximum (FWHM) of a microvessel was improved from 133μm to 35μm, which is smaller than the ultrasound wavelength (73μm). The proposed method was further tested in anin vivohuman liver data. Results showed the reconstructed super-resolved images could resolve a microvessel of nearly 170μm (FWHM). Adjacent microvessels with a distance of 670μm, which cannot be resolved with power Doppler imaging, can be well-separated with the proposed method. Improved contrast ratios using the proposed method were shown compared with that of the conventional deep-ULM method. Additionally, the processing time to reconstruct a high-resolution ultrasound frame with an image size of 1024 × 512 pixels was around 16 ms, comparable to state-of-the-art deep-ULMs.