A new deep learning method for displacement tracking from ultrasound RF signals of vascular walls

Robust tracking of deformable anatomical structures with severe occlusions using deformable geometrical primitives

BACKGROUND AND OBJECTIVE

Surgical robotics tends to develop cognitive control architectures to provide certain degree of autonomy to improve patient safety and surgery outcomes, while decreasing the required surgeons' cognitive load dedicated to low level decisions. Cognition needs workspace perception, which is an essential step towards automatic decision-making and task planning capabilities. Robust and accurate detection and tracking in minimally invasive surgery suffers from limited visibility, occlusions, anatomy deformations and camera movements.

METHOD

This paper develops a robust methodology to detect and track anatomical structures in real time to be used in automatic control of robotic systems and augmented reality. The work focuses on the experimental validation in highly challenging surgery: fetoscopic repair of Open Spina Bifida. The proposed method is based on two sequential steps: first, selection of relevant points (contour) using a Convolutional Neural Network and, second, reconstruction of the anatomical shape by means of deformable geometric primitives.

RESULTS

The methodology performance was validated with different scenarios. Synthetic scenario tests, designed for extreme validation conditions, demonstrate the safety margin offered by the methodology with respect to the nominal conditions during surgery. Real scenario experiments have demonstrated the validity of the method in terms of accuracy, robustness and computational efficiency.

CONCLUSIONS

This paper presents a robust anatomical structure detection in present of abrupt camera movements, severe occlusions and deformations. Even though the paper focuses on a case study, Open Spina Bifida, the methodology is applicable in all anatomies which contours can be approximated by geometric primitives. The methodology is designed to provide effective inputs to cognitive robotic control and augmented reality systems that require accurate tracking of sensitive anatomies.

A lung disease diagnosis algorithm based on 2D spectral features of ultrasound RF signals

Lung diseases are commonly diagnosed based on clinical pathological indications criteria and radiological imaging tools (e.g., X-rays and CT). During a pandemic like COVID-19, the use of ultrasound imaging devices has broadened for emergency examinations by taking their unique advantages such as portability, real-time detection, easy operation and no radiation. This provides a rapid, safe, and cost-effective imaging modality for screening lung diseases. However, the current pulmonary ultrasound diagnosis mainly relies on the subjective assessments of sonographers, which has high requirements for the operator's professional ability and clinical experience. In this study, we proposed an objective and quantifiable algorithm for the diagnosis of lung diseases that utilizes two-dimensional (2D) spectral features of ultrasound radiofrequency (RF) signals. The ultrasound data samples consisted of a set of RF signal frames, which were collected by professional sonographers. In each case, a region of interest of uniform size was delineated along the pleural line. The standard deviation curve of the 2D spatial spectrum was calculated and smoothed. A linear fit was applied to the high-frequency segment of the processed data curve, and the slope of the fitted line was defined as the frequency spectrum standard deviation slope (FSSDS). Based on the current data, the method exhibited a superior diagnostic sensitivity of 98% and an accuracy of 91% for the identification of lung diseases. The area under the curve obtained by the current method exceeded the results obtained that interpreted by professional sonographers, which indicated that the current method could provide strong support for the clinical ultrasound diagnosis of lung diseases.

Deep learning algorithm using bispectrum analysis energy feature maps based on ultrasound radiofrequency signals to detect breast cancer

Background

Ultrasonography is an important imaging method for clinical breast cancer screening. As the original echo signals of ultrasonography, ultrasound radiofrequency (RF) signals provide abundant tissue macroscopic and microscopic information and have important development and utilization value in breast cancer detection.

Methods

In this study, we proposed a deep learning method based on bispectrum analysis feature maps to process RF signals and realize breast cancer detection. The bispectrum analysis energy feature maps with frequency subdivision were first proposed and applied to breast cancer detection in this study. Our deep learning network was based on a weight sharing network framework for the input of multiple feature maps. A feature map attention module was designed for multiple feature maps input of the network to adaptively learn both feature maps and features that were conducive to classification. We also designed a similarity constraint factor, learning the similarity and difference between feature maps by cosine distance.

Results

The experiment results showed that the areas under the receiver operating characteristic curves of our proposed method in the validation set and two independent test sets for benign and malignant breast tumor classification were 0.913, 0.900, and 0.885, respectively. The performance of the model combining four ultrasound bispectrum analysis energy feature maps in breast cancer detection was superior to that of the model using an ultrasound grayscale image and the model using a single bispectrum analysis energy feature map in this study.

Conclusion

The combination of deep learning technology and our proposed ultrasound bispectrum analysis energy feature maps effectively realized breast cancer detection and was an efficient method of feature extraction and utilization of ultrasound RF signals.

Unsupervised deep learning-based displacement estimation for vascular elasticity imaging applications

Objective. Arterial wall stiffness can provide valuable information on the proper function of the cardiovascular system. Ultrasound elasticity imaging techniques have shown great promise as a low-cost and non-invasive tool to enable localized maps of arterial wall stiffness. Such techniques rely upon motion detection algorithms that provide arterial wall displacement estimation.Approach. In this study, we propose an unsupervised deep learning-based approach, originally proposed for image registration, in order to enable improved quality arterial wall displacement estimation at high temporal and spatial resolutions. The performance of the proposed network was assessed through phantom experiments, where various models were trained by using ultrasound RF signals, or B-mode images, as well as different loss functions.Main results. Using the mean square error (MSE) for the training process provided the highest signal-to-noise ratio when training on the B-modes images (30.36 ± 1.14 dB) and highest contrast-to-noise ratio when training on the RF signals (32.84 ± 1.89 dB). In addition, training the model on RF signals demonstrated the capability of providing accurate localized pulse wave velocity (PWV) maps, with a mean relative error (MREPWV) of 3.32 ± 1.80% and anR2 of 0.97 ± 0.03. Finally, the developed model was tested in human common carotid arteriesin vivo, providing accurate tracking of the distension pulse wave propagation, with an MREPWV= 3.86 ± 2.69% andR2 = 0.95 ± 0.03.Significance. In conclusion, a novel displacement estimation approach was presented, showing promise in improving vascular elasticity imaging techniques.

Vascular wall motion detection models based on long short-term memory in plane-wave-based ultrasound imaging

Objective.Vascular wall motion can be used to diagnose cardiovascular diseases. In this study, long short-term memory (LSTM) neural networks were used to track vascular wall motion in plane-wave-based ultrasound imaging.Approach.The proposed LSTM and convolutional LSTM (ConvLSTM) models were trained using ultrasound data from simulations and tested experimentally using a tissue-mimicking vascular phantom and anin vivostudy using a carotid artery. The performance of the models in the simulation was evaluated using the mean square error from axial and lateral motions and compared with the cross-correlation (XCorr) method. Statistical analysis was performed using the Bland-Altman plot, Pearson correlation coefficient, and linear regression in comparison with the manually annotated ground truth.Main results.For thein vivodata, the median error and 95% limit of agreement from the Bland-Altman analysis were (0.01, 0.13), (0.02, 0.19), and (0.03, 0.18), the Pearson correlation coefficients were 0.97, 0.94, and 0.94, respectively, and the linear equations were 0.89x+ 0.02, 0.84x+ 0.03, and 0.88x+ 0.03 from linear regression for the ConvLSTM model, LSTM model, and XCorr method, respectively. In the longitudinal and transverse views of the carotid artery, the LSTM-based models outperformed the XCorr method. Overall, the ConvLSTM model was superior to the LSTM model and XCorr method.Significance.This study demonstrated that vascular wall motion can be tracked accurately and precisely using plane-wave-based ultrasound imaging and the proposed LSTM-based models.

Osteoporosis Diagnostic Model Using a Multichannel Convolutional Neural Network Based on Quantitative Ultrasound Radiofrequency Signal

Quantitative ultrasound (QUS) is a promising screening method for osteoporosis. In this study, a new method to improve the diagnostic accuracy of QUS was established in which a multichannel convolutional neural network (MCNN) processes the raw radiofrequency (RF) signal of QUS. The improvement in the diagnostic accuracy of osteoporosis using this new method was evaluated by comparison with the conventional speed of sound (SOS) method. Dual-energy X-ray absorptiometry was used as the diagnostic standard. After being trained, validated and tested in a data set consisting of 274 participants, the MCNN model could significantly raise the accuracy of osteoporosis diagnosis compared with the SOS method. The adjusted MCNN model performed even better when adjusted by age, height and weight data. The sensitivity, specificity and accuracy of the adjusted MCNN method for osteoporosis diagnosis were 80.86%, 84.23% and 83.05%, respectively; the corresponding values for SOS were 50.60%, 73.68% and 66.67%. The area under the receiver operating characteristic curve of the adjusted MCNN method was also higher than that of SOS (0.846 vs. 0.679). In conclusion, our study indicates that the MCNN method may be more accurate than the conventional SOS method. The MCNN tool and ultrasound RF signal analysis are promising future developmental directions for QUS in screening for osteoporosis.

Therapeutic Ultrasound Effects on Human Induced Pluripotent Stem Cell Cardiomyocytes Measured Optically and with Spectral Ultrasound

To the best of our knowledge, therapeutic ultrasound (TUS) is thus far an unexplored means of delivering mechanical stimulation to cardiomyocyte cultures, which is necessary to engineer a more mature cardiomyocyte phenotype in vitro. Spectral ultrasound (SUS) may provide a way to non-invasively, non-disruptively and inexpensively monitor growth and change in cell cultures over long periods. Compared with other measurement methods, SUS as an acoustic measurement tool will not be affected by an acoustic therapy, unlike electrical measurement methods, in which motion caused by acoustic therapy can affect measurements. Further SUS has the potential to provide functional as well as morphological information in cell cultures. Human induced pluripotent stem cell cardiomyocytes (iPS-CMs) were imaged with calcium fluorescence microscopy while TUS was being applied. TUS was applied at 600 kHz and 1, 3.4 and 6 W/cm² for a continuous 1 s pulse. Measures of the instantaneous beat frequency, repolarization rate and calcium spike amplitude were calculated from the fluorescence data. At 600 kHz, TUS at 1 and 6 W/cm² had significant effects on the shortening of both the repolarization rate and instantaneous beat rate of the iPS-CMs (p < 0.05), while TUS at 3.4 and 6 W/cm² had significant effects on the shortening of the calcium spike amplitude (p < 0.05). Three SUS measures and one gray-level measure were captured from the iPS-CM monolayers while they were simultaneously being imaged with calcium-labeled confocal microscopy. The gray-level measure performed the best of all SUS measures; however, it was not reliable enough to produce a consistent determination of the beat rate of the cell. Finally, SUS measures were captured using three different transducers while simultaneously applying TUS. A center-of-mass (COM) measure calculated from the wavelet transform scalogram of the time-averaged radiofrequency data revealed that SUS was able to detect a change in the frequency content of the reflected ultrasound at 1 and 6 W/cm² before and after ultrasound application (p < 0.05), showing promise for the ability of SUS to measure changes in the beating behavior of iPS-CMs. Overall, SUS is promising as a method for constant monitoring of dynamic cell and tissue culture and growth.

Dense Convolutional Network and Its Application in Medical Image Analysis

Dense convolutional network (DenseNet) is a hot topic in deep learning research in recent years, which has good applications in medical image analysis. In this paper, DenseNet is summarized from the following aspects. First, the basic principle of DenseNet is introduced; second, the development of DenseNet is summarized and analyzed from five aspects: broaden DenseNet structure, lightweight DenseNet structure, dense unit, dense connection mode, and attention mechanism; finally, the application research of DenseNet in the field of medical image analysis is summarized from three aspects: pattern recognition, image segmentation, and object detection. The network structures of DenseNet are systematically summarized in this paper, which has certain positive significance for the research and development of DenseNet.

Displacement detection with sub-pixel accuracy and high spatial resolution using deep learning

PURPOSE

The purpose of this study was to detect two dimensional and sub-pixel displacement with high spatial resolution using an ultrasonic diagnostic apparatus. Conventional displacement detection methods assume neighborhood uniformity and cannot achieve both high spatial resolution and sub-pixel displacement detection.

METHODS

A deep-learning network that utilizes ultrasound images and output displacement distribution was developed. The network structure was constructed by modifying FlowNet2, a widely used network for optical flow estimation, and a training dataset was developed using ultrasound image simulation. Detection accuracy and spatial resolution were evaluated via simulated ultrasound images, and the clinical usefulness was evaluated with ultrasound images of the liver exposed to high-intensity-focused ultrasound (HIFU). These results were compared to the Lucas-Kanade method, a conventional sub-pixel displacement detection method.

RESULTS

For a displacement within ± 40 µm (± 0.6 pixels), a pixel size of 67 µm, and signal noise of 1%, the accuracy was above 0.5 µm and 0.2 µm, the precision was above 0.4 µm and 0.3 µm, and the spatial resolution was 1.1 mm and 0.8 mm for the lateral and axial displacements, respectively. These improvements were also observed in the experimental data. Visualization of the lateral displacement distribution, which determines the edge of the treated lesion using HIFU, was also realized.

CONCLUSION

Two-dimensional and sub-pixel displacement detection with high spatial resolution was realized using a deep-learning methodology. The proposed method enabled the monitoring of small and local tissue deformations induced by HIFU exposure.