Ultrasound Image Segmentation: A Deeply Supervised Network With Attention to Boundaries

Automatic Segmentation and Alignment of Uterine Shapes from 3D Ultrasound Data

BACKGROUND

The uterus is the most important organ in the female reproductive system. Its shape plays a critical role in fertility and pregnancy outcomes. Advances in medical imaging, such as 3D ultrasound, have significantly improved the exploration of the female genital tract, thereby enhancing gynecological healthcare. Despite well-documented data for organs like the liver and heart, large-scale studies on the uterus are lacking. Existing classifications, such as VCUAM and ESHRE/ESGE, provide different definitions for normal uterine shapes but are not based on real-world measurements. Moreover, the lack of comprehensive datasets significantly hinders research in this area. Our research, part of the larger NURSE study, aims to fill this gap by establishing the shape of a normal uterus using real-world 3D vaginal ultrasound scans. This will facilitate research into uterine shape abnormalities associated with infertility and recurrent miscarriages.

METHODS

We developed an automated system for the segmentation and alignment of uterine shapes from 3D ultrasound data, which consists of two steps: automatic segmentation of the uteri in 3D ultrasound scans using deep learning techniques, and alignment of the resulting shapes with standard geometrical approaches, enabling the extraction of the normal shape for future analysis. The system was trained and validated on a comprehensive dataset of 3D ultrasound images from multiple medical centers. Its performance was evaluated by comparing the automated results with manual annotations provided by expert clinicians.

RESULTS

The presented approach demonstrated high accuracy in segmenting and aligning uterine shapes from 3D ultrasound data. The segmentation achieved an average Dice similarity coefficient (DSC) of 0.90. Our method for aligning uterine shapes showed minimal translation and rotation errors compared to traditional methods, with the preliminary average shape exhibiting characteristics consistent with expert findings of a normal uterus.

CONCLUSION

We have presented an approach to automatically segment and align uterine shapes from 3D ultrasound data. We trained a deep learning nnU-Net model that achieved high accuracy and proposed an alignment method using a combination of standard geometrical techniques. Additionally, we have created a publicly available dataset of 3D transvaginal ultrasound volumes with manual annotations of uterine cavities to support further research and development in this field. The dataset and the trained models are available at https://github.com/UL-FRI-LGM/UterUS.

A transformer-guided cross-modality adaptive feature fusion framework for esophageal gross tumor volume segmentation

BACKGROUND AND OBJECTIVE

Accurate segmentation of esophageal gross tumor volume (GTV) indirectly enhances the efficacy of radiotherapy for patients with esophagus cancer. In this domain, learning-based methods have been employed to fuse cross-modality positron emission tomography (PET) and computed tomography (CT) images, aiming to improve segmentation accuracy. This fusion is essential as it combines functional metabolic information from PET with anatomical information from CT, providing complementary information. While the existing three-dimensional (3D) segmentation method has achieved state-of-the-art (SOTA) performance, it typically relies on pure-convolution architectures, limiting its ability to capture long-range spatial dependencies due to convolution's confinement to a local receptive field. To address this limitation and further enhance esophageal GTV segmentation performance, this work proposes a transformer-guided cross-modality adaptive feature fusion network, referred to as TransAttPSNN, which is based on cross-modality PET/CT scans.

METHODS

Specifically, we establish an attention progressive semantically-nested network (AttPSNN) by incorporating the convolutional attention mechanism into the progressive semantically-nested network (PSNN). Subsequently, we devise a plug-and-play transformer-guided cross-modality adaptive feature fusion model, which is inserted between the multi-scale feature counterparts of a two-stream AttPSNN backbone (one for the PET modality flow and another for the CT modality flow), resulting in the proposed TransAttPSNN architecture.

RESULTS

Through extensive four-fold cross-validation experiments on the clinical PET/CT cohort. The proposed approach acquires a Dice similarity coefficient (DSC) of 0.76 ± 0.13, a Hausdorff distance (HD) of 9.38 ± 8.76 mm, and a Mean surface distance (MSD) of 1.13 ± 0.94 mm, outperforming the SOTA competing methods. The qualitative results show a satisfying consistency with the lesion areas.

CONCLUSIONS

The devised transformer-guided cross-modality adaptive feature fusion module integrates the strengths of PET and CT, effectively enhancing the segmentation performance of esophageal GTV. The proposed TransAttPSNN has further advanced the research of esophageal GTV segmentation.

Uncertainty-aware refinement framework for ovarian tumor segmentation in CECT volume

BACKGROUND

Ovarian cancer is a highly lethal gynecological disease. Accurate and automated segmentation of ovarian tumors in contrast-enhanced computed tomography (CECT) images is crucial in the radiotherapy treatment of ovarian cancer, enabling radiologists to evaluate cancer progression and develop timely therapeutic plans. However, automatic ovarian tumor segmentation is challenging due to factors such as inhomogeneous background, ambiguous tumor boundaries, and imbalanced foreground-background, all of which contribute to high predictive uncertainty for a segmentation model.

PURPOSE

To tackle these challenges, we propose an uncertainty-aware refinement framework that aims to estimate and refine regions with high predictive uncertainty for accurate ovarian tumor segmentation in CECT images.

METHODS

To this end, we first employ an approximate Bayesian network to detect coarse regions of interest (ROIs) of both ovarian tumors and uncertain regions. These ROIs allow a subsequent segmentation network to narrow down the search area for tumors and prioritize uncertain regions, resulting in precise segmentation of ovarian tumors. Meanwhile, the framework integrates two guidance modules that learn two implicit functions capable of mapping query features sampled according to their uncertainty to organ or boundary manifolds, guiding the segmentation network to facilitate information encoding of uncertain regions.

RESULTS

Firstly, 367 CECT images are collected from the same hospital for experiments. Dice score, Jaccard, Recall, Positive predictive value (PPV), 95% Hausdorff distance (HD95) and Average symmetric surface distance (ASSD) for the testing group of 77 cases are 86.31%, 73.93%, 83.95%, 86.03%, 15.17  mm and 2.57  mm, all of which are significantly better than that of the other state-of-the-art models. And results of visual comparison shows that the compared methods have more mis-segmentation than our method. Furthermore, our method achieves a Dice score that is at least 20% higher than the Dice scores of other compared methods when tumor volumes are less than 20 cm3 $^3$ , indicating better recognition ability to small regions by our method. And then, 38 CECT images are collected from another hospital to form an external testing group. Our approach consistently outperform the compared methods significantly, with the external testing group exhibiting substantial improvements across key evaluation metrics: Dice score (83.74%), Jaccard (69.55%), Recall (82.12%), PPV (81.61%), HD95 (12.31 mm), and ASSD (2.32 mm), robustly establishing its superior performance.

CONCLUSIONS

Experimental results demonstrate that the framework significantly outperforms the compared state-of-the-art methods, with decreased under- or over-segmentation and better small tumor identification. It has the potential for clinical application.

Artificial intelligence-based speckle featurization and localization for ultrasound speckle tracking velocimetry

Deep learning-based super-resolution ultrasound (DL-SRU) framework has been successful in improving spatial resolution and measuring the velocity field information of a blood flows by localizing and tracking speckle signals of red blood cells (RBCs) without using any contrast agents. However, DL-SRU can localize only a small part of the speckle signals of blood flow owing to ambiguity problems encountered in the classification of blood flow signals from ultrasound B-mode images and the building up of suitable datasets required for training artificial neural networks, as well as the structural limitations of the neural network itself. An artificial intelligence-based speckle featurization and localization (AI-SFL) framework is proposed in this study. It includes a machine learning-based algorithm for classifying blood flow signals from ultrasound B-mode images, dimensionality reduction for featurizing speckle patterns of the classified blood flow signals by approximating them with quantitative values. A novel and robust neural network (ResSU-net) is trained using the online data generation (ODG) method and the extracted speckle features. The super-resolution performance of the proposed AI-SFL and ODG method is evaluated and compared with the results of previous U-net and conventional data augmentation methods under in silico conditions. The predicted locations of RBCs by the AI-SFL and DL-SRU for speckle patterns of blood flow are applied to a PTV algorithm to measure quantitative velocity fields of the flow. Finally, the feasibility of the proposed AI-SFL framework for measuring real blood flows is verified under in vivo conditions.

Deep Learning Paradigm and Its Bias for Coronary Artery Wall Segmentation in Intravascular Ultrasound Scans: A Closer Look

BACKGROUND AND MOTIVATION

Coronary artery disease (CAD) has the highest mortality rate; therefore, its diagnosis is vital. Intravascular ultrasound (IVUS) is a high-resolution imaging solution that can image coronary arteries, but the diagnosis software via wall segmentation and quantification has been evolving. In this study, a deep learning (DL) paradigm was explored along with its bias.

METHODS

Using a PRISMA model, 145 best UNet-based and non-UNet-based methods for wall segmentation were selected and analyzed for their characteristics and scientific and clinical validation. This study computed the coronary wall thickness by estimating the inner and outer borders of the coronary artery IVUS cross-sectional scans. Further, the review explored the bias in the DL system for the first time when it comes to wall segmentation in IVUS scans. Three bias methods, namely (i) ranking, (ii) radial, and (iii) regional area, were applied and compared using a Venn diagram. Finally, the study presented explainable AI (XAI) paradigms in the DL framework.

FINDINGS AND CONCLUSIONS

UNet provides a powerful paradigm for the segmentation of coronary walls in IVUS scans due to its ability to extract automated features at different scales in encoders, reconstruct the segmented image using decoders, and embed the variants in skip connections. Most of the research was hampered by a lack of motivation for XAI and pruned AI (PAI) models. None of the UNet models met the criteria for bias-free design. For clinical assessment and settings, it is necessary to move from a paper-to-practice approach.

Deep Learning Approaches for Automatic Quality Assurance of Magnetic Resonance Images Using ACR Phantom

BACKGROUND

In recent years, there has been a growing trend towards utilizing Artificial Intelligence (AI) and machine learning techniques in medical imaging, including for the purpose of automating quality assurance. In this research, we aimed to develop and evaluate various deep learning-based approaches for automatic quality assurance of Magnetic Resonance (MR) images using the American College of Radiology (ACR) standards.

METHODS

The study involved the development, optimization, and testing of custom convolutional neural network (CNN) models. Additionally, popular pre-trained models such as VGG16, VGG19, ResNet50, InceptionV3, EfficientNetB0, and EfficientNetB5 were trained and tested. The use of pre-trained models, particularly those trained on the ImageNet dataset, for transfer learning was also explored. Two-class classification models were employed for assessing spatial resolution and geometric distortion, while an approach classifying the image into 10 classes representing the number of visible spokes was used for the low contrast.

RESULTS

Our results showed that deep learning-based methods can be effectively used for MR image quality assurance and can improve the performance of these models. The low contrast test was one of the most challenging tests within the ACR phantom.

CONCLUSIONS

Overall, for geometric distortion and spatial resolution, all of the deep learning models tested produced prediction accuracy of 80% or higher. The study also revealed that training the models from scratch performed slightly better compared to transfer learning. For the low contrast, our investigation emphasized the adaptability and potential of deep learning models. The custom CNN models excelled in predicting the number of visible spokes, achieving commendable accuracy, recall, precision, and F1 scores.

Anatomically Guided Cross-Domain Repair and Screening for Ultrasound Fetal Biometry

Ultrasound based estimation of fetal biometry is extensively used to diagnose prenatal abnormalities and to monitor fetal growth, for which accurate segmentation of the fetal anatomy is a crucial prerequisite. Although deep neural network-based models have achieved encouraging results on this task, inevitable distribution shifts in ultrasound images can still result in severe performance drop in real world deployment scenarios. In this article, we propose a complete ultrasound fetal examination system to deal with this troublesome problem by repairing and screening the anatomically implausible results. Our system consists of three main components: A routine segmentation network, a fetal anatomical key points guided repair network, and a shape-coding based selective screener. Guided by the anatomical key points, our repair network has stronger cross-domain repair capabilities, which can substantially improve the outputs of the segmentation network. By quantifying the distance between an arbitrary segmentation mask to its corresponding anatomical shape class, the proposed shape-coding based selective screener can then effectively reject the entire implausible results that cannot be fully repaired. Extensive experiments demonstrate that our proposed framework has strong anatomical guarantee and outperforms other methods in three different cross-domain scenarios.

TS-GCN: A novel tumor segmentation method integrating transformer and GCN

As one of the critical branches of medical image processing, the task of segmentation of breast cancer tumors is of great importance for planning surgical interventions, radiotherapy and chemotherapy. Breast cancer tumor segmentation faces several challenges, including the inherent complexity and heterogeneity of breast tissue, the presence of various imaging artifacts and noise in medical images, low contrast between the tumor region and healthy tissue, and inconsistent size of the tumor region. Furthermore, the existing segmentation methods may not fully capture the rich spatial and contextual information in small-sized regions in breast images, leading to suboptimal performance. In this paper, we propose a novel breast tumor segmentation method, called the transformer and graph convolutional neural (TS-GCN) network, for medical imaging analysis. Specifically, we designed a feature aggregation network to fuse the features extracted from the transformer, GCN and convolutional neural network (CNN) networks. The CNN extract network is designed for the image's local deep feature, and the transformer and GCN networks can better capture the spatial and context dependencies among pixels in images. By leveraging the strengths of three feature extraction networks, our method achieved superior segmentation performance on the BUSI dataset and dataset B. The TS-GCN showed the best performance on several indexes, with Acc of 0.9373, Dice of 0.9058, IoU of 0.7634, F1 score of 0.9338, and AUC of 0.9692, which outperforms other state-of-the-art methods. The research of this segmentation method provides a promising future for medical image analysis and diagnosis of other diseases.

Intelligent contour extraction approach for accurate segmentation of medical ultrasound images

Introduction: Accurate contour extraction in ultrasound images is of great interest for image-guided organ interventions and disease diagnosis. Nevertheless, it remains a problematic issue owing to the missing or ambiguous outline between organs (i.e., prostate and kidney) and surrounding tissues, the appearance of shadow artifacts, and the large variability in the shape of organs. Methods: To address these issues, we devised a method that includes four stages. In the first stage, the data sequence is acquired using an improved adaptive selection principal curve method, in which a limited number of radiologist defined data points are adopted as the prior. The second stage then uses an enhanced quantum evolution network to help acquire the optimal neural network. The third stage involves increasing the precision of the experimental outcomes after training the neural network, while using the data sequence as the input. In the final stage, the contour is smoothed using an explicable mathematical formula explained by the model parameters of the neural network. Results: Our experiments showed that our approach outperformed other current methods, including hybrid and Transformer-based deep-learning methods, achieving an average Dice similarity coefficient, Jaccard similarity coefficient, and accuracy of 95.7 ± 2.4%, 94.6 ± 2.6%, and 95.3 ± 2.6%, respectively. Discussion: This work develops an intelligent contour extraction approach on ultrasound images. Our approach obtained more satisfactory outcome compared with recent state-of-the-art approaches . The knowledge of precise boundaries of the organ is significant for the conservation of risk structures. Our developed approach has the potential to enhance disease diagnosis and therapeutic outcomes.

Deep learning for the rapid automatic segmentation of forearm muscle boundaries from ultrasound datasets

Ultrasound (US) is widely used in the clinical diagnosis and treatment of musculoskeletal diseases. However, the low efficiency and non-uniformity of artificial recognition hinder the application and popularization of US for this purpose. Herein, we developed an automatic muscle boundary segmentation tool for US image recognition and tested its accuracy and clinical applicability. Our dataset was constructed from a total of 465 US images of the flexor digitorum superficialis (FDS) from 19 participants (10 men and 9 women, age 27.4 ± 6.3 years). We used the U-net model for US image segmentation. The U-net output often includes several disconnected regions. Anatomically, the target muscle usually only has one connected region. Based on this principle, we designed an algorithm written in C++ to eliminate redundantly connected regions of outputs. The muscle boundary images generated by the tool were compared with those obtained by professionals and junior physicians to analyze their accuracy and clinical applicability. The dataset was divided into five groups for experimentation, and the average Dice coefficient, recall, and accuracy, as well as the intersection over union (IoU) of the prediction set in each group were all about 90%. Furthermore, we propose a new standard to judge the segmentation results. Under this standard, 99% of the total 150 predicted images by U-net are excellent, which is very close to the segmentation result obtained by professional doctors. In this study, we developed an automatic muscle segmentation tool for US-guided muscle injections. The accuracy of the recognition of the muscle boundary was similar to that of manual labeling by a specialist sonographer, providing a reliable auxiliary tool for clinicians to shorten the US learning cycle, reduce the clinical workload, and improve injection safety.

Creation of the Forearm 3D-Model with Veins from Transversal Ultrasonography Image Sequence

This study developed an automatic detection algorithm of vessel and skin regions in a transversal ultrasonography image on the arm. We also developed an algorithm to generate a 3D model from detected areas to assist vein puncture. In the algorithm, the vessel's candidate regions in the ultrasonography image were detected using U-Net or Mask R-CNN, which are a kind of deep learning method for segmentation. Then vessel regions were selected among the candidates based on continuous properties in an image sequence. The skin regions were also detected. The 3D polygon data was created from paired pixels in sequential images. The experiments demonstrated that Mask R-CNN could correctly estimate the branch of vessel which were difficult to identify accurate region separately using U-Net, and achieved an overall IoU of 80%. The confirmation experiment of 3D model demonstrated that generated model have enough feasibility for assessment of appropriate veins and locations for puncture.Clinical relevance-The developed 3D model generation from ultrasonography images will be useful for support to identify the appropriate veins for puncture.

CNN-based automatic segmentations and radiomics feature reliability on contrast-enhanced ultrasound images for renal tumors

Objective

To investigate the feasibility and efficiency of automatic segmentation of contrast-enhanced ultrasound (CEUS) images in renal tumors by convolutional neural network (CNN) based models and their further application in radiomic analysis.

Materials and methods

From 94 pathologically confirmed renal tumor cases, 3355 CEUS images were extracted and randomly divided into training set (3020 images) and test set (335 images). According to the histological subtypes of renal cell carcinoma, the test set was further split into clear cell renal cell carcinoma (ccRCC) set (225 images), renal angiomyolipoma (AML) set (77 images) and set of other subtypes (33 images). Manual segmentation was the gold standard and serves as ground truth. Seven CNN-based models including DeepLabV3+, UNet, UNet++, UNet3+, SegNet, MultilResUNet and Attention UNet were used for automatic segmentation. Python 3.7.0 and Pyradiomics package 3.0.1 were used for radiomic feature extraction. Performance of all approaches was evaluated by the metrics of mean intersection over union (mIOU), dice similarity coefficient (DSC), precision, and recall. Reliability and reproducibility of radiomics features were evaluated by the Pearson coefficient and the intraclass correlation coefficient (ICC).

Results

All seven CNN-based models achieved good performance with the mIOU, DSC, precision and recall ranging between 81.97%-93.04%, 78.67%-92.70%, 93.92%-97.56%, and 85.29%-95.17%, respectively. The average Pearson coefficients ranged from 0.81 to 0.95, and the average ICCs ranged from 0.77 to 0.92. The UNet++ model showed the best performance with the mIOU, DSC, precision and recall of 93.04%, 92.70%, 97.43% and 95.17%, respectively. For ccRCC, AML and other subtypes, the reliability and reproducibility of radiomic analysis derived from automatically segmented CEUS images were excellent, with the average Pearson coefficients of 0.95, 0.96 and 0.96, and the average ICCs for different subtypes were 0.91, 0.93 and 0.94, respectively.

Conclusion

This retrospective single-center study showed that the CNN-based models had good performance on automatic segmentation of CEUS images for renal tumors, especially the UNet++ model. The radiomics features extracted from automatically segmented CEUS images were feasible and reliable, and further validation by multi-center research is necessary.

Real-Time Deep Recognition of Standardized Liver Ultrasound Scan Locations

Liver ultrasound (US) plays a critical role in diagnosing liver diseases. However, it is often difficult for examiners to accurately identify the liver segments captured in US images due to patient variability and the complexity of US images. Our study aim is automatic, real-time recognition of standardized US scans coordinated with reference liver segments to guide examiners. We propose a novel deep hierarchical architecture for classifying liver US images into 11 standardized US scans, which has yet to be properly established due to excessive variability and complexity. We address this problem based on a hierarchical classification of 11 US scans with different features applied to individual hierarchies as well as a novel feature space proximity analysis for handling ambiguous US images. Experiments were performed using US image datasets obtained from a hospital setting. To evaluate the performance under patient variability, we separated the training and testing datasets into distinct patient groups. The experimental results show that the proposed method achieved an F1-score of more than 93%, which is more than sufficient for a tool to guide examiners. The superior performance of the proposed hierarchical architecture was demonstrated by comparing its performance with that of non-hierarchical architecture.

Skin lesion segmentation using two-phase cross-domain transfer learning framework

BACKGROUND AND OBJECTIVE

Deep learning (DL) models have been used for medical imaging for a long time but they did not achieve their full potential in the past because of insufficient computing power and scarcity of training data. In recent years, we have seen substantial growth in DL networks because of improved technology and an abundance of data. However, previous studies indicate that even a well-trained DL algorithm may struggle to generalize data from multiple sources because of domain shifts. Additionally, ineffectiveness of basic data fusion methods, complexity of segmentation target and low interpretability of current DL models limit their use in clinical decisions. To meet these challenges, we present a new two-phase cross-domain transfer learning system for effective skin lesion segmentation from dermoscopic images.

METHODS

Our system is based on two significant technical inventions. We examine a two- phase cross-domain transfer learning approach, including model-level and data-level transfer learning, by fine-tuning the system on two datasets, MoleMap and ImageNet. We then present nSknRSUNet, a high-performing DL network, for skin lesion segmentation using broad receptive fields and spatial edge attention feature fusion. We examine the trained model's generalization capabilities on skin lesion segmentation to quantify these two inventions. We cross-examine the model using two skin lesion image datasets, MoleMap and HAM10000, obtained from varied clinical contexts.

RESULTS

At data-level transfer learning for the HAM10000 dataset, the proposed model obtained 94.63% of DSC and 99.12% accuracy. In cross-examination at data-level transfer learning for the Molemap dataset, the proposed model obtained 93.63% of DSC and 97.01% of accuracy.

CONCLUSION

Numerous experiments reveal that our system produces excellent performance and improves upon state-of-the-art methods on both qualitative and quantitative measures.

SCCNet: Self-correction boundary preservation with a dynamic class prior filter for high-variability ultrasound image segmentation

The highly ambiguous nature of boundaries and similar objects is difficult to address in some ultrasound image segmentation tasks, such as neck muscle segmentation, leading to unsatisfactory performance. Thus, this paper proposes a two-stage network called SCCNet (self-correction context network) using a self-correction boundary preservation module and class-context filter to alleviate these problems. The proposed self-correction boundary preservation module uses a dynamic key boundary point (KBP) map to increase the capability of iteratively discriminating ambiguous boundary points segments, and the predicted segmentation map from one stage is used to obtain a dynamic class prior filter to improve the segmentation performance at Stage 2. Finally, three datasets, Neck Muscle, CAMUS and Thyroid, are used to demonstrate that our proposed SCCNet outperforms other state-of-the art methods, such as BPBnet, DSNnet, and RAGCnet. Our proposed network shows at least a 1.2-3.7% improvement on the three datasets, Neck Muscle, Thyroid, and CAMUS. The source code is available at https://github.com/lijixing0425/SCCNet.

Prostate Ultrasound Image Segmentation Based on DSU-Net

In recent years, the incidence of prostate cancer in the male population has been increasing year by year. Transrectal ultrasound (TRUS) is an important means of prostate cancer diagnosis. The accurate segmentation of the prostate in TRUS images can assist doctors in needle biopsy and surgery and is also the basis for the accurate identification of prostate cancer. Due to the asymmetric shape and blurred boundary line of the prostate in TRUS images, it is difficult to obtain accurate segmentation results with existing segmentation methods. Therefore, a prostate segmentation method called DSU-Net is proposed in this paper. This proposed method replaces the basic convolution in the U-Net model with the improved convolution combining shear transformation and deformable convolution, making the network more sensitive to border features and more suitable for prostate segmentation tasks. Experiments show that DSU-Net has higher accuracy than other existing traditional segmentation methods.

3D pyramidal densely connected network with cross-frame uncertainty guidance for intravascular ultrasound sequence segmentation

Objective. Automatic extraction of external elastic membrane border (EEM) and lumen-intima border (LIB) in intravascular ultrasound (IVUS) sequences aids atherosclerosis diagnosis. Existing IVUS segmentation networks ignored longitudinal relations among sequential images and neglected that IVUS images of different vascular conditions vary largely in intricacy and informativeness. As a result, they suffered from performance degradation in complicated parts in IVUS sequences.Approach. In this paper, we develop a 3D Pyramidal Densely-connected Network (PDN) with Adaptive learning and post-Correction guided by a novel cross-frame uncertainty (CFU). The proposed method is named PDN-AC. Specifically, the PDN enables the longitudinal information exploitation and the effective perception of size-varied vessel regions in IVUS samples, by pyramidally connecting multi-scale 3D dilated convolutions. Additionally, the CFU enhances the robustness of the method to complicated pathology from the frame-level (f-CFU) and pixel-level (p-CFU) via exploiting cross-frame knowledge in IVUS sequences. The f-CFU weighs the complexity of IVUS frames and steers an adaptive sampling during the PDN training. The p-CFU visualizes uncertain pixels probably misclassified by the PDN and guides an active contour-based post-correction.Main results. Human and animal experiments were conducted on IVUS datasets acquired from atherosclerosis patients and pigs. Results showed that the f-CFU weighted adaptive sampling reduced the Hausdorff distance (HD) by 10.53%/7.69% in EEM/LIB detection. Improvements achieved by the p-CFU guided post-correction were 2.94%/5.56%.Significance. The PDN-AC attained mean Jaccard values of 0.90/0.87 and HD values of 0.33/0.34 mm in EEM/LIB detection, preferable to state-of-the-art IVUS segmentation methods.

Dense-PSP-UNet: A neural network for fast inference liver ultrasound segmentation

Liver Ultrasound (US) or sonography is popularly used because of its real-time output, low-cost, ease-of-use, portability, and non-invasive nature. Segmentation of real-time liver US is essential for diagnosing and analyzing liver conditions (e.g., hepatocellular carcinoma (HCC)), assisting the surgeons/radiologists in therapeutic procedures. In this paper, we propose a method using a modified Pyramid Scene Parsing (PSP) module in tuned neural network backbones to achieve real-time segmentation without compromising the segmentation accuracy. Considering widespread noise in US data and its impact on outcomes, we study the impact of pre-processing and the influence of loss functions on segmentation performance. We have tested our method after annotating a publicly available US dataset containing 2400 images of 8 healthy volunteers (link to the annotated dataset is provided); the results show that the Dense-PSP-UNet model achieves a high Dice coefficient of 0.913±0.024 while delivering a real-time performance of 37 frames per second (FPS).

An attention base U-net for parotid tumor autosegmentation

A parotid neoplasm is an uncommon condition that only accounts for less than 3% of all head and neck cancers, and they make up less than 0.3% of all new cancers diagnosed annually. Due to their nonspecific imaging features and heterogeneous nature, accurate preoperative diagnosis remains a challenge. Automatic parotid tumor segmentation may help physicians evaluate these tumors. Two hundred eighty-five patients diagnosed with benign or malignant parotid tumors were enrolled in this study. Parotid and tumor tissues were segmented by 3 radiologists on T1-weighted (T1w), T2-weighted (T2w) and T1-weighted contrast-enhanced (T1wC) MR images. These images were randomly divided into two datasets, including a training dataset (90%) and an validation dataset (10%). A 10-fold cross-validation was performed to assess the performance. An attention base U-net for parotid tumor autosegmentation was created on the MRI T1w, T2 and T1wC images. The results were evaluated in a separate dataset, and the mean Dice similarity coefficient (DICE) for both parotids was 0.88. The mean DICE for left and right tumors was 0.85 and 0.86, respectively. These results indicate that the performance of this model corresponds with the radiologist's manual segmentation. In conclusion, an attention base U-net for parotid tumor autosegmentation may assist physicians to evaluate parotid gland tumors.

Ultrasonic breast tumor extraction based on adversarial mechanism and active contour

BACKGROUND AND OBJECTIVE

Breast cancer is a high incidence of gynecological diseases; breast ultrasound screening can effectively reduce the mortality rate of breast cancer. In breast ultrasound images, the localization and segmentation of tumor lesions are important steps for the extraction of lesions, which helps clinicians evaluate breast lesions quantitatively and makes better clinical diagnosis of the disease. However, the segmentation of breast lesions is difficult due to the blurred and uneven edges of some lesions. In this paper, we propose a segmentation framework combining active contour module and deep learning adversarial mechanism and apply it for the segmentation of breast tumor lesions.

METHOD

We use a conditional adversarial network as the main framework. The generator is a segmentation network consisting of a Deformed U-Net and an active contour module. Here, the Deformed U-Net performs pixel-level segmentation for breast ultrasound images. The active contour module refines the tumor lesion edges, and the refined result provides loss information for Deformed U-Net. Therefore, the Deformed U-Net can better classify the edge pixels. The discriminator is the Markov discriminator; this discriminator provides loss feedback for the segmentation network. We cross-train the discriminator and segmentation network to implement Adversarial Mechanism for getting a more optimized segmentation network.

RESULTS

The segmentation performance of the segmentation network for breast ultrasound images is improved by adding a Markov discriminator to provide discriminant loss training. The proposed method for segmenting the tumor lesions in breast ultrasound image obtains dice coefficient: 89.7%, accuracy: 98.1%, precision: 86.3%, mean-intersection-over-union: 82.2%, recall: 94.7%, specificity: 98.5% and F1score: 89.7%.

CONCLUSION

Comparing with traditional methods, the proposed method gives better performance. The experimental results show that the proposed method can effectively segment the lesions in breast ultrasound images, and then assist doctors to realize the diagnosis of breast lesions.

Structure boundary-preserving U-Net for prostate ultrasound image segmentation

Prostate cancer diagnosis is performed under ultrasound-guided puncture for pathological cell extraction. However, determining accurate prostate location remains a challenge from two aspects: (1) prostate boundary in ultrasound images is always ambiguous; (2) the delineation of radiologists always occupies multiple pixels, leading to many disturbing points around the actual contour. We proposed a boundary structure-preserving U-Net (BSP U-Net) in this paper to achieve precise prostate contour. BSP U-Net incorporates prostate shape prior to traditional U-Net. The prior shape is built by the key point selection module, which is an active shape model-based method. Then, the module plugs into the traditional U-Net structure network to achieve prostate segmentation. The experiments were conducted on two datasets: PH2 + ISBI 2016 challenge and our private prostate ultrasound dataset. The results on PH2 + ISBI 2016 challenge achieved a Dice similarity coefficient (DSC) of 95.94% and a Jaccard coefficient (JC) of 88.58%. The results of prostate contour based on our method achieved a higher pixel accuracy of 97.05%, a mean intersection over union of 93.65%, a DSC of 92.54%, and a JC of 93.16%. The experimental results show that the proposed BSP U-Net has good performance on PH2 + ISBI 2016 challenge and prostate ultrasound image segmentation and outperforms other state-of-the-art methods.

Aux-MVNet: Auxiliary Classifier-Based Multi-View Convolutional Neural Network for Maxillary Sinusitis Diagnosis on Paranasal Sinuses View

Computed tomography (CT) is undoubtedly the most reliable and the only method for accurate diagnosis of sinusitis, while X-ray has long been used as the first imaging technique for early detection of sinusitis symptoms. More importantly, radiography plays a key role in determining whether or not a CT examination should be performed for further evaluation. In order to simplify the diagnostic process of paranasal sinus view and moreover to avoid the use of CT scans which have disadvantages such as high radiation dose, high cost, and high time consumption, this paper proposed a multi-view CNN able to faithfully estimate the severity of sinusitis. In this study, a multi-view convolutional neural network (CNN) is proposed which is able to accurately estimate the severity of sinusitis by analyzing only radiographs consisting of Waters’ view and Caldwell’s view without the aid of CT scans. The proposed network is designed as a cascaded architecture, and can simultaneously provide decisions for maxillary sinus localization and sinusitis classification. We obtained an average area under the curve (AUC) of 0.722 for maxillary sinusitis classification, and an AUC of 0.750 and 0.700 for the left and right maxillary sinusitis, respectively, using the proposed network.

TypeSeg: A type-aware encoder-decoder network for multi-type ultrasound images co-segmentation

PURPOSE

As a portable and radiation-free imaging modality, ultrasound can be easily used to image various types of tissue structures. It is important to develop a method which supports the multi-type ultrasound images co-segmentation. However, state-of-the-art ultrasound segmentation methods commonly only focus on the single type images or ignore the type-aware information.

METHODS

To solve the above problem, this work proposes a novel type-aware encoder-decoder network (TypeSeg) for the multi-type ultrasound images co-segmentation. First, we develop a type-aware metric learning module to find an optimum latent feature space where the ultrasound images of the same types are close and that of the different types are separated by a certain margin. Second, depending on the extracted features, a decision module decides whether the input ultrasound images have the common tissue type or not, and the encoder-decoder network produces a segmentation mask accordingly.

RESULTS

We evaluate the performance of the proposed TypeSeg model on the ultrasound dataset that contains four types of tissues. The proposed TypeSeg model achieves the overall best results with the mean IOU score of 87.51% ± 3.93% for the multi-type ultrasound images.

CONCLUSION

The experimental results indicate that the proposed method outperforms all the compared state-of-the-art algorithms for the multi-type ultrasound images co-segmentation task.

Edge-Enhancement DenseNet for X-ray Fluoroscopy Image Denoising in Cardiac Electrophysiology Procedures

Purpose

Reducing X‐ray dose increases safety in cardiac electrophysiology procedures but also increases image noise and artifacts which may affect the discernibility of devices and anatomical cues. Previous denoising methods based on convolutional neural networks (CNNs) have shown improvements in the quality of low‐dose X‐ray fluoroscopy images but may compromise clinically important details required by cardiologists.

Methods

In order to obtain denoised X‐ray fluoroscopy images whilst preserving details, we propose a novel deep‐learning‐based denoising framework, namely edge‐enhancement densenet (EEDN), in which an attention‐awareness edge‐enhancement module is designed to increase edge sharpness. In this framework, a CNN‐based denoiser is first used to generate an initial denoising result. Contours representing edge information are then extracted using an attention block and a group of interacted ultra‐dense blocks for edge feature representation. Finally, the initial denoising result and enhanced edges are combined to generate the final X‐ray image. The proposed denoising framework was tested on a total of 3262 clinical images taken from 100 low‐dose X‐ray sequences acquired from 20 patients. The performance was assessed by pairwise voting from five cardiologists as well as quantitative indicators. Furthermore, we evaluated our technique's effect on catheter detection using 416 images containing coronary sinus catheters in order to examine its influence as a pre‐processing tool.

Results

The average signal‐to‐noise ratio of X‐ray images denoised with EEDN was 24.5, which was 2.2 times higher than that of the original images. The accuracy of catheter detection from EEDN denoised sequences showed no significant difference compared with their original counterparts. Moreover, EEDN received the highest average votes in our clinician assessment when compared to our existing technique and the original images.

Conclusion

The proposed deep learning‐based framework shows promising capability for denoising interventional X‐ray fluoroscopy images. The results from the catheter detection show that the network does not affect the results of such an algorithm when used as a pre‐processing step. The extensive qualitative and quantitative evaluations suggest that the network may be of benefit to reduce radiation dose when applied in real time in the catheter laboratory.

Weakly-supervised deep learning for ultrasound diagnosis of breast cancer

Conventional deep learning (DL) algorithm requires full supervision of annotating the region of interest (ROI) that is laborious and often biased. We aimed to develop a weakly-supervised DL algorithm that diagnosis breast cancer at ultrasound without image annotation. Weakly-supervised DL algorithms were implemented with three networks (VGG16, ResNet34, and GoogLeNet) and trained using 1000 unannotated US images (500 benign and 500 malignant masses). Two sets of 200 images (100 benign and 100 malignant masses) were used for internal and external validation sets. For comparison with fully-supervised algorithms, ROI annotation was performed manually and automatically. Diagnostic performances were calculated as the area under the receiver operating characteristic curve (AUC). Using the class activation map, we determined how accurately the weakly-supervised DL algorithms localized the breast masses. For internal validation sets, the weakly-supervised DL algorithms achieved excellent diagnostic performances, with AUC values of 0.92–0.96, which were not statistically different (all Ps > 0.05) from those of fully-supervised DL algorithms with either manual or automated ROI annotation (AUC, 0.92–0.96). For external validation sets, the weakly-supervised DL algorithms achieved AUC values of 0.86–0.90, which were not statistically different (Ps > 0.05) or higher (P = 0.04, VGG16 with automated ROI annotation) from those of fully-supervised DL algorithms (AUC, 0.84–0.92). In internal and external validation sets, weakly-supervised algorithms could localize 100% of malignant masses, except for ResNet34 (98%). The weakly-supervised DL algorithms developed in the present study were feasible for US diagnosis of breast cancer with well-performing localization and differential diagnosis.

3D U-Net Improves Automatic Brain Extraction for Isotropic Rat Brain Magnetic Resonance Imaging Data

Brain extraction is a critical pre-processing step in brain magnetic resonance imaging (MRI) analytical pipelines. In rodents, this is often achieved by manually editing brain masks slice-by-slice, a time-consuming task where workloads increase with higher spatial resolution datasets. We recently demonstrated successful automatic brain extraction via a deep-learning-based framework, U-Net, using 2D convolutions. However, such an approach cannot make use of the rich 3D spatial-context information from volumetric MRI data. In this study, we advanced our previously proposed U-Net architecture by replacing all 2D operations with their 3D counterparts and created a 3D U-Net framework. We trained and validated our model using a recently released CAMRI rat brain database acquired at isotropic spatial resolution, including T2-weighted turbo-spin-echo structural MRI and T2*-weighted echo-planar-imaging functional MRI. The performance of our 3D U-Net model was compared with existing rodent brain extraction tools, including Rapid Automatic Tissue Segmentation, Pulse-Coupled Neural Network, SHape descriptor selected External Regions after Morphologically filtering, and our previously proposed 2D U-Net model. 3D U-Net demonstrated superior performance in Dice, Jaccard, center-of-mass distance, Hausdorff distance, and sensitivity. Additionally, we demonstrated the reliability of 3D U-Net under various noise levels, evaluated the optimal training sample sizes, and disseminated all source codes publicly, with a hope that this approach will benefit rodent MRI research community. Significant Methodological Contribution: We proposed a deep-learning-based framework to automatically identify the rodent brain boundaries in MRI. With a fully 3D convolutional network model, 3D U-Net, our proposed method demonstrated improved performance compared to current automatic brain extraction methods, as shown in several qualitative metrics (Dice, Jaccard, PPV, SEN, and Hausdorff). We trust that this tool will avoid human bias and streamline pre-processing steps during 3D high resolution rodent brain MRI data analysis. The software developed herein has been disseminated freely to the community.

Deep learning-based motion tracking using ultrasound images

PURPOSE

Ultrasound (US) imaging is an established imaging modality capable of offering video-rate volumetric images without ionizing radiation. It has the potential for intra-fraction motion tracking in radiation therapy. In this study, a deep learning-based method has been developed to tackle the challenges in motion tracking using US imaging.

METHODS

We present a Markov-like network, which is implemented via generative adversarial networks, to extract features from sequential US frames (one tracked frame followed by untracked frames) and thereby estimate a set of deformation vector fields (DVFs) through the registration of the tracked frame and the untracked frames. The positions of the landmarks in the untracked frames are finally determined by shifting landmarks in the tracked frame according to the estimated DVFs. The performance of the proposed method was evaluated on the testing dataset by calculating the tracking error (TE) between the predicted and ground truth landmarks on each frame.

RESULTS

The proposed method was evaluated using the MICCAI CLUST 2015 dataset which was collected using seven US scanners with eight types of transducers and the Cardiac Acquisitions for Multi-structure Ultrasound Segmentation (CAMUS) dataset which was acquired using GE Vivid E95 ultrasound scanners. The CLUST dataset contains 63 2D and 22 3D US image sequences respectively from 42 and 18 subjects, and the CAMUS dataset includes 2D US images from 450 patients. On CLUST dataset, our proposed method achieved a mean tracking error of 0.70 ± 0.38 mm for the 2D sequences and 1.71 ± 0.84 mm for the 3D sequences for those public available annotations. And on CAMUS dataset, a mean tracking error of 0.54 ± 1.24 mm for the landmarks in the left atrium was achieved.

CONCLUSIONS

A novel motion tracking algorithm using US images based on modern deep learning techniques has been demonstrated in this study. The proposed method can offer millimeter-level tumor motion prediction in real time, which has the potential to be adopted into routine tumor motion management in radiation therapy.

ApodNet: Learning for High Frame Rate Synthetic Transmit Aperture Ultrasound Imaging

Two-way dynamic focusing in synthetic transmit aperture (STA) beamforming can benefit high-quality ultrasound imaging with higher lateral spatial resolution and contrast resolution. However, STA requires the complete dataset for beamforming in a relatively low frame rate and transmit power. This paper proposes a deep-learning architecture to achieve high frame rate STA imaging with two-way dynamic focusing. The network consists of an encoder and a joint decoder. The encoder trains a set of binary weights as the apodizations of the high-frame-rate plane wave transmissions. In this respect, we term our network ApodNet. The decoder can recover the complete dataset from the acquired channel data to achieve dynamic transmit focusing. We evaluate the proposed method by simulations at different levels of noise and in-vivo experiments on the human biceps brachii and common carotid artery. The experimental results demonstrate that ApodNet provides a promising strategy for high frame rate STA imaging, obtaining comparable lateral resolution and contrast resolution with four-times higher frame rate than conventional STA imaging in the in-vivo experiments. Particularly, ApodNet improves contrast resolution of the hypoechoic targets with much shorter computational time when compared with other high-frame-rate methods in both simulations and in-vivo experiments.

A Generalized Approach for Automatic 3-D Geometry Assessment of Blood Vessels in Transverse Ultrasound Images Using Convolutional Neural Networks

Accurate 3-D geometries of arteries and veins are important clinical data for diagnosis of arterial disease and intervention planning. Automatic segmentation of vessels in the transverse view suffers from the low lateral resolution and contrast. Convolutional neural networks are a promising tool for automatic segmentation of medical images, outperforming the traditional segmentation methods with high robustness. In this study, we aim to create a general, robust, and accurate method to segment the lumen-wall boundary of healthy central and peripheral vessels in large field-of-view freehand ultrasound (US) datasets. Data were acquired using the freehand US, in combination with a probe tracker. A total of ±36 000 cross-sectional images, acquired in the common, internal, and external carotid artery ( N = 37 ), in the radial, ulnar artery, and cephalic vein ( N = 12 ), and in the femoral artery ( N = 5 ) were included. To create masks (of the lumen) for training data, a conventional automatic segmentation method was used. The neural networks were trained on: 1) data of all vessels and 2) the carotid artery only. The performance was compared and tested using an open-access dataset. The recall, precision, DICE, and intersection over union (IoU) were calculated. Overall, segmentation was successful in the carotid and peripheral arteries. The Multires U-net architecture performs best overall with DICE = 0.93 when trained on the total dataset. Future studies will focus on the inclusion of vascular pathologies.

Automated Segmentation of Median Nerve in Dynamic Sonography Using Deep Learning: Evaluation of Model Performance

There is an emerging trend to employ dynamic sonography in the diagnosis of entrapment neuropathy, which exhibits aberrant spatiotemporal characteristics of the entrapped nerve when adjacent tissues move. However, the manual tracking of the entrapped nerve in consecutive images demands tons of human labors and impedes its popularity clinically. Here we evaluated the performance of automated median nerve segmentation in dynamic sonography using a variety of deep learning models pretrained with ImageNet, including DeepLabV3+, U-Net, FPN, and Mask-R-CNN. Dynamic ultrasound images of the median nerve at across wrist level were acquired from 52 subjects diagnosed as carpal tunnel syndrome when they moved their fingers. The videos of 16 subjects exhibiting diverse appearance and that of the remaining 36 subjects were used for model test and training, respectively. The centroid, circularity, perimeter, and cross section area of the median nerve in individual frame were automatically determined from the inferred nerve. The model performance was evaluated by the score of intersection over union (IoU) between the annotated and model-predicted data. We found that both DeepLabV3+ and Mask R-CNN predicted median nerve the best with averaged IOU scores close to 0.83, which indicates the feasibility of automated median nerve segmentation in dynamic sonography using deep learning.

Catheter position prediction using deep-learning-based multi-atlas registration for high-dose rate prostate brachytherapy

PURPOSE

High-dose-rate (HDR) prostate brachytherapy involves treatment catheter placement, which is currently empirical and physician dependent. The lack of proper catheter placement guidance during the procedure has left the physicians to rely on a heuristic thinking-while-doing technique, which may cause large catheter placement variation and increased plan quality uncertainty. Therefore, the achievable dose distribution could not be quantified prior to the catheter placement. To overcome this challenge, we proposed a learning-based method to provide HDR catheter placement guidance for prostate cancer patients undergoing HDR brachytherapy.

METHODS

The proposed framework consists of deformable registration via registration network (Reg-Net), multi-atlas ranking, and catheter regression. To model the global spatial relationship among multiple organs, binary masks of the prostate and organs-at-risk are transformed into distance maps, which describe the distance of each local voxel to the organ surfaces. For a new patient, the generated distance map is used as fixed image. Reg-Net is utilized to deformably register the distance maps from multi-atlas set to match this patient's distance map and then bring catheter maps from multi-atlas to this patient via spatial transformation. Several criteria, namely prostate volume similarity, multi-organ semantic image similarity, and catheter position criteria (far from the urethra and within the partial prostate), are used for multi-atlas ranking. The top-ranked atlas' deformed catheter positions are selected as the predicted catheter positions for this patient. Finally, catheter regression is used to refine the final catheter positions. A retrospective study on 90 patients with a fivefold cross-validation scheme was used to evaluate the proposed method's feasibility. In order to investigate the impact of plan quality from the predicted catheter pattern, we optimized the source dwell position and time for both the clinical catheter pattern and predicted catheter pattern with the same optimization settings. Comparisons of clinically relevant dose volume histogram (DVH) metrics were completed.

RESULTS

For all patients, on average, both the clinical plan dose and predicted plan dose meet the common dose constraints when prostate dose coverage is kept at V100 = 95%. The plans from the predicted catheter pattern have slightly higher hotspot in terms of V150 by 5.0% and V200 by 2.9% on average. For bladder V75, rectum V75, and urethra V125, the average difference is close to zero, and the range of most patients is within ±1 cc.

CONCLUSION

We developed a new catheter placement prediction method for HDR prostate brachytherapy based on a deep-learning-based multi-atlas registration algorithm. It has great clinical potential since it can provide catheter location estimation prior to catheter placement, which could reduce the dependence on physicians' experience in catheter implantation and improve the quality of prostate HDR treatment plans. This approach merits further clinical evaluation and validation as a method of quality control for HDR prostate brachytherapy.

Assessment of hip displacement in children with cerebral palsy using machine learning approach

Manual measurements of migration percentage (MP) on pelvis radiographs for assessing hip displacement are subjective and time consuming. A deep learning approach using convolution neural networks (CNNs) to automatically measure the MP was proposed. The pre-trained Inception ResNet v2 was fine tuned to detect locations of the eight reference landmarks used for MP measurements. A second network, fine-tuned MobileNetV2, was trained on the regions of interest to obtain more precise landmarks' coordinates. The MP was calculated from the final estimated landmarks' locations. A total of 122 radiographs were divided into 57 for training, 10 for validation, and 55 for testing. The mean absolute difference (MAD) and intra-class correlation coefficient (ICC [2,1]) of the comparison for the MP on 110 measurements (left and right hips) were 4.5 [Formula: see text] 4.3% (95% CI, 3.7-5.3%) and 0.91, respectively. Sensitivity and specificity were 87.8% and 93.4% for the classification of hip displacement (MP-threshold of 30%), and 63.2% and 94.5% for the classification of surgery-needed hips (MP-threshold of 40%). The prediction results were returned within 5 s. The developed fine-tuned CNNs detected the landmarks and provided automatic MP measurements with high accuracy and excellent reliability, which can assist clinicians to diagnose hip displacement in children with CP.

Improved real-time segmentation of Intravascular Ultrasound images using coordinate-aware fully convolutional networks

Segmentation of Intravascular Ultrasound (IVUS) images into Lumen and Media (interior and exterior) artery vessel walls is highly clinically relevant in the diagnosis and treatment of cardiovascular diseases such as atherosclerosis. When fused with position data, such segmentations also play a key role in reconstructing 3D representations of arteries. Automated segmentation in real-time is known to be a difficult image analysis problem, primarily due to artefacts commonly present in IVUS ultrasound images such as shadows, guide-wire effects, and side-branches. An additional challenge is the limited amount of expert labelled IVUS data, which limits the application of many well-performing deep learning models from other domains. To exploit the circular layered structure of the artery in B-Mode images, we propose a multi-class fully convolutional semantic segmentation network based on a minimal U-Net architecture augmented with learned translation dependence in the polar domain. The coordinate awareness in the multi-class segmentation allows the model to exploit relative spatial context about the interior and exterior vessel walls which are simply separable in polar coordinates. After training on 109 expert-labelled examples, our model significantly outperforms the state-of-the art in terms of mean Jaccard Measure (0.91 vs. 0.89) and Hausdorff distance (0.32 mm vs. 0.48 mm) on Media segmentation, and reaches equivalent performance on Lumen segmentation when evaluated on a standard publicly available dataset of 326 IVUS B-Mode images captured by 20 Mhz ultrasound probes. Using an order of magnitude fewer trainable parameters than the previous state-of-the-art, our model runs over 50 times faster and is able to execute in only 3 ms on a common GPU, achieving both leading accuracy and practical real-time performance.

Advanced deep learning methodology for accurate, real-time segmentation of high-resolution intravascular ultrasound images

AIMS

The aim of this study is to develop and validate a deep learning (DL) methodology capable of automated and accurate segmentation of intravascular ultrasound (IVUS) image sequences in real-time.

METHODS AND RESULTS

IVUS segmentation was performed by two experts who manually annotated the external elastic membrane (EEM) and lumen borders in the end-diastolic frames of 197 IVUS sequences portraying the native coronary arteries of 65 patients. The IVUS sequences of 177 randomly-selected vessels were used to train and optimise a novel DL model for the segmentation of IVUS images. Validation of the developed methodology was performed in 20 vessels using the estimations of two expert analysts as the reference standard. The mean difference for the EEM, lumen and plaque area between the DL-methodology and the analysts was ≤0.23mm² (standard deviation ≤0.85mm²), while the Hausdorff and mean distance differences for the EEM and lumen borders was ≤0.19 mm (standard deviation≤0.17 mm). The agreement between DL and experts was similar to experts' agreement (Williams Index ranges: 0.754-1.061) with similar results in frames portraying calcific plaques or side branches.

CONCLUSIONS

The developed DL-methodology appears accurate and capable of segmenting high-resolution real-world IVUS datasets. These features are expected to facilitate its broad adoption and enhance the applications of IVUS in clinical practice and research.

Cross-Model Attention-Guided Tumor Segmentation for 3D Automated Breast Ultrasound (ABUS) Images

Tumor segmentation in 3D automated breast ultrasound (ABUS) plays an important role in breast disease diagnosis and surgical planning. However, automatic segmentation of tumors in 3D ABUS images is still challenging, due to the large tumor shape and size variations, and uncertain tumor locations among patients. In this paper, we develop a novel cross-model attention-guided tumor segmentation network with a hybrid loss for 3D ABUS images. Specifically, we incorporate the tumor location into a segmentation network by combining an improved 3D Mask R-CNN head into V-Net as an end-to-end architecture. Furthermore, we introduce a cross-model attention mechanism that is able to aggregate the segmentation probability map from the improved 3D Mask R-CNN to each feature extraction level in the V-Net. Then, we design a hybrid loss to balance the contribution of each part in the proposed cross-model segmentation network. We conduct extensive experiments on 170 3D ABUS from 107 patients. Experimental results show that our method outperforms other state-of-the-art methods, by achieving the Dice similarity coefficient (DSC) of 64.57%, Jaccard coefficient (JC) of 53.39%, recall (REC) of 64.43%, precision (PRE) of 74.51%, 95th Hausdorff distance (95HD) of 11.91mm, and average surface distance (ASD) of 4.63mm. Our code will be available online (https://github.com/zhouyuegithub/CMVNet).

Male pelvic CT multi-organ segmentation using synthetic MRI-aided dual pyramid networks

The delineation of the prostate and organs-at-risk (OARs) is fundamental to prostate radiation treatment planning, but is currently labor-intensive and observer-dependent. We aimed to develop an automated computed tomography (CT)-based multi-organ (bladder, prostate, rectum, left and right femoral heads (RFHs)) segmentation method for prostate radiation therapy treatment planning. The proposed method uses synthetic MRIs (sMRIs) to offer superior soft-tissue information for male pelvic CT images. Cycle-consistent adversarial networks (CycleGAN) were used to generate CT-based sMRIs. Dual pyramid networks (DPNs) extracted features from both CTs and sMRIs. A deep attention strategy was integrated into the DPNs to select the most relevant features from both CTs and sMRIs to identify organ boundaries. The CT-based sMRI generated from our previously trained CycleGAN and its corresponding CT images were inputted to the proposed DPNs to provide complementary information for pelvic multi-organ segmentation. The proposed method was trained and evaluated using datasets from 140 patients with prostate cancer, and were then compared against state-of-art methods. The Dice similarity coefficients and mean surface distances between our results and ground truth were 0.95 ± 0.05, 1.16 ± 0.70 mm; 0.88 ± 0.08, 1.64 ± 1.26 mm; 0.90 ± 0.04, 1.27 ± 0.48 mm; 0.95 ± 0.04, 1.08 ± 1.29 mm; and 0.95 ± 0.04, 1.11 ± 1.49 mm for bladder, prostate, rectum, left and RFHs, respectively. Mean center of mass distances was within 3 mm for all organs. Our results performed significantly better than those of competing methods in most evaluation metrics. We demonstrated the feasibility of sMRI-aided DPNs for multi-organ segmentation on pelvic CT images, and its superiority over other networks. The proposed method could be used in routine prostate cancer radiotherapy treatment planning to rapidly segment the prostate and standard OARs.

Dual attention enhancement feature fusion network for segmentation and quantitative analysis of paediatric echocardiography

Paediatric echocardiography is a standard method for screening congenital heart disease (CHD). The segmentation of paediatric echocardiography is essential for subsequent extraction of clinical parameters and interventional planning. However, it remains a challenging task due to (1) the considerable variation of key anatomic structures, (2) the poor lateral resolution affecting accurate boundary definition, (3) the existence of speckle noise and artefacts in echocardiographic images. In this paper, we propose a novel deep network to address these challenges comprehensively. We first present a dual-path feature extraction module (DP-FEM) to extract rich features via a channel attention mechanism. A high- and low-level feature fusion module (HL-FFM) is devised based on spatial attention, which selectively fuses rich semantic information from high-level features with spatial cues from low-level features. In addition, a hybrid loss is designed to deal with pixel-level misalignment and boundary ambiguities. Based on the segmentation results, we derive key clinical parameters for diagnosis and treatment planning. We extensively evaluate the proposed method on 4,485 two-dimensional (2D) paediatric echocardiograms from 127 echocardiographic videos. The proposed method consistently achieves better segmentation performance than other state-of-the-art methods, whichdemonstratesfeasibility for automatic segmentation and quantitative analysis of paediatric echocardiography. Our code is publicly available at https://github.com/end-of-the-century/Cardiac.

Thyroid gland delineation in noncontrast-enhanced CTs using deep convolutional neural networks

The purpose of this study is to develop a deep learning method for thyroid delineation with high accuracy, efficiency, and robustness in noncontrast-enhanced head and neck CTs. The cross-sectional analysis consisted of six tests, including randomized cross-validation and hold-out experiments, tests of prediction accuracy between cancer and benign and cross-gender analysis were performed to evaluate the proposed deep-learning-based performance method. CT images of 1977 patients with suspected thyroid carcinoma were retrospectively investigated. The automatically segmented thyroid gland volume was compared against physician-approved clinical contours using metrics, the Pearson correlation and Bland-Altman analysis. Quantitative metrics included: the Dice similarity coefficient (DSC), sensitivity, specificity, Jaccard index (JAC), Hausdorff distance (HD), mean surface distance (MSD), residual mean square distance (RMSD) and the center of mass distance (CMD). The robustness of the proposed method was further tested using the nonparametric Kruskal-Wallis test to assess the equality of distribution of DSC values. The proposed method's accuracy remained high through all the tests, with the median DSC, JAC, sensitivity and specificity higher than 0.913, 0.839, 0.856 and 0.979, respectively. The proposed method also resulted in median MSD, RMSD, HD and CMD, of less than 0.31 mm, 0.48 mm, 2.06 mm and 0.50 mm, respectively. The MSD and RMSD were 0.40 ± 0.29 mm and 0.70 ± 0.46 mm, respectively. Concurrent testing of the proposed method with 3D U-Net and V-Net showed that the proposed method had significantly improved performance. The proposed deep-learning method achieved accurate and robust performance through six cross-sectional analysis tests.

A deep learning methodology for the automated detection of end-diastolic frames in intravascular ultrasound images

Coronary luminal dimensions change during the cardiac cycle. However, contemporary volumetric intravascular ultrasound (IVUS) analysis is performed in non-gated images as existing methods to acquire gated or to retrospectively gate IVUS images have failed to dominate in research. We developed a novel deep learning (DL)-methodology for end-diastolic frame detection in IVUS and compared its efficacy against expert analysts and a previously established methodology using electrocardiographic (ECG)-estimations as reference standard. Near-infrared spectroscopy-IVUS (NIRS-IVUS) data were prospectively acquired from 20 coronary arteries and co-registered with the concurrent ECG-signal to identify end-diastolic frames. A DL-methodology which takes advantage of changes in intensity of corresponding pixels in consecutive NIRS-IVUS frames and consists of a network model designed in a bidirectional gated-recurrent-unit (Bi-GRU) structure was trained to detect end-diastolic frames. The efficacy of the DL-methodology in identifying end-diastolic frames was compared with two expert analysts and a conventional image-based (CIB)-methodology that relies on detecting vessel movement to estimate phases of the cardiac cycle. A window of ± 100 ms from the ECG estimations was used to define accurate end-diastolic frames detection. The ECG-signal identified 3,167 end-diastolic frames. The mean difference between DL and ECG estimations was 3 ± 112 ms while the mean differences between the 1st-analyst and ECG, 2nd-analyst and ECG and CIB-methodology and ECG were 86 ± 192 ms, 78 ± 183 ms and 59 ± 207 ms, respectively. The DL-methodology was able to accurately detect 80.4%, while the two analysts and the CIB-methodology detected 39.0%, 43.4% and 42.8% of end-diastolic frames, respectively (P < 0.05). The DL-methodology can identify NIRS-IVUS end-diastolic frames accurately and should be preferred over expert analysts and CIB-methodologies, which have limited efficacy.

Global guidance network for breast lesion segmentation in ultrasound images

Automatic breast lesion segmentation in ultrasound helps to diagnose breast cancer, which is one of the dreadful diseases that affect women globally. Segmenting breast regions accurately from ultrasound image is a challenging task due to the inherent speckle artifacts, blurry breast lesion boundaries, and inhomogeneous intensity distributions inside the breast lesion regions. Recently, convolutional neural networks (CNNs) have demonstrated remarkable results in medical image segmentation tasks. However, the convolutional operations in a CNN often focus on local regions, which suffer from limited capabilities in capturing long-range dependencies of the input ultrasound image, resulting in degraded breast lesion segmentation accuracy. In this paper, we develop a deep convolutional neural network equipped with a global guidance block (GGB) and breast lesion boundary detection (BD) modules for boosting the breast ultrasound lesion segmentation. The GGB utilizes the multi-layer integrated feature map as a guidance information to learn the long-range non-local dependencies from both spatial and channel domains. The BD modules learn additional breast lesion boundary map to enhance the boundary quality of a segmentation result refinement. Experimental results on a public dataset and a collected dataset show that our network outperforms other medical image segmentation methods and the recent semantic segmentation methods on breast ultrasound lesion segmentation. Moreover, we also show the application of our network on the ultrasound prostate segmentation, in which our method better identifies prostate regions than state-of-the-art networks.

Artificial intelligence in musculoskeletal ultrasound imaging

Ultrasonography (US) is noninvasive and offers real-time, low-cost, and portable imaging that facilitates the rapid and dynamic assessment of musculoskeletal components. Significant technological improvements have contributed to the increasing adoption of US for musculoskeletal assessments, as artificial intelligence (AI)-based computer-aided detection and computer-aided diagnosis are being utilized to improve the quality, efficiency, and cost of US imaging. This review provides an overview of classical machine learning techniques and modern deep learning approaches for musculoskeletal US, with a focus on the key categories of detection and diagnosis of musculoskeletal disorders, predictive analysis with classification and regression, and automated image segmentation. Moreover, we outline challenges and a range of opportunities for AI in musculoskeletal US practice.

Automatic quantification of myocardium and pericardial fat from coronary computed tomography angiography: a multicenter study

OBJECTIVES

To develop a deep learning-based method for simultaneous myocardium and pericardial fat quantification from coronary computed tomography angiography (CCTA) for the diagnosis and treatment of cardiovascular disease (CVD).

METHODS

We retrospectively identified CCTA data obtained between May 2008 and July 2018 in a multicenter (six centers) CVD study. The proposed method was evaluated on 422 patients' data by two studies. The first overall study involves training model on CVD patients and testing on non-CVD patients, as well as training on non-CVD patients and testing on CVD patients. The second study was performed using the leave-center-out approach. The method performance was evaluated using Dice similarity coefficient (DSC), Jaccard index (JAC), 95% Hausdorff distance (HD95), mean surface distance (MSD), residual mean square distance (RMSD), and the center of mass distance (CMD). The robustness of the proposed method was tested using the nonparametric Kruskal-Wallis test and post hoc test to assess the equality of distribution of DSC values among different tests.

RESULTS

The automatic segmentation achieved a strong correlation with contour (ICC and R > 0.97, p value < 0.001 throughout all tests). The accuracy of the proposed method remained high through all the tests, with the median DSC higher than 0.88 for pericardial fat and 0.96 for myocardium. The proposed method also resulted in mean MSD, RMSD, HD95, and CMD of less than 1.36 mm for pericardial fat and 1.00 mm for myocardium.

CONCLUSIONS

The proposed deep learning-based segmentation method enables accurate simultaneous quantification of myocardium and pericardial fat in a multicenter study.

KEY POINTS

• Deep learning-based myocardium and pericardial fat segmentation method tested on 422 patients' coronary computed tomography angiography in a multicenter study. • The proposed method provides segmentations with high volumetric accuracy (ICC and R > 0.97, p value < 0.001) and similar shape as manual annotation by experienced radiologists (median Dice similarity coefficient ≥ 0.88 for pericardial fat and 0.96 for myocardium).

Extracting Membrane Borders in IVUS Images Using a Multi-Scale Feature Aggregated U-Net

Automatic extraction of the lumen-intima border (LIB) and the media-adventitia border (MAB) in intravascular ultrasound (IVUS) images is of high clinical interest. Despite the superior performance achieved by deep neural networks (DNNs) on various medical image segmentation tasks, there are few applications to IVUS images. The complicated pathological presentation and the lack of enough annotation in IVUS datasets make the learning process challenging. Several existing networks designed for IVUS segmentation train two groups of weights to detect the MAB and LIB separately. In this paper, we propose a multi-scale feature aggregated U-Net (MFAU-Net) to extract two membrane borders simultaneously. The MFAU-Net integrates multi-scale inputs, the deep supervision, and a bi-directional convolutional long short-term memory (BConvLSTM) unit. It is designed to sufficiently learn features from complicated IVUS images through a small number of training samples. Trained and tested on the publicly available IVUS datasets, the MFAU-Net achieves both 0.90 Jaccard measure (JM) for the MAB and LIB detection on 20 MHz dataset. The corresponding metrics on 40 MHz dataset are 0.85 and 0.84 JM respectively. Comparative evaluations with state-of-the-art published results demonstrate the competitiveness of the proposed MFAU-Net.

Head and neck multi-organ auto-segmentation on CT images aided by synthetic MRI

PURPOSE

Because the manual contouring process is labor-intensive and time-consuming, segmentation of organs-at-risk (OARs) is a weak link in radiotherapy treatment planning process. Our goal was to develop a synthetic MR (sMR)-aided dual pyramid network (DPN) for rapid and accurate head and neck multi-organ segmentation in order to expedite the treatment planning process.

METHODS

Forty-five patients' CT, MR, and manual contours pairs were included as our training dataset. Nineteen OARs were target organs to be segmented. The proposed sMR-aided DPN method featured a deep attention strategy to effectively segment multiple organs. The performance of sMR-aided DPN method was evaluated using five metrics, including Dice similarity coefficient (DSC), Hausdorff distance 95% (HD95), mean surface distance (MSD), residual mean square distance (RMSD), and volume difference. Our method was further validated using the 2015 head and neck challenge data.

RESULTS

The contours generated by the proposed method closely resemble the ground truth manual contours, as evidenced by encouraging quantitative results in terms of DSC using the 2015 head and neck challenge data. Mean DSC values of 0.91 ± 0.02, 0.73 ± 0.11, 0.96 ± 0.01, 0.78 ± 0.09/0.78 ± 0.11, 0.88 ± 0.04/0.88 ± 0.06 and 0.86 ± 0.08/0.85 ± 0.1 were achieved for brain stem, chiasm, mandible, left/right optic nerve, left/right parotid, and left/right submandibular, respectively.

CONCLUSIONS

We demonstrated the feasibility of sMR-aided DPN for head and neck multi-organ delineation on CT images. Our method has shown superiority over the other methods on the 2015 head and neck challenge data results. The proposed method could significantly expedite the treatment planning process by rapidly segmenting multiple OARs.

Estimation of Ultrasound Echogenicity Map from B-Mode Images Using Convolutional Neural Network

In ultrasound B-mode imaging, speckle noises decrease the accuracy of estimation of tissue echogenicity of imaged targets from the amplitude of the echo signals. In addition, since the granular size of the speckle pattern is affected by the point spread function (PSF) of the imaging system, the resolution of B-mode image remains limited, and the boundaries of tissue structures often become blurred. This study proposed a convolutional neural network (CNN) to remove speckle noises together with improvement of image spatial resolution to reconstruct ultrasound tissue echogenicity map. The CNN model is trained using in silico simulation dataset and tested with experimentally acquired images. Results indicate that the proposed CNN method can effectively eliminate the speckle noises in the background of the B-mode images while retaining the contours and edges of the tissue structures. The contrast and the contrast-to-noise ratio of the reconstructed echogenicity map increased from 0.22/2.72 to 0.33/44.14, and the lateral and axial resolutions also improved from 5.9/2.4 to 2.9/2.0, respectively. Compared with other post-processing filtering methods, the proposed CNN method provides better approximation to the original tissue echogenicity by completely removing speckle noises and improving the image resolution together with the capability for real-time implementation.