Comparing deep learning-based automatic segmentation of breast masses to expert interobserver variability in ultrasound imaging

Personalized Deep Learning Model for Clinical Target Volume on Daily Cone Beam Computed Tomography in Breast Cancer Patients

Purpose

Herein, we developed a deep learning algorithm to improve the segmentation of the clinical target volume (CTV) on daily cone beam computed tomography (CBCT) scans in breast cancer radiation therapy. By leveraging the Intentional Deep Overfit Learning (IDOL) framework, we aimed to enhance personalized image-guided radiation therapy based on patient-specific learning.

Methods and Materials

We used 240 CBCT scans from 100 breast cancer patients and employed a 2-stage training approach. The first stage involved training a novel general deep learning model (Swin UNETR, UNET, and SegResNET) on 90 patients. The second stage used intentional overfitting on the remaining 10 patients for patient-specific CBCT outputs. Quantitative evaluation was conducted using the Dice Similarity Coefficient (DSC), Hausdorff Distance (HD), mean surface distance (MSD), and independent samples t test with expert contours on CBCT scans from the first to 15th fractions.

Results

IDOL integration significantly improved CTV segmentation, particularly with the Swin UNETR model (P values < .05). Using patient-specific data, IDOL enhanced the DSC, HD, and MSD metrics. The average DSC for the 15th fraction improved from 0.9611 to 0.9819, the average HD decreased from 4.0118 mm to 1.3935 mm, and the average MSD decreased from 0.8723 to 0.4603. Incorporating CBCT scans from the initial treatments and first to third fractions further improved results, with an average DSC of 0.9850, an average HD of 1.2707 mm, and an average MSD of 0.4076 for the 15th fraction, closely aligning with physician-drawn contours.

Conclusion

Compared with a general model, our patient-specific deep learning-based training algorithm significantly improved CTV segmentation accuracy of CBCT scans in patients with breast cancer. This approach, coupled with continuous deep learning training using daily CBCT scans, demonstrated enhanced CTV delineation accuracy and efficiency. Future studies should explore the adaptability of the IDOL framework to diverse deep learning models, data sets, and cancer sites.

Enhancing Breast Cancer Diagnosis: A Nomogram Model Integrating AI Ultrasound and Clinical Factors

PURPOSE

A novel nomogram incorporating artificial intelligence (AI) and clinical features for enhanced ultrasound prediction of benign and malignant breast masses.

MATERIALS AND METHODS

This study analyzed 340 breast masses identified through ultrasound in 308 patients. The masses were divided into training (n = 260) and validation (n = 80) groups. The AI-based analysis employed the Samsung Ultrasound AI system (S-detect). Univariate and multivariate analyses were conducted to construct nomograms using logistic regression. The AI-Nomogram was based solely on AI results, while the ClinAI- Nomogram incorporated additional clinical factors. Both nomograms underwent internal validation with 1000 bootstrap resamples and external validation using the independent validation group. Performance was evaluated by analyzing the area under the receiver operating characteristic (ROC) curve (AUC) and calibration curves.

RESULTS

The ClinAI-Nomogram, which incorporates patient age, AI-based mass size, and AI-based diagnosis, outperformed an existing AI-Nomogram in differentiating benign from malignant breast masses. The ClinAI-Nomogram surpassed the AI-Nomogram in predicting malignancy with significantly higher AUC scores in both training (0.873, 95% CI: 0.830-0.917 vs. 0.792, 95% CI: 0.748-0.836; p = 0.016) and validation phases (0.847, 95% CI: 0.763-0.932 vs. 0.770, 95% CI: 0.709-0.833; p < 0.001). Calibration curves further revealed excellent agreement between the ClinAI-Nomogram's predicted probabilities and actual observed risks of malignancy.

CONCLUSION

The ClinAI- Nomogram, combining AI alongside clinical data, significantly enhanced the differentiation of benign and malignant breast masses in clinical AI-facilitated ultrasound examinations.

Assessing breast disease with deep learning model using bimodal bi-view ultrasound images and clinical information

Breast cancer is the second leading cause of carcinoma-linked death in women. We developed a multi-modal deep-learning model (BreNet) to differentiate breast cancer from benign lesions. BreNet was constructed and trained on 10,108 images from one center and tested on 3,762 images from two centers in three steps. The diagnostic ability of BreNet was first compared with that of six radiologists; a BreNet-aided scheme was constructed to improve the diagnostic ability of the radiologists; and the diagnosis of real-world radiologists' scheme was then compared with the BreNet-aided scheme. The diagnostic performance of BreNet was superior to that of the radiologists (area under the curve [AUC]: 0.996 vs. 0.841). BreNet-aided scheme increased the pooled AUC of the radiologists from 0.841 to 0.934 for reviewing images, and from 0.892 to 0.934 in the real-world test. The use of BreNet significantly enhances the diagnostic ability of radiologists in the detection of breast cancer.

Cellpose as a reliable method for single-cell segmentation of autofluorescence microscopy images

Autofluorescence microscopy uses intrinsic sources of molecular contrast to provide cellular-level information without extrinsic labels. However, traditional cell segmentation tools are often optimized for high signal-to-noise ratio (SNR) images, such as fluorescently labeled cells, and unsurprisingly perform poorly on low SNR autofluorescence images. Therefore, new cell segmentation tools are needed for autofluorescence microscopy. Cellpose is a deep learning network that is generalizable across diverse cell microscopy images and automatically segments single cells to improve throughput and reduce inter-human biases. This study aims to validate Cellpose for autofluorescence imaging, specifically from multiphoton intensity images of NAD(P)H. Manually segmented nuclear masks of NAD(P)H images were used to train new Cellpose models. These models were applied to PANC-1 cells treated with metabolic inhibitors and patient-derived cancer organoids (across 9 patients) treated with chemotherapies. These datasets include co-registered fluorescence lifetime imaging microscopy (FLIM) of NAD(P)H and FAD, so fluorescence decay parameters and the optical redox ratio (ORR) were compared between masks generated by the new Cellpose model and manual segmentation. The Dice score between repeated manually segmented masks was significantly lower than that of repeated Cellpose masks (p<0.0001) indicating greater reproducibility between Cellpose masks. There was also a high correlation (R2>0.9) between Cellpose and manually segmented masks for the ORR, mean NAD(P)H lifetime, and mean FAD lifetime across 2D and 3D cell culture treatment conditions. Masks generated from Cellpose and manual segmentation also maintain similar means, variances, and effect sizes between treatments for the ORR and FLIM parameters. Overall, Cellpose provides a fast, reliable, reproducible, and accurate method to segment single cells in autofluorescence microscopy images such that functional changes in cells are accurately captured in both 2D and 3D culture.

The effect of image resolution on convolutional neural networks in breast ultrasound

Purpose

The objective of this research was to investigate the efficacy of various parameter combinations of Convolutional Neural Networks (CNNs) models, namely MobileNet and DenseNet121, and different input image resolutions (REZs) ranging from 64×64 to 512×512 pixels, for diagnosing breast cancer.

Materials and methods

During the period of June 2015 to November 2020, two hospitals were involved in the collection of two-dimensional ultrasound breast images for this retrospective multicenter study. The diagnostic performance of the computer models MobileNet and DenseNet 121 was compared at different resolutions.

Results

The results showed that MobileNet had the best breast cancer diagnosis performance at 320×320pixel REZ and DenseNet121 had the best breast cancer diagnosis performance at 448×448pixel REZ.

Conclusion

Our study reveals a significant correlation between image resolution and breast cancer diagnosis accuracy. Through the comparison of MobileNet and DenseNet121, it is highlighted that lightweight neural networks (LW-CNNs) can achieve model performance similar to or even slightly better than large neural networks models (HW-CNNs) in ultrasound images, and LW-CNNs' prediction time per image is lower.

An efficient image segmentation method based on expectation maximization and Salp swarm algorithm

Multilevel image thresholding using Expectation Maximization (EM) is an efficient method for image segmentation. However, it has two weaknesses: 1) EM is a greedy algorithm and cannot jump out of local optima. 2) it cannot guarantee the number of required classes while estimating the histogram by Gaussian Mixture Models (GMM). in this paper, to overcome these shortages, a novel thresholding approach by combining EM and Salp Swarm Algorithm (SSA) is developed. SSA suggests potential points to the EM algorithm to fly to a better position. Moreover, a new mechanism is considered to maintain the number of desired clusters. Twenty-four medical test images are selected and examined by standard metrics such as PSNR and FSIM. The proposed method is compared with the traditional EM algorithm, and an average improvement of 5.27% in PSNR values and 2.01% in FSIM values were recorded. Also, the proposed approach is compared with four existing segmentation techniques by using CT scan images that Qatar University has collected. Experimental results depict that the proposed method obtains the first rank in terms of PSNR and the second rank in terms of FSIM. It has been observed that the proposed technique performs better performance in the segmentation result compared to other considered state-of-the-art methods.

Breast tumor localization and segmentation using machine learning techniques: Overview of datasets, findings, and methods

The Global Cancer Statistics 2020 reported breast cancer (BC) as the most common diagnosis of cancer type. Therefore, early detection of such type of cancer would reduce the risk of death from it. Breast imaging techniques are one of the most frequently used techniques to detect the position of cancerous cells or suspicious lesions. Computer-aided diagnosis (CAD) is a particular generation of computer systems that assist experts in detecting medical image abnormalities. In the last decades, CAD has applied deep learning (DL) and machine learning approaches to perform complex medical tasks in the computer vision area and improve the ability to make decisions for doctors and radiologists. The most popular and widely used technique of image processing in CAD systems is segmentation which consists of extracting the region of interest (ROI) through various techniques. This research provides a detailed description of the main categories of segmentation procedures which are classified into three classes: supervised, unsupervised, and DL. The main aim of this work is to provide an overview of each of these techniques and discuss their pros and cons. This will help researchers better understand these techniques and assist them in choosing the appropriate method for a given use case.

Fully automatic tumor segmentation of breast ultrasound images with deep learning

BACKGROUND

Breast ultrasound (BUS) imaging is one of the most prevalent approaches for the detection of breast cancers. Tumor segmentation of BUS images can facilitate doctors in localizing tumors and is a necessary step for computer-aided diagnosis systems. While the majority of clinical BUS scans are normal ones without tumors, segmentation approaches such as U-Net often predict mass regions for these images. Such false-positive problem becomes serious if a fully automatic artificial intelligence system is used for routine screening.

METHODS

In this study, we proposed a novel model which is more suitable for routine BUS screening. The model contains a classification branch that determines whether the image is normal or with tumors, and a segmentation branch that outlines tumors. Two branches share the same encoder network. We also built a new dataset that contains 1600 BUS images from 625 patients for training and a testing dataset with 130 images from 120 patients for testing. The dataset is the largest one with pixel-wise masks manually segmented by experienced radiologists. Our code is available at https://github.com/szhangNJU/BUS_segmentation.

RESULTS

The area under the receiver operating characteristic curve (AUC) for classifying images into normal/abnormal categories was 0.991. The dice similarity coefficient (DSC) for segmentation of mass regions was 0.898, better than the state-of-the-art models. Testing on an external dataset gave a similar performance, demonstrating a good transferability of our model. Moreover, we simulated the use of the model in actual clinic practice by processing videos recorded during BUS scans; the model gave very low false-positive predictions on normal images without sacrificing sensitivities for images with tumors.

CONCLUSIONS

Our model achieved better segmentation performance than the state-of-the-art models and showed a good transferability on an external test set. The proposed deep learning architecture holds potential for use in fully automatic BUS health screening.

Efficient System for Delimitation of Benign and Malignant Breast Masses

In this study, a high-performing scheme is introduced to delimit benign and malignant masses in breast ultrasound images. The proposal is built upon by the Nonlocal Means filter for image quality improvement, an Intuitionistic Fuzzy C-Means local clustering algorithm for superpixel generation with high adherence to the edges, and the DBSCAN algorithm for the global clustering of those superpixels in order to delimit masses' regions. The empirical study was performed using two datasets, both with benign and malignant breast tumors. The quantitative results with respect to the BUSI dataset were JSC≥0.907, DM≥0.913, HD≥7.025, and MCR≤6.431 for benign masses and JSC≥0.897, DM≥0.900, HD≥8.666, and MCR≤8.016 for malignant ones, while the MID dataset resulted in JSC≥0.890, DM≥0.905, HD≥8.370, and MCR≤7.241 along with JSC≥0.881, DM≥0.898, HD≥8.865, and MCR≤7.808 for benign and malignant masses, respectively. These numerical results revealed that our proposal outperformed all the evaluated comparative state-of-the-art methods in mass delimitation. This is confirmed by the visual results since the segmented regions had a better edge delimitation.

Convolutional Neural Network for Predicting Thyroid Cancer Based on Ultrasound Elastography Image of Perinodular Region

We aimed to develop deep learning models based on perinodular regions' shear-wave elastography (SWE) images and ultrasound (US) images of thyroid nodules (TNs) and determine their performances in predicting thyroid cancer. A total of 1747 American College of Radiology Thyroid Imaging Reporting & Data System 4 (TR4) thyroid nodules (TNs) in 1582 patients were included in this retrospective study. US images, SWE images, and 2 quantitative SWE parameters (maximum elasticity of TNs; 5-point average maximum elasticity of TNs) were obtained. Based on US and SWE images of TNs and perinodular tissue, respectively, 7 single-image convolutional neural networks (CNN) models [US, internal SWE, 0.5 mm SWE, 1.0 mm SWE, 1.5 mm SWE, 2.0 mm SWE of perinodular tissue, and whole SWE region of interest (ROI) image] and another 6 fusional-image CNN models (US + internal SWE, US + 0.5 mm SWE, US + 1.0 mm SWE, US + 1.5 mm SWE, US + 2.0 mm SWE, US + ROI SWE) were established using RestNet18. All of the CNN models and quantitative SWE parameters were built on a training cohort (1247 TNs) and evaluated on a validation cohort (500 TNs). In predicting thyroid cancer, US + 2.0 mm SWE image CNN model obtained the highest area under the curve in 10 mm < TNs ≤ 20 mm (0.95 for training; 0.92 for validation) and TNs > 20 mm (0.95 for training; 0.92 for validation), while US + 1.0 mm SWE image CNN model obtained the highest area under the curve in TNs ≤ 10 mm (0.95 for training; 0.92 for validation). The CNN models based on the fusion of SWE segmentation images and US images improve the radiological diagnostic accuracy of thyroid cancer.

Accurate assessment of the lung sliding artefact on lung ultrasonography using a deep learning approach

Pneumothorax is a potentially life-threatening condition that can be rapidly and accurately assessed via the lung sliding artefact generated using lung ultrasound (LUS). Access to LUS is challenged by user dependence and shortage of training. Image classification using deep learning methods can automate interpretation in LUS and has not been thoroughly studied for lung sliding. Using a labelled LUS dataset from 2 academic hospitals, clinical B-mode (also known as brightness or two-dimensional mode) videos featuring both presence and absence of lung sliding were transformed into motion (M) mode images. These images were subsequently used to train a deep neural network binary classifier that was evaluated using a holdout set comprising 15% of the total data. Grad-CAM explanations were examined. Our binary classifier using the EfficientNetB0 architecture was trained using 2535 LUS clips from 614 patients. When evaluated on a test set of data uninvolved in training (540 clips from 124 patients), the model performed with a sensitivity of 93.5%, specificity of 87.3% and an area under the receiver operating characteristic curve (AUC) of 0.973. Grad-CAM explanations confirmed the model's focus on relevant regions on M-mode images. Our solution accurately distinguishes between the presence and absence of lung sliding artefacts on LUS.

Deep learning-based transcription factor activity for stratification of breast cancer patients

Transcription factors directly bind to DNA and regulate the expression of the gene, causing epigenetic modification of the DNA. They often mediate epigenetic parameters of transcriptional and posttranscriptional mechanisms, and their expression activities can be used to characterize genomic aberrations in cancer cell. In this study, the activity profile of transcription factors inferred by VIPER algorithm. The autoencoder model was applied for compressing the transcription factor activity profile for obtaining more useful transformed features for stratifying patients into two different breast cancer subtypes. The deep learning-based subtypes exhibited superior prognostic value and yielded better risk-stratification than the transcription factor activity-based method. Importantly, according to transformed features, a deep neural network was constructed to predict the subtypes, and achieved the accuracy of 94.98% and area under the ROC curve of 0.9663, respectively. The proposed subtypes were found to be significantly associated with immune infiltration, tumor immunogenicity and so on. Furthermore, the ceRNA network was constructed for the breast cancer subtypes. Besides, 11 master regulators were found to be associated with patients in cluster 1. Given the robustness performance of our deep learning model over multiple breast cancer cohorts, we expected this model may be useful in the area of prognosis prediction and lead some possibility for personalized medicine in breast cancer patients.

Recent Advances in Machine Learning Applied to Ultrasound Imaging

Machine learning (ML) methods are pervading an increasing number of fields of application because of their capacity to effectively solve a wide variety of challenging problems. The employment of ML techniques in ultrasound imaging applications started several years ago but the scientific interest in this issue has increased exponentially in the last few years. The present work reviews the most recent (2019 onwards) implementations of machine learning techniques for two of the most popular ultrasound imaging fields, medical diagnostics and non-destructive evaluation. The former, which covers the major part of the review, was analyzed by classifying studies according to the human organ investigated and the methodology (e.g., detection, segmentation, and/or classification) adopted, while for the latter, some solutions to the detection/classification of material defects or particular patterns are reported. Finally, the main merits of machine learning that emerged from the study analysis are summarized and discussed.

MBANet: Multi-branch aware network for kidney ultrasound images segmentation

Due to the influence of kidney morphology, heterogeneous structure and image quality, segmenting kidney in ultrasound images is challenging. To alleviate this challenge, we proposed a novel deep neural network architecture, namely Multi-branch Aware Network (MBANet), to segment kidney accurately and robustly. MBANet mainly consists of multi-scale feature pyramid (MSFP), multi-branch encoders (MBE) and master decoder. The design of MSFP can make the network more accessible to different kinds of class details at different scales. The information exchange between MBE can reduce the loss of feature information and improve the segmentation accuracy of the network. In addition, we designed a multi-scale fusion block (MFBlock) in the MBE to further extract and fuse more refined multi-scale image information. In order to further improve the robustness of MBANet, this paper also designed a step-by-step training mechanism. We validated the proposed approach and compared to several state-of-the-art approaches on the same kidney ultrasound datasets using six quantitative metrics. The results of our method on the six indicators of pixel accuracy (PA), intersection over union (IoU), precision, recall, specificity and F1-score (F1) are 98.83%, 92.38%, 97.10%, 95.03%, 99.46% and 0.9601, respectively. Compared with the comparison method, the average values on the six indicators are improved by about 2%. The evaluation results and segmentation results demonstrate that the proposed approach achieves the best overall performance on kidney ultrasound images segmentation.