Interobserver variability in identification of breast tumors in MRI and its implications for prognostic biomarkers and radiogenomics

BreastDM: A DCE-MRI dataset for breast tumor image segmentation and classification

Dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) has shown high sensitivity to diagnose breast cancer. However, few computer-aided algorithms focus on employing DCE-MR images for breast cancer diagnosis due to the lack of publicly available DCE-MRI datasets. To address this issue, our work releases a new DCE-MRI dataset called BreastDM for breast tumor segmentation and classification. In particular, a dataset of 232 patients selected with DCE-MR images for benign and malignant cases is established. Each case consists of three types of sequences: pre-contrast, post-contrast, and subtraction sequences. To show the difficulty of breast DCE-MRI tumor image segmentation and classification tasks, benchmarks are achieved by state-of-the-art image segmentation and classification algorithms, including conventional hand-crafted based methods and recently-emerged deep learning-based methods. More importantly, a local-global cross attention fusion network (LG-CAFN) is proposed to further improve the performance of breast tumor images classification. Specifically, LG-CAFN achieved the highest accuracy (88.20%, 83.93%) and AUC value (0.9154,0.8826) in both groups of experiments. Extensive experiments are conducted to present strong baselines based on various typical image segmentation and classification algorithms. Experiment results also demonstrate the superiority of the proposed LG-CAFN to other breast tumor images classification methods. The related dataset and evaluation codes are publicly available at smallboy-code/Breast-cancer-dataset.

Robustness of radiomics to variations in segmentation methods in multimodal brain MRI

Radiomics in neuroimaging uses fully automatic segmentation to delineate the anatomical areas for which radiomic features are computed. However, differences among these segmentation methods affect radiomic features to an unknown extent. A scan-rescan dataset (n = 46) of T1-weighted and diffusion tensor images was used. Subjects were split into a sleep-deprivation and a control group. Scans were segmented using four segmentation methods from which radiomic features were computed. First, we measured segmentation agreement using the Dice-coefficient. Second, robustness and reproducibility of radiomic features were measured using the intraclass correlation coefficient (ICC). Last, difference in predictive power was assessed using the Friedman-test on performance in a radiomics-based sleep deprivation classification application. Segmentation agreement was generally high (interquartile range = 0.77–0.90) and median feature robustness to segmentation method variation was higher (ICC > 0.7) than scan-rescan reproducibility (ICC 0.3–0.8). However, classification performance differed significantly among segmentation methods (p < 0.001) ranging from 77 to 84%. Accuracy was higher for more recent deep learning-based segmentation methods. Despite high agreement among segmentation methods, subtle differences significantly affected radiomic features and their predictive power. Consequently, the effect of differences in segmentation methods should be taken into account when designing and evaluating radiomics-based research methods.

Comparative Multicentric Evaluation of Inter-Observer Variability in Manual and Automatic Segmentation of Neuroblastic Tumors in Magnetic Resonance Images

Simple Summary

Tumor segmentation is a key step in oncologic imaging processing and is a time-consuming process usually performed manually by radiologists. To facilitate it, there is growing interest in applying deep-learning segmentation algorithms. Thus, we explore the variability between two observers performing manual segmentation and use the state-of-the-art deep learning architecture nnU-Net to develop a model to detect and segment neuroblastic tumors on MR images. We were able to show that the variability between nnU-Net and manual segmentation is similar to the inter-observer variability in manual segmentation. Furthermore, we compared the time needed to manually segment the tumors from scratch with the time required for the automatic model to segment the same cases, with posterior human validation with manual adjustment when needed.

Abstract

Tumor segmentation is one of the key steps in imaging processing. The goals of this study were to assess the inter-observer variability in manual segmentation of neuroblastic tumors and to analyze whether the state-of-the-art deep learning architecture nnU-Net can provide a robust solution to detect and segment tumors on MR images. A retrospective multicenter study of 132 patients with neuroblastic tumors was performed. Dice Similarity Coefficient (DSC) and Area Under the Receiver Operating Characteristic Curve (AUC ROC) were used to compare segmentation sets. Two more metrics were elaborated to understand the direction of the errors: the modified version of False Positive (FPRm) and False Negative (FNR) rates. Two radiologists manually segmented 46 tumors and a comparative study was performed. nnU-Net was trained-tuned with 106 cases divided into five balanced folds to perform cross-validation. The five resulting models were used as an ensemble solution to measure training (n = 106) and validation (n = 26) performance, independently. The time needed by the model to automatically segment 20 cases was compared to the time required for manual segmentation. The median DSC for manual segmentation sets was 0.969 (±0.032 IQR). The median DSC for the automatic tool was 0.965 (±0.018 IQR). The automatic segmentation model achieved a better performance regarding the FPRm. MR images segmentation variability is similar between radiologists and nnU-Net. Time leverage when using the automatic model with posterior visual validation and manual adjustment corresponds to 92.8%.

Transparency of deep neural networks for medical image analysis: A review of interpretability methods

Artificial Intelligence (AI) has emerged as a useful aid in numerous clinical applications for diagnosis and treatment decisions. Deep neural networks have shown the same or better performance than clinicians in many tasks owing to the rapid increase in the available data and computational power. In order to conform to the principles of trustworthy AI, it is essential that the AI system be transparent, robust, fair, and ensure accountability. Current deep neural solutions are referred to as black-boxes due to a lack of understanding of the specifics concerning the decision-making process. Therefore, there is a need to ensure the interpretability of deep neural networks before they can be incorporated into the routine clinical workflow. In this narrative review, we utilized systematic keyword searches and domain expertise to identify nine different types of interpretability methods that have been used for understanding deep learning models for medical image analysis applications based on the type of generated explanations and technical similarities. Furthermore, we report the progress made towards evaluating the explanations produced by various interpretability methods. Finally, we discuss limitations, provide guidelines for using interpretability methods and future directions concerning the interpretability of deep neural networks for medical imaging analysis.

Radiomics in breast MRI: current progress toward clinical application in the era of artificial intelligence

Breast magnetic resonance imaging (MRI) is the most sensitive imaging modality for breast cancer diagnosis and is widely used clinically. Dynamic contrast-enhanced MRI is the basis for breast MRI, but ultrafast images, T2-weighted images, and diffusion-weighted images are also taken to improve the characteristics of the lesion. Such multiparametric MRI with numerous morphological and functional data poses new challenges to radiologists, and thus, new tools for reliable, reproducible, and high-volume quantitative assessments are warranted. In this context, radiomics, which is an emerging field of research involving the conversion of digital medical images into mineable data for clinical decision-making and outcome prediction, has been gaining ground in oncology. Recent development in artificial intelligence has promoted radiomics studies in various fields including breast cancer treatment and numerous studies have been conducted. However, radiomics has shown a translational gap in clinical practice, and many issues remain to be solved. In this review, we will outline the steps of radiomics workflow and investigate clinical application of radiomics focusing on breast MRI based on published literature, as well as current discussion about limitations and challenges in radiomics.

Radiomics: A Primer for Breast Radiologists

Radiomics has a long-standing history in breast imaging with computer-aided detection (CAD) for screening mammography developed in the late 20th century. Although conventional CAD had widespread adoption, the clinical benefits for experienced breast radiologists were debatable due to high false-positive marks and subsequent increased recall rates. The dramatic growth in recent years of artificial intelligence-based analysis, including machine learning and deep learning, has provided numerous opportunities for improved modern radiomics work in breast imaging. There has been extensive radiomics work in mammography, digital breast tomosynthesis, MRI, ultrasound, PET-CT, and combined multimodality imaging. Specific radiomics outcomes of interest have been diverse, including CAD, prediction of response to neoadjuvant therapy, lesion classification, and survival, among other outcomes. Additionally, the radiogenomics subfield that correlates radiomics features with genetics has been very proliferative, in parallel with the clinical validation of breast cancer molecular subtypes and gene expression assays. Despite the promise of radiomics, there are important challenges related to image normalization, limited large unbiased data sets, and lack of external validation. Much of the radiomics work to date has been exploratory using single-institution retrospective series for analysis, but several promising lines of investigation have made the leap to clinical practice with commercially available products. As a result, breast radiologists will increasingly be incorporating radiomics-based tools into their daily practice in the near future. Therefore, breast radiologists must have a broad understanding of the scope, applications, and limitations of radiomics work.

Dedicated Axillary MRI-Based Radiomics Analysis for the Prediction of Axillary Lymph Node Metastasis in Breast Cancer

Radiomics features may contribute to increased diagnostic performance of MRI in the prediction of axillary lymph node metastasis. The objective of the study was to predict preoperative axillary lymph node metastasis in breast cancer using clinical models and radiomics models based on T2-weighted (T2W) dedicated axillary MRI features with node-by-node analysis. From August 2012 until October 2014, all women who had undergone dedicated axillary 3.0T T2W MRI, followed by axillary surgery, were retrospectively identified, and available clinical data were collected. All axillary lymph nodes were manually delineated on the T2W MR images, and quantitative radiomics features were extracted from the delineated regions. Data were partitioned patient-wise to train 100 models using different splits for the training and validation cohorts to account for multiple lymph nodes per patient and class imbalance. Features were selected in the training cohorts using recursive feature elimination with repeated 5-fold cross-validation, followed by the development of random forest models. The performance of the models was assessed using the area under the curve (AUC). A total of 75 women (median age, 61 years; interquartile range, 51-68 years) with 511 axillary lymph nodes were included. On final pathology, 36 (7%) of the lymph nodes had metastasis. A total of 105 original radiomics features were extracted from the T2W MR images. Each cohort split resulted in a different number of lymph nodes in the training cohorts and a different set of selected features. Performance of the 100 clinical and radiomics models showed a wide range of AUC values between 0.41-0.74 and 0.48-0.89 in the training cohorts, respectively, and between 0.30-0.98 and 0.37-0.99 in the validation cohorts, respectively. With these results, it was not possible to obtain a final prediction model. Clinical characteristics and dedicated axillary MRI-based radiomics with node-by-node analysis did not contribute to the prediction of axillary lymph node metastasis in breast cancer based on data where variations in acquisition and reconstruction parameters were not addressed.

A Practical Guide to Artificial Intelligence-Based Image Analysis in Radiology

The use of artificial intelligence (AI) is a powerful tool for image analysis that is increasingly being evaluated by radiology professionals. However, due to the fact that these methods have been developed for the analysis of nonmedical image data and data structure in radiology departments is not "AI ready", implementing AI in radiology is not straightforward. The purpose of this review is to guide the reader through the pipeline of an AI project for automated image analysis in radiology and thereby encourage its implementation in radiology departments. At the same time, this review aims to enable readers to critically appraise articles on AI-based software in radiology.

MRI-based radiomics in breast cancer: feature robustness with respect to inter-observer segmentation variability

Radiomics is an emerging field using the extraction of quantitative features from medical images for tissue characterization. While MRI-based radiomics is still at an early stage, it showed some promising results in studies focusing on breast cancer patients in improving diagnoses and therapy response assessment. Nevertheless, the use of radiomics raises a number of issues regarding feature quantification and robustness. Therefore, our study aim was to determine the robustness of radiomics features extracted by two commonly used radiomics software with respect to variability in manual breast tumor segmentation on MRI. A total of 129 histologically confirmed breast tumors were segmented manually in three dimensions on the first post-contrast T1-weighted MR exam by four observers: a dedicated breast radiologist, a resident, a Ph.D. candidate, and a medical student. Robust features were assessed using the intraclass correlation coefficient (ICC > 0.9). The inter-observer variability was evaluated by the volumetric Dice Similarity Coefficient (DSC). The mean DSC for all tumors was 0.81 (range 0.19–0.96), indicating a good spatial overlap of the segmentations based on observers of varying expertise. In total, 41.6% (552/1328) and 32.8% (273/833) of all RadiomiX and Pyradiomics features, respectively, were identified as robust and were independent of inter-observer manual segmentation variability.

Deep learning for identifying radiogenomic associations in breast cancer

RATIONALE AND OBJECTIVES

To determine whether deep learning models can distinguish between breast cancer molecular subtypes based on dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI).

MATERIALS AND METHODS

In this institutional review board-approved single-center study, we analyzed DCE-MR images of 270 patients at our institution. Lesions of interest were identified by radiologists. The task was to automatically determine whether the tumor is of the Luminal A subtype or of another subtype based on the MR image patches representing the tumor. Three different deep learning approaches were used to classify the tumor according to their molecular subtypes: learning from scratch where only tumor patches were used for training, transfer learning where networks pre-trained on natural images were fine-tuned using tumor patches, and off-the-shelf deep features where the features extracted by neural networks trained on natural images were used for classification with a support vector machine. Network architectures utilized in our experiments were GoogleNet, VGG, and CIFAR. We used 10-fold crossvalidation method for validation and area under the receiver operating characteristic (AUC) as the measure of performance.

RESULTS

The best AUC performance for distinguishing molecular subtypes was 0.65 (95% CI:[0.57,0.71]) and was achieved by the off-the-shelf deep features approach. The highest AUC performance for training from scratch was 0.58 (95% CI:[0.51,0.64]) and the best AUC performance for transfer learning was 0.60 (95% CI:[0.52,0.65]) respectively. For the off-the-shelf approach, the features extracted from the fully connected layer performed the best.

CONCLUSION

Deep learning may play a role in discovering radiogenomic associations in breast cancer.

Association of distant recurrence-free survival with algorithmically extracted MRI characteristics in breast cancer

BACKGROUND

While important in diagnosis of breast cancer, the scientific assessment of the role of imaging in prognosis of outcomes and treatment planning is limited.

PURPOSE

To evaluate the potential of using quantitative imaging variables for stratifying risk of distant recurrence in breast cancer patients.

STUDY TYPE

Retrospective.

POPULATION

In all, 892 female invasive breast cancer patients.

SEQUENCE

Dynamic contrast-enhanced MRI with field strength 1.5 T and 3 T.

ASSESSMENT

Computer vision algorithms were applied to extract a comprehensive set of 529 imaging features quantifying size, shape, enhancement patterns, and heterogeneity of the tumors and the surrounding tissue. Using a development set with 446 cases, we selected 20 imaging features with high prognostic value.

STATISTICAL TESTS

We evaluated the imaging features using an independent test set with 446 cases. The principal statistical measure was a concordance index between individual imaging features and patient distant recurrence-free survival (DRFS).

RESULTS

The strongest association with DRFS that persisted after controlling for known prognostic clinical and pathology variables was found for signal enhancement ratio (SER) partial tumor volume (concordance index [C] = 0.768, 95% confidence interval [CI]: 0.679-0.856), tumor major axis length (C = 0.742, 95% CI: 0.650-0.834), kurtosis of the SER map within tumor (C = 0.640, 95% CI: 0.521-0.760), tumor cluster shade (C = 0.313, 95% CI: 0.216-0.410), and washin rate information measure of correlation (C = 0.702, 95% CI: 0.601-0.803).

DATA CONCLUSION

Quantitative assessment of breast cancer features seen in a routine breast MRI might be able to be used for assessment of risk of distant recurrence.

LEVEL OF EVIDENCE

4 Technical Efficacy: Stage 6 J. Magn. Reson. Imaging 2019.

Breast cancer MRI radiomics: An overview of algorithmic features and impact of inter-reader variability in annotating tumors

PURPOSE

To review features used in MRI radiomics of breast cancer and study the inter-reader stability of the features.

METHODS

We implemented 529 algorithmic features that can be extracted from tumor and fibroglandular tissue (FGT) in breast MRIs. The features were identified based on a review of the existing literature with consideration of their usage, prognostic ability, and uniqueness. The set was then extended so that it comprehensively describes breast cancer imaging characteristics. The features were classified into 10 groups based on the type of data used to extract them and the type of calculation being performed. For the assessment of inter-reader variability, four fellowship-trained readers annotated tumors on preoperative dynamic contrast-enhanced MRIs for 50 breast cancer patients. Based on the annotations, an algorithm automatically segmented the image and extracted all features resulting in one set of features for each reader. For a given feature, the inter-reader stability was defined as the intraclass correlation coefficient (ICC) computed using the feature values obtained through all readers for all cases.

RESULTS

The average inter-reader stability for all features was 0.8474 (95% CI: 0.8068-0.8858). The mean inter-reader stability was lower for tumor-based features (0.6348, 95% CI: 0.5391-0.7257) than FGT-based features (0.9984, 95% CI: 0.9970-0.9992). The feature group with the highest inter-reader stability quantifies breast and FGT volume. The feature group with the lowest inter-reader stability quantifies variations in tumor enhancement.

CONCLUSIONS

Breast MRI radiomics features widely vary in terms of their stability in the presence of inter-reader variability. Appropriate measures need to be taken for reducing this variability in tumor-based radiomics.

A study of association of Oncotype DX recurrence score with DCE-MRI characteristics using multivariate machine learning models

PURPOSE

To determine whether multivariate machine learning models of algorithmically assessed magnetic resonance imaging (MRI) features from breast cancer patients are associated with Oncotype DX (ODX) test recurrence scores.

METHODS

A set of 261 female patients with invasive breast cancer, pre-operative dynamic contrast enhanced magnetic resonance (DCE-MR) images and available ODX score at our institution was identified. A computer algorithm extracted a comprehensive set of 529 features from the DCE-MR images of these patients. The set of patients was divided into a training set and a test set. Using the training set we developed two machine learning-based models to discriminate (1) high ODX scores from intermediate and low ODX scores, and (2) high and intermediate ODX scores from low ODX scores. The performance of these models was evaluated on the independent test set.

RESULTS

High against low and intermediate ODX scores were predicted by the multivariate model with AUC 0.77 (95% CI 0.56-0.98, p < 0.003). Low against intermediate and high ODX score was predicted with AUC 0.51 (95% CI 0.41-0.61, p = 0.75).

CONCLUSION

A moderate association between imaging and ODX score was identified. The evaluated models currently do not warrant replacement of ODX with imaging alone.

Can algorithmically assessed MRI features predict which patients with a preoperative diagnosis of ductal carcinoma in situ are upstaged to invasive breast cancer?

PURPOSE

To assess the ability of algorithmically assessed magnetic resonance imaging (MRI) features to predict the likelihood of upstaging to invasive cancer in newly diagnosed ductal carcinoma in situ (DCIS).

MATERIALS AND METHODS

We identified 131 patients at our institution from 2000-2014 with a core needle biopsy-confirmed diagnosis of pure DCIS, a 1.5 or 3T preoperative bilateral breast MRI with nonfat-saturated T1 -weighted MRI sequences, no preoperative therapy before breast MRI, and no prior history of breast cancer. A fellowship-trained radiologist identified the lesion on each breast MRI using a bounding box. Twenty-nine imaging features were then computed automatically using computer algorithms based on the radiologist's annotation.

RESULTS

The rate of upstaging of DCIS to invasive cancer in our study was 26.7% (35/131). Out of all imaging variables tested, the information measure of correlation 1, which quantifies spatial dependency in neighboring voxels of the tumor, showed the highest predictive value of upstaging with an area under the curve (AUC) = 0.719 (95% confidence interval [CI]: 0.609-0.829). This feature was statistically significant after adjusting for tumor size (P < 0.001).

CONCLUSION

Automatically assessed MRI features may have a role in triaging which patients with a preoperative diagnosis of DCIS are at highest risk for occult invasive disease.

LEVEL OF EVIDENCE

4 Technical Efficacy: Stage 3 J. Magn. Reson. Imaging 2017;46:1332-1340.