A publicly available deep learning model and dataset for segmentation of breast, fibroglandular tissue, and vessels in breast MRI

Breast density, or the amount of fibroglandular tissue (FGT) relative to the overall breast volume, increases the risk of developing breast cancer. Although previous studies have utilized deep learning to assess breast density, the limited public availability of data and quantitative tools hinders the development of better assessment tools. Our objective was to (1) create and share a large dataset of pixel-wise annotations according to well-defined criteria, and (2) develop, evaluate, and share an automated segmentation method for breast, FGT, and blood vessels using convolutional neural networks. We used the Duke Breast Cancer MRI dataset to randomly select 100 MRI studies and manually annotated the breast, FGT, and blood vessels for each study. Model performance was evaluated using the Dice similarity coefficient (DSC). The model achieved DSC values of 0.92 for breast, 0.86 for FGT, and 0.65 for blood vessels on the test set. The correlation between our model's predicted breast density and the manually generated masks was 0.95. The correlation between the predicted breast density and qualitative radiologist assessment was 0.75. Our automated models can accurately segment breast, FGT, and blood vessels using pre-contrast breast MRI data. The data and the models were made publicly available.

Computed Tomography Volumetrics for Size Matching in Lung Transplantation for Restrictive Disease

BACKGROUND

There is no consensus on the optimal allograft sizing strategy for lung transplantation in restrictive lung disease. Current methods that are based on predicted total lung capacity (pTLC) ratios do not account for the diminutive recipient chest size. The study investigators hypothesized that a new sizing ratio incorporating preoperative recipient computed tomographic lung volumes (CTVol) would be associated with postoperative outcomes.

METHODS

A retrospective single-institution study was conducted of adults undergoing primary bilateral lung transplantation between January 2016 and July 2020 for restrictive lung disease. CTVol was computed for recipients by using advanced segmentation software. Two sizing ratios were calculated: pTLC ratio (pTLCdonor/pTLCrecipient) and a new volumetric ratio (pTLCdonor/CTVolrecipient). Patients were divided into reference, oversized, and undersized groups on the basis of ratio quintiles, and multivariable models were used to assess the effect of the ratios on primary graft dysfunction and survival.

RESULTS

CTVol was successfully acquired in 218 of 220 (99.1%) patients. In adjusted analysis, undersizing on the basis of the volumetric ratio was independently associated with decreased primary graft dysfunction grade 2 or 3 within 72 hours (odds ratio, 0.42; 95% CI, 0.20-0.87; P =.02). The pTLC ratio was not significantly associated with primary graft dysfunction. Oversizing on the basis of the volumetric ratio was independently associated with an increased risk of death (hazard ratio, 2.27; 95% CI, 1.04-4.99; P =.04], whereas the pTLC ratio did not have a significant survival association.

CONCLUSIONS

Using computed tomography-acquired lung volumes for donor-recipient size matching in lung transplantation is feasible with advanced segmentation software. This method may be more predictive of outcome compared with current sizing methods, which use gender and height only.

SWSSL: Sliding Window-Based Self-Supervised Learning for Anomaly Detection in High-Resolution Images

Anomaly detection (AD) aims to determine if an instance has properties different from those seen in normal cases. The success of this technique depends on how well a neural network learns from normal instances. We observe that the learning difficulty scales exponentially with the input resolution, making it infeasible to apply AD to high-resolution images. Resizing them to a lower resolution is a compromising solution and does not align with clinical practice where the diagnosis could depend on image details. In this work, we propose to train the network and perform inference at the patch level, through the sliding window algorithm. This simple operation allows the network to receive high-resolution images but introduces additional training difficulties, including inconsistent image structure and higher variance. We address these concerns by setting the network's objective to learn augmentation-invariant features. We further study the augmentation function in the context of medical imaging. In particular, we observe that the resizing operation, a key augmentation in general computer vision literature, is detrimental to detection accuracy, and the inverting operation can be beneficial. We also propose a new module that encourages the network to learn from adjacent patches to boost detection performance. Extensive experiments are conducted on breast tomosynthesis and chest X-ray datasets and our method improves 8.03% and 5.66% AUC on image-level classification respectively over the current leading techniques. The experimental results demonstrate the effectiveness of our approach.

Improving Image Classification of Knee Radiographs: An Automated Image Labeling Approach

Large numbers of radiographic images are available in musculoskeletal radiology practices which could be used for training of deep learning models for diagnosis of knee abnormalities. However, those images do not typically contain readily available labels due to limitations of human annotations. The purpose of our study was to develop an automated labeling approach that improves the image classification model to distinguish normal knee images from those with abnormalities or prior arthroplasty. The automated labeler was trained on a small set of labeled data to automatically label a much larger set of unlabeled data, further improving the image classification performance for knee radiographic diagnosis. We used BioBERT and EfficientNet as the feature extraction backbone of the labeler and imaging model, respectively. We developed our approach using 7382 patients and validated it on a separate set of 637 patients. The final image classification model, trained using both manually labeled and pseudo-labeled data, had the higher weighted average AUC (WA-AUC 0.903) value and higher AUC values among all classes (normal AUC 0.894; abnormal AUC 0.896, arthroplasty AUC 0.990) compared to the baseline model (WA-AUC = 0.857; normal AUC 0.842; abnormal AUC 0.848, arthroplasty AUC 0.987), trained using only manually labeled data. Statistical tests show that the improvement is significant on normal (p value < 0.002), abnormal (p value < 0.001), and WA-AUC (p value = 0.001). Our findings demonstrated that the proposed automated labeling approach significantly improves the performance of image classification for radiographic knee diagnosis, allowing for facilitating patient care and curation of large knee datasets.

Segment anything model for medical image analysis: An experimental study

Training segmentation models for medical images continues to be challenging due to the limited availability of data annotations. Segment Anything Model (SAM) is a foundation model trained on over 1 billion annotations, predominantly for natural images, that is intended to segment user-defined objects of interest in an interactive manner. While the model performance on natural images is impressive, medical image domains pose their own set of challenges. Here, we perform an extensive evaluation of SAM's ability to segment medical images on a collection of 19 medical imaging datasets from various modalities and anatomies. In our experiments, we generated point and box prompts for SAM using a standard method that simulates interactive segmentation. We report the following findings: (1) SAM's performance based on single prompts highly varies depending on the dataset and the task, from IoU=0.1135 for spine MRI to IoU=0.8650 for hip X-ray. (2) Segmentation performance appears to be better for well-circumscribed objects with prompts with less ambiguity such as the segmentation of organs in computed tomography and poorer in various other scenarios such as the segmentation of brain tumors. (3) SAM performs notably better with box prompts than with point prompts. (4) SAM outperforms similar methods RITM, SimpleClick, and FocalClick in almost all single-point prompt settings. (5) When multiple-point prompts are provided iteratively, SAM's performance generally improves only slightly while other methods' performance improves to the level that surpasses SAM's point-based performance. We also provide several illustrations for SAM's performance on all tested datasets, iterative segmentation, and SAM's behavior given prompt ambiguity. We conclude that SAM shows impressive zero-shot segmentation performance for certain medical imaging datasets, but moderate to poor performance for others. SAM has the potential to make a significant impact in automated medical image segmentation in medical imaging, but appropriate care needs to be applied when using it. Code for evaluation SAM is made publicly available at https://github.com/mazurowski-lab/segment-anything-medical-evaluation.

MRI-based Deep Learning Assessment of Amyloid, Tau, and Neurodegeneration Biomarker Status across the Alzheimer Disease Spectrum

Background PET can be used for amyloid-tau-neurodegeneration (ATN) classification in Alzheimer disease, but incurs considerable cost and exposure to ionizing radiation. MRI currently has limited use in characterizing ATN status. Deep learning techniques can detect complex patterns in MRI data and have potential for noninvasive characterization of ATN status. Purpose To use deep learning to predict PET-determined ATN biomarker status using MRI and readily available diagnostic data. Materials and Methods MRI and PET data were retrospectively collected from the Alzheimer's Disease Imaging Initiative. PET scans were paired with MRI scans acquired within 30 days, from August 2005 to September 2020. Pairs were randomly split into subsets as follows: 70% for training, 10% for validation, and 20% for final testing. A bimodal Gaussian mixture model was used to threshold PET scans into positive and negative labels. MRI data were fed into a convolutional neural network to generate imaging features. These features were combined in a logistic regression model with patient demographics, APOE gene status, cognitive scores, hippocampal volumes, and clinical diagnoses to classify each ATN biomarker component as positive or negative. Area under the receiver operating characteristic curve (AUC) analysis was used for model evaluation. Feature importance was derived from model coefficients and gradients. Results There were 2099 amyloid (mean patient age, 75 years ± 10 [SD]; 1110 male), 557 tau (mean patient age, 75 years ± 7; 280 male), and 2768 FDG PET (mean patient age, 75 years ± 7; 1645 male) and MRI pairs. Model AUCs for the test set were as follows: amyloid, 0.79 (95% CI: 0.74, 0.83); tau, 0.73 (95% CI: 0.58, 0.86); and neurodegeneration, 0.86 (95% CI: 0.83, 0.89). Within the networks, high gradients were present in key temporal, parietal, frontal, and occipital cortical regions. Model coefficients for cognitive scores, hippocampal volumes, and APOE status were highest. Conclusion A deep learning algorithm predicted each component of PET-determined ATN status with acceptable to excellent efficacy using MRI and other available diagnostic data. © RSNA, 2023 Supplemental material is available for this article.

Duke Liver Dataset: A Publicly Available Liver MRI Dataset with Liver Segmentation Masks and Series Labels

The Duke Liver Dataset contains 2146 abdominal MRI series from 105 patients, including a majority with cirrhotic features, and 310 image series with corresponding manually segmented liver masks.

Feasibility of predicting a screening digital breast tomosynthesis recall using features extracted from the electronic medical record

PURPOSE

Tools to predict a screening mammogram recall at the time of scheduling could improve patient care. We extracted patient demographic and breast care history information within the electronic medical record (EMR) for women undergoing digital breast tomosynthesis (DBT) to identify which factors were associated with a screening recall recommendation.

METHOD

In 2018, 21,543 women aged 40 years or greater who underwent screening DBT at our institution were identified. Demographic information and breast care factors were extracted automatically from the EMR. The primary outcome was a screening recall recommendation of BI-RADS 0. A multivariable logistic regression model was built and included age, race, ethnicity groups, family breast cancer history, personal breast cancer history, surgical breast cancer history, recall history, and days since last available screening mammogram.

RESULTS

Multiple factors were associated with a recall on the multivariable model: history of breast cancer surgery (OR: 2.298, 95% CI: 1.854, 2.836); prior recall within the last five years (vs no prior, OR: 0.768, 95% CI: 0.687, 0.858); prior screening mammogram within 0-18 months (vs no prior, OR: 0.601, 95% CI: 0.520, 0.691), prior screening mammogram within 18-30 months (vs no prior, OR: 0.676, 95% CI: 0.520, 0.691); and age (normalized OR: 0.723, 95% CI: 0.690, 0.758).

CONCLUSIONS

It is feasible to predict a DBT screening recall recommendation using patient demographics and breast care factors that can be extracted automatically from the EMR.

Multistep Automated Data Labelling Procedure (MADLaP) for thyroid nodules on ultrasound: An artificial intelligence approach for automating image annotation

Machine learning (ML) for diagnosis of thyroid nodules on ultrasound is an active area of research. However, ML tools require large, well-labeled datasets, the curation of which is time-consuming and labor-intensive. The purpose of our study was to develop and test a deep-learning-based tool to facilitate and automate the data annotation process for thyroid nodules; we named our tool Multistep Automated Data Labelling Procedure (MADLaP). MADLaP was designed to take multiple inputs including pathology reports, ultrasound images, and radiology reports. Using multiple step-wise 'modules' including rule-based natural language processing, deep-learning-based imaging segmentation, and optical character recognition, MADLaP automatically identified images of a specific thyroid nodule and correctly assigned a pathology label. The model was developed using a training set of 378 patients across our health system and tested on a separate set of 93 patients. Ground truths for both sets were selected by an experienced radiologist. Performance metrics including yield (how many labeled images the model produced) and accuracy (percentage correct) were measured using the test set. MADLaP achieved a yield of 63 % and an accuracy of 83 %. The yield progressively increased as the input data moved through each module, while accuracy peaked part way through. Error analysis showed that inputs from certain examination sites had lower accuracy (40 %) than the other sites (90 %, 100 %). MADLaP successfully created curated datasets of labeled ultrasound images of thyroid nodules. While accurate, the relatively suboptimal yield of MADLaP exposed some challenges when trying to automatically label radiology images from heterogeneous sources. The complex task of image curation and annotation could be automated, allowing for enrichment of larger datasets for use in machine learning development.

Unsupervised anomaly localization in high-resolution breast scans using deep pluralistic image completion

Automated tumor detection in Digital Breast Tomosynthesis (DBT) is a difficult task due to natural tumor rarity, breast tissue variability, and high resolution. Given the scarcity of abnormal images and the abundance of normal images for this problem, an anomaly detection/localization approach could be well-suited. However, most anomaly localization research in machine learning focuses on non-medical datasets, and we find that these methods fall short when adapted to medical imaging datasets. The problem is alleviated when we solve the task from the image completion perspective, in which the presence of anomalies can be indicated by a discrepancy between the original appearance and its auto-completion conditioned on the surroundings. However, there are often many valid normal completions given the same surroundings, especially in the DBT dataset, making this evaluation criterion less precise. To address such an issue, we consider pluralistic image completion by exploring the distribution of possible completions instead of generating fixed predictions. This is achieved through our novel application of spatial dropout on the completion network during inference time only, which requires no additional training cost and is effective at generating diverse completions. We further propose minimum completion distance (MCD), a new metric for detecting anomalies, thanks to these stochastic completions. We provide theoretical as well as empirical support for the superiority over existing methods of using the proposed method for anomaly localization. On the DBT dataset, our model outperforms other state-of-the-art methods by at least 10% AUROC for pixel-level detection.

Deep learning for classification of thyroid nodules on ultrasound: validation on an independent dataset

OBJECTIVES

The purpose is to apply a previously validated deep learning algorithm to a new thyroid nodule ultrasound image dataset and compare its performances with radiologists.

METHODS

Prior study presented an algorithm which is able to detect thyroid nodules and then make malignancy classifications with two ultrasound images. A multi-task deep convolutional neural network was trained from 1278 nodules and originally tested with 99 separate nodules. The results were comparable with that of radiologists. The algorithm was further tested with 378 nodules imaged with ultrasound machines from different manufacturers and product types than the training cases. Four experienced radiologists were requested to evaluate the nodules for comparison with deep learning.

RESULTS

The Area Under Curve (AUC) of the deep learning algorithm and four radiologists were calculated with parametric, binormal estimation. For the deep learning algorithm, the AUC was 0.69 (95% CI: 0.64-0.75). The AUC of radiologists were 0.63 (95% CI: 0.59-0.67), 0.66 (95% CI:0.61-0.71), 0.65 (95% CI: 0.60-0.70), and 0.63 (95%CI: 0.58-0.67).

CONCLUSION

In the new testing dataset, the deep learning algorithm achieved similar performances with all four radiologists. The relative performance difference between the algorithm and the radiologists is not significantly affected by the difference of ultrasound scanner.

Deep Learning for Breast MRI Style Transfer with Limited Training Data

In this work we introduce a novel medical image style transfer method, StyleMapper, that can transfer medical scans to an unseen style with access to limited training data. This is made possible by training our model on unlimited possibilities of simulated random medical imaging styles on the training set, making our work more computationally efficient when compared with other style transfer methods. Moreover, our method enables arbitrary style transfer: transferring images to styles unseen in training. This is useful for medical imaging, where images are acquired using different protocols and different scanner models, resulting in a variety of styles that data may need to be transferred between. Our model disentangles image content from style and can modify an image's style by simply replacing the style encoding with one extracted from a single image of the target style, with no additional optimization required. This also allows the model to distinguish between different styles of images, including among those that were unseen in training. We propose a formal description of the proposed model. Experimental results on breast magnetic resonance images indicate the effectiveness of our method for style transfer. Our style transfer method allows for the alignment of medical images taken with different scanners into a single unified style dataset, allowing for the training of other downstream tasks on such a dataset for tasks such as classification, object detection and others.

A Competition, Benchmark, Code, and Data for Using Artificial Intelligence to Detect Lesions in Digital Breast Tomosynthesis

IMPORTANCE

An accurate and robust artificial intelligence (AI) algorithm for detecting cancer in digital breast tomosynthesis (DBT) could significantly improve detection accuracy and reduce health care costs worldwide.

OBJECTIVES

To make training and evaluation data for the development of AI algorithms for DBT analysis available, to develop well-defined benchmarks, and to create publicly available code for existing methods.

DESIGN, SETTING, AND PARTICIPANTS

This diagnostic study is based on a multi-institutional international grand challenge in which research teams developed algorithms to detect lesions in DBT. A data set of 22 032 reconstructed DBT volumes was made available to research teams. Phase 1, in which teams were provided 700 scans from the training set, 120 from the validation set, and 180 from the test set, took place from December 2020 to January 2021, and phase 2, in which teams were given the full data set, took place from May to July 2021.

MAIN OUTCOMES AND MEASURES

The overall performance was evaluated by mean sensitivity for biopsied lesions using only DBT volumes with biopsied lesions; ties were broken by including all DBT volumes.

RESULTS

A total of 8 teams participated in the challenge. The team with the highest mean sensitivity for biopsied lesions was the NYU B-Team, with 0.957 (95% CI, 0.924-0.984), and the second-place team, ZeDuS, had a mean sensitivity of 0.926 (95% CI, 0.881-0.964). When the results were aggregated, the mean sensitivity for all submitted algorithms was 0.879; for only those who participated in phase 2, it was 0.926.

CONCLUSIONS AND RELEVANCE

In this diagnostic study, an international competition produced algorithms with high sensitivity for using AI to detect lesions on DBT images. A standardized performance benchmark for the detection task using publicly available clinical imaging data was released, with detailed descriptions and analyses of submitted algorithms accompanied by a public release of their predictions and code for selected methods. These resources will serve as a foundation for future research on computer-assisted diagnosis methods for DBT, significantly lowering the barrier of entry for new researchers.

Anomaly Detection of Calcifications in Mammography Based on 11,000 Negative Cases

In mammography, calcifications are one of the most common signs of breast cancer. Detection of such lesions is an active area of research for computer-aided diagnosis and machine learning algorithms. Due to limited numbers of positive cases, many supervised detection models suffer from overfitting and fail to generalize. We present a one-class, semi-supervised framework using a deep convolutional autoencoder trained with over 50,000 images from 11,000 negative-only cases. Since the model learned from only normal breast parenchymal features, calcifications produced large signals when comparing the residuals between input and reconstruction output images. As a key advancement, a structural dissimilarity index was used to suppress non-structural noises. Our selected model achieved pixel-based AUROC of 0.959 and AUPRC of 0.676 during validation, where calcification masks were defined in a semi-automated process. Although not trained directly on any cancers, detection performance of calcification lesions on 1,883 testing images (645 malignant and 1238 negative) achieved 75% sensitivity at 2.5 false positives per image. Performance plateaued early when trained with only a fraction of the cases, and greater model complexity or a larger dataset did not improve performance. This study demonstrates the potential of this anomaly detection approach to detect mammographic calcifications in a semi-supervised manner with efficient use of a small number of labeled images, and may facilitate new clinical applications such as computer-aided triage and quality improvement.

Multi-label annotation of text reports from computed tomography of the chest, abdomen, and pelvis using deep learning

Background

There is progress to be made in building artificially intelligent systems to detect abnormalities that are not only accurate but can handle the true breadth of findings that radiologists encounter in body (chest, abdomen, and pelvis) computed tomography (CT). Currently, the major bottleneck for developing multi-disease classifiers is a lack of manually annotated data. The purpose of this work was to develop high throughput multi-label annotators for body CT reports that can be applied across a variety of abnormalities, organs, and disease states thereby mitigating the need for human annotation.

Methods

We used a dictionary approach to develop rule-based algorithms (RBA) for extraction of disease labels from radiology text reports. We targeted three organ systems (lungs/pleura, liver/gallbladder, kidneys/ureters) with four diseases per system based on their prevalence in our dataset. To expand the algorithms beyond pre-defined keywords, attention-guided recurrent neural networks (RNN) were trained using the RBA-extracted labels to classify reports as being positive for one or more diseases or normal for each organ system. Alternative effects on disease classification performance were evaluated using random initialization or pre-trained embedding as well as different sizes of training datasets. The RBA was tested on a subset of 2158 manually labeled reports and performance was reported as accuracy and F-score. The RNN was tested against a test set of 48,758 reports labeled by RBA and performance was reported as area under the receiver operating characteristic curve (AUC), with 95% CIs calculated using the DeLong method.

Results

Manual validation of the RBA confirmed 91–99% accuracy across the 15 different labels. Our models extracted disease labels from 261,229 radiology reports of 112,501 unique subjects. Pre-trained models outperformed random initialization across all diseases. As the training dataset size was reduced, performance was robust except for a few diseases with a relatively small number of cases. Pre-trained classification AUCs reached > 0.95 for all four disease outcomes and normality across all three organ systems.

Conclusions

Our label-extracting pipeline was able to encompass a variety of cases and diseases in body CT reports by generalizing beyond strict rules with exceptional accuracy. The method described can be easily adapted to enable automated labeling of hospital-scale medical data sets for training image-based disease classifiers.

3D Pyramid Pooling Network for Abdominal MRI Series Classification

Recognizing and organizing different series in an MRI examination is important both for clinical review and research, but it is poorly addressed by the current generation of picture archiving and communication systems (PACSs) and post-processing workstations. In this paper, we study the problem of using deep convolutional neural networks for automatic classification of abdominal MRI series to one of many series types. Our contributions are three-fold. First, we created a large abdominal MRI dataset containing 3717 MRI series including 188,665 individual images, derived from liver examinations. 30 different series types are represented in this dataset. The dataset was annotated by consensus readings from two radiologists. Both the MRIs and the annotations were made publicly available. Second, we proposed a 3D pyramid pooling network, which can elegantly handle abdominal MRI series with varied sizes of each dimension, and achieved state-of-the-art classification performance. Third, we performed the first ever comparison between the algorithm and the radiologists on an additional dataset and had several meaningful findings.

Prediction of Upstaging in Ductal Carcinoma in Situ Based on Mammographic Radiomic Features

Background Improving diagnosis of ductal carcinoma in situ (DCIS) before surgery is important in choosing optimal patient management strategies. However, patients may harbor occult invasive disease not detected until definitive surgery. Purpose To assess the performance and clinical utility of mammographic radiomic features in the prediction of occult invasive cancer among women diagnosed with DCIS on the basis of core biopsy findings. Materials and Methods In this Health Insurance Portability and Accountability Act-compliant retrospective study, digital magnification mammographic images were collected from women who underwent breast core-needle biopsy for calcifications that was performed at a single institution between September 2008 and April 2017 and yielded a diagnosis of DCIS. The database query was directed at asymptomatic women with calcifications without a mass, architectural distortion, asymmetric density, or palpable disease. Logistic regression with regularization was used. Differences across training and internal test set by upstaging rate, age, lesion size, and estrogen and progesterone receptor status were assessed by using the Kruskal-Wallis or χ² test. Results The study consisted of 700 women with DCIS (age range, 40-89 years; mean age, 59 years ± 10 [standard deviation]), including 114 with lesions (16.3%) upstaged to invasive cancer at subsequent surgery. The sample was split randomly into 400 women for the training set and 300 for the testing set (mean ages: training set, 59 years ± 10; test set, 59 years ± 10; P = .85). A total of 109 radiomic and four clinical features were extracted. The best model on the test set by using all radiomic and clinical features helped predict upstaging with an area under the receiver operating characteristic curve of 0.71 (95% CI: 0.62, 0.79). For a fixed high sensitivity (90%), the model yielded a specificity of 22%, a negative predictive value of 92%, and an odds ratio of 2.4 (95% CI: 1.8, 3.2). High specificity (90%) corresponded to a sensitivity of 37%, positive predictive value of 41%, and odds ratio of 5.0 (95% CI: 2.8, 9.0). Conclusion Machine learning models that use radiomic features applied to mammographic calcifications may help predict upstaging of ductal carcinoma in situ, which can refine clinical decision making and treatment planning. © RSNA, 2022.

Classification of Multiple Diseases on Body CT Scans Using Weakly Supervised Deep Learning

PURPOSE

To design multidisease classifiers for body CT scans for three different organ systems using automatically extracted labels from radiology text reports.

MATERIALS AND METHODS

This retrospective study included a total of 12 092 patients (mean age, 57 years ± 18 [standard deviation]; 6172 women) for model development and testing. Rule-based algorithms were used to extract 19 225 disease labels from 13 667 body CT scans performed between 2012 and 2017. Using a three-dimensional DenseVNet, three organ systems were segmented: lungs and pleura, liver and gallbladder, and kidneys and ureters. For each organ system, a three-dimensional convolutional neural network classified each as no apparent disease or for the presence of four common diseases, for a total of 15 different labels across all three models. Testing was performed on a subset of 2158 CT volumes relative to 2875 manually derived reference labels from 2133 patients (mean age, 58 years ± 18; 1079 women). Performance was reported as area under the receiver operating characteristic curve (AUC), with 95% CIs calculated using the DeLong method.

RESULTS

Manual validation of the extracted labels confirmed 91%-99% accuracy across the 15 different labels. AUCs for lungs and pleura labels were as follows: atelectasis, 0.77 (95% CI: 0.74, 0.81); nodule, 0.65 (95% CI: 0.61, 0.69); emphysema, 0.89 (95% CI: 0.86, 0.92); effusion, 0.97 (95% CI: 0.96, 0.98); and no apparent disease, 0.89 (95% CI: 0.87, 0.91). AUCs for liver and gallbladder were as follows: hepatobiliary calcification, 0.62 (95% CI: 0.56, 0.67); lesion, 0.73 (95% CI: 0.69, 0.77); dilation, 0.87 (95% CI: 0.84, 0.90); fatty, 0.89 (95% CI: 0.86, 0.92); and no apparent disease, 0.82 (95% CI: 0.78, 0.85). AUCs for kidneys and ureters were as follows: stone, 0.83 (95% CI: 0.79, 0.87); atrophy, 0.92 (95% CI: 0.89, 0.94); lesion, 0.68 (95% CI: 0.64, 0.72); cyst, 0.70 (95% CI: 0.66, 0.73); and no apparent disease, 0.79 (95% CI: 0.75, 0.83).

CONCLUSION

Weakly supervised deep learning models were able to classify diverse diseases in multiple organ systems from CT scans.Keywords: CT, Diagnosis/Classification/Application Domain, Semisupervised Learning, Whole-Body Imaging© RSNA, 2022.

Normalization of breast MRIs using cycle-consistent generative adversarial networks

OBJECTIVES

Dynamic Contrast Enhanced-Magnetic Resonance Imaging (DCE-MRI) is widely used to complement ultrasound examinations and x-ray mammography for early detection and diagnosis of breast cancer. However, images generated by various MRI scanners (e.g., GE Healthcare, and Siemens) differ both in intensity and noise distribution, preventing algorithms trained on MRIs from one scanner to generalize to data from other scanners. In this work, we propose a method to solve this problem by normalizing images between various scanners.

METHODS

MRI normalization is challenging because it requires normalizing intensity values and mapping noise distributions between scanners. We utilize a cycle-consistent generative adversarial network to learn a bidirectional mapping and perform normalization between MRIs produced by GE Healthcare and Siemens scanners in an unpaired setting. Initial experiments demonstrate that the traditional CycleGAN architecture struggles to preserve the anatomical structures of the breast during normalization. Thus, we propose two technical innovations in order to preserve both the shape of the breast as well as the tissue structures within the breast. First, we incorporate mutual information loss during training in order to ensure anatomical consistency. Second, we propose a modified discriminator architecture that utilizes a smaller field-of-view to ensure the preservation of finer details in the breast tissue.

RESULTS

Quantitative and qualitative evaluations show that the second innovation consistently preserves the breast shape and tissue structures while also performing the proper intensity normalization and noise distribution mapping.

CONCLUSION

Our results demonstrate that the proposed model can successfully learn a bidirectional mapping and perform normalization between MRIs produced by different vendors, potentially enabling improved diagnosis and detection of breast cancer. All the data used in this study are publicly available at https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=70226903.

A Data Set and Deep Learning Algorithm for the Detection of Masses and Architectural Distortions in Digital Breast Tomosynthesis Images

IMPORTANCE

Breast cancer screening is among the most common radiological tasks, with more than 39 million examinations performed each year. While it has been among the most studied medical imaging applications of artificial intelligence, the development and evaluation of algorithms are hindered by the lack of well-annotated, large-scale publicly available data sets.

OBJECTIVES

To curate, annotate, and make publicly available a large-scale data set of digital breast tomosynthesis (DBT) images to facilitate the development and evaluation of artificial intelligence algorithms for breast cancer screening; to develop a baseline deep learning model for breast cancer detection; and to test this model using the data set to serve as a baseline for future research.

DESIGN, SETTING, AND PARTICIPANTS

In this diagnostic study, 16 802 DBT examinations with at least 1 reconstruction view available, performed between August 26, 2014, and January 29, 2018, were obtained from Duke Health System and analyzed. From the initial cohort, examinations were divided into 4 groups and split into training and test sets for the development and evaluation of a deep learning model. Images with foreign objects or spot compression views were excluded. Data analysis was conducted from January 2018 to October 2020.

EXPOSURES

Screening DBT.

MAIN OUTCOMES AND MEASURES

The detection algorithm was evaluated with breast-based free-response receiver operating characteristic curve and sensitivity at 2 false positives per volume.

RESULTS

The curated data set contained 22 032 reconstructed DBT volumes that belonged to 5610 studies from 5060 patients with a mean (SD) age of 55 (11) years and 5059 (100.0%) women. This included 4 groups of studies: (1) 5129 (91.4%) normal studies; (2) 280 (5.0%) actionable studies, for which where additional imaging was needed but no biopsy was performed; (3) 112 (2.0%) benign biopsied studies; and (4) 89 studies (1.6%) with cancer. Our data set included masses and architectural distortions that were annotated by 2 experienced radiologists. Our deep learning model reached breast-based sensitivity of 65% (39 of 60; 95% CI, 56%-74%) at 2 false positives per DBT volume on a test set of 460 examinations from 418 patients.

CONCLUSIONS AND RELEVANCE

The large, diverse, and curated data set presented in this study could facilitate the development and evaluation of artificial intelligence algorithms for breast cancer screening by providing data for training as well as a common set of cases for model validation. The performance of the model developed in this study showed that the task remains challenging; its performance could serve as a baseline for future model development.

Do We Expect More from Radiology AI than from Radiologists?

The expectations of radiology artificial intelligence do not match expectations of radiologists in terms of performance and explainability.

A generative adversarial network-based abnormality detection using only normal images for model training with application to digital breast tomosynthesis

Deep learning has shown tremendous potential in the task of object detection in images. However, a common challenge with this task is when only a limited number of images containing the object of interest are available. This is a particular issue in cancer screening, such as digital breast tomosynthesis (DBT), where less than 1% of cases contain cancer. In this study, we propose a method to train an inpainting generative adversarial network to be used for cancer detection using only images that do not contain cancer. During inference, we removed a part of the image and used the network to complete the removed part. A significant error in completing an image part was considered an indication that such location is unexpected and thus abnormal. A large dataset of DBT images used in this study was collected at Duke University. It consisted of 19,230 reconstructed volumes from 4348 patients. Cancerous masses and architectural distortions were marked with bounding boxes by radiologists. Our experiments showed that the locations containing cancer were associated with a notably higher completion error than the non-cancer locations (mean error ratio of 2.77). All data used in this study has been made publicly available by the authors.

Deep learning-based algorithm for assessment of knee osteoarthritis severity in radiographs matches performance of radiologists

A fully-automated deep learning algorithm matched performance of radiologists in assessment of knee osteoarthritis severity in radiographs using the Kellgren-Lawrence grading system.

PURPOSE

To develop an automated deep learning-based algorithm that jointly uses Posterior-Anterior (PA) and Lateral (LAT) views of knee radiographs to assess knee osteoarthritis severity according to the Kellgren-Lawrence grading system.

MATERIALS AND METHODS

We used a dataset of 9739 exams from 2802 patients from Multicenter Osteoarthritis Study (MOST). The dataset was divided into a training set of 2040 patients, a validation set of 259 patients and a test set of 503 patients. A novel deep learning-based method was utilized for assessment of knee OA in two steps: (1) localization of knee joints in the images, (2) classification according to the KL grading system. Our method used both PA and LAT views as the input to the model. The scores generated by the algorithm were compared to the grades provided in the MOST dataset for the entire test set as well as grades provided by 5 radiologists at our institution for a subset of the test set.

RESULTS

The model obtained a multi-class accuracy of 71.90% on the entire test set when compared to the ratings provided in the MOST dataset. The quadratic weighted Kappa coefficient for this set was 0.9066. The average quadratic weighted Kappa between all pairs of radiologists from our institution who took part in the study was 0.748. The average quadratic-weighted Kappa between the algorithm and the radiologists at our institution was 0.769.

CONCLUSION

The proposed model performed demonstrated equivalency of KL classification to MSK radiologists, but clearly superior reproducibility. Our model also agreed with radiologists at our institution to the same extent as the radiologists with each other. The algorithm could be used to provide reproducible assessment of knee osteoarthritis severity.

Machine-learning-based multiple abnormality prediction with large-scale chest computed tomography volumes

Machine learning models for radiology benefit from large-scale data sets with high quality labels for abnormalities. We curated and analyzed a chest computed tomography (CT) data set of 36,316 volumes from 19,993 unique patients. This is the largest multiply-annotated volumetric medical imaging data set reported. To annotate this data set, we developed a rule-based method for automatically extracting abnormality labels from free-text radiology reports with an average F-score of 0.976 (min 0.941, max 1.0). We also developed a model for multi-organ, multi-disease classification of chest CT volumes that uses a deep convolutional neural network (CNN). This model reached a classification performance of AUROC >0.90 for 18 abnormalities, with an average AUROC of 0.773 for all 83 abnormalities, demonstrating the feasibility of learning from unfiltered whole volume CT data. We show that training on more labels improves performance significantly: for a subset of 9 labels - nodule, opacity, atelectasis, pleural effusion, consolidation, mass, pericardial effusion, cardiomegaly, and pneumothorax - the model's average AUROC increased by 10% when the number of training labels was increased from 9 to all 83. All code for volume preprocessing, automated label extraction, and the volume abnormality prediction model is publicly available. The 36,316 CT volumes and labels will also be made publicly available pending institutional approval.

Using the American College of Radiology Thyroid Imaging Reporting and Data System at the Point of Care: Sonographer Performance and Interobserver Variability

The purpose of this study was to assess inter-observer variability and performance when sonographers assign features to thyroid nodules on ultrasound using the American College of Radiology (ACR) Thyroid Imaging Reporting and Data System (TI-RADS). Fifteen sonographers retrospectively evaluated 100 thyroid nodules and assigned features to each nodule according to ACR TI-RADS lexicon. Ratings were compared with one another and to a gold standard using Fleiss' and Cohen's kappa statistics, respectively. Sonographers were also asked subjective questions regarding their comfort level assessing each feature, and opinions were compared with performance using a mixed effects model. Sonographers demonstrated only slight agreement for margin (κ = 0.18, 95% confidence interval [CI]: 0.16-0.20) and large comet tail artifact (κ = 0.08, 95% CI: 0.06-0.10) but better performance for macrocalcification (κ = 0.41, 95% CI: 0.39-0.43) and no echogenic foci (κ = 0.52, 95% CI: 0.50-0.54). Sonographer comfort level with different feature assignments did not statistically correlate with performance for a given feature. In conclusion, sonographers using ACR TI-RADS to assign thyroid nodule features on ultrasound demonstrate a range of agreement across features, with margin and large comet tail artifact showing the most variability. These results highlight potential areas of focus for sonographer education efforts as ACR TI-RADS continues to be implemented in radiology departments.

Deep Learning-Based Segmentation of Nodules in Thyroid Ultrasound: Improving Performance by Utilizing Markers Present in the Images

Computer-aided segmentation of thyroid nodules in ultrasound imaging could assist in their accurate characterization. In this study, using data for 1278 nodules, we proposed and evaluated two methods for deep learning-based segmentation of thyroid nodules that utilize calipers present in the images. The first method used approximate nodule masks generated based on the calipers. The second method combined manual annotations with automatic guidance by the calipers. When only approximate nodule masks were used for training, the achieved Dice similarity coefficient (DSC) was 85.1%. The performance of a network trained using manual annotations was DSC = 90.4%. When the guidance by the calipers was added, the performance increased to DSC = 93.1%. An increase in the number of cases used for training resulted in increased performance for all methods. The proposed method utilizing the guidance by calipers matched the performance of the network that did not use it with a reduced number of manually annotated training cases.

Breast Cancer Radiogenomics: Current Status and Future Directions

Radiogenomics is an area of research that aims to identify associations between imaging phenotypes ("radio-") and tumor genome ("-genomics"). Breast cancer radiogenomics research in particular has been an especially prolific area of investigation in recent years as evidenced by the wide number and variety of publications and conferences presentations. To date, research has primarily been focused on dynamic contrast enhanced pre-operative breast MRI and breast cancer molecular subtypes, but investigations have extended to all breast imaging modalities as well as multiple additional genetic markers including those that are commercially available. Furthermore, both human and computer-extracted features as well as deep learning techniques have been explored. This review will summarize the specific imaging modalities used in radiogenomics analysis, describe the methods of extracting imaging features, and present the types of genomics, molecular, and related information used for analysis. Finally, the limitations and future directions of breast cancer radiogenomics research will be discussed.

Deep Radiogenomics of Lower-Grade Gliomas: Convolutional Neural Networks Predict Tumor Genomic Subtypes Using MR Images

PURPOSE

To employ deep learning to predict genomic subtypes of lower-grade glioma (LLG) tumors based on their appearance at MRI.

MATERIALS AND METHODS

Imaging data from The Cancer Imaging Archive and genomic data from The Cancer Genome Atlas from 110 patients from five institutions with lower-grade gliomas (World Health Organization grade II and III) were used in this study. A convolutional neural network was trained to predict tumor genomic subtype based on the MRI of the tumor. Two different deep learning approaches were tested: training from random initialization and transfer learning. Deep learning models were pretrained on glioblastoma MRI, instead of natural images, to determine if performance was improved for the detection of LGGs. The models were evaluated using area under the receiver operating characteristic curve (AUC) with cross-validation. Imaging data and annotations used in this study are publicly available.

RESULTS

The best performing model was based on transfer learning from glioblastoma MRI. It achieved AUC of 0.730 (95% confidence interval [CI]: 0.605, 0.844) for discriminating cluster-of-clusters 2 from others. For the same task, a network trained from scratch achieved an AUC of 0.680 (95% CI: 0.538, 0.811), whereas a model pretrained on natural images achieved an AUC of 0.640 (95% CI: 0.521, 0.763).

CONCLUSION

These findings show the potential of utilizing deep learning to identify relationships between cancer imaging and cancer genomics in LGGs. However, more accurate models are needed to justify clinical use of such tools, which might be obtained using substantially larger training datasets.Supplemental material is available for this article.© RSNA, 2020.

Artificial Intelligence in Radiology: Some Ethical Considerations for Radiologists and Algorithm Developers

As artificial intelligence (AI) is finding its place in radiology, it is important to consider how to guide the research and clinical implementation in a way that will be most beneficial to patients. Although there are multiple aspects of this issue, I consider a specific one: a potential misalignment of the self-interests of radiologists and AI developers with the best interests of the patients. Radiologists know that supporting research into AI and advocating for its adoption in clinical settings could diminish their employment opportunities and reduce respect for their profession. This provides an incentive to oppose AI in various ways. AI developers have an incentive to hype their discoveries to gain attention. This could provide short-term personal gains, however, it could also create a distrust toward the field if it became apparent that the state of the art was far from where it was promised to be. The future research and clinical implementation of AI in radiology will be partially determined by radiologist and AI researchers. Therefore, it is very important that we recognize our own personal motivations and biases and act responsibly to ensure the highest benefit of the AI transformation to the patients.

Prediction of Upstaged Ductal Carcinoma In Situ Using Forced Labeling and Domain Adaptation

OBJECTIVE

The goal of this study is to use adjunctive classes to improve a predictive model whose performance is limited by the common problems of small numbers of primary cases, high feature dimensionality, and poor class separability. Specifically, our clinical task is to use mammographic features to predict whether ductal carcinoma in situ (DCIS) identified at needle core biopsy will be later upstaged or shown to contain invasive breast cancer.

METHODS

To improve the prediction of pure DCIS (negative) versus upstaged DCIS (positive) cases, this study considers the adjunctive roles of two related classes: atypical ductal hyperplasia (ADH), a non-cancer type of breast abnormity, and invasive ductal carcinoma (IDC), with 113 computer vision based mammographic features extracted from each case. To improve the baseline Model A's classification of pure vs. upstaged DCIS, we designed three different strategies (Models B, C, D) with different ways of embedding features or inputs.

RESULTS

Based on ROC analysis, the baseline Model A performed with AUC of 0.614 (95% CI, 0.496-0.733). All three new models performed better than the baseline, with domain adaptation (Model D) performing the best with an AUC of 0.697 (95% CI, 0.595-0.797).

CONCLUSION

We improved the prediction performance of DCIS upstaging by embedding two related pathology classes in different training phases.

SIGNIFICANCE

The three new strategies of embedding related class data all outperformed the baseline model, thus demonstrating not only feature similarities among these different classes, but also the potential for improving classification by using other related classes.

Management of Thyroid Nodules Seen on US Images: Deep Learning May Match Performance of Radiologists

BackgroundManagement of thyroid nodules may be inconsistent between different observers and time consuming for radiologists. An artificial intelligence system that uses deep learning may improve radiology workflow for management of thyroid nodules.PurposeTo develop a deep learning algorithm that uses thyroid US images to decide whether a thyroid nodule should undergo a biopsy and to compare the performance of the algorithm with the performance of radiologists who adhere to American College of Radiology (ACR) Thyroid Imaging Reporting and Data System (TI-RADS).Materials and MethodsIn this retrospective analysis, studies in patients referred for US with subsequent fine-needle aspiration or with surgical histologic analysis used as the standard were evaluated. The study period was from August 2006 to May 2010. A multitask deep convolutional neural network was trained to provide biopsy recommendations for thyroid nodules on the basis of two orthogonal US images as the input. In the training phase, the deep learning algorithm was first evaluated by using 10-fold cross-validation. Internal validation was then performed on an independent set of 99 consecutive nodules. The sensitivity and specificity of the algorithm were compared with a consensus of three ACR TI-RADS committee experts and nine other radiologists, all of whom interpreted thyroid US images in clinical practice.ResultsIncluded were 1377 thyroid nodules in 1230 patients with complete imaging data and conclusive cytologic or histologic diagnoses. For the 99 test nodules, the proposed deep learning algorithm achieved 13 of 15 (87%: 95% confidence interval [CI]: 67%, 100%) sensitivity, the same as expert consensus (P > .99) and higher than five of nine radiologists. The specificity of the deep learning algorithm was 44 of 84 (52%; 95% CI: 42%, 62%), which was similar to expert consensus (43 of 84; 51%; 95% CI: 41%, 62%; P = .91) and higher than seven of nine other radiologists. The mean sensitivity and specificity for the nine radiologists was 83% (95% CI: 64%, 98%) and 48% (95% CI: 37%, 59%), respectively.ConclusionSensitivity and specificity of a deep learning algorithm for thyroid nodule biopsy recommendations was similar to that of expert radiologists who used American College of Radiology Thyroid Imaging and Reporting Data System guidelines.© RSNA, 2019Online supplemental material is available for this article.

Using Artificial Intelligence to Revise ACR TI-RADS Risk Stratification of Thyroid Nodules: Diagnostic Accuracy and Utility

Background Risk stratification systems for thyroid nodules are often complicated and affected by low specificity. Continual improvement of these systems is necessary to reduce the number of unnecessary thyroid biopsies. Purpose To use artificial intelligence (AI) to optimize the American College of Radiology (ACR) Thyroid Imaging Reporting and Data System (TI-RADS). Materials and Methods A total of 1425 biopsy-proven thyroid nodules from 1264 consecutive patients (1026 women; mean age, 52.9 years [range, 18-93 years]) were evaluated retrospectively. Expert readers assigned points based on five ACR TI-RADS categories (composition, echogenicity, shape, margin, echogenic foci), and a genetic AI algorithm was applied to a training set (1325 nodules). Point and pathologic data were used to create an optimized scoring system (hereafter, AI TI-RADS). Performance of the systems was compared by using a test set of the final 100 nodules with interpretations from the expert reader, eight nonexpert readers, and an expert panel. Initial performance of AI TI-RADS was calculated by using a test for differences between binomial proportions. Additional comparisons across readers were conducted by using bootstrapping; diagnostic performance was assessed by using area under the receiver operating curve. Results AI TI-RADS assigned new point values for eight ACR TI-RADS features. Six features were assigned zero points, which simplified categorization. By using expert reader data, the diagnostic performance of ACR TI-RADS and AI TI-RADS was area under the receiver operating curve of 0.91 and 0.93, respectively. For the same expert, specificity of AI TI-RADS (65%, 55 of 85) was higher (P < .001) than that of ACR TI-RADS (47%, 40 of 85). For the eight nonexpert radiologists, mean specificity for AI TI-RADS (55%) was also higher (P < .001) than that of ACR TI-RADS (48%). An interactive AI TI-RADS calculator can be viewed at http://deckard.duhs.duke.edu/∼ai-ti-rads . Conclusion An artificial intelligence-optimized Thyroid Imaging Reporting and Data System (TI-RADS) validates the American College of Radiology TI-RADS while slightly improving specificity and maintaining sensitivity. Additionally, it simplifies feature assignments, which may improve ease of use. © RSNA, 2019 Online supplemental material is available for this article.

Association of genomic subtypes of lower-grade gliomas with shape features automatically extracted by a deep learning algorithm

Recent analysis identified distinct genomic subtypes of lower-grade glioma tumors which are associated with shape features. In this study, we propose a fully automatic way to quantify tumor imaging characteristics using deep learning-based segmentation and test whether these characteristics are predictive of tumor genomic subtypes. We used preoperative imaging and genomic data of 110 patients from 5 institutions with lower-grade gliomas from The Cancer Genome Atlas. Based on automatic deep learning segmentations, we extracted three features which quantify two-dimensional and three-dimensional characteristics of the tumors. Genomic data for the analyzed cohort of patients consisted of previously identified genomic clusters based on IDH mutation and 1p/19q co-deletion, DNA methylation, gene expression, DNA copy number, and microRNA expression. To analyze the relationship between the imaging features and genomic clusters, we conducted the Fisher exact test for 10 hypotheses for each pair of imaging feature and genomic subtype. To account for multiple hypothesis testing, we applied a Bonferroni correction. P-values lower than 0.005 were considered statistically significant. We found the strongest association between RNASeq clusters and the bounding ellipsoid volume ratio (p < 0.0002) and between RNASeq clusters and margin fluctuation (p < 0.005). In addition, we identified associations between bounding ellipsoid volume ratio and all tested molecular subtypes (p < 0.02) as well as between angular standard deviation and RNASeq cluster (p < 0.02). In terms of automatic tumor segmentation that was used to generate the quantitative image characteristics, our deep learning algorithm achieved a mean Dice coefficient of 82% which is comparable to human performance.

Deep learning for identifying radiogenomic associations in breast cancer

RATIONALE AND OBJECTIVES

To determine whether deep learning models can distinguish between breast cancer molecular subtypes based on dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI).

MATERIALS AND METHODS

In this institutional review board-approved single-center study, we analyzed DCE-MR images of 270 patients at our institution. Lesions of interest were identified by radiologists. The task was to automatically determine whether the tumor is of the Luminal A subtype or of another subtype based on the MR image patches representing the tumor. Three different deep learning approaches were used to classify the tumor according to their molecular subtypes: learning from scratch where only tumor patches were used for training, transfer learning where networks pre-trained on natural images were fine-tuned using tumor patches, and off-the-shelf deep features where the features extracted by neural networks trained on natural images were used for classification with a support vector machine. Network architectures utilized in our experiments were GoogleNet, VGG, and CIFAR. We used 10-fold crossvalidation method for validation and area under the receiver operating characteristic (AUC) as the measure of performance.

RESULTS

The best AUC performance for distinguishing molecular subtypes was 0.65 (95% CI:[0.57,0.71]) and was achieved by the off-the-shelf deep features approach. The highest AUC performance for training from scratch was 0.58 (95% CI:[0.51,0.64]) and the best AUC performance for transfer learning was 0.60 (95% CI:[0.52,0.65]) respectively. For the off-the-shelf approach, the features extracted from the fully connected layer performed the best.

CONCLUSION

Deep learning may play a role in discovering radiogenomic associations in breast cancer.

Deep learning in radiology: An overview of the concepts and a survey of the state of the art with focus on MRI

Deep learning is a branch of artificial intelligence where networks of simple interconnected units are used to extract patterns from data in order to solve complex problems. Deep-learning algorithms have shown groundbreaking performance in a variety of sophisticated tasks, especially those related to images. They have often matched or exceeded human performance. Since the medical field of radiology mainly relies on extracting useful information from images, it is a very natural application area for deep learning, and research in this area has rapidly grown in recent years. In this article, we discuss the general context of radiology and opportunities for application of deep-learning algorithms. We also introduce basic concepts of deep learning, including convolutional neural networks. Then, we present a survey of the research in deep learning applied to radiology. We organize the studies by the types of specific tasks that they attempt to solve and review a broad range of deep-learning algorithms being utilized. Finally, we briefly discuss opportunities and challenges for incorporating deep learning in the radiology practice of the future. Level of Evidence: 3 Technical Efficacy: Stage 1 J. Magn. Reson. Imaging 2019;49:939-954.

Hierarchical Convolutional Neural Networks for Segmentation of Breast Tumors in MRI With Application to Radiogenomics

Breast tumor segmentation based on dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) is a challenging problem and an active area of research. Particular challenges, similarly as in other segmentation problems, include the class-imbalance problem as well as confounding background in DCE-MR images. To address these issues, we propose a mask-guided hierarchical learning (MHL) framework for breast tumor segmentation via fully convolutional networks (FCN). Specifically, we first develop an FCN model to generate a 3D breast mask as the region of interest (ROI) for each image, to remove confounding information from input DCE-MR images. We then design a two-stage FCN model to perform coarse-to-fine segmentation for breast tumors. Particularly, we propose a Dice-Sensitivity-like loss function and a reinforcement sampling strategy to handle the class-imbalance problem. To precisely identify locations of tumors that underwent a biopsy, we further propose an FCN model to detect two landmarks located at two nipples. We finally selected the biopsied tumor based on both identified landmarks and segmentations. We validate our MHL method on 272 patients, achieving a mean Dice similarity coefficient (DSC) of 0.72 which is comparable to mutual DSC between expert radiologists. Using the segmented biopsied tumors, we also demonstrate that the automatically generated masks can be applied to radiogenomics and can identify luminal A subtype from other molecular subtypes with the similar accuracy with the analysis based on semi-manual tumor segmentation.

Association of distant recurrence-free survival with algorithmically extracted MRI characteristics in breast cancer

BACKGROUND

While important in diagnosis of breast cancer, the scientific assessment of the role of imaging in prognosis of outcomes and treatment planning is limited.

PURPOSE

To evaluate the potential of using quantitative imaging variables for stratifying risk of distant recurrence in breast cancer patients.

STUDY TYPE

Retrospective.

POPULATION

In all, 892 female invasive breast cancer patients.

SEQUENCE

Dynamic contrast-enhanced MRI with field strength 1.5 T and 3 T.

ASSESSMENT

Computer vision algorithms were applied to extract a comprehensive set of 529 imaging features quantifying size, shape, enhancement patterns, and heterogeneity of the tumors and the surrounding tissue. Using a development set with 446 cases, we selected 20 imaging features with high prognostic value.

STATISTICAL TESTS

We evaluated the imaging features using an independent test set with 446 cases. The principal statistical measure was a concordance index between individual imaging features and patient distant recurrence-free survival (DRFS).

RESULTS

The strongest association with DRFS that persisted after controlling for known prognostic clinical and pathology variables was found for signal enhancement ratio (SER) partial tumor volume (concordance index [C] = 0.768, 95% confidence interval [CI]: 0.679-0.856), tumor major axis length (C = 0.742, 95% CI: 0.650-0.834), kurtosis of the SER map within tumor (C = 0.640, 95% CI: 0.521-0.760), tumor cluster shade (C = 0.313, 95% CI: 0.216-0.410), and washin rate information measure of correlation (C = 0.702, 95% CI: 0.601-0.803).

DATA CONCLUSION

Quantitative assessment of breast cancer features seen in a routine breast MRI might be able to be used for assessment of risk of distant recurrence.

LEVEL OF EVIDENCE

4 Technical Efficacy: Stage 6 J. Magn. Reson. Imaging 2019.

Machine learning-based prediction of future breast cancer using algorithmically measured background parenchymal enhancement on high-risk screening MRI

BACKGROUND

Preliminary work has demonstrated that background parenchymal enhancement (BPE) assessed by radiologists is predictive of future breast cancer in women undergoing high-risk screening MRI. Algorithmically assessed measures of BPE offer a more precise and reproducible means of measuring BPE than human readers and thus might improve the predictive performance of future cancer development.

PURPOSE

To determine if algorithmically extracted imaging features of BPE on screening breast MRI in high-risk women are associated with subsequent development of cancer.

STUDY TYPE

Case-control study.

POPULATION

In all, 133 women at high risk for developing breast cancer; 46 of these patients developed breast cancer subsequently over a follow-up period of 2 years.

FIELD STRENGTH/SEQUENCE

5 T or 3.0 T T₁ -weighted precontrast fat-saturated and nonfat-saturated sequences and postcontrast nonfat-saturated sequences.

ASSESSMENT

Automatic features of BPE were extracted with a computer algorithm. Subjective BPE scores from five breast radiologists (blinded to clinical outcomes) were also available.

STATISTICAL TESTS

Leave-one-out crossvalidation for a multivariate logistic regression model developed using the automatic features and receiver operating characteristic (ROC) analysis were performed to calculate the area under the curve (AUC). Comparison of automatic features and subjective features was performed using a generalized regression model and the P-value was obtained. Odds ratios for automatic and subjective features were compared.

RESULTS

The multivariate model discriminated patients who developed cancer from the patients who did not, with an AUC of 0.70 (95% confidence interval: 0.60-0.79, P < 0.001). The imaging features remained independently predictive of subsequent development of cancer (P < 0.003) when compared with the subjective BPE assessment of the readers.

DATA CONCLUSION

Automatically extracted BPE measurements may potentially be used to further stratify risk in patients undergoing high-risk screening MRI.

LEVEL OF EVIDENCE

3 Technical Efficacy: Stage 5 J. Magn. Reson. Imaging 2019;50:456-464.

Relationship between Background Parenchymal Enhancement on High-risk Screening MRI and Future Breast Cancer Risk

RATIONALE AND OBJECTIVES

To determine if background parenchymal enhancement (BPE) on screening breast magnetic resonance imaging (MRI) in high-risk women correlates with future cancer.

MATERIALS AND METHODS

All screening breast MRIs (n = 1039) in high-risk women at our institution from August 1, 2004, to July 30, 2013, were identified. Sixty-one patients who subsequently developed breast cancer were matched 1:2 by age and high-risk indication with patients who did not develop breast cancer (n = 122). Five fellowship-trained breast radiologists independently recorded the BPE. The median reader BPE for each case was calculated and compared between the cancer and control cohorts.

RESULTS

Cancer cohort patients were high-risk because of a history of radiation therapy (10%, 6 of 61), high-risk lesion (18%, 11 of 61), or breast cancer (30%, 18 of 61); BRCA mutation (18%, 11 of 61); or family history (25%, 15 of 61). Subsequent malignancies were invasive ductal carcinoma (64%, 39 of 61), ductal carcinoma in situ (30%, 18 of 61) and invasive lobular carcinoma (7%, 4of 61). BPE was significantly higher in the cancer cohort than in the control cohort (P = 0.01). Women with mild, moderate, or marked BPE were 2.5 times more likely to develop breast cancer than women with minimal BPE (odds ratio = 2.5, 95% confidence interval: 1.3-4.8, P = .005). There was fair interreader agreement (κ = 0.39).

CONCLUSIONS

High-risk women with greater than minimal BPE at screening MRI have increased risk of future breast cancer.

Intra-tumor molecular heterogeneity in breast cancer: definitions of measures and association with distant recurrence-free survival

PURPOSE

The purpose of the study was to define quantitative measures of intra-tumor heterogeneity in breast cancer based on histopathology data gathered from multiple samples on individual patients and determine their association with distant recurrence-free survival (DRFS).

METHODS

We collected data from 971 invasive breast cancers, from 1st January 2000 to 23rd March 2014, that underwent repeat tumor sampling at our institution. We defined and calculated 31 measures of intra-tumor heterogeneity including ER, PR, and HER2 immunohistochemistry (IHC), proliferation, EGFR IHC, grade, and histology. For each heterogeneity measure, Cox proportional hazards models were used to determine whether patients with heterogeneous disease had different distant recurrence-free survival (DRFS) than those with homogeneous disease.

RESULTS

The presence of heterogeneity in ER percentage staining was prognostic of reduced DRFS with a hazard ratio of 4.26 (95% CI 2.22-8.18, p < 0.00002). It remained significant after controlling for the ER status itself (p < 0.00062) and for patients that had chemotherapy (p < 0.00032). Most of the heterogeneity measures did not show any association with DRFS despite the considerable sample size.

CONCLUSIONS

Intra-tumor heterogeneity of ER receptor status may be a predictor of patient DRFS. Histopathologic data from multiple tissue samples may offer a view of tumor heterogeneity and assess recurrence risk.

Multivariate machine learning models for prediction of pathologic response to neoadjuvant therapy in breast cancer using MRI features: a study using an independent validation set

PURPOSE

To determine whether a multivariate machine learning-based model using computer-extracted features of pre-treatment dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) can predict pathologic complete response (pCR) to neoadjuvant therapy (NAT) in breast cancer patients.

METHODS

Institutional review board approval was obtained for this retrospective study of 288 breast cancer patients at our institution who received NAT and had a pre-treatment breast MRI. A comprehensive set of 529 radiomic features was extracted from each patient's pre-treatment MRI. The patients were divided into equal groups to form a training set and an independent test set. Two multivariate machine learning models (logistic regression and a support vector machine) based on imaging features were trained to predict pCR in (a) all patients with NAT, (b) patients with neoadjuvant chemotherapy (NACT), and (c) triple-negative or human epidermal growth factor receptor 2-positive (TN/HER2+) patients who had NAT. The multivariate models were tested using the independent test set, and the area under the receiver operating characteristics (ROC) curve (AUC) was calculated.

RESULTS

Out of the 288 patients, 64 achieved pCR. The AUC values for predicting pCR in TN/HER+ patients who received NAT were significant (0.707, 95% CI 0.582-0.833, p < 0.002).

CONCLUSIONS

The multivariate models based on pre-treatment MRI features were able to predict pCR in TN/HER2+ patients.

Abdominal Radiography With Digital Tomosynthesis: An Alternative to Computed Tomography for Identification of Urinary Calculi?

OBJECTIVE

To compare the accuracy of plain abdominal radiography (kidneys, ureter, and bladder [KUB]) with digital tomosynthesis (DT) to noncontrast computed tomography (NCCT), the gold standard imaging modality for urinary stones. Due to radiation and cost concerns, KUB is often used for diagnosis and follow-up of nephrolithiasis. DT, a novel technique that produces high-quality radiographs with less radiation and/or cost than low-dose NCCT, has not been assessed in this situation.

MATERIALS AND METHODS

Seven fresh tissue cadavers were implanted with stones of known size and/or composition and imaged with KUB, DT, and NCCT. Four blinded readers (2 urologists, 2 radiologists) evaluated KUBs for presence and/or location of calculi. They then re-evaluated with addition of tomograms to assess additional value. After a memory extinction period, readers evaluated NCCT images. Accuracy of detection was determined using nearest-neighbor match with generalized linear mixed modeling.

RESULTS

Total of 59 stones were identified on reference read. Overall, NCCT and DT were both superior to KUB alone (P < .001) while the difference between DT and NCCT was not significant (P = .06). When evaluating uric acid stones, NCCT and DT outperformed KUB (P < .01 and P < .05, respectively) while DT and NCCT were similar (P = .16). Intrarenal stones were better evaluated on DT and NCCT (P < .001 compared to KUB), while DT and NCCT were similar (P = 1.00). Accuracy was lower than anticipated across modalities due to use of the cadaver model.

CONCLUSION

Our study demonstrates DT is superior to KUB for identification of intrarenal calculi and could replace routine use of KUB or NCCT for detecting renal stones, even those composed of uric acid.

Breast cancer MRI radiomics: An overview of algorithmic features and impact of inter-reader variability in annotating tumors

PURPOSE

To review features used in MRI radiomics of breast cancer and study the inter-reader stability of the features.

METHODS

We implemented 529 algorithmic features that can be extracted from tumor and fibroglandular tissue (FGT) in breast MRIs. The features were identified based on a review of the existing literature with consideration of their usage, prognostic ability, and uniqueness. The set was then extended so that it comprehensively describes breast cancer imaging characteristics. The features were classified into 10 groups based on the type of data used to extract them and the type of calculation being performed. For the assessment of inter-reader variability, four fellowship-trained readers annotated tumors on preoperative dynamic contrast-enhanced MRIs for 50 breast cancer patients. Based on the annotations, an algorithm automatically segmented the image and extracted all features resulting in one set of features for each reader. For a given feature, the inter-reader stability was defined as the intraclass correlation coefficient (ICC) computed using the feature values obtained through all readers for all cases.

RESULTS

The average inter-reader stability for all features was 0.8474 (95% CI: 0.8068-0.8858). The mean inter-reader stability was lower for tumor-based features (0.6348, 95% CI: 0.5391-0.7257) than FGT-based features (0.9984, 95% CI: 0.9970-0.9992). The feature group with the highest inter-reader stability quantifies breast and FGT volume. The feature group with the lowest inter-reader stability quantifies variations in tumor enhancement.

CONCLUSIONS

Breast MRI radiomics features widely vary in terms of their stability in the presence of inter-reader variability. Appropriate measures need to be taken for reducing this variability in tumor-based radiomics.

A study of association of Oncotype DX recurrence score with DCE-MRI characteristics using multivariate machine learning models

PURPOSE

To determine whether multivariate machine learning models of algorithmically assessed magnetic resonance imaging (MRI) features from breast cancer patients are associated with Oncotype DX (ODX) test recurrence scores.

METHODS

A set of 261 female patients with invasive breast cancer, pre-operative dynamic contrast enhanced magnetic resonance (DCE-MR) images and available ODX score at our institution was identified. A computer algorithm extracted a comprehensive set of 529 features from the DCE-MR images of these patients. The set of patients was divided into a training set and a test set. Using the training set we developed two machine learning-based models to discriminate (1) high ODX scores from intermediate and low ODX scores, and (2) high and intermediate ODX scores from low ODX scores. The performance of these models was evaluated on the independent test set.

RESULTS

High against low and intermediate ODX scores were predicted by the multivariate model with AUC 0.77 (95% CI 0.56-0.98, p < 0.003). Low against intermediate and high ODX score was predicted with AUC 0.51 (95% CI 0.41-0.61, p = 0.75).

CONCLUSION

A moderate association between imaging and ODX score was identified. The evaluated models currently do not warrant replacement of ODX with imaging alone.

Prediction of Occult Invasive Disease in Ductal Carcinoma in Situ Using Deep Learning Features

PURPOSE

The aim of this study was to determine whether deep features extracted from digital mammograms using a pretrained deep convolutional neural network are prognostic of occult invasive disease for patients with ductal carcinoma in situ (DCIS) on core needle biopsy.

METHODS

In this retrospective study, digital mammographic magnification views were collected for 99 subjects with DCIS at biopsy, 25 of which were subsequently upstaged to invasive cancer. A deep convolutional neural network model that was pretrained on nonmedical images (eg, animals, plants, instruments) was used as the feature extractor. Through a statistical pooling strategy, deep features were extracted at different levels of convolutional layers from the lesion areas, without sacrificing the original resolution or distorting the underlying topology. A multivariate classifier was then trained to predict which tumors contain occult invasive disease. This was compared with the performance of traditional "handcrafted" computer vision (CV) features previously developed specifically to assess mammographic calcifications. The generalization performance was assessed using Monte Carlo cross-validation and receiver operating characteristic curve analysis.

RESULTS

Deep features were able to distinguish DCIS with occult invasion from pure DCIS, with an area under the receiver operating characteristic curve of 0.70 (95% confidence interval, 0.68-0.73). This performance was comparable with the handcrafted CV features (area under the curve = 0.68; 95% confidence interval, 0.66-0.71) that were designed with prior domain knowledge.

CONCLUSIONS

Despite being pretrained on only nonmedical images, the deep features extracted from digital mammograms demonstrated comparable performance with handcrafted CV features for the challenging task of predicting DCIS upstaging.

Abstract GS5-04: Prediction of occult invasive disease in ductal carcinoma in situ using deep learning features

Background: Deep learning, especially deep convolutional neural network (CNN), has emerged as a promising approach for many image recognition or classification tasks, demonstrating human or even superhuman performance. Used as feature extractor, some pre-trained CNN models can match or surpass the performance of domain-specific, “handcrafted” features. In this study, we aim to determine whether deep features extracted from digital mammograms using a pre-trained deep CNN are prognostic of occult invasive disease for patients with ductal carcinoma in situ (DCIS) on core needle biopsy.

Materials and Methods: In this retrospective study, we collected digital mammography magnification views for 99 subjects with DCIS at biopsy, 25 of which were subsequently upstaged to invasive cancer. We utilized a deep CNN model that was pre-trained on non-medical images (e.g., animals, plants, instruments) as the feature extractor. Through a statistical pooling strategy, we extracted deep features at different levels of convolutional layers from the lesion areas, without sacrificing the original resolution or distorting the underlying topology. A multivariate classifier was then trained to predict which tumors contain occult invasive disease. This was compared to the performance of traditional “handcrafted” computer vision (CV) features previously developed specifically to assess mammographic calcifications. The generalization performance was assessed using Monte Carlo cross validation and receiver operating characteristic (ROC) curve analysis.

Results: Deep features were able to distinguish DCIS with occult invasion from pure DCIS, with an area under the ROC curve (AUC-ROC) equal to 0.70 (95% CI: 0.68-0.73). This performance was comparable to the "handcrafted" CV features (AUC-ROC = 0.68, 95% CI: 0.66-0.71) that were designed with prior domain knowledge.

Conclusion: In spite of being pre-trained on only non-medical images, the deep features extracted from digital mammograms demonstrated comparable performance to "handcrafted" CV features for the challenging task of predicting DCIS upstaging.

Acknowledgments: This work was supported in part by NIH/NCI R01-CQA185138 and DOD Breast Cancer Research Program W81XWH-14-1-0473.

Citation Format: Lo JY, Grimm LJ, Mazurowski MA, Baker JA, Marks JR, King LM, Maley CC, Hwang E-SS, Shi B. Prediction of occult invasive disease in ductal carcinoma in situ using deep learning features [abstract]. In: Proceedings of the 2017 San Antonio Breast Cancer Symposium; 2017 Dec 5-9; San Antonio, TX. Philadelphia (PA): AACR; Cancer Res 2018;78(4 Suppl):Abstract nr GS5-04.

Deep learning for segmentation of brain tumors: Impact of cross-institutional training and testing

BACKGROUND AND PURPOSE

Convolutional neural networks (CNNs) are commonly used for segmentation of brain tumors. In this work, we assess the effect of cross-institutional training on the performance of CNNs.

METHODS

We selected 44 glioblastoma (GBM) patients from two institutions in The Cancer Imaging Archive dataset. The images were manually annotated by outlining each tumor component to form ground truth. To automatically segment the tumors in each patient, we trained three CNNs: (a) one using data for patients from the same institution as the test data, (b) one using data for the patients from the other institution and (c) one using data for the patients from both of the institutions. The performance of the trained models was evaluated using Dice similarity coefficients as well as Average Hausdorff Distance between the ground truth and automatic segmentations. The 10-fold cross-validation scheme was used to compare the performance of different approaches.

RESULTS

Performance of the model significantly decreased (P < 0.0001) when it was trained on data from a different institution (dice coefficients: 0.68 ± 0.19 and 0.59 ± 0.19) as compared to training with data from the same institution (dice coefficients: 0.72 ± 0.17 and 0.76 ± 0.12). This trend persisted for segmentation of the entire tumor as well as its individual components.

CONCLUSIONS

There is a very strong effect of selecting data for training on performance of CNNs in a multi-institutional setting. Determination of the reasons behind this effect requires additional comprehensive investigation.

Effects of MRI scanner parameters on breast cancer radiomics

PURPOSE

To assess the impact of varying magnetic resonance imaging (MRI) scanner parameters on the extraction of algorithmic features in breast MRI radiomics studies.

METHODS

In this retrospective study, breast imaging data for 272 patients were analyzed with magnetic resonance (MR) images. From the MR images, we assembled and implemented 529 algorithmic features of breast tumors and fibrograndular tissue (FGT). We divided the features into 10 groups based on the type of data used for the feature extraction and the nature of the extracted information. Three scanner parameters were considered: scanner manufacturer, scanner magnetic field strength, and slice thickness. We assessed the impact of each of the scanner parameters on each of the feature by testing whether the feature values are systematically diverse for different values of these scanner parameters. A two-sample t-test has been used to establish whether the impact of a scanner parameter on values of a feature is significant and receiver operating characteristics have been used for to establish the extent of that effect.

RESULTS

On average, higher proportion (69% FGT versus 20% tumor) of FGT related features were affected by the three scanner parameters. Of all feature groups and scanner parameters, the feature group related to the variation in FGT enhancement was found to be the most sensitive to the scanner manufacturer (AUC = 0.81 ± 0.14).

CONCLUSIONS

Features involving calculations from FGT are particularly sensitive to the scanner parameters.