101
|
Fashandi H, Kuling G, Lu Y, Wu H, Martel AL. An investigation of the effect of fat suppression and dimensionality on the accuracy of breast MRI segmentation using U-nets. Med Phys 2019; 46:1230-1244. [PMID: 30609062 DOI: 10.1002/mp.13375] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/29/2018] [Revised: 10/17/2018] [Accepted: 12/11/2018] [Indexed: 01/17/2023] Open
Abstract
PURPOSE Accurate segmentation of the breast is required for breast density estimation and the assessment of background parenchymal enhancement, both of which have been shown to be related to breast cancer risk. The MRI breast segmentation task is challenging, and recent work has demonstrated that convolutional neural networks perform well for this task. In this study, we have investigated the performance of several two-dimensional (2D) U-Net and three-dimensional (3D) U-Net configurations using both fat-suppressed and nonfat-suppressed images. We have also assessed the effect of changing the number and quality of the ground truth segmentations. MATERIALS AND METHODS We designed eight studies to investigate the effect of input types and the dimensionality of the U-Net operations for the breast MRI segmentation. Our training data contained 70 whole breast volumes of T1-weighted sequences without fat suppression (WOFS) and with fat suppression (FS). For each subject, we registered the WOFS and FS volumes together before manually segmenting the breast to generate ground truth. We compared four different input types to the U-nets: WOFS, FS, MIXED (WOFS and FS images treated as separate samples), and MULTI (WOFS and FS images combined into a single multichannel image). We trained 2D U-Nets and 3D U-Nets with these data, which resulted in our eight studies (2D-WOFS, 3D-WOFS, 2D-FS, 3D-FS, 2D-MIXED, 3D-MIXED, 2D-MULTI, and 3D-MULT). For each of these studies, we performed a systematic grid search to tune the hyperparameters of the U-Nets. A separate validation set with 15 whole breast volumes was used for hyperparameter tuning. We performed Kruskal-Walis test on the results of our hyperparameter tuning and did not find a statistically significant difference in the ten top models of each study. For this reason, we chose the best model as the model with the highest mean dice similarity coefficient (DSC) value on the validation set. The reported test results are the results of the top model of each study on our test set which contained 19 whole breast volumes annotated by three readers fused with the STAPLE algorithm. We also investigated the effect of the quality of the training annotations and the number of training samples for this task. RESULTS The study with the highest average DSC result was 3D-MULTI with 0.96 ± 0.02. The second highest average is 2D WOFS (0.96 ± 0.03), and the third is 2D MULTI (0.96 ± 0.03). We performed the Kruskal-Wallis one-way ANOVA test with Dunn's multiple comparison tests using Bonferroni P-value correction on the results of the selected model of each study and found that 3D-MULTI, 2D-MULTI, 3D-WOFS, 2D-WOFS, 2D-FS, and 3D-FS were not statistically different in their distributions, which indicates that comparable results could be obtained in fat-suppressed and nonfat-suppressed volumes and that there is no significant difference between the 3D and 2D approach. Our results also suggested that the networks trained on single sequence images or multiple sequence images organized in multichannel images perform better than the models trained on a mixture of volumes from different sequences. Our investigation of the size of the training set revealed that training a U-Net in this domain only requires a modest amount of training data and results obtained with 49 and 70 training datasets were not significantly different. CONCLUSIONS To summarize, we investigated the use of 2D U-Nets and 3D U-Nets for breast volume segmentation in T1 fat-suppressed and without fat-suppressed volumes. Although our highest score was obtained in the 3D MULTI study, when we took advantage of information in both fat-suppressed and nonfat-suppressed volumes and their 3D structure, all of the methods we explored gave accurate segmentations with an average DSC on >94% demonstrating that the U-Net is a robust segmentation method for breast MRI volumes.
Collapse
Affiliation(s)
- Homa Fashandi
- Physical Sciences, Sunnybrook Research Institute, Toronto, Ontario, M4N 3M5, Canada
| | - Gregory Kuling
- Department of Medical Biophysics, University of Toronto, Toronto, Ontario, M5G 1L7, Canada
| | - Yingli Lu
- Physical Sciences, Sunnybrook Research Institute, Toronto, Ontario, M4N 3M5, Canada
| | - Hongbo Wu
- Department of Medical Biophysics, University of Toronto, Toronto, Ontario, M5G 1L7, Canada
| | - Anne L Martel
- Physical Sciences, Sunnybrook Research Institute, Toronto, Ontario, M4N 3M5, Canada.,Department of Medical Biophysics, University of Toronto, Toronto, Ontario, M5G 1L7, Canada
| |
Collapse
|
102
|
Soffer S, Ben-Cohen A, Shimon O, Amitai MM, Greenspan H, Klang E. Convolutional Neural Networks for Radiologic Images: A Radiologist's Guide. Radiology 2019; 290:590-606. [PMID: 30694159 DOI: 10.1148/radiol.2018180547] [Citation(s) in RCA: 275] [Impact Index Per Article: 55.0] [Reference Citation Analysis] [Abstract] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/06/2023]
Abstract
Deep learning has rapidly advanced in various fields within the past few years and has recently gained particular attention in the radiology community. This article provides an introduction to deep learning technology and presents the stages that are entailed in the design process of deep learning radiology research. In addition, the article details the results of a survey of the application of deep learning-specifically, the application of convolutional neural networks-to radiologic imaging that was focused on the following five major system organs: chest, breast, brain, musculoskeletal system, and abdomen and pelvis. The survey of the studies is followed by a discussion about current challenges and future trends and their potential implications for radiology. This article may be used as a guide for radiologists planning research in the field of radiologic image analysis using convolutional neural networks.
Collapse
Affiliation(s)
- Shelly Soffer
- From the Department of Diagnostic Imaging, Sheba Medical Center, Emek HaEla St 1, Ramat Gan, Israel (S.S., M.M.A., E.K.); Faculty of Engineering, Department of Biomedical Engineering, Medical Image Processing Laboratory, Tel Aviv University, Tel Aviv, Israel (A.B., H.G.); and Sackler School of Medicine, Tel Aviv University, Tel Aviv, Israel (S.S., O.S.)
| | - Avi Ben-Cohen
- From the Department of Diagnostic Imaging, Sheba Medical Center, Emek HaEla St 1, Ramat Gan, Israel (S.S., M.M.A., E.K.); Faculty of Engineering, Department of Biomedical Engineering, Medical Image Processing Laboratory, Tel Aviv University, Tel Aviv, Israel (A.B., H.G.); and Sackler School of Medicine, Tel Aviv University, Tel Aviv, Israel (S.S., O.S.)
| | - Orit Shimon
- From the Department of Diagnostic Imaging, Sheba Medical Center, Emek HaEla St 1, Ramat Gan, Israel (S.S., M.M.A., E.K.); Faculty of Engineering, Department of Biomedical Engineering, Medical Image Processing Laboratory, Tel Aviv University, Tel Aviv, Israel (A.B., H.G.); and Sackler School of Medicine, Tel Aviv University, Tel Aviv, Israel (S.S., O.S.)
| | - Michal Marianne Amitai
- From the Department of Diagnostic Imaging, Sheba Medical Center, Emek HaEla St 1, Ramat Gan, Israel (S.S., M.M.A., E.K.); Faculty of Engineering, Department of Biomedical Engineering, Medical Image Processing Laboratory, Tel Aviv University, Tel Aviv, Israel (A.B., H.G.); and Sackler School of Medicine, Tel Aviv University, Tel Aviv, Israel (S.S., O.S.)
| | - Hayit Greenspan
- From the Department of Diagnostic Imaging, Sheba Medical Center, Emek HaEla St 1, Ramat Gan, Israel (S.S., M.M.A., E.K.); Faculty of Engineering, Department of Biomedical Engineering, Medical Image Processing Laboratory, Tel Aviv University, Tel Aviv, Israel (A.B., H.G.); and Sackler School of Medicine, Tel Aviv University, Tel Aviv, Israel (S.S., O.S.)
| | - Eyal Klang
- From the Department of Diagnostic Imaging, Sheba Medical Center, Emek HaEla St 1, Ramat Gan, Israel (S.S., M.M.A., E.K.); Faculty of Engineering, Department of Biomedical Engineering, Medical Image Processing Laboratory, Tel Aviv University, Tel Aviv, Israel (A.B., H.G.); and Sackler School of Medicine, Tel Aviv University, Tel Aviv, Israel (S.S., O.S.)
| |
Collapse
|
103
|
Bratt A, Kim J, Pollie M, Beecy AN, Tehrani NH, Codella N, Perez-Johnston R, Palumbo MC, Alakbarli J, Colizza W, Drexler IR, Azevedo CF, Kim RJ, Devereux RB, Weinsaft JW. Machine learning derived segmentation of phase velocity encoded cardiovascular magnetic resonance for fully automated aortic flow quantification. J Cardiovasc Magn Reson 2019; 21:1. [PMID: 30612574 PMCID: PMC6322266 DOI: 10.1186/s12968-018-0509-0] [Citation(s) in RCA: 48] [Impact Index Per Article: 9.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2018] [Accepted: 11/18/2018] [Indexed: 02/08/2023] Open
Abstract
BACKGROUND Phase contrast (PC) cardiovascular magnetic resonance (CMR) is widely employed for flow quantification, but analysis typically requires time consuming manual segmentation which can require human correction. Advances in machine learning have markedly improved automated processing, but have yet to be applied to PC-CMR. This study tested a novel machine learning model for fully automated analysis of PC-CMR aortic flow. METHODS A machine learning model was designed to track aortic valve borders based on neural network approaches. The model was trained in a derivation cohort encompassing 150 patients who underwent clinical PC-CMR then compared to manual and commercially-available automated segmentation in a prospective validation cohort. Further validation testing was performed in an external cohort acquired from a different site/CMR vendor. RESULTS Among 190 coronary artery disease patients prospectively undergoing CMR on commercial scanners (84% 1.5T, 16% 3T), machine learning segmentation was uniformly successful, requiring no human intervention: Segmentation time was < 0.01 min/case (1.2 min for entire dataset); manual segmentation required 3.96 ± 0.36 min/case (12.5 h for entire dataset). Correlations between machine learning and manual segmentation-derived flow approached unity (r = 0.99, p < 0.001). Machine learning yielded smaller absolute differences with manual segmentation than did commercial automation (1.85 ± 1.80 vs. 3.33 ± 3.18 mL, p < 0.01): Nearly all (98%) of cases differed by ≤5 mL between machine learning and manual methods. Among patients without advanced mitral regurgitation, machine learning correlated well (r = 0.63, p < 0.001) and yielded small differences with cine-CMR stroke volume (∆ 1.3 ± 17.7 mL, p = 0.36). Among advanced mitral regurgitation patients, machine learning yielded lower stroke volume than did volumetric cine-CMR (∆ 12.6 ± 20.9 mL, p = 0.005), further supporting validity of this method. Among the external validation cohort (n = 80) acquired using a different CMR vendor, the algorithm yielded equivalently small differences (∆ 1.39 ± 1.77 mL, p = 0.4) and high correlations (r = 0.99, p < 0.001) with manual segmentation, including similar results in 20 patients with bicuspid or stenotic aortic valve pathology (∆ 1.71 ± 2.25 mL, p = 0.25). CONCLUSION Fully automated machine learning PC-CMR segmentation performs robustly for aortic flow quantification - yielding rapid segmentation, small differences with manual segmentation, and identification of differential forward/left ventricular volumetric stroke volume in context of concomitant mitral regurgitation. Findings support use of machine learning for analysis of large scale CMR datasets.
Collapse
Affiliation(s)
- Alex Bratt
- Department of Radiology, Weill Cornell Medicine, 525 E 68th St, New York, NY 10065 USA
| | - Jiwon Kim
- Department of Radiology, Weill Cornell Medicine, 525 E 68th St, New York, NY 10065 USA
- Greenberg Division of Cardiology, Department of Medicine, Weill Cornell Medicine, 525 E 68th St, New York, NY 10065 USA
| | - Meridith Pollie
- Greenberg Division of Cardiology, Department of Medicine, Weill Cornell Medicine, 525 E 68th St, New York, NY 10065 USA
| | - Ashley N. Beecy
- Greenberg Division of Cardiology, Department of Medicine, Weill Cornell Medicine, 525 E 68th St, New York, NY 10065 USA
| | - Nathan H. Tehrani
- Greenberg Division of Cardiology, Department of Medicine, Weill Cornell Medicine, 525 E 68th St, New York, NY 10065 USA
| | - Noel Codella
- IBM TJ Watson Research Center, 1101 Kitchawan Rd, Yorktown Heights, NY 10598 USA
| | | | - Maria Chiara Palumbo
- Greenberg Division of Cardiology, Department of Medicine, Weill Cornell Medicine, 525 E 68th St, New York, NY 10065 USA
| | - Javid Alakbarli
- Greenberg Division of Cardiology, Department of Medicine, Weill Cornell Medicine, 525 E 68th St, New York, NY 10065 USA
| | - Wayne Colizza
- Department of Radiology, Weill Cornell Medicine, 525 E 68th St, New York, NY 10065 USA
| | - Ian R. Drexler
- Department of Radiology, Weill Cornell Medicine, 525 E 68th St, New York, NY 10065 USA
| | - Clerio F. Azevedo
- Duke Cardiovascular Magnetic Resonance Center, 10 Duke Medicine Circle, Durham, NC 27710 USA
| | - Raymond J. Kim
- Duke Cardiovascular Magnetic Resonance Center, 10 Duke Medicine Circle, Durham, NC 27710 USA
| | - Richard B. Devereux
- Greenberg Division of Cardiology, Department of Medicine, Weill Cornell Medicine, 525 E 68th St, New York, NY 10065 USA
| | - Jonathan W. Weinsaft
- Department of Radiology, Weill Cornell Medicine, 525 E 68th St, New York, NY 10065 USA
- Greenberg Division of Cardiology, Department of Medicine, Weill Cornell Medicine, 525 E 68th St, New York, NY 10065 USA
- Memorial Sloan Kettering Cancer Center, 1275 York Ave, New York, NY 10065 USA
- Weill Cornell Medical College, 525 East 68th Street, New York, NY 10021 USA
| |
Collapse
|
104
|
Artificial Intelligence for Breast MRI in 2008-2018: A Systematic Mapping Review. AJR Am J Roentgenol 2019; 212:280-292. [PMID: 30601029 DOI: 10.2214/ajr.18.20389] [Citation(s) in RCA: 36] [Impact Index Per Article: 7.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/22/2022]
Abstract
OBJECTIVE The purpose of this study is to review literature from the past decade on applications of artificial intelligence (AI) to breast MRI. MATERIALS AND METHODS In June 2018, a systematic search of the literature was performed to identify articles on the use of AI in breast MRI. For each article identified, the surname of the first author, year of publication, journal of publication, Web of Science Core Collection journal category, country of affiliation of the first author, study design, dataset, study aim(s), AI methods used, and, when available, diagnostic performance were recorded. RESULTS Sixty-seven studies, 58 (87%) of which had a retrospective design, were analyzed. When journal categories were considered, 36% of articles were identified as being included in the radiology and imaging journal category. Contrast-enhanced sequences were used for most AI applications (n = 50; 75%) and, on occasion, were combined with other MRI sequences (n = 8; 12%). Four main clinical aims were addressed: breast lesion classification (n = 36; 54%), image processing (n = 14; 21%), prognostic imaging (n = 9; 13%), and response to neoadjuvant therapy (n = 8; 12%). Artificial neural networks, support vector machines, and clustering were the most frequently used algorithms, accounting for 66%. The performance achieved and the most frequently used techniques were then analyzed according to specific clinical aims. Supervised learning algorithms were primarily used for lesion characterization, with the AUC value from ROC analysis ranging from 0.74 to 0.98 (median, 0.87) and with that from prognostic imaging ranging from 0.62 to 0.88 (median, 0.80), whereas unsupervised learning was mainly used for image processing purposes. CONCLUSION Interest in the application of advanced AI methods to breast MRI is growing worldwide. Although this growth is encouraging, the current performance of AI applications in breast MRI means that such applications are still far from being incorporated into clinical practice.
Collapse
|
105
|
Sahiner B, Pezeshk A, Hadjiiski LM, Wang X, Drukker K, Cha KH, Summers RM, Giger ML. Deep learning in medical imaging and radiation therapy. Med Phys 2019; 46:e1-e36. [PMID: 30367497 PMCID: PMC9560030 DOI: 10.1002/mp.13264] [Citation(s) in RCA: 372] [Impact Index Per Article: 74.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2018] [Revised: 09/18/2018] [Accepted: 10/09/2018] [Indexed: 12/15/2022] Open
Abstract
The goals of this review paper on deep learning (DL) in medical imaging and radiation therapy are to (a) summarize what has been achieved to date; (b) identify common and unique challenges, and strategies that researchers have taken to address these challenges; and (c) identify some of the promising avenues for the future both in terms of applications as well as technical innovations. We introduce the general principles of DL and convolutional neural networks, survey five major areas of application of DL in medical imaging and radiation therapy, identify common themes, discuss methods for dataset expansion, and conclude by summarizing lessons learned, remaining challenges, and future directions.
Collapse
Affiliation(s)
- Berkman Sahiner
- DIDSR/OSEL/CDRH U.S. Food and Drug AdministrationSilver SpringMD20993USA
| | - Aria Pezeshk
- DIDSR/OSEL/CDRH U.S. Food and Drug AdministrationSilver SpringMD20993USA
| | | | - Xiaosong Wang
- Imaging Biomarkers and Computer‐aided Diagnosis LabRadiology and Imaging SciencesNIH Clinical CenterBethesdaMD20892‐1182USA
| | - Karen Drukker
- Department of RadiologyUniversity of ChicagoChicagoIL60637USA
| | - Kenny H. Cha
- DIDSR/OSEL/CDRH U.S. Food and Drug AdministrationSilver SpringMD20993USA
| | - Ronald M. Summers
- Imaging Biomarkers and Computer‐aided Diagnosis LabRadiology and Imaging SciencesNIH Clinical CenterBethesdaMD20892‐1182USA
| | | |
Collapse
|
106
|
Automated Detection and Segmentation of Nonmass-Enhancing Breast Tumors with Dynamic Contrast-Enhanced Magnetic Resonance Imaging. CONTRAST MEDIA & MOLECULAR IMAGING 2018; 2018:5308517. [PMID: 30647551 PMCID: PMC6311739 DOI: 10.1155/2018/5308517] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/31/2018] [Accepted: 09/16/2018] [Indexed: 01/27/2023]
Abstract
Nonmass-enhancing (NME) lesions constitute a diagnostic challenge in dynamic contrast-enhanced magnetic resonance imaging (DCE-MRI) of the breast. Computer-aided diagnosis (CAD) systems provide physicians with advanced tools for analysis, assessment, and evaluation that have a significant impact on the diagnostic performance. Here, we propose a new approach to address the challenge of NME lesion detection and segmentation, taking advantage of independent component analysis (ICA) to extract data-driven dynamic lesion characterizations. A set of independent sources was obtained from the DCE-MRI dataset of breast cancer patients, and the dynamic behavior of the different tissues was described by multiple dynamic curves, together with a set of eigenimages describing the scores for each voxel. A new test image is projected onto the independent source space using the unmixing matrix, and each voxel is classified by a support vector machine (SVM) that has already been trained with manually delineated data. A solution to the high false-positive rate problem is proposed by controlling the SVM hyperplane location, outperforming previously published approaches.
Collapse
|
107
|
Application of Radiomics and Decision Support Systems for Breast MR Differential Diagnosis. COMPUTATIONAL AND MATHEMATICAL METHODS IN MEDICINE 2018; 2018:7417126. [PMID: 30344618 PMCID: PMC6174735 DOI: 10.1155/2018/7417126] [Citation(s) in RCA: 16] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/28/2018] [Revised: 07/24/2018] [Accepted: 09/04/2018] [Indexed: 01/17/2023]
Abstract
Over the years, MR systems have evolved from imaging modalities to advanced computational systems producing a variety of numerical parameters that can be used for the noninvasive preoperative assessment of breast pathology. Furthermore, the combination with state-of-the-art image analysis methods provides a plethora of quantifiable imaging features, termed radiomics that increases diagnostic accuracy towards individualized therapy planning. More importantly, radiomics can now be complemented by the emerging deep learning techniques for further process automation and correlation with other clinical data which facilitate the monitoring of treatment response, as well as the prediction of patient's outcome, by means of unravelling of the complex underlying pathophysiological mechanisms which are reflected in tissue phenotype. The scope of this review is to provide applications and limitations of radiomics towards the development of clinical decision support systems for breast cancer diagnosis and prognosis.
Collapse
|
108
|
Abraham B, Nair MS. Computer-aided classification of prostate cancer grade groups from MRI images using texture features and stacked sparse autoencoder. Comput Med Imaging Graph 2018; 69:60-68. [PMID: 30205334 DOI: 10.1016/j.compmedimag.2018.08.006] [Citation(s) in RCA: 45] [Impact Index Per Article: 7.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/29/2017] [Revised: 06/06/2018] [Accepted: 08/22/2018] [Indexed: 12/26/2022]
Abstract
A novel method to determine the Grade Group (GG) in prostate cancer (PCa) using multi-parametric magnetic resonance imaging (mpMRI) biomarkers is investigated in this paper. In this method, high-level features are extracted from hand-crafted texture features using a deep network of stacked sparse autoencoders (SSAE) and classified them using a softmax classifier (SMC). Transaxial T2 Weighted (T2W), Apparent Diffusion Coefficient (ADC) and high B-Value Diffusion-Weighted (BVAL) images obtained from PROSTATEx-2 2017 challenge dataset are used in this technique. The method was evaluated on the challenge dataset composed of a training set of 112 lesions and a test set of 70 lesions. It achieved a quadratic-weighted Kappa score of 0.2772 on evaluation using test dataset of the challenge. It also reached a Positive Predictive Value (PPV) of 80% in predicting PCa with GG > 1. The method achieved first place in the challenge, winning over 43 methods submitted by 21 groups. A 3-fold cross-validation using training data of the challenge was further performed and the method achieved a quadratic-weighted kappa score of 0.2326 and Positive Predictive Value (PPV) of 80.26% in predicting PCa with GG > 1. Even though the training dataset is a highly imbalanced one, the method was able to achieve a fair kappa score. Being one of the pioneer methods which attempted to classify prostate cancer into 5 grade groups from MRI images, it could serve as a base method for further investigations and improvements.
Collapse
Affiliation(s)
- Bejoy Abraham
- Department of Computer Science, University of Kerala, Kariavattom, Thiruvananthapuram 695581, Kerala, India.
| | - Madhu S Nair
- Department of Computer Science, Cochin University of Science and Technology, Kochi 682022, Kerala, India
| |
Collapse
|
109
|
Deep Learning and Medical Diagnosis: A Review of Literature. MULTIMODAL TECHNOLOGIES AND INTERACTION 2018. [DOI: 10.3390/mti2030047] [Citation(s) in RCA: 166] [Impact Index Per Article: 27.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/11/2022] Open
Abstract
In this review the application of deep learning for medical diagnosis is addressed. A thorough analysis of various scientific articles in the domain of deep neural networks application in the medical field has been conducted. More than 300 research articles were obtained, and after several selection steps, 46 articles were presented in more detail. The results indicate that convolutional neural networks (CNN) are the most widely represented when it comes to deep learning and medical image analysis. Furthermore, based on the findings of this article, it can be noted that the application of deep learning technology is widespread, but the majority of applications are focused on bioinformatics, medical diagnosis and other similar fields.
Collapse
|
110
|
Vreemann S, van Zelst JCM, Schlooz-Vries M, Bult P, Hoogerbrugge N, Karssemeijer N, Gubern-Mérida A, Mann RM. The added value of mammography in different age-groups of women with and without BRCA mutation screened with breast MRI. Breast Cancer Res 2018; 20:84. [PMID: 30075794 PMCID: PMC6091096 DOI: 10.1186/s13058-018-1019-6] [Citation(s) in RCA: 27] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/11/2017] [Accepted: 07/10/2018] [Indexed: 12/22/2022] Open
Abstract
Background Breast magnetic resonance imaging (MRI) is the most sensitive imaging method for breast cancer detection and is therefore offered as a screening technique to women at increased risk of developing breast cancer. However, mammography is currently added from the age of 30 without proven benefits. The purpose of this study is to investigate the added cancer detection of mammography when breast MRI is available, focusing on the value in women with and without BRCA mutation, and in the age groups above and below 50 years. Methods This retrospective single-center study evaluated 6553 screening rounds in 2026 women at increased risk of breast cancer (1 January 2003 to 1 January 2014). Risk category (BRCA mutation versus others at increased risk of breast cancer), age at examination, recall, biopsy, and histopathological diagnosis were recorded. Cancer yield, false positive recall rate (FPR), and false positive biopsy rate (FPB) were calculated using generalized estimating equations for separate age categories (< 40, 40–50, 50–60, ≥ 60 years). Numbers of screens needed to detect an additional breast cancer with mammography (NSN) were calculated for the subgroups. Results Of a total of 125 screen-detected breast cancers, 112 were detected by MRI and 66 by mammography: 13 cancers were solely detected by mammography, including 8 cases of ductal carcinoma in situ. In BRCA mutation carriers, 3 of 61 cancers were detected only on mammography, while in other women 10 of 64 cases were detected with mammography alone. While 77% of mammography-detected-only cancers were detected in women ≥ 50 years of age, mammography also added more to the FPR in these women. Below 50 years the number of mammographic examinations needed to find an MRI-occult cancer was 1427. Conclusions Mammography is of limited added value in terms of cancer detection when breast MRI is available for women of all ages who are at increased risk. While the benefit appears slightly larger in women over 50 years of age without BRCA mutation, there is also a substantial increase in false positive findings in these women.
Collapse
Affiliation(s)
- Suzan Vreemann
- Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Geert Grooteplein 10, 6525 GA, Nijmegen, the Netherlands.
| | - Jan C M van Zelst
- Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Geert Grooteplein 10, 6525 GA, Nijmegen, the Netherlands
| | | | - Peter Bult
- Department of Pathology, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Nicoline Hoogerbrugge
- Department of Human Genetics, Radboud University Medical Center, Nijmegen, the Netherlands
| | - Nico Karssemeijer
- Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Geert Grooteplein 10, 6525 GA, Nijmegen, the Netherlands
| | - Albert Gubern-Mérida
- Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Geert Grooteplein 10, 6525 GA, Nijmegen, the Netherlands
| | - Ritse M Mann
- Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Geert Grooteplein 10, 6525 GA, Nijmegen, the Netherlands
| |
Collapse
|
111
|
Computer-Aided Diagnosis Scheme for Determining Histological Classification of Breast Lesions on Ultrasonographic Images Using Convolutional Neural Network. Diagnostics (Basel) 2018; 8:diagnostics8030048. [PMID: 30044441 PMCID: PMC6163984 DOI: 10.3390/diagnostics8030048] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/28/2018] [Revised: 07/18/2018] [Accepted: 07/23/2018] [Indexed: 11/16/2022] Open
Abstract
It can be difficult for clinicians to accurately discriminate among histological classifications of breast lesions on ultrasonographic images. The purpose of this study was to develop a computer-aided diagnosis (CADx) scheme for determining histological classifications of breast lesions using a convolutional neural network (CNN). Our database consisted of 578 breast ultrasonographic images. It included 287 malignant (217 invasive carcinomas and 70 noninvasive carcinomas) and 291 benign lesions (111 cysts and 180 fibroadenomas). In this study, the CNN constructed from four convolutional layers, three batch-normalization layers, four pooling layers, and two fully connected layers was employed for distinguishing between the four different types of histological classifications for lesions. The classification accuracies for histological classifications with our CNN model were 83.9⁻87.6%, which were substantially higher than those with our previous method (55.7⁻79.3%) using hand-crafted features and a classifier. The area under the curve with our CNN model was 0.976, whereas that with our previous method was 0.939 (p = 0.0001). Our CNN model would be useful in differential diagnoses of breast lesions as a diagnostic aid.
Collapse
|
112
|
Classification of Computed Tomography Images in Different Slice Positions Using Deep Learning. JOURNAL OF HEALTHCARE ENGINEERING 2018; 2018:1753480. [PMID: 30123439 PMCID: PMC6079460 DOI: 10.1155/2018/1753480] [Citation(s) in RCA: 19] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 03/21/2018] [Accepted: 05/22/2018] [Indexed: 12/14/2022]
Abstract
This study aimed at elucidating the relationship between the number of computed tomography (CT) images, including data concerning the accuracy of models and contrast enhancement for classifying the images. We enrolled 1539 patients who underwent contrast or noncontrast CT imaging, followed by dividing the CT imaging dataset for creating classification models into 10 classes for brain, neck, chest, abdomen, and pelvis with contrast-enhanced and plain imaging. The number of images prepared in each class were 100, 500, 1000, 2000, 3000, 4000, 5000, 6000, 7000, 8000, 9000, and 10,000. Accordingly, the names of datasets were defined as 0.1K, 0.5K, 1K, 2K, 3K, 4K, 5K, 6K, 7K, 8K, 9K, and 10K, respectively. We subsequently created and evaluated the models and compared the convolutional neural network (CNN) architecture between AlexNet and GoogLeNet. The time required for training models of AlexNet was lesser than that for GoogLeNet. The best overall accuracy for the classification of 10 classes was 0.721 with the 10K dataset of GoogLeNet. Furthermore, the best overall accuracy for the classification of the slice position without contrast media was 0.862 with the 2K dataset of AlexNet.
Collapse
|
113
|
Meyer P, Noblet V, Mazzara C, Lallement A. Survey on deep learning for radiotherapy. Comput Biol Med 2018; 98:126-146. [PMID: 29787940 DOI: 10.1016/j.compbiomed.2018.05.018] [Citation(s) in RCA: 149] [Impact Index Per Article: 24.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/03/2018] [Revised: 05/15/2018] [Accepted: 05/15/2018] [Indexed: 12/17/2022]
Abstract
More than 50% of cancer patients are treated with radiotherapy, either exclusively or in combination with other methods. The planning and delivery of radiotherapy treatment is a complex process, but can now be greatly facilitated by artificial intelligence technology. Deep learning is the fastest-growing field in artificial intelligence and has been successfully used in recent years in many domains, including medicine. In this article, we first explain the concept of deep learning, addressing it in the broader context of machine learning. The most common network architectures are presented, with a more specific focus on convolutional neural networks. We then present a review of the published works on deep learning methods that can be applied to radiotherapy, which are classified into seven categories related to the patient workflow, and can provide some insights of potential future applications. We have attempted to make this paper accessible to both radiotherapy and deep learning communities, and hope that it will inspire new collaborations between these two communities to develop dedicated radiotherapy applications.
Collapse
Affiliation(s)
- Philippe Meyer
- Department of Medical Physics, Paul Strauss Center, Strasbourg, France.
| | | | | | | |
Collapse
|
114
|
Burt JR, Torosdagli N, Khosravan N, RaviPrakash H, Mortazi A, Tissavirasingham F, Hussein S, Bagci U. Deep learning beyond cats and dogs: recent advances in diagnosing breast cancer with deep neural networks. Br J Radiol 2018; 91:20170545. [PMID: 29565644 DOI: 10.1259/bjr.20170545] [Citation(s) in RCA: 60] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022] Open
Abstract
Deep learning has demonstrated tremendous revolutionary changes in the computing industry and its effects in radiology and imaging sciences have begun to dramatically change screening paradigms. Specifically, these advances have influenced the development of computer-aided detection and diagnosis (CAD) systems. These technologies have long been thought of as "second-opinion" tools for radiologists and clinicians. However, with significant improvements in deep neural networks, the diagnostic capabilities of learning algorithms are approaching levels of human expertise (radiologists, clinicians etc.), shifting the CAD paradigm from a "second opinion" tool to a more collaborative utility. This paper reviews recently developed CAD systems based on deep learning technologies for breast cancer diagnosis, explains their superiorities with respect to previously established systems, defines the methodologies behind the improved achievements including algorithmic developments, and describes remaining challenges in breast cancer screening and diagnosis. We also discuss possible future directions for new CAD models that continue to change as artificial intelligence algorithms evolve.
Collapse
Affiliation(s)
- Jeremy R Burt
- 1 Department of Radiology, Florida Hospital , Orlando, FL , USA.,2 Department of Computer Science, Center for Research in Computer Vision, University of Central Florida (UCF) , Orlando, FL , USA
| | - Neslisah Torosdagli
- 2 Department of Computer Science, Center for Research in Computer Vision, University of Central Florida (UCF) , Orlando, FL , USA
| | - Naji Khosravan
- 2 Department of Computer Science, Center for Research in Computer Vision, University of Central Florida (UCF) , Orlando, FL , USA
| | - Harish RaviPrakash
- 2 Department of Computer Science, Center for Research in Computer Vision, University of Central Florida (UCF) , Orlando, FL , USA
| | - Aliasghar Mortazi
- 2 Department of Computer Science, Center for Research in Computer Vision, University of Central Florida (UCF) , Orlando, FL , USA
| | | | - Sarfaraz Hussein
- 2 Department of Computer Science, Center for Research in Computer Vision, University of Central Florida (UCF) , Orlando, FL , USA
| | - Ulas Bagci
- 2 Department of Computer Science, Center for Research in Computer Vision, University of Central Florida (UCF) , Orlando, FL , USA
| |
Collapse
|
115
|
Vreemann S, Gubern-Mérida A, Schlooz-Vries MS, Bult P, van Gils CH, Hoogerbrugge N, Karssemeijer N, Mann RM. Influence of Risk Category and Screening Round on the Performance of an MR Imaging and Mammography Screening Program in Carriers of the BRCA Mutation and Other Women at Increased Risk. Radiology 2018; 286:443-451. [DOI: 10.1148/radiol.2017170458] [Citation(s) in RCA: 38] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
Affiliation(s)
- Suzan Vreemann
- From the Departments of Radiology and Nuclear Medicine (S.V., A.G.M., N.K., R.M.M.), Surgery (M.S.S.V.), Pathology (P.B.), and Human Genetics (N.H.), Radboud University Medical Centre, Geert Grooteplein 10, 6525 GA Nijmegen, the Netherlands; and Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands (C.H.v.G.)
| | - Albert Gubern-Mérida
- From the Departments of Radiology and Nuclear Medicine (S.V., A.G.M., N.K., R.M.M.), Surgery (M.S.S.V.), Pathology (P.B.), and Human Genetics (N.H.), Radboud University Medical Centre, Geert Grooteplein 10, 6525 GA Nijmegen, the Netherlands; and Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands (C.H.v.G.)
| | - Margrethe S. Schlooz-Vries
- From the Departments of Radiology and Nuclear Medicine (S.V., A.G.M., N.K., R.M.M.), Surgery (M.S.S.V.), Pathology (P.B.), and Human Genetics (N.H.), Radboud University Medical Centre, Geert Grooteplein 10, 6525 GA Nijmegen, the Netherlands; and Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands (C.H.v.G.)
| | - Peter Bult
- From the Departments of Radiology and Nuclear Medicine (S.V., A.G.M., N.K., R.M.M.), Surgery (M.S.S.V.), Pathology (P.B.), and Human Genetics (N.H.), Radboud University Medical Centre, Geert Grooteplein 10, 6525 GA Nijmegen, the Netherlands; and Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands (C.H.v.G.)
| | - Carla H. van Gils
- From the Departments of Radiology and Nuclear Medicine (S.V., A.G.M., N.K., R.M.M.), Surgery (M.S.S.V.), Pathology (P.B.), and Human Genetics (N.H.), Radboud University Medical Centre, Geert Grooteplein 10, 6525 GA Nijmegen, the Netherlands; and Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands (C.H.v.G.)
| | - Nicoline Hoogerbrugge
- From the Departments of Radiology and Nuclear Medicine (S.V., A.G.M., N.K., R.M.M.), Surgery (M.S.S.V.), Pathology (P.B.), and Human Genetics (N.H.), Radboud University Medical Centre, Geert Grooteplein 10, 6525 GA Nijmegen, the Netherlands; and Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands (C.H.v.G.)
| | - Nico Karssemeijer
- From the Departments of Radiology and Nuclear Medicine (S.V., A.G.M., N.K., R.M.M.), Surgery (M.S.S.V.), Pathology (P.B.), and Human Genetics (N.H.), Radboud University Medical Centre, Geert Grooteplein 10, 6525 GA Nijmegen, the Netherlands; and Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands (C.H.v.G.)
| | - Ritse M. Mann
- From the Departments of Radiology and Nuclear Medicine (S.V., A.G.M., N.K., R.M.M.), Surgery (M.S.S.V.), Pathology (P.B.), and Human Genetics (N.H.), Radboud University Medical Centre, Geert Grooteplein 10, 6525 GA Nijmegen, the Netherlands; and Julius Center for Health Sciences and Primary Care, University Medical Center Utrecht, Utrecht, the Netherlands (C.H.v.G.)
| |
Collapse
|
116
|
Vreemann S, Gubern-Merida A, Lardenoije S, Bult P, Karssemeijer N, Pinker K, Mann RM. The frequency of missed breast cancers in women participating in a high-risk MRI screening program. Breast Cancer Res Treat 2018; 169:323-331. [PMID: 29383629 PMCID: PMC5945731 DOI: 10.1007/s10549-018-4688-z] [Citation(s) in RCA: 22] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/07/2017] [Accepted: 01/21/2018] [Indexed: 12/19/2022]
Abstract
Purpose To evaluate the frequency of missed cancers on breast MRI in women participating in a high-risk screening program. Methods Patient files from women who participated in an increased risk mammography and MRI screening program (2003–2014) were coupled to the Dutch National Cancer Registry. For each cancer detected, we determined whether an MRI scan was available (0–24 months before cancer detection), which was reported to be negative. These negative MRI scans were in consensus re-evaluated by two dedicated breast radiologists, with knowledge of the cancer location. Cancers were scored as invisible, minimal sign, or visible. Additionally, BI-RADS scores, background parenchymal enhancement, and image quality (IQ; perfect, sufficient, bad) were determined. Results were stratified by detection mode (mammography, MRI, interval cancers, or cancers in prophylactic mastectomies) and patient characteristics (presence of BRCA mutation, age, menopausal state). Results Negative prior MRI scans were available for 131 breast cancers. Overall 31% of cancers were visible at the initially negative MRI scan and 34% of cancers showed a minimal sign. The presence of a BRCA mutation strongly reduced the likelihood of visible findings in the last negative MRI (19 vs. 46%, P < 0.001). Less than perfect IQ increased the likelihood of visible findings and minimal signs in the negative MRI (P = 0.021). Conclusion This study shows that almost one-third of cancers detected in a high-risk screening program are already visible at the last negative MRI scan, and even more in women without BRCA mutations. Regular auditing and double reading for breast MRI screening is warranted.
Collapse
Affiliation(s)
- S. Vreemann
- Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Geert Grooteplein 10, 6525 GA Nijmegen, The Netherlands
| | - A. Gubern-Merida
- Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Geert Grooteplein 10, 6525 GA Nijmegen, The Netherlands
| | - S. Lardenoije
- Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Geert Grooteplein 10, 6525 GA Nijmegen, The Netherlands
| | - P. Bult
- Department of Pathology, Radboud University Medical Center, Nijmegen, The Netherlands
| | - N. Karssemeijer
- Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Geert Grooteplein 10, 6525 GA Nijmegen, The Netherlands
| | - K. Pinker
- Division of Biomedical Imaging and Image-guided Therapy, Division of Molecular and Gender Imaging, Medical University Vienna, Vienna, Austria
- Molecular Imaging and Therapy Service, Department of Radiology, Memorial Sloan Kettering Cancer Center, New York, USA
| | - R. M. Mann
- Department of Radiology and Nuclear Medicine, Radboud University Medical Center, Geert Grooteplein 10, 6525 GA Nijmegen, The Netherlands
| |
Collapse
|
117
|
Dalmış MU, Vreemann S, Kooi T, Mann RM, Karssemeijer N, Gubern-Mérida A. Fully automated detection of breast cancer in screening MRI using convolutional neural networks. J Med Imaging (Bellingham) 2018; 5:014502. [PMID: 29340287 PMCID: PMC5763014 DOI: 10.1117/1.jmi.5.1.014502] [Citation(s) in RCA: 26] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2017] [Accepted: 12/18/2017] [Indexed: 11/14/2022] Open
Abstract
Current computer-aided detection (CADe) systems for contrast-enhanced breast MRI rely on both spatial information obtained from the early-phase and temporal information obtained from the late-phase of the contrast enhancement. However, late-phase information might not be available in a screening setting, such as in abbreviated MRI protocols, where acquisition is limited to early-phase scans. We used deep learning to develop a CADe system that exploits the spatial information obtained from the early-phase scans. This system uses three-dimensional (3-D) morphological information in the candidate locations and the symmetry information arising from the enhancement differences of the two breasts. We compared the proposed system to a previously developed system, which uses the full dynamic breast MRI protocol. For training and testing, we used 385 MRI scans, containing 161 malignant lesions. Performance was measured by averaging the sensitivity values between 1/8-eight false positives. In our experiments, the proposed system obtained a significantly ([Formula: see text]) higher average sensitivity ([Formula: see text]) compared with that of the previous CADe system ([Formula: see text]). In conclusion, we developed a CADe system that is able to exploit the spatial information obtained from the early-phase scans and can be used in screening programs where abbreviated MRI protocols are used.
Collapse
Affiliation(s)
- Mehmet Ufuk Dalmış
- Radboud University Medical Center (RadboudUMC), Diagnostic Image Analysis Group (DIAG) Nijmegen, The Netherlands
| | - Suzan Vreemann
- Radboud University Medical Center (RadboudUMC), Diagnostic Image Analysis Group (DIAG) Nijmegen, The Netherlands
| | - Thijs Kooi
- Radboud University Medical Center (RadboudUMC), Diagnostic Image Analysis Group (DIAG) Nijmegen, The Netherlands
| | - Ritse M. Mann
- Radboud University Medical Center (RadboudUMC), Diagnostic Image Analysis Group (DIAG) Nijmegen, The Netherlands
| | - Nico Karssemeijer
- Radboud University Medical Center (RadboudUMC), Diagnostic Image Analysis Group (DIAG) Nijmegen, The Netherlands
| | - Albert Gubern-Mérida
- Radboud University Medical Center (RadboudUMC), Diagnostic Image Analysis Group (DIAG) Nijmegen, The Netherlands
| |
Collapse
|
118
|
Litjens G, Kooi T, Bejnordi BE, Setio AAA, Ciompi F, Ghafoorian M, van der Laak JAWM, van Ginneken B, Sánchez CI. A survey on deep learning in medical image analysis. Med Image Anal 2017; 42:60-88. [PMID: 28778026 DOI: 10.1016/j.media.2017.07.005] [Citation(s) in RCA: 4325] [Impact Index Per Article: 617.9] [Reference Citation Analysis] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2017] [Revised: 07/24/2017] [Accepted: 07/25/2017] [Indexed: 02/07/2023]
Affiliation(s)
- Geert Litjens
- Diagnostic Image Analysis Group, Radboud University Medical Center, Nijmegen, The Netherlands.
| | - Thijs Kooi
- Diagnostic Image Analysis Group, Radboud University Medical Center, Nijmegen, The Netherlands
| | | | | | - Francesco Ciompi
- Diagnostic Image Analysis Group, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Mohsen Ghafoorian
- Diagnostic Image Analysis Group, Radboud University Medical Center, Nijmegen, The Netherlands
| | | | - Bram van Ginneken
- Diagnostic Image Analysis Group, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Clara I Sánchez
- Diagnostic Image Analysis Group, Radboud University Medical Center, Nijmegen, The Netherlands
| |
Collapse
|
119
|
Tseng HH, Luo Y, Cui S, Chien JT, Ten Haken RK, Naqa IE. Deep reinforcement learning for automated radiation adaptation in lung cancer. Med Phys 2017; 44:6690-6705. [PMID: 29034482 DOI: 10.1002/mp.12625] [Citation(s) in RCA: 98] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2017] [Revised: 08/25/2017] [Accepted: 10/02/2017] [Indexed: 12/12/2022] Open
Abstract
PURPOSE To investigate deep reinforcement learning (DRL) based on historical treatment plans for developing automated radiation adaptation protocols for nonsmall cell lung cancer (NSCLC) patients that aim to maximize tumor local control at reduced rates of radiation pneumonitis grade 2 (RP2). METHODS In a retrospective population of 114 NSCLC patients who received radiotherapy, a three-component neural networks framework was developed for deep reinforcement learning (DRL) of dose fractionation adaptation. Large-scale patient characteristics included clinical, genetic, and imaging radiomics features in addition to tumor and lung dosimetric variables. First, a generative adversarial network (GAN) was employed to learn patient population characteristics necessary for DRL training from a relatively limited sample size. Second, a radiotherapy artificial environment (RAE) was reconstructed by a deep neural network (DNN) utilizing both original and synthetic data (by GAN) to estimate the transition probabilities for adaptation of personalized radiotherapy patients' treatment courses. Third, a deep Q-network (DQN) was applied to the RAE for choosing the optimal dose in a response-adapted treatment setting. This multicomponent reinforcement learning approach was benchmarked against real clinical decisions that were applied in an adaptive dose escalation clinical protocol. In which, 34 patients were treated based on avid PET signal in the tumor and constrained by a 17.2% normal tissue complication probability (NTCP) limit for RP2. The uncomplicated cure probability (P+) was used as a baseline reward function in the DRL. RESULTS Taking our adaptive dose escalation protocol as a blueprint for the proposed DRL (GAN + RAE + DQN) architecture, we obtained an automated dose adaptation estimate for use at ∼2/3 of the way into the radiotherapy treatment course. By letting the DQN component freely control the estimated adaptive dose per fraction (ranging from 1-5 Gy), the DRL automatically favored dose escalation/de-escalation between 1.5 and 3.8 Gy, a range similar to that used in the clinical protocol. The same DQN yielded two patterns of dose escalation for the 34 test patients, but with different reward variants. First, using the baseline P+ reward function, individual adaptive fraction doses of the DQN had similar tendencies to the clinical data with an RMSE = 0.76 Gy; but adaptations suggested by the DQN were generally lower in magnitude (less aggressive). Second, by adjusting the P+ reward function with higher emphasis on mitigating local failure, better matching of doses between the DQN and the clinical protocol was achieved with an RMSE = 0.5 Gy. Moreover, the decisions selected by the DQN seemed to have better concordance with patients eventual outcomes. In comparison, the traditional temporal difference (TD) algorithm for reinforcement learning yielded an RMSE = 3.3 Gy due to numerical instabilities and lack of sufficient learning. CONCLUSION We demonstrated that automated dose adaptation by DRL is a feasible and a promising approach for achieving similar results to those chosen by clinicians. The process may require customization of the reward function if individual cases were to be considered. However, development of this framework into a fully credible autonomous system for clinical decision support would require further validation on larger multi-institutional datasets.
Collapse
Affiliation(s)
- Huan-Hsin Tseng
- Department of Radiation Oncology, University of Michigan, Ann Arbor, MI, USA
| | - Yi Luo
- Department of Radiation Oncology, University of Michigan, Ann Arbor, MI, USA
| | - Sunan Cui
- Department of Radiation Oncology, University of Michigan, Ann Arbor, MI, USA
| | - Jen-Tzung Chien
- Department of Radiation Oncology, University of Michigan, Ann Arbor, MI, USA.,Department of Electrical and Computer Engineering, National Chiao Tung University, Hsinchu, Taiwan
| | - Randall K Ten Haken
- Department of Radiation Oncology, University of Michigan, Ann Arbor, MI, USA
| | - Issam El Naqa
- Department of Radiation Oncology, University of Michigan, Ann Arbor, MI, USA
| |
Collapse
|
120
|
Kwon K, Kim D, Park H. A parallel MR imaging method using multilayer perceptron. Med Phys 2017; 44:6209-6224. [DOI: 10.1002/mp.12600] [Citation(s) in RCA: 89] [Impact Index Per Article: 12.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/01/2017] [Revised: 09/17/2017] [Accepted: 09/18/2017] [Indexed: 11/08/2022] Open
Affiliation(s)
- Kinam Kwon
- Department of Electrical Engineering; Korea Advanced Institute of Science and Technology (KAIST); Daejeon South Korea
| | - Dongchan Kim
- College of Medicine; Gachon University; Incheon South Korea
| | - HyunWook Park
- Department of Electrical Engineering; Korea Advanced Institute of Science and Technology (KAIST); Daejeon South Korea
| |
Collapse
|
121
|
Korfiatis P, Kline TL, Lachance DH, Parney IF, Buckner JC, Erickson BJ. Residual Deep Convolutional Neural Network Predicts MGMT Methylation Status. J Digit Imaging 2017; 30:622-628. [PMID: 28785873 PMCID: PMC5603430 DOI: 10.1007/s10278-017-0009-z] [Citation(s) in RCA: 96] [Impact Index Per Article: 13.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/19/2022] Open
Abstract
Predicting methylation of the O6-methylguanine methyltransferase (MGMT) gene status utilizing MRI imaging is of high importance since it is a predictor of response and prognosis in brain tumors. In this study, we compare three different residual deep neural network (ResNet) architectures to evaluate their ability in predicting MGMT methylation status without the need for a distinct tumor segmentation step. We found that the ResNet50 (50 layers) architecture was the best performing model, achieving an accuracy of 94.90% (+/- 3.92%) for the test set (classification of a slice as no tumor, methylated MGMT, or non-methylated). ResNet34 (34 layers) achieved 80.72% (+/- 13.61%) while ResNet18 (18 layers) accuracy was 76.75% (+/- 20.67%). ResNet50 performance was statistically significantly better than both ResNet18 and ResNet34 architectures (p < 0.001). We report a method that alleviates the need of extensive preprocessing and acts as a proof of concept that deep neural architectures can be used to predict molecular biomarkers from routine medical images.
Collapse
Affiliation(s)
- Panagiotis Korfiatis
- Department of Radiology, Mayo Clinic, 200 1st Street SW, Rochester, MN, 55905, USA
| | - Timothy L Kline
- Department of Radiology, Mayo Clinic, 200 1st Street SW, Rochester, MN, 55905, USA
| | - Daniel H Lachance
- Department of Neurology, Mayo Clinic, 200 1st Street SW, Rochester, MN, 55905, USA
| | - Ian F Parney
- Department of Neurologic Surgery, Mayo Clinic, 200 1st Street SW, Rochester, MN, 55905, USA
| | - Jan C Buckner
- Department of Medical Oncology, Mayo Clinic, 200 1st Street SW, Rochester, MN, 55905, USA
| | - Bradley J Erickson
- Department of Radiology, Mayo Clinic, 200 1st Street SW, Rochester, MN, 55905, USA.
| |
Collapse
|
122
|
Bejnordi BE, Zuidhof G, Balkenhol M, Hermsen M, Bult P, van Ginneken B, Karssemeijer N, Litjens G, van der Laak J. Context-aware stacked convolutional neural networks for classification of breast carcinomas in whole-slide histopathology images. J Med Imaging (Bellingham) 2017; 4:044504. [PMID: 29285517 PMCID: PMC5729919 DOI: 10.1117/1.jmi.4.4.044504] [Citation(s) in RCA: 87] [Impact Index Per Article: 12.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/09/2017] [Accepted: 11/14/2017] [Indexed: 12/31/2022] Open
Abstract
Currently, histopathological tissue examination by a pathologist represents the gold standard for breast lesion diagnostics. Automated classification of histopathological whole-slide images (WSIs) is challenging owing to the wide range of appearances of benign lesions and the visual similarity of ductal carcinoma in-situ (DCIS) to invasive lesions at the cellular level. Consequently, analysis of tissue at high resolutions with a large contextual area is necessary. We present context-aware stacked convolutional neural networks (CNN) for classification of breast WSIs into normal/benign, DCIS, and invasive ductal carcinoma (IDC). We first train a CNN using high pixel resolution to capture cellular level information. The feature responses generated by this model are then fed as input to a second CNN, stacked on top of the first. Training of this stacked architecture with large input patches enables learning of fine-grained (cellular) details and global tissue structures. Our system is trained and evaluated on a dataset containing 221 WSIs of hematoxylin and eosin stained breast tissue specimens. The system achieves an AUC of 0.962 for the binary classification of nonmalignant and malignant slides and obtains a three-class accuracy of 81.3% for classification of WSIs into normal/benign, DCIS, and IDC, demonstrating its potential for routine diagnostics.
Collapse
Affiliation(s)
- Babak Ehteshami Bejnordi
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Nijmegen, The Netherlands
| | - Guido Zuidhof
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Pathology, Nijmegen, The Netherlands
| | - Maschenka Balkenhol
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Pathology, Nijmegen, The Netherlands
| | - Meyke Hermsen
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Pathology, Nijmegen, The Netherlands
| | - Peter Bult
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Pathology, Nijmegen, The Netherlands
| | - Bram van Ginneken
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Nijmegen, The Netherlands
| | - Nico Karssemeijer
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Nijmegen, The Netherlands
| | - Geert Litjens
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Nijmegen, The Netherlands
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Pathology, Nijmegen, The Netherlands
| | - Jeroen van der Laak
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Radiology and Nuclear Medicine, Nijmegen, The Netherlands
- Radboud University Medical Center, Diagnostic Image Analysis Group, Department of Pathology, Nijmegen, The Netherlands
| |
Collapse
|
123
|
Affiliation(s)
- Jihoon Jeong
- Advisor, Lunit Inc., Seoul, Korea
- Partner, Digital Healthcare Partners, Seoul, Korea
- Senior Teaching Fellow, Department of Media and Communications, Kyung Hee Cyber University, Seoul, Korea
| |
Collapse
|