1
|
Wang J, Sourlos N, Zheng S, van der Velden N, Pelgrim GJ, Vliegenthart R, van Ooijen P. Preparing CT imaging datasets for deep learning in lung nodule analysis: Insights from four well-known datasets. Heliyon 2023; 9:e17104. [PMID: 37484314 PMCID: PMC10361226 DOI: 10.1016/j.heliyon.2023.e17104] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2023] [Revised: 06/06/2023] [Accepted: 06/07/2023] [Indexed: 07/25/2023] Open
Abstract
BACKGROUND Deep learning is an important means to realize the automatic detection, segmentation, and classification of pulmonary nodules in computed tomography (CT) images. An entire CT scan cannot directly be used by deep learning models due to image size, image format, image dimensionality, and other factors. Between the acquisition of the CT scan and feeding the data into the deep learning model, there are several steps including data use permission, data access and download, data annotation, and data preprocessing. This paper aims to recommend a complete and detailed guide for researchers who want to engage in interdisciplinary lung nodule research of CT images and Artificial Intelligence (AI) engineering. METHODS The data preparation pipeline used the following four popular large-scale datasets: LIDC-IDRI (Lung Image Database Consortium image collection), LUNA16 (Lung Nodule Analysis 2016), NLST (National Lung Screening Trial) and NELSON (The Dutch-Belgian Randomized Lung Cancer Screening Trial). The dataset preparation is presented in chronological order. FINDINGS The different data preparation steps before deep learning were identified. These include both more generic steps and steps dedicated to lung nodule research. For each of these steps, the required process, necessity, and example code or tools for actual implementation are provided. DISCUSSION AND CONCLUSION Depending on the specific research question, researchers should be aware of the various preparation steps required and carefully select datasets, data annotation methods, and image preprocessing methods. Moreover, it is vital to acknowledge that each auxiliary tool or code has its specific scope of use and limitations. This paper proposes a standardized data preparation process while clearly demonstrating the principles and sequence of different steps. A data preparation pipeline can be quickly realized by following these proposed steps and implementing the suggested example codes and tools.
Collapse
Affiliation(s)
- Jingxuan Wang
- Department of Radiology, University of Groningen, University Medical Center of Groningen, 9713GZ, Groningen, the Netherlands
| | - Nikos Sourlos
- Department of Radiology, University of Groningen, University Medical Center of Groningen, 9713GZ, Groningen, the Netherlands
| | - Sunyi Zheng
- School of Engineering, Westlake University, Xihu District, 310030, Hangzhou, China
| | - Nils van der Velden
- Department of Radiology, University of Groningen, University Medical Center of Groningen, 9713GZ, Groningen, the Netherlands
| | - Gert Jan Pelgrim
- Department of Radiology, University of Groningen, University Medical Center of Groningen, 9713GZ, Groningen, the Netherlands
| | - Rozemarijn Vliegenthart
- Department of Radiology, University of Groningen, University Medical Center of Groningen, 9713GZ, Groningen, the Netherlands
- Data Science Center in Health (DASH), University of Groningen, University Medical Center of Groningen, 9713GZ, Groningen, the Netherlands
| | - Peter van Ooijen
- Department of Radiation Oncology, University of Groningen, University Medical Center of Groningen, 9713GZ, Groningen, the Netherlands
- Data Science Center in Health (DASH), University of Groningen, University Medical Center of Groningen, 9713GZ, Groningen, the Netherlands
| |
Collapse
|
2
|
Shiri I, Salimi Y, Pakbin M, Hajianfar G, Avval AH, Sanaat A, Mostafaei S, Akhavanallaf A, Saberi A, Mansouri Z, Askari D, Ghasemian M, Sharifipour E, Sandoughdaran S, Sohrabi A, Sadati E, Livani S, Iranpour P, Kolahi S, Khateri M, Bijari S, Atashzar MR, Shayesteh SP, Khosravi B, Babaei MR, Jenabi E, Hasanian M, Shahhamzeh A, Foroghi Ghomi SY, Mozafari A, Teimouri A, Movaseghi F, Ahmari A, Goharpey N, Bozorgmehr R, Shirzad-Aski H, Mortazavi R, Karimi J, Mortazavi N, Besharat S, Afsharpad M, Abdollahi H, Geramifar P, Radmard AR, Arabi H, Rezaei-Kalantari K, Oveisi M, Rahmim A, Zaidi H. COVID-19 prognostic modeling using CT radiomic features and machine learning algorithms: Analysis of a multi-institutional dataset of 14,339 patients. Comput Biol Med 2022; 145:105467. [PMID: 35378436 PMCID: PMC8964015 DOI: 10.1016/j.compbiomed.2022.105467] [Citation(s) in RCA: 22] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2022] [Revised: 03/24/2022] [Accepted: 03/26/2022] [Indexed: 12/16/2022]
Abstract
BACKGROUND We aimed to analyze the prognostic power of CT-based radiomics models using data of 14,339 COVID-19 patients. METHODS Whole lung segmentations were performed automatically using a deep learning-based model to extract 107 intensity and texture radiomics features. We used four feature selection algorithms and seven classifiers. We evaluated the models using ten different splitting and cross-validation strategies, including non-harmonized and ComBat-harmonized datasets. The sensitivity, specificity, and area under the receiver operating characteristic curve (AUC) were reported. RESULTS In the test dataset (4,301) consisting of CT and/or RT-PCR positive cases, AUC, sensitivity, and specificity of 0.83 ± 0.01 (CI95%: 0.81-0.85), 0.81, and 0.72, respectively, were obtained by ANOVA feature selector + Random Forest (RF) classifier. Similar results were achieved in RT-PCR-only positive test sets (3,644). In ComBat harmonized dataset, Relief feature selector + RF classifier resulted in the highest performance of AUC, reaching 0.83 ± 0.01 (CI95%: 0.81-0.85), with a sensitivity and specificity of 0.77 and 0.74, respectively. ComBat harmonization did not depict statistically significant improvement compared to a non-harmonized dataset. In leave-one-center-out, the combination of ANOVA feature selector and RF classifier resulted in the highest performance. CONCLUSION Lung CT radiomics features can be used for robust prognostic modeling of COVID-19. The predictive power of the proposed CT radiomics model is more reliable when using a large multicentric heterogeneous dataset, and may be used prospectively in clinical setting to manage COVID-19 patients.
Collapse
Affiliation(s)
- Isaac Shiri
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, 1211, Switzerland
| | - Yazdan Salimi
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, 1211, Switzerland
| | - Masoumeh Pakbin
- Imaging Department, Qom University of Medical Sciences, Qum, Iran
| | - Ghasem Hajianfar
- Rajaie Cardiovascular, Medical & Research Center, Iran University of Medical Science, Tehran, Iran
| | | | - Amirhossein Sanaat
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, 1211, Switzerland
| | - Shayan Mostafaei
- Division of Clinical Geriatrics, Department of Neurobiology, Care Sciences and Society, Karolinska Institutet, Stockholm, Sweden
| | - Azadeh Akhavanallaf
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, 1211, Switzerland
| | - Abdollah Saberi
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, 1211, Switzerland
| | - Zahra Mansouri
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, 1211, Switzerland
| | - Dariush Askari
- Department of Radiology Technology, Shahid Beheshti University of Medical Sciences, Tehran, Iran
| | - Mohammadreza Ghasemian
- Department of Radiology, Shahid Beheshti Hospital, Qom University of Medical Sciences, Qum, Iran
| | - Ehsan Sharifipour
- Neuroscience Research Center, Qom University of Medical Sciences, Qum, Iran
| | - Saleh Sandoughdaran
- Men's Health and Reproductive Health Research Center, Shahid Beheshti University of Medical Sciences, Tehran, Iran
| | - Ahmad Sohrabi
- Cancer Control Research Center, Cancer Control Foundation, Iran University of Medical Sciences, Tehran, Iran
| | - Elham Sadati
- Department of Medical Physics, Faculty of Medical Sciences, Tarbiat Modares University, Tehran, Iran
| | - Somayeh Livani
- Clinical Research Development Unit (CRDU), Sayad Shirazi Hospital, Golestan University of Medical Sciences, Gorgan, Iran
| | - Pooya Iranpour
- Medical Imaging Research Center, Department of Radiology, Shiraz University of Medical Sciences, Shiraz, Iran
| | - Shahriar Kolahi
- Department of Radiology, School of Medicine, Advanced Diagnostic and Interventional Radiology Research Center (ADIR), Imam Khomeini Hospital, Tehran University of Medical Sciences, Tehran, Iran
| | - Maziar Khateri
- Department of Medical Radiation Engineering, Science and Research Branch, Islamic Azad University, Tehran, Tehran, Iran
| | - Salar Bijari
- Department of Medical Physics, Faculty of Medical Sciences, Tarbiat Modares University, Tehran, Iran
| | - Mohammad Reza Atashzar
- Department of Immunology, School of Medicine, Fasa University of Medical Sciences, Fasa, Iran
| | - Sajad P Shayesteh
- Department of Physiology, Pharmacology and Medical Physics, Alborz University of Medical Sciences, Karaj, Iran
| | - Bardia Khosravi
- Department of Radiology, Shariati Hospital, Tehran University of Medical Sciences, Tehran, Iran
| | - Mohammad Reza Babaei
- Department of Interventional Radiology, Firouzgar Hospital, Iran University of Medical Sciences, Tehran, Iran
| | - Elnaz Jenabi
- Research Centre for Nuclear Medicine, Shariati Hospital, Tehran University of Medical Sciences, Tehran, Iran
| | - Mohammad Hasanian
- Department of Radiology, Arak University of Medical Sciences, Arak, Iran
| | - Alireza Shahhamzeh
- Clinical Research Development Center, Qom University of Medical Sciences, Qum, Iran
| | - Seyaed Yaser Foroghi Ghomi
- Clinical Research Development Center, Shahid Beheshti Hospital, Qom University Of Medical Sciences, Qom, Iran
| | - Abolfazl Mozafari
- Department of Medical Sciences, Qom Branch, Islamic Azad University, Qum, Iran
| | - Arash Teimouri
- Medical Imaging Research Center, Department of Radiology, Shiraz University of Medical Sciences, Shiraz, Iran
| | - Fatemeh Movaseghi
- Department of Medical Sciences, Qom Branch, Islamic Azad University, Qum, Iran
| | - Azin Ahmari
- Ayatolah Khansary Hospital, Arak University of Medical Sciences, Arak, Iran
| | - Neda Goharpey
- Department of Radiation Oncology, Shohadaye Tajrish Hospital, Shahid Beheshti University of Medical Sciences, Tehran, Iran
| | - Rama Bozorgmehr
- Clinical Research Development Unit, Shohadaye Tajrish Hospital, Shahid Beheshti University of Medical Sciences, Tehran, Iran
| | | | - Roozbeh Mortazavi
- Department of Internal Medicine, Shiraz University of Medical Sciences, Shiraz, Iran
| | - Jalal Karimi
- Department of Infectious Disease, School of Medicine, Fasa University of Medical Sciences, Fasa, Iran
| | - Nazanin Mortazavi
- Dental Research Center, Golestan University of Medical Sciences, Gorgan, Iran
| | - Sima Besharat
- Golestan Research Center of Gastroenterology and Hepatology, Golestan University of Medical Sciences, Gorgan, Iran
| | - Mandana Afsharpad
- Cancer Control Research Center, Cancer Control Foundation, Iran University of Medical Sciences, Tehran, Iran
| | - Hamid Abdollahi
- Department of Radiologic Technology, Faculty of Allied Medical Sciences, Kerman University of Medical Sciences, Kerman, Iran
| | - Parham Geramifar
- Research Centre for Nuclear Medicine, Shariati Hospital, Tehran University of Medical Sciences, Tehran, Iran
| | - Amir Reza Radmard
- Department of Radiology, Shariati Hospital, Tehran University of Medical Sciences, Tehran, Iran
| | - Hossein Arabi
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, 1211, Switzerland
| | - Kiara Rezaei-Kalantari
- Rajaie Cardiovascular, Medical & Research Center, Iran University of Medical Science, Tehran, Iran
| | - Mehrdad Oveisi
- Comprehensive Cancer Centre, School of Cancer & Pharmaceutical Sciences, Faculty of Life Sciences & Medicine, King's College London, London, United Kingdom
| | - Arman Rahmim
- Departments of Radiology and Physics, University of British Columbia, Vancouver, BC, Canada; Department of Integrative Oncology, BC Cancer Research Institute, Vancouver, BC, Canada
| | - Habib Zaidi
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, 1211, Switzerland; Geneva University Neurocenter, Geneva University, Geneva, Switzerland; Department of Nuclear Medicine and Molecular Imaging, University of Groningen, University Medical Center Groningen, Groningen, Netherlands; Department of Nuclear Medicine, University of Southern Denmark, Odense, Denmark.
| |
Collapse
|
3
|
Jungmann F, Müller L, Hahn F, Weustenfeld M, Dapper AK, Mähringer-Kunz A, Graafen D, Düber C, Schafigh D, Pinto Dos Santos D, Mildenberger P, Kloeckner R. Commercial AI solutions in detecting COVID-19 pneumonia in chest CT: not yet ready for clinical implementation? Eur Radiol 2022; 32:3152-3160. [PMID: 34950973 PMCID: PMC8700707 DOI: 10.1007/s00330-021-08409-4] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2021] [Revised: 09/16/2021] [Accepted: 10/08/2021] [Indexed: 11/29/2022]
Abstract
OBJECTIVES In response to the COVID-19 pandemic, many researchers have developed artificial intelligence (AI) tools to differentiate COVID-19 pneumonia from other conditions in chest CT. However, in many cases, performance has not been clinically validated. The aim of this study was to evaluate the performance of commercial AI solutions in differentiating COVID-19 pneumonia from other lung conditions. METHODS Four commercial AI solutions were evaluated on a dual-center clinical dataset consisting of 500 CT studies; COVID-19 pneumonia was microbiologically proven in 50 of these. Sensitivity, specificity, positive and negative predictive values, and AUC were calculated. In a subgroup analysis, the performance of the AI solutions in differentiating COVID-19 pneumonia from other conditions was evaluated in CT studies with ground-glass opacities (GGOs). RESULTS Sensitivity and specificity ranges were 62-96% and 31-80%, respectively. Negative and positive predictive values ranged between 82-99% and 19-25%, respectively. AUC was in the range 0.54-0.79. In CT studies with GGO, sensitivity remained unchanged. However, specificity was lower, and ranged between 15 and 53%. AUC for studies with GGO was in the range 0.54-0.69. CONCLUSIONS This study highlights the variable specificity and low positive predictive value of AI solutions in diagnosing COVID-19 pneumonia in chest CT. However, one solution yielded acceptable values for sensitivity. Thus, with further improvement, commercial AI solutions currently under development have the potential to be integrated as alert tools in clinical routine workflow. Randomized trials are needed to assess the true benefits and also potential harms of the use of AI in image analysis. KEY POINTS • Commercial AI solutions achieved a sensitivity and specificity ranging from 62 to 96% and from 31 to 80%, respectively, in identifying patients suspicious for COVID-19 in a clinical dataset. • Sensitivity remained within the same range, while specificity was even lower in subgroup analysis of CT studies with ground-glass opacities, and interrater agreement between the commercial AI solutions was minimal to nonexistent. • Thus, commercial AI solutions have the potential to be integrated as alert tools for the detection of patients with lung changes suspicious for COVID-19 pneumonia in a clinical routine workflow, if further improvement is made.
Collapse
Affiliation(s)
- Florian Jungmann
- Department of Diagnostic and Interventional Radiology, University Medical Center of the Johannes Gutenberg University Mainz, Langenbeckst. 1, 55131, Mainz, Germany.
| | - Lukas Müller
- Department of Diagnostic and Interventional Radiology, University Medical Center of the Johannes Gutenberg University Mainz, Langenbeckst. 1, 55131, Mainz, Germany
| | - Felix Hahn
- Department of Diagnostic and Interventional Radiology, University Medical Center of the Johannes Gutenberg University Mainz, Langenbeckst. 1, 55131, Mainz, Germany
| | - Maximilian Weustenfeld
- Department of Diagnostic and Interventional Radiology, University Medical Center of the Johannes Gutenberg University Mainz, Langenbeckst. 1, 55131, Mainz, Germany
| | - Ann-Kathrin Dapper
- Department of Neuroradiology, University Medical Center of the Johannes Gutenberg University Mainz, Mainz, Germany
| | - Aline Mähringer-Kunz
- Department of Diagnostic and Interventional Radiology, University Medical Center of the Johannes Gutenberg University Mainz, Langenbeckst. 1, 55131, Mainz, Germany
| | - Dirk Graafen
- Department of Diagnostic and Interventional Radiology, University Medical Center of the Johannes Gutenberg University Mainz, Langenbeckst. 1, 55131, Mainz, Germany
| | - Christoph Düber
- Department of Diagnostic and Interventional Radiology, University Medical Center of the Johannes Gutenberg University Mainz, Langenbeckst. 1, 55131, Mainz, Germany
| | - Darius Schafigh
- Department of Radiology, University Hospital of Cologne, Cologne, Germany
| | | | - Peter Mildenberger
- Department of Diagnostic and Interventional Radiology, University Medical Center of the Johannes Gutenberg University Mainz, Langenbeckst. 1, 55131, Mainz, Germany
| | - Roman Kloeckner
- Department of Diagnostic and Interventional Radiology, University Medical Center of the Johannes Gutenberg University Mainz, Langenbeckst. 1, 55131, Mainz, Germany
| |
Collapse
|
4
|
Alyasseri ZAA, Al‐Betar MA, Doush IA, Awadallah MA, Abasi AK, Makhadmeh SN, Alomari OA, Abdulkareem KH, Adam A, Damasevicius R, Mohammed MA, Zitar RA. Review on COVID-19 diagnosis models based on machine learning and deep learning approaches. EXPERT SYSTEMS 2022; 39:e12759. [PMID: 34511689 PMCID: PMC8420483 DOI: 10.1111/exsy.12759] [Citation(s) in RCA: 59] [Impact Index Per Article: 29.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/29/2021] [Revised: 05/17/2021] [Accepted: 06/07/2021] [Indexed: 05/02/2023]
Abstract
COVID-19 is the disease evoked by a new breed of coronavirus called the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). Recently, COVID-19 has become a pandemic by infecting more than 152 million people in over 216 countries and territories. The exponential increase in the number of infections has rendered traditional diagnosis techniques inefficient. Therefore, many researchers have developed several intelligent techniques, such as deep learning (DL) and machine learning (ML), which can assist the healthcare sector in providing quick and precise COVID-19 diagnosis. Therefore, this paper provides a comprehensive review of the most recent DL and ML techniques for COVID-19 diagnosis. The studies are published from December 2019 until April 2021. In general, this paper includes more than 200 studies that have been carefully selected from several publishers, such as IEEE, Springer and Elsevier. We classify the research tracks into two categories: DL and ML and present COVID-19 public datasets established and extracted from different countries. The measures used to evaluate diagnosis methods are comparatively analysed and proper discussion is provided. In conclusion, for COVID-19 diagnosing and outbreak prediction, SVM is the most widely used machine learning mechanism, and CNN is the most widely used deep learning mechanism. Accuracy, sensitivity, and specificity are the most widely used measurements in previous studies. Finally, this review paper will guide the research community on the upcoming development of machine learning for COVID-19 and inspire their works for future development. This review paper will guide the research community on the upcoming development of ML and DL for COVID-19 and inspire their works for future development.
Collapse
Affiliation(s)
- Zaid Abdi Alkareem Alyasseri
- Center for Artificial Intelligence Technology, Faculty of Information Science and TechnologyUniversiti Kebangsaan MalaysiaBangiMalaysia
- ECE Department‐Faculty of EngineeringUniversity of KufaNajafIraq
| | - Mohammed Azmi Al‐Betar
- Artificial Intelligence Research Center (AIRC)Ajman UniversityAjmanUnited Arab Emirates
- Department of Information TechnologyAl‐Huson University College, Al‐Balqa Applied UniversityIrbidJordan
| | - Iyad Abu Doush
- Computing Department, College of Engineering and Applied SciencesAmerican University of KuwaitSalmiyaKuwait
- Computer Science DepartmentYarmouk UniversityIrbidJordan
| | - Mohammed A. Awadallah
- Artificial Intelligence Research Center (AIRC)Ajman UniversityAjmanUnited Arab Emirates
- Department of Computer ScienceAl‐Aqsa UniversityGazaPalestine
| | - Ammar Kamal Abasi
- Artificial Intelligence Research Center (AIRC)Ajman UniversityAjmanUnited Arab Emirates
- School of Computer SciencesUniversiti Sains MalaysiaPenangMalaysia
| | - Sharif Naser Makhadmeh
- Artificial Intelligence Research Center (AIRC)Ajman UniversityAjmanUnited Arab Emirates
- Faculty of Information TechnologyMiddle East UniversityAmmanJordan
| | | | | | - Afzan Adam
- Center for Artificial Intelligence Technology, Faculty of Information Science and TechnologyUniversiti Kebangsaan MalaysiaBangiMalaysia
| | | | - Mazin Abed Mohammed
- College of Computer Science and Information TechnologyUniversity of AnbarAnbarIraq
| | - Raed Abu Zitar
- Sorbonne Center of Artificial IntelligenceSorbonne University‐Abu DhabiAbu DhabiUnited Arab Emirates
| |
Collapse
|
5
|
Dhont J, Wolfs C, Verhaegen F. Automatic coronavirus disease 2019 diagnosis based on chest radiography and deep learning - Success story or dataset bias? Med Phys 2022; 49:978-987. [PMID: 34951033 PMCID: PMC9015341 DOI: 10.1002/mp.15419] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/20/2021] [Revised: 12/03/2021] [Accepted: 12/03/2021] [Indexed: 11/11/2022] Open
Abstract
PURPOSE Over the last 2 years, the artificial intelligence (AI) community has presented several automatic screening tools for coronavirus disease 2019 (COVID-19) based on chest radiography (CXR), with reported accuracies often well over 90%. However, it has been noted that many of these studies have likely suffered from dataset bias, leading to overly optimistic results. The purpose of this study was to thoroughly investigate to what extent biases have influenced the performance of a range of previously proposed and promising convolutional neural networks (CNNs), and to determine what performance can be expected with current CNNs on a realistic and unbiased dataset. METHODS Five CNNs for COVID-19 positive/negative classification were implemented for evaluation, namely VGG19, ResNet50, InceptionV3, DenseNet201, and COVID-Net. To perform both internal and cross-dataset evaluations, four datasets were created. The first dataset Valencian Region Medical Image Bank (BIMCV) followed strict reverse transcriptase-polymerase chain reaction (RT-PCR) test criteria and was created from a single reliable open access databank, while the second dataset (COVIDxB8) was created through a combination of six online CXR repositories. The third and fourth datasets were created by combining the opposing classes from the BIMCV and COVIDxB8 datasets. To decrease inter-dataset variability, a pre-processing workflow of resizing, normalization, and histogram equalization were applied to all datasets. Classification performance was evaluated on unseen test sets using precision and recall. A qualitative sanity check was performed by evaluating saliency maps displaying the top 5%, 10%, and 20% most salient segments in the input CXRs, to evaluate whether the CNNs were using relevant information for decision making. In an additional experiment and to further investigate the origin of potential dataset bias, all pixel values outside the lungs were set to zero through automatic lung segmentation before training and testing. RESULTS When trained and evaluated on the single online source dataset (BIMCV), the performance of all CNNs is relatively low (precision: 0.65-0.72, recall: 0.59-0.71), but remains relatively consistent during external evaluation (precision: 0.58-0.82, recall: 0.57-0.72). On the contrary, when trained and internally evaluated on the combinatory datasets, all CNNs performed well across all metrics (precision: 0.94-1.00, recall: 0.77-1.00). However, when subsequently evaluated cross-dataset, results dropped substantially (precision: 0.10-0.61, recall: 0.04-0.80). For all datasets, saliency maps revealed the CNNs rarely focus on areas inside the lungs for their decision-making. However, even when setting all pixel values outside the lungs to zero, classification performance does not change and dataset bias remains. CONCLUSIONS Results in this study confirm that when trained on a combinatory dataset, CNNs tend to learn the origin of the CXRs rather than the presence or absence of disease, a behavior known as short-cut learning. The bias is shown to originate from differences in overall pixel values rather than embedded text or symbols, despite consistent image pre-processing. When trained on a reliable, and realistic single-source dataset in which non-lung pixels have been masked, CNNs currently show limited sensitivity (<70%) for COVID-19 infection in CXR, questioning their use as a reliable automatic screening tool.
Collapse
Affiliation(s)
- Jennifer Dhont
- Department of Radiation Oncology (Maastro)GROW School for OncologyMaastricht University Medical Centre+Maastrichtthe Netherlands
| | - Cecile Wolfs
- Department of Radiation Oncology (Maastro)GROW School for OncologyMaastricht University Medical Centre+Maastrichtthe Netherlands
| | - Frank Verhaegen
- Department of Radiation Oncology (Maastro)GROW School for OncologyMaastricht University Medical Centre+Maastrichtthe Netherlands
| |
Collapse
|
6
|
Bouchareb Y, Moradi Khaniabadi P, Al Kindi F, Al Dhuhli H, Shiri I, Zaidi H, Rahmim A. Artificial intelligence-driven assessment of radiological images for COVID-19. Comput Biol Med 2021; 136:104665. [PMID: 34343890 PMCID: PMC8291996 DOI: 10.1016/j.compbiomed.2021.104665] [Citation(s) in RCA: 32] [Impact Index Per Article: 10.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2021] [Revised: 07/11/2021] [Accepted: 07/17/2021] [Indexed: 12/24/2022]
Abstract
Artificial Intelligence (AI) methods have significant potential for diagnosis and prognosis of COVID-19 infections. Rapid identification of COVID-19 and its severity in individual patients is expected to enable better control of the disease individually and at-large. There has been remarkable interest by the scientific community in using imaging biomarkers to improve detection and management of COVID-19. Exploratory tools such as AI-based models may help explain the complex biological mechanisms and provide better understanding of the underlying pathophysiological processes. The present review focuses on AI-based COVID-19 studies as applies to chest x-ray (CXR) and computed tomography (CT) imaging modalities, and the associated challenges. Explicit radiomics, deep learning methods, and hybrid methods that combine both deep learning and explicit radiomics have the potential to enhance the ability and usefulness of radiological images to assist clinicians in the current COVID-19 pandemic. The aims of this review are: first, to outline COVID-19 AI-analysis workflows, including acquisition of data, feature selection, segmentation methods, feature extraction, and multi-variate model development and validation as appropriate for AI-based COVID-19 studies. Secondly, existing limitations of AI-based COVID-19 analyses are discussed, highlighting potential improvements that can be made. Finally, the impact of AI and radiomics methods and the associated clinical outcomes are summarized. In this review, pipelines that include the key steps for AI-based COVID-19 signatures identification are elaborated. Sample size, non-standard imaging protocols, segmentation, availability of public COVID-19 databases, combination of imaging and clinical information and full clinical validation remain major limitations and challenges. We conclude that AI-based assessment of CXR and CT images has significant potential as a viable pathway for the diagnosis, follow-up and prognosis of COVID-19.
Collapse
Affiliation(s)
- Yassine Bouchareb
- Department of Radiology and Molecular Imaging, College of Medicine and Health Science, Sultan Qaboos University, PO. Box 35, Al Khod, Muscat, 123, Oman.
| | - Pegah Moradi Khaniabadi
- Department of Radiology and Molecular Imaging, College of Medicine and Health Science, Sultan Qaboos University, PO. Box 35, Al Khod, Muscat, 123, Oman.
| | | | - Humoud Al Dhuhli
- Department of Radiology and Molecular Imaging, College of Medicine and Health Science, Sultan Qaboos University, PO. Box 35, Al Khod, Muscat, 123, Oman
| | - Isaac Shiri
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, CH-1211 Geneva 4, Switzerland
| | - Habib Zaidi
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, CH-1211 Geneva 4, Switzerland; Geneva University Neurocenter, Geneva University, Geneva, Switzerland; Department of Nuclear Medicine and Molecular Imaging, University of Groningen, University Medical Center Groningen, Groningen, Netherlands; Department of Nuclear Medicine, University of Southern Denmark, Odense, Denmark
| | - Arman Rahmim
- Departments of Radiology and Physics, University of British Columbia, Vancouver, BC, Canada; Department of Integrative Oncology, BC Cancer Research Institute, Vancouver, BC, Canada
| |
Collapse
|
7
|
Chan JH, Li C. Learning from imbalanced COVID-19 chest X-ray (CXR) medical imaging data. Methods 2021; 202:31-39. [PMID: 34090971 DOI: 10.1016/j.ymeth.2021.06.002] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/02/2021] [Revised: 05/07/2021] [Accepted: 06/01/2021] [Indexed: 12/24/2022] Open
Abstract
The trendy task of digital medical image analysis has been continually evolving. It has been an area of prominent and growing importance from both research and deployment perspectives. Nonetheless, it is necessary to realize that the use of algorithms, methodology, as well as the source of medical image data, must be strictly scrutinized. As the COVID-19 pandemic has been gripping much of the world recently, there has been much efforts gone into developing affordable testing for the masses, and it has been shown that the established and widely available chest X-rays (CXR) images may be used as a screening criteria for assistive diagnosis purpose. Thanks to the dedicated work by various individuals and organizations, publicly available CXR of COVID-19 subjects are available for analytic usage. We have also provided a publicly available CXR dataset on the Kaggle platform. As a case study, this paper presents a systematic approach to learn from a typically imbalanced set of CXR images, which consists of a limited number of publicly available COVID-19 images. Our results show that we are able to outperform the top finishers in a related Kaggle multi-class CXR challenge. The proposed methodology should be able to help guide medical personnel in obtaining a robust diagnosis model to discern COVID-19 from other conditions confidently.
Collapse
Affiliation(s)
- Jonathan H Chan
- Innovative Cognitive Computing (IC2) Research Center, School of Information Technology, King Mongkut's University of Technology Thonburi, Bangkok, Thailand.
| | - Chenqi Li
- Division of Engineering Science, Faculty of Applied Science and Engineering, University of Toronto, Toronto, Canada.
| |
Collapse
|
8
|
Yao JC, Wang T, Hou GH, Ou D, Li W, Zhu QD, Chen WC, Yang C, Wang LJ, Wang LP, Fan LY, Shi KY, Zhang J, Xu D, Li YQ. AI detection of mild COVID-19 pneumonia from chest CT scans. Eur Radiol 2021; 31:7192-7201. [PMID: 33738595 PMCID: PMC7971359 DOI: 10.1007/s00330-021-07797-x] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2020] [Revised: 01/11/2021] [Accepted: 02/16/2021] [Indexed: 11/12/2022]
Abstract
Objectives An artificial intelligence model was adopted to identify mild COVID-19 pneumonia from computed tomography (CT) volumes, and its diagnostic performance was then evaluated. Methods In this retrospective multicenter study, an atrous convolution-based deep learning model was established for the computer-assisted diagnosis of mild COVID-19 pneumonia. The dataset included 2087 chest CT exams collected from four hospitals between 1 January 2019 and 31 May 2020. The true positive rate, true negative rate, receiver operating characteristic curve, area under the curve (AUC) and convolutional feature map were used to evaluate the model. Results The proposed deep learning model was trained on 1538 patients and tested on an independent testing cohort of 549 patients. The overall sensitivity was 91.5% (195/213; p < 0.001, 95% CI: 89.2–93.9%), the overall specificity was 90.5% (304/336; p < 0.001, 95% CI: 88.0–92.9%) and the general AUC value was 0.955 (p < 0.001). Conclusions A deep learning model can accurately detect COVID-19 and serve as an important supplement to the COVID-19 reverse transcription–polymerase chain reaction (RT-PCR) test. Key Points • The implementation of a deep learning model to identify mild COVID-19 pneumonia was confirmed to be effective and feasible. • The strategy of using a binary code instead of the region of interest label to identify mild COVID-19 pneumonia was verified. • This AI model can assist in the early screening of COVID-19 without interfering with normal clinical examinations. Supplementary Information The online version contains supplementary material available at 10.1007/s00330-021-07797-x.
Collapse
Affiliation(s)
- Jin-Cao Yao
- Cancer Hospital of the University of Chinese Academy of Sciences (Zhejiang Cancer Hospital), No. 1 of East Banshan Road, Hangzhou, Zhejiang Province, China.,Institute of Basic Medicine and Cancer (IBMC), Chinese Academy of Sciences, Hangzhou, China
| | - Tao Wang
- Department of Respiratory and Critical Care Medicine, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, China
| | - Guang-Hua Hou
- Department of Infection Medicine, Huangpi People's Hospital of Jianghan University, Wuhan, China
| | - Di Ou
- Cancer Hospital of the University of Chinese Academy of Sciences (Zhejiang Cancer Hospital), No. 1 of East Banshan Road, Hangzhou, Zhejiang Province, China.,Institute of Basic Medicine and Cancer (IBMC), Chinese Academy of Sciences, Hangzhou, China
| | - Wei Li
- Cancer Hospital of the University of Chinese Academy of Sciences (Zhejiang Cancer Hospital), No. 1 of East Banshan Road, Hangzhou, Zhejiang Province, China.,Institute of Basic Medicine and Cancer (IBMC), Chinese Academy of Sciences, Hangzhou, China
| | - Qiao-Dan Zhu
- Cancer Hospital of the University of Chinese Academy of Sciences (Zhejiang Cancer Hospital), No. 1 of East Banshan Road, Hangzhou, Zhejiang Province, China.,Institute of Basic Medicine and Cancer (IBMC), Chinese Academy of Sciences, Hangzhou, China
| | - Wen-Cong Chen
- Department of Biostatistics, Vanderbilt University Medical Center, Nashville, USA
| | - Chen Yang
- Cancer Hospital of the University of Chinese Academy of Sciences (Zhejiang Cancer Hospital), No. 1 of East Banshan Road, Hangzhou, Zhejiang Province, China.,Institute of Basic Medicine and Cancer (IBMC), Chinese Academy of Sciences, Hangzhou, China
| | - Li-Jing Wang
- Cancer Hospital of the University of Chinese Academy of Sciences (Zhejiang Cancer Hospital), No. 1 of East Banshan Road, Hangzhou, Zhejiang Province, China.,Institute of Basic Medicine and Cancer (IBMC), Chinese Academy of Sciences, Hangzhou, China
| | - Li-Ping Wang
- Cancer Hospital of the University of Chinese Academy of Sciences (Zhejiang Cancer Hospital), No. 1 of East Banshan Road, Hangzhou, Zhejiang Province, China.,Institute of Basic Medicine and Cancer (IBMC), Chinese Academy of Sciences, Hangzhou, China
| | - Lin-Yin Fan
- Cancer Hospital of the University of Chinese Academy of Sciences (Zhejiang Cancer Hospital), No. 1 of East Banshan Road, Hangzhou, Zhejiang Province, China.,Institute of Basic Medicine and Cancer (IBMC), Chinese Academy of Sciences, Hangzhou, China
| | - Kai-Yuan Shi
- Cancer Hospital of the University of Chinese Academy of Sciences (Zhejiang Cancer Hospital), No. 1 of East Banshan Road, Hangzhou, Zhejiang Province, China.,Institute of Basic Medicine and Cancer (IBMC), Chinese Academy of Sciences, Hangzhou, China
| | - Jie Zhang
- Cancer Hospital of the University of Chinese Academy of Sciences (Zhejiang Cancer Hospital), No. 1 of East Banshan Road, Hangzhou, Zhejiang Province, China.,Institute of Basic Medicine and Cancer (IBMC), Chinese Academy of Sciences, Hangzhou, China
| | - Dong Xu
- Cancer Hospital of the University of Chinese Academy of Sciences (Zhejiang Cancer Hospital), No. 1 of East Banshan Road, Hangzhou, Zhejiang Province, China. .,Institute of Basic Medicine and Cancer (IBMC), Chinese Academy of Sciences, Hangzhou, China.
| | - Ya-Qing Li
- Cancer Hospital of the University of Chinese Academy of Sciences (Zhejiang Cancer Hospital), No. 1 of East Banshan Road, Hangzhou, Zhejiang Province, China. .,Institute of Basic Medicine and Cancer (IBMC), Chinese Academy of Sciences, Hangzhou, China. .,Department of Respiratory Medicine, Zhejiang Provincial People's Hospital, Hangzhou, China.
| |
Collapse
|