1
|
Fayyad J, Alijani S, Najjaran H. Empirical validation of Conformal Prediction for trustworthy skin lesions classification. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 253:108231. [PMID: 38820714 DOI: 10.1016/j.cmpb.2024.108231] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/17/2024] [Revised: 03/15/2024] [Accepted: 05/14/2024] [Indexed: 06/02/2024]
Abstract
BACKGROUND AND OBJECTIVE Uncertainty quantification is a pivotal field that contributes to realizing reliable and robust systems. It becomes instrumental in fortifying safe decisions by providing complementary information, particularly within high-risk applications. existing studies have explored various methods that often operate under specific assumptions or necessitate substantial modifications to the network architecture to effectively account for uncertainties. The objective of this paper is to study Conformal Prediction, an emerging distribution-free uncertainty quantification technique, and provide a comprehensive understanding of the advantages and limitations inherent in various methods within the medical imaging field. METHODS In this study, we developed Conformal Prediction, Monte Carlo Dropout, and Evidential Deep Learning approaches to assess uncertainty quantification in deep neural networks. The effectiveness of these methods is evaluated using three public medical imaging datasets focused on detecting pigmented skin lesions and blood cell types. RESULTS The experimental results demonstrate a significant enhancement in uncertainty quantification with the utilization of the Conformal Prediction method, surpassing the performance of the other two methods. Furthermore, the results present insights into the effectiveness of each uncertainty method in handling Out-of-Distribution samples from domain-shifted datasets. Our code is available at: github.com/jfayyad/ConformalDx. CONCLUSIONS Our conclusion highlights a robust and consistent performance of conformal prediction across diverse testing conditions. This positions it as the preferred choice for decision-making in safety-critical applications.
Collapse
Affiliation(s)
- Jamil Fayyad
- University of Victoria, 800 Finnerty Road, Victoria, V8P 5C2, BC, Canada; Cognia AI, 2031 Store street, Victoria, V8T 5L9, BC, Canada.
| | - Shadi Alijani
- University of Victoria, 800 Finnerty Road, Victoria, V8P 5C2, BC, Canada; Cognia AI, 2031 Store street, Victoria, V8T 5L9, BC, Canada.
| | - Homayoun Najjaran
- University of Victoria, 800 Finnerty Road, Victoria, V8P 5C2, BC, Canada; Cognia AI, 2031 Store street, Victoria, V8T 5L9, BC, Canada.
| |
Collapse
|
2
|
Schott B, Pinchuk D, Santoro-Fernandes V, Klaneček Ž, Rivetti L, Deatsch A, Perlman S, Li Y, Jeraj R. Uncertainty quantification via localized gradients for deep learning-based medical image assessments. Phys Med Biol 2024; 69:155015. [PMID: 38981594 DOI: 10.1088/1361-6560/ad611d] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/12/2024] [Accepted: 07/09/2024] [Indexed: 07/11/2024]
Abstract
Objective.Deep learning models that aid in medical image assessment tasks must be both accurate and reliable to be deployed within clinical settings. While deep learning models have been shown to be highly accurate across a variety of tasks, measures that indicate the reliability of these models are less established. Increasingly, uncertainty quantification (UQ) methods are being introduced to inform users on the reliability of model outputs. However, most existing methods cannot be augmented to previously validated models because they are not post hoc, and they change a model's output. In this work, we overcome these limitations by introducing a novel post hoc UQ method, termedLocal Gradients UQ, and demonstrate its utility for deep learning-based metastatic disease delineation.Approach.This method leverages a trained model's localized gradient space to assess sensitivities to trained model parameters. We compared the Local Gradients UQ method to non-gradient measures defined using model probability outputs. The performance of each uncertainty measure was assessed in four clinically relevant experiments: (1) response to artificially degraded image quality, (2) comparison between matched high- and low-quality clinical images, (3) false positive (FP) filtering, and (4) correspondence with physician-rated disease likelihood.Main results.(1) Response to artificially degraded image quality was enhanced by the Local Gradients UQ method, where the median percent difference between matching lesions in non-degraded and most degraded images was consistently higher for the Local Gradients uncertainty measure than the non-gradient uncertainty measures (e.g. 62.35% vs. 2.16% for additive Gaussian noise). (2) The Local Gradients UQ measure responded better to high- and low-quality clinical images (p< 0.05 vsp> 0.1 for both non-gradient uncertainty measures). (3) FP filtering performance was enhanced by the Local Gradients UQ method when compared to the non-gradient methods, increasing the area under the receiver operating characteristic curve (ROC AUC) by 20.1% and decreasing the false positive rate by 26%. (4) The Local Gradients UQ method also showed more favorable correspondence with physician-rated likelihood for malignant lesions by increasing ROC AUC for correspondence with physician-rated disease likelihood by 16.2%.Significance. In summary, this work introduces and validates a novel gradient-based UQ method for deep learning-based medical image assessments to enhance user trust when using deployed clinical models.
Collapse
Affiliation(s)
- Brayden Schott
- Department of Medical Physics, School of Medicine and Public Health, University of Wisconsin, Madison, WI, United States of America
| | - Dmitry Pinchuk
- Department of Medical Physics, School of Medicine and Public Health, University of Wisconsin, Madison, WI, United States of America
| | - Victor Santoro-Fernandes
- Department of Medical Physics, School of Medicine and Public Health, University of Wisconsin, Madison, WI, United States of America
| | - Žan Klaneček
- Faculty of Mathematics and Physics, University of Ljubljana, Ljubljana, Slovenia
| | - Luciano Rivetti
- Faculty of Mathematics and Physics, University of Ljubljana, Ljubljana, Slovenia
| | - Alison Deatsch
- Department of Medical Physics, School of Medicine and Public Health, University of Wisconsin, Madison, WI, United States of America
| | - Scott Perlman
- Department of Radiology, Section of Nuclear Medicine, School of Medicine and Public Health, University of Wisconsin, Madison, WI, United States of America
| | - Yixuan Li
- Department of Computer Sciences, School of Computer, Data, & Information Sciences, University of Wisconsin, Madison, WI, United States of America
| | - Robert Jeraj
- Department of Medical Physics, School of Medicine and Public Health, University of Wisconsin, Madison, WI, United States of America
- Faculty of Mathematics and Physics, University of Ljubljana, Ljubljana, Slovenia
| |
Collapse
|
3
|
Luosang G, Wang Z, Liu J, Zeng F, Yi Z, Wang J. Automated Quality Assessment of Medical Images in Echocardiography Using Neural Networks with Adaptive Ranking and Structure-Aware Learning. Int J Neural Syst 2024:2450054. [PMID: 38984421 DOI: 10.1142/s0129065724500540] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/11/2024]
Abstract
The quality of medical images is crucial for accurately diagnosing and treating various diseases. However, current automated methods for assessing image quality are based on neural networks, which often focus solely on pixel distortion and overlook the significance of complex structures within the images. This study introduces a novel neural network model designed explicitly for automated image quality assessment that addresses pixel and semantic distortion. The model introduces an adaptive ranking mechanism enhanced with contrast sensitivity weighting to refine the detection of minor variances in similar images for pixel distortion assessment. More significantly, the model integrates a structure-aware learning module employing graph neural networks. This module is adept at deciphering the intricate relationships between an image's semantic structure and quality. When evaluated on two ultrasound imaging datasets, the proposed method outshines existing leading models in performance. Additionally, it boasts seamless integration into clinical workflows, enabling real-time image quality assessment, crucial for precise disease diagnosis and treatment.
Collapse
Affiliation(s)
- Gadeng Luosang
- Machine Intelligence Laboratory, College of Computer Science, Sichuan University, Chengdu 610065, P. R. China
- College of Information Science and Technology, Tibet University, Lhasa 850000, P. R. China
| | - Zhihua Wang
- College of Computer Science and Technology, Zhejiang University, Hangzhou 310058, P. R. China
- Anhui Kunlong Kangxin Medical, Technology Company Limited, Anhui 230000, P. R. China
| | - Jian Liu
- Department of Ultrasound, Clinical Medical College, The First Affiliated Hospital of Chengdu Medical College, Chengdu 610599, P. R. China
| | - Fanxin Zeng
- Department of Clinical Research Center, Dazhou Central Hospital, Sichuan 635099, P. R. China
| | - Zhang Yi
- Machine Intelligence Laboratory, College of Computer Science, Sichuan University, Chengdu 610065, P. R. China
| | - Jianyong Wang
- Machine Intelligence Laboratory, College of Computer Science, Sichuan University, Chengdu 610065, P. R. China
| |
Collapse
|
4
|
Wang H, Jin Q, Li S, Liu S, Wang M, Song Z. A comprehensive survey on deep active learning in medical image analysis. Med Image Anal 2024; 95:103201. [PMID: 38776841 DOI: 10.1016/j.media.2024.103201] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2023] [Revised: 04/25/2024] [Accepted: 05/06/2024] [Indexed: 05/25/2024]
Abstract
Deep learning has achieved widespread success in medical image analysis, leading to an increasing demand for large-scale expert-annotated medical image datasets. Yet, the high cost of annotating medical images severely hampers the development of deep learning in this field. To reduce annotation costs, active learning aims to select the most informative samples for annotation and train high-performance models with as few labeled samples as possible. In this survey, we review the core methods of active learning, including the evaluation of informativeness and sampling strategy. For the first time, we provide a detailed summary of the integration of active learning with other label-efficient techniques, such as semi-supervised, self-supervised learning, and so on. We also summarize active learning works that are specifically tailored to medical image analysis. Additionally, we conduct a thorough comparative analysis of the performance of different AL methods in medical image analysis with experiments. In the end, we offer our perspectives on the future trends and challenges of active learning and its applications in medical image analysis. An accompanying paper list and code for the comparative analysis is available in https://github.com/LightersWang/Awesome-Active-Learning-for-Medical-Image-Analysis.
Collapse
Affiliation(s)
- Haoran Wang
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai 200032, China; Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, China
| | - Qiuye Jin
- Computational Bioscience Research Center (CBRC), King Abdullah University of Science and Technology (KAUST), Thuwal 23955, Saudi Arabia
| | - Shiman Li
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai 200032, China; Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, China
| | - Siyu Liu
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai 200032, China; Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, China
| | - Manning Wang
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai 200032, China; Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, China.
| | - Zhijian Song
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai 200032, China; Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, China.
| |
Collapse
|
5
|
Huang L, Ruan S, Xing Y, Feng M. A review of uncertainty quantification in medical image analysis: Probabilistic and non-probabilistic methods. Med Image Anal 2024; 97:103223. [PMID: 38861770 DOI: 10.1016/j.media.2024.103223] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/09/2023] [Revised: 03/16/2024] [Accepted: 05/27/2024] [Indexed: 06/13/2024]
Abstract
The comprehensive integration of machine learning healthcare models within clinical practice remains suboptimal, notwithstanding the proliferation of high-performing solutions reported in the literature. A predominant factor hindering widespread adoption pertains to an insufficiency of evidence affirming the reliability of the aforementioned models. Recently, uncertainty quantification methods have been proposed as a potential solution to quantify the reliability of machine learning models and thus increase the interpretability and acceptability of the results. In this review, we offer a comprehensive overview of the prevailing methods proposed to quantify the uncertainty inherent in machine learning models developed for various medical image tasks. Contrary to earlier reviews that exclusively focused on probabilistic methods, this review also explores non-probabilistic approaches, thereby furnishing a more holistic survey of research pertaining to uncertainty quantification for machine learning models. Analysis of medical images with the summary and discussion on medical applications and the corresponding uncertainty evaluation protocols are presented, which focus on the specific challenges of uncertainty in medical image analysis. We also highlight some potential future research work at the end. Generally, this review aims to allow researchers from both clinical and technical backgrounds to gain a quick and yet in-depth understanding of the research in uncertainty quantification for medical image analysis machine learning models.
Collapse
Affiliation(s)
- Ling Huang
- Saw Swee Hock School of Public Health, National University of Singapore, Singapore
| | - Su Ruan
- Quantif, LITIS, University of Rouen Normandy, France.
| | - Yucheng Xing
- Saw Swee Hock School of Public Health, National University of Singapore, Singapore
| | - Mengling Feng
- Saw Swee Hock School of Public Health, National University of Singapore, Singapore; Institute of Data Science, National University of Singapore, Singapore
| |
Collapse
|
6
|
Lambert B, Forbes F, Doyle S, Dehaene H, Dojat M. Trustworthy clinical AI solutions: A unified review of uncertainty quantification in Deep Learning models for medical image analysis. Artif Intell Med 2024; 150:102830. [PMID: 38553168 DOI: 10.1016/j.artmed.2024.102830] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2023] [Revised: 02/28/2024] [Accepted: 03/01/2024] [Indexed: 04/02/2024]
Abstract
The full acceptance of Deep Learning (DL) models in the clinical field is rather low with respect to the quantity of high-performing solutions reported in the literature. End users are particularly reluctant to rely on the opaque predictions of DL models. Uncertainty quantification methods have been proposed in the literature as a potential solution, to reduce the black-box effect of DL models and increase the interpretability and the acceptability of the result by the final user. In this review, we propose an overview of the existing methods to quantify uncertainty associated with DL predictions. We focus on applications to medical image analysis, which present specific challenges due to the high dimensionality of images and their variable quality, as well as constraints associated with real-world clinical routine. Moreover, we discuss the concept of structural uncertainty, a corpus of methods to facilitate the alignment of segmentation uncertainty estimates with clinical attention. We then discuss the evaluation protocols to validate the relevance of uncertainty estimates. Finally, we highlight the open challenges for uncertainty quantification in the medical field.
Collapse
Affiliation(s)
- Benjamin Lambert
- Univ. Grenoble Alpes, Inserm, U1216, Grenoble Institut des Neurosciences, Grenoble, 38000, France; Pixyl Research and Development Laboratory, Grenoble, 38000, France
| | - Florence Forbes
- Univ. Grenoble Alpes, Inria, CNRS, Grenoble INP, LJK, Grenoble, 38000, France
| | - Senan Doyle
- Pixyl Research and Development Laboratory, Grenoble, 38000, France
| | - Harmonie Dehaene
- Pixyl Research and Development Laboratory, Grenoble, 38000, France
| | - Michel Dojat
- Univ. Grenoble Alpes, Inserm, U1216, Grenoble Institut des Neurosciences, Grenoble, 38000, France.
| |
Collapse
|
7
|
Fernandez-Martín C, Silva-Rodriguez J, Kiraz U, Morales S, Janssen EAM, Naranjo V. Uninformed Teacher-Student for hard-samples distillation in weakly supervised mitosis localization. Comput Med Imaging Graph 2024; 112:102328. [PMID: 38244279 DOI: 10.1016/j.compmedimag.2024.102328] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/12/2023] [Revised: 11/02/2023] [Accepted: 12/12/2023] [Indexed: 01/22/2024]
Abstract
BACKGROUND AND OBJECTIVE Mitotic activity is a crucial biomarker for diagnosing and predicting outcomes for different types of cancers, particularly breast cancer. However, manual mitosis counting is challenging and time-consuming for pathologists, with moderate reproducibility due to biopsy slide size, low mitotic cell density, and pattern heterogeneity. In recent years, deep learning methods based on convolutional neural networks (CNNs) have been proposed to address these limitations. Nonetheless, these methods have been hampered by the available data labels, which usually consist only of the centroids of mitosis, and by the incoming noise from annotated hard negatives. As a result, complex algorithms with multiple stages are often required to refine the labels at the pixel level and reduce the number of false positives. METHODS This article presents a novel weakly supervised approach for mitosis detection that utilizes only image-level labels on histological hematoxylin and eosin (H&E) images, avoiding the need for complex labeling scenarios. Also, an Uninformed Teacher-Student (UTS) pipeline is introduced to detect and distill hard samples by comparing weakly supervised localizations and the annotated centroids, using strong augmentations to enhance uncertainty. Additionally, an automatic proliferation score is proposed that mimicks the pathologist-annotated mitotic activity index (MAI). The proposed approach is evaluated on three publicly available datasets for mitosis detection on breast histology samples, and two datasets for mitotic activity counting in whole-slide images. RESULTS The proposed framework achieves competitive performance with relevant prior literature in all the datasets used for evaluation without explicitly using the mitosis location information during training. This approach challenges previous methods that rely on strong mitosis location information and multiple stages to refine false positives. Furthermore, the proposed pipeline for hard-sample distillation demonstrates promising dataset-specific improvements. Concretely, when the annotation has not been thoroughly refined by multiple pathologists, the UTS model offers improvements of up to ∼4% in mitosis localization, thanks to the detection and distillation of uncertain cases. Concerning the mitosis counting task, the proposed automatic proliferation score shows a moderate positive correlation with the MAI annotated by pathologists at the biopsy level on two external datasets. CONCLUSIONS The proposed Uninformed Teacher-Student pipeline leverages strong augmentations to distill uncertain samples and measure dissimilarities between predicted and annotated mitosis. Results demonstrate the feasibility of the weakly supervised approach and highlight its potential as an objective evaluation tool for tumor proliferation.
Collapse
Affiliation(s)
- Claudio Fernandez-Martín
- Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, HUMAN-tech, Universitat Politècnica de València, Valencia, Spain.
| | | | - Umay Kiraz
- Department of Chemistry, Bioscience and Environmental Engineering, University of Stavanger, Stavanger, Norway; Department of Pathology, Stavanger University Hospital, Stavanger, Norway
| | - Sandra Morales
- Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, HUMAN-tech, Universitat Politècnica de València, Valencia, Spain
| | - Emiel A M Janssen
- Department of Chemistry, Bioscience and Environmental Engineering, University of Stavanger, Stavanger, Norway; Department of Pathology, Stavanger University Hospital, Stavanger, Norway
| | - Valery Naranjo
- Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, HUMAN-tech, Universitat Politècnica de València, Valencia, Spain
| |
Collapse
|
8
|
Yue G, Zhuo G, Yan W, Zhou T, Tang C, Yang P, Wang T. Boundary uncertainty aware network for automated polyp segmentation. Neural Netw 2024; 170:390-404. [PMID: 38029720 DOI: 10.1016/j.neunet.2023.11.050] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/20/2023] [Revised: 07/15/2023] [Accepted: 11/22/2023] [Indexed: 12/01/2023]
Abstract
Recently, leveraging deep neural networks for automated colorectal polyp segmentation has emerged as a hot topic due to the favored advantages in evading the limitations of visual inspection, e.g., overwork and subjectivity. However, most existing methods do not pay enough attention to the uncertain areas of colonoscopy images and often provide unsatisfactory segmentation performance. In this paper, we propose a novel boundary uncertainty aware network (BUNet) for precise and robust colorectal polyp segmentation. Specifically, considering that polyps vary greatly in size and shape, we first adopt a pyramid vision transformer encoder to learn multi-scale feature representations. Then, a simple yet effective boundary exploration module (BEM) is proposed to explore boundary cues from the low-level features. To make the network focus on the ambiguous area where the prediction score is biased to neither the foreground nor the background, we further introduce a boundary uncertainty aware module (BUM) that explores error-prone regions from the high-level features with the assistance of boundary cues provided by the BEM. Through the top-down hybrid deep supervision, our BUNet implements coarse-to-fine polyp segmentation and finally localizes polyp regions precisely. Extensive experiments on five public datasets show that BUNet is superior to thirteen competing methods in terms of both effectiveness and generalization ability.
Collapse
Affiliation(s)
- Guanghui Yue
- National-Reginoal Key Technology Engineering Laboratory for Medical Ultrasound, Guangdong Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Marshall Laboratory of Biomedical Engineering, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen 518060, China
| | - Guibin Zhuo
- National-Reginoal Key Technology Engineering Laboratory for Medical Ultrasound, Guangdong Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Marshall Laboratory of Biomedical Engineering, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen 518060, China
| | - Weiqing Yan
- School of Computer and Control Engineering, Yantai University, Yantai 264005, China
| | - Tianwei Zhou
- College of Management, Shenzhen University, Shenzhen 518060, China.
| | - Chang Tang
- School of Computer Science, China University of Geosciences, Wuhan 430074, China
| | - Peng Yang
- National-Reginoal Key Technology Engineering Laboratory for Medical Ultrasound, Guangdong Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Marshall Laboratory of Biomedical Engineering, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen 518060, China
| | - Tianfu Wang
- National-Reginoal Key Technology Engineering Laboratory for Medical Ultrasound, Guangdong Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Marshall Laboratory of Biomedical Engineering, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen 518060, China
| |
Collapse
|
9
|
Dasegowda G, Bizzo BC, Gupta RV, Kaviani P, Ebrahimian S, Ricciardelli D, Abedi-Tari F, Neumark N, Digumarthy SR, Kalra MK, Dreyer KJ. Radiologist-Trained AI Model for Identifying Suboptimal Chest-Radiographs. Acad Radiol 2023; 30:2921-2930. [PMID: 37019698 DOI: 10.1016/j.acra.2023.03.006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/09/2023] [Revised: 02/28/2023] [Accepted: 03/06/2023] [Indexed: 04/05/2023]
Abstract
RATIONALE AND OBJECTIVES Suboptimal chest radiographs (CXR) can limit interpretation of critical findings. Radiologist-trained AI models were evaluated for differentiating suboptimal(sCXR) and optimal(oCXR) chest radiographs. MATERIALS AND METHODS Our IRB-approved study included 3278 CXRs from adult patients (mean age 55 ± 20 years) identified from a retrospective search of CXR in radiology reports from 5 sites. A chest radiologist reviewed all CXRs for the cause of suboptimality. The de-identified CXRs were uploaded into an AI server application for training and testing 5 AI models. The training set consisted of 2202 CXRs (n = 807 oCXR; n = 1395 sCXR) while 1076 CXRs (n = 729 sCXR; n = 347 oCXR) were used for testing. Data were analyzed with the Area under the curve (AUC) for the model's ability to classify oCXR and sCXR correctly. RESULTS For the two-class classification into sCXR or oCXR from all sites, for CXR with missing anatomy, AI had sensitivity, specificity, accuracy, and AUC of 78%, 95%, 91%, 0.87(95% CI 0.82-0.92), respectively. AI identified obscured thoracic anatomy with 91% sensitivity, 97% specificity, 95% accuracy, and 0.94 AUC (95% CI 0.90-0.97). Inadequate exposure with 90% sensitivity, 93% specificity, 92% accuracy, and AUC of 0.91 (95% CI 0.88-0.95). The presence of low lung volume was identified with 96% sensitivity, 92% specificity, 93% accuracy, and 0.94 AUC (95% CI 0.92-0.96). The sensitivity, specificity, accuracy, and AUC of AI in identifying patient rotation were 92%, 96%, 95%, and 0.94 (95% CI 0.91-0.98), respectively. CONCLUSION The radiologist-trained AI models can accurately classify optimal and suboptimal CXRs. Such AI models at the front end of radiographic equipment can enable radiographers to repeat sCXRs when necessary.
Collapse
Affiliation(s)
- Giridhar Dasegowda
- Department of Radiology, Massachusetts General Hospital and Harvard Medical School, 75 Blossom Court, Suite 248, Boston, MA 02114; Mass General Brigham Data Science Office (DSO), 100 Cambridge St, Boston, MA, US 02114
| | - Bernardo C Bizzo
- Department of Radiology, Massachusetts General Hospital and Harvard Medical School, 75 Blossom Court, Suite 248, Boston, MA 02114; Mass General Brigham Data Science Office (DSO), 100 Cambridge St, Boston, MA, US 02114
| | - Reya V Gupta
- Department of Radiology, Massachusetts General Hospital and Harvard Medical School, 75 Blossom Court, Suite 248, Boston, MA 02114
| | - Parisa Kaviani
- Department of Radiology, Massachusetts General Hospital and Harvard Medical School, 75 Blossom Court, Suite 248, Boston, MA 02114; Mass General Brigham Data Science Office (DSO), 100 Cambridge St, Boston, MA, US 02114
| | - Shadi Ebrahimian
- Department of Radiology, Massachusetts General Hospital and Harvard Medical School, 75 Blossom Court, Suite 248, Boston, MA 02114; Mass General Brigham Data Science Office (DSO), 100 Cambridge St, Boston, MA, US 02114
| | - Debra Ricciardelli
- Department of Radiology, Massachusetts General Hospital and Harvard Medical School, 75 Blossom Court, Suite 248, Boston, MA 02114
| | - Faezeh Abedi-Tari
- Department of Radiology, Massachusetts General Hospital and Harvard Medical School, 75 Blossom Court, Suite 248, Boston, MA 02114
| | - Nir Neumark
- Mass General Brigham Data Science Office (DSO), 100 Cambridge St, Boston, MA, US 02114
| | - Subba R Digumarthy
- Department of Radiology, Massachusetts General Hospital and Harvard Medical School, 75 Blossom Court, Suite 248, Boston, MA 02114; Mass General Brigham Data Science Office (DSO), 100 Cambridge St, Boston, MA, US 02114
| | - Mannudeep K Kalra
- Department of Radiology, Massachusetts General Hospital and Harvard Medical School, 75 Blossom Court, Suite 248, Boston, MA 02114; Mass General Brigham Data Science Office (DSO), 100 Cambridge St, Boston, MA, US 02114.
| | - Keith J Dreyer
- Department of Radiology, Massachusetts General Hospital and Harvard Medical School, 75 Blossom Court, Suite 248, Boston, MA 02114; Mass General Brigham Data Science Office (DSO), 100 Cambridge St, Boston, MA, US 02114
| |
Collapse
|
10
|
Bhat S, Mansoor A, Georgescu B, Panambur AB, Ghesu FC, Islam S, Packhäuser K, Rodríguez-Salas D, Grbic S, Maier A. AUCReshaping: improved sensitivity at high-specificity. Sci Rep 2023; 13:21097. [PMID: 38036602 PMCID: PMC10689839 DOI: 10.1038/s41598-023-48482-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2023] [Accepted: 11/27/2023] [Indexed: 12/02/2023] Open
Abstract
The evaluation of deep-learning (DL) systems typically relies on the Area under the Receiver-Operating-Curve (AU-ROC) as a performance metric. However, AU-ROC, in its holistic form, does not sufficiently consider performance within specific ranges of sensitivity and specificity, which are critical for the intended operational context of the system. Consequently, two systems with identical AU-ROC values can exhibit significantly divergent real-world performance. This issue is particularly pronounced in the context of anomaly detection tasks, a commonly employed application of DL systems across various research domains, including medical imaging, industrial automation, manufacturing, cyber security, fraud detection, and drug research, among others. The challenge arises from the heavy class imbalance in training datasets, with the abnormality class often incurring a considerably higher misclassification cost compared to the normal class. Traditional DL systems address this by adjusting the weighting of the cost function or optimizing for specific points along the ROC curve. While these approaches yield reasonable results in many cases, they do not actively seek to maximize performance for the desired operating point. In this study, we introduce a novel technique known as AUCReshaping, designed to reshape the ROC curve exclusively within the specified sensitivity and specificity range, by optimizing sensitivity at a predetermined specificity level. This reshaping is achieved through an adaptive and iterative boosting mechanism that allows the network to focus on pertinent samples during the learning process. We primarily investigated the impact of AUCReshaping in the context of abnormality detection tasks, specifically in Chest X-Ray (CXR) analysis, followed by breast mammogram and credit card fraud detection tasks. The results reveal a substantial improvement, ranging from 2 to 40%, in sensitivity at high-specificity levels for binary classification tasks.
Collapse
Affiliation(s)
- Sheethal Bhat
- Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, 91058, Erlangen, Germany.
- Digital Technology and Innovation, Siemens Healthineers, Erlangen, Germany.
| | - Awais Mansoor
- Digital Technology and Innovation, Siemens Medical Solutions, Princeton, NJ, 08540, USA
| | - Bogdan Georgescu
- Digital Technology and Innovation, Siemens Medical Solutions, Princeton, NJ, 08540, USA
| | - Adarsh B Panambur
- Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, 91058, Erlangen, Germany
- Digital Technology and Innovation, Siemens Healthineers, Erlangen, Germany
| | - Florin C Ghesu
- Digital Technology and Innovation, Siemens Medical Solutions, Princeton, NJ, 08540, USA
| | - Saahil Islam
- Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, 91058, Erlangen, Germany
- Digital Technology and Innovation, Siemens Healthineers, Erlangen, Germany
| | - Kai Packhäuser
- Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, 91058, Erlangen, Germany
| | - Dalia Rodríguez-Salas
- Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, 91058, Erlangen, Germany
| | - Sasa Grbic
- Digital Technology and Innovation, Siemens Medical Solutions, Princeton, NJ, 08540, USA
| | - Andreas Maier
- Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, 91058, Erlangen, Germany
| |
Collapse
|
11
|
Pepe A, Egger J, Codari M, Willemink MJ, Gsaxner C, Li J, Roth PM, Schmalstieg D, Mistelbauer G, Fleischmann D. Automated cross-sectional view selection in CT angiography of aortic dissections with uncertainty awareness and retrospective clinical annotations. Comput Biol Med 2023; 165:107365. [PMID: 37647783 DOI: 10.1016/j.compbiomed.2023.107365] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/09/2023] [Revised: 07/20/2023] [Accepted: 08/12/2023] [Indexed: 09/01/2023]
Abstract
Surveillance imaging of patients with chronic aortic diseases, such as aneurysms and dissections, relies on obtaining and comparing cross-sectional diameter measurements along the aorta at predefined aortic landmarks, over time. The orientation of the cross-sectional measuring planes at each landmark is currently defined manually by highly trained operators. Centerline-based approaches are unreliable in patients with chronic aortic dissection, because of the asymmetric flow channels, differences in contrast opacification, and presence of mural thrombus, making centerline computations or measurements difficult to generate and reproduce. In this work, we present three alternative approaches - INS, MCDS, MCDbS - based on convolutional neural networks and uncertainty quantification methods to predict the orientation (ϕ,θ) of such cross-sectional planes. For the monitoring of chronic aortic dissections, we show how a dataset of 162 CTA volumes with overall 3273 imperfect manual annotations routinely collected in a clinic can be efficiently used to accomplish this task, despite the presence of non-negligible interoperator variabilities in terms of mean absolute error (MAE) and 95% limits of agreement (LOA). We show how, despite the large limits of agreement in the training data, the trained model provides faster and more reproducible results than either an expert user or a centerline method. The remaining disagreement lies within the variability produced by three independent expert annotators and matches the current state of the art, providing a similar error, but in a fraction of the time.
Collapse
Affiliation(s)
- Antonio Pepe
- Graz University of Technology, Institute of Computer Graphics and Vision, Inffeldgasse 16/II, 8010 Graz, Austria; Stanford University, School of Medicine, 3D and Quantitative Imaging Lab, 300 Pasteur Drive Stanford, CA 94305, USA; Computer Algorithms for Médicine (Café) Laboratory, Graz, Austria.
| | - Jan Egger
- Computer Algorithms for Médicine (Café) Laboratory, Graz, Austria; University Medicine Essen, Institute for AI in Medicine (IKIM), Girardetstraße 2, 45131 Essen, Germany.
| | - Marina Codari
- Stanford University, School of Medicine, 3D and Quantitative Imaging Lab, 300 Pasteur Drive Stanford, CA 94305, USA.
| | - Martin J Willemink
- Stanford University, School of Medicine, 3D and Quantitative Imaging Lab, 300 Pasteur Drive Stanford, CA 94305, USA.
| | - Christina Gsaxner
- Graz University of Technology, Institute of Computer Graphics and Vision, Inffeldgasse 16/II, 8010 Graz, Austria; Computer Algorithms for Médicine (Café) Laboratory, Graz, Austria.
| | - Jianning Li
- Computer Algorithms for Médicine (Café) Laboratory, Graz, Austria; University Medicine Essen, Institute for AI in Medicine (IKIM), Girardetstraße 2, 45131 Essen, Germany.
| | - Peter M Roth
- Graz University of Technology, Institute of Computer Graphics and Vision, Inffeldgasse 16/II, 8010 Graz, Austria.
| | - Dieter Schmalstieg
- Graz University of Technology, Institute of Computer Graphics and Vision, Inffeldgasse 16/II, 8010 Graz, Austria.
| | - Gabriel Mistelbauer
- Stanford University, School of Medicine, 3D and Quantitative Imaging Lab, 300 Pasteur Drive Stanford, CA 94305, USA.
| | - Dominik Fleischmann
- Stanford University, School of Medicine, 3D and Quantitative Imaging Lab, 300 Pasteur Drive Stanford, CA 94305, USA.
| |
Collapse
|
12
|
Chen X, Wang X, Lv J, Qin G, Zhou Z. An integrated network based on 2D/3D feature correlations for benign-malignant tumor classification and uncertainty estimation in digital breast tomosynthesis. Phys Med Biol 2023; 68:175046. [PMID: 37582379 DOI: 10.1088/1361-6560/acf092] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/12/2023] [Accepted: 08/15/2023] [Indexed: 08/17/2023]
Abstract
Objective.Classification of benign and malignant tumors is important for the early diagnosis of breast cancer. Over the last decade, digital breast tomosynthesis (DBT) has gradually become an effective imaging modality for breast cancer diagnosis due to its ability to generate three-dimensional (3D) visualizations. However, computer-aided diagnosis (CAD) systems based on 3D images require high computational costs and time. Furthermore, there is considerable redundant information in 3D images. Most CAD systems are designed based on 2D images, which may lose the spatial depth information of tumors. In this study, we propose a 2D/3D integrated network for the diagnosis of benign and malignant breast tumors.Approach.We introduce a correlation strategy to describe feature correlations between slices in 3D volumes, corresponding to the tissue relationship and spatial depth features of tumors. The correlation strategy can be used to extract spatial features with little computational cost. In the prediction stage, 3D spatial correlation features and 2D features are both used for classification.Main results.Experimental results demonstrate that our proposed framework achieves higher accuracy and reliability than pure 2D or 3D models. Our framework has a high area under the curve of 0.88 and accuracy of 0.82. The parameter size of the feature extractor in our framework is only 35% of that of the 3D models. In reliability evaluations, our proposed model is more reliable than pure 2D or 3D models because of its effective and nonredundant features.Significance.This study successfully combines 3D spatial correlation features and 2D features for the diagnosis of benign and malignant breast tumors in DBT. In addition to high accuracy and low computational cost, our model is more reliable and can output uncertainty value. From this point of view, the proposed method has the potential to be applied in clinic.
Collapse
Affiliation(s)
- Xi Chen
- School of Information and Communications Engineering, Xi'an Jiaotong University, Xi'an, 710049, Shaanxi, People's Republic of China
| | - Xiaoyu Wang
- School of Information and Communications Engineering, Xi'an Jiaotong University, Xi'an, 710049, Shaanxi, People's Republic of China
| | - Jiahuan Lv
- School of Information and Communications Engineering, Xi'an Jiaotong University, Xi'an, 710049, Shaanxi, People's Republic of China
| | - Genggeng Qin
- Department of Radiology, Nanfang Hospital, Southern Medical University, Guangzhou, 510515, Guangdong, People's Republic of China
| | - Zhiguo Zhou
- Department of Biostatistics and Data Science, University of Kansas Medical Center, Kansas City, KS-66160, United States of America
| |
Collapse
|
13
|
Dubey AK, Chabert GL, Carriero A, Pasche A, Danna PSC, Agarwal S, Mohanty L, Sharma N, Yadav S, Jain A, Kumar A, Kalra MK, Sobel DW, Laird JR, Singh IM, Singh N, Tsoulfas G, Fouda MM, Alizad A, Kitas GD, Khanna NN, Viskovic K, Kukuljan M, Al-Maini M, El-Baz A, Saba L, Suri JS. Ensemble Deep Learning Derived from Transfer Learning for Classification of COVID-19 Patients on Hybrid Deep-Learning-Based Lung Segmentation: A Data Augmentation and Balancing Framework. Diagnostics (Basel) 2023; 13:diagnostics13111954. [PMID: 37296806 DOI: 10.3390/diagnostics13111954] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/04/2023] [Revised: 05/22/2023] [Accepted: 05/29/2023] [Indexed: 06/12/2023] Open
Abstract
BACKGROUND AND MOTIVATION Lung computed tomography (CT) techniques are high-resolution and are well adopted in the intensive care unit (ICU) for COVID-19 disease control classification. Most artificial intelligence (AI) systems do not undergo generalization and are typically overfitted. Such trained AI systems are not practical for clinical settings and therefore do not give accurate results when executed on unseen data sets. We hypothesize that ensemble deep learning (EDL) is superior to deep transfer learning (TL) in both non-augmented and augmented frameworks. METHODOLOGY The system consists of a cascade of quality control, ResNet-UNet-based hybrid deep learning for lung segmentation, and seven models using TL-based classification followed by five types of EDL's. To prove our hypothesis, five different kinds of data combinations (DC) were designed using a combination of two multicenter cohorts-Croatia (80 COVID) and Italy (72 COVID and 30 controls)-leading to 12,000 CT slices. As part of generalization, the system was tested on unseen data and statistically tested for reliability/stability. RESULTS Using the K5 (80:20) cross-validation protocol on the balanced and augmented dataset, the five DC datasets improved TL mean accuracy by 3.32%, 6.56%, 12.96%, 47.1%, and 2.78%, respectively. The five EDL systems showed improvements in accuracy of 2.12%, 5.78%, 6.72%, 32.05%, and 2.40%, thus validating our hypothesis. All statistical tests proved positive for reliability and stability. CONCLUSION EDL showed superior performance to TL systems for both (a) unbalanced and unaugmented and (b) balanced and augmented datasets for both (i) seen and (ii) unseen paradigms, validating both our hypotheses.
Collapse
Affiliation(s)
- Arun Kumar Dubey
- Bharati Vidyapeeth's College of Engineering, New Delhi 110063, India
| | - Gian Luca Chabert
- Department of Radiology, Azienda Ospedaliero Universitaria (A.O.U.), 09123 Cagliari, Italy
| | - Alessandro Carriero
- Department of Radiology, Azienda Ospedaliero Universitaria (A.O.U.), 09123 Cagliari, Italy
| | - Alessio Pasche
- Department of Radiology, "Maggiore della Carità" Hospital, University of Piemonte Orientale, Via Solaroli 17, 28100 Novara, Italy
| | - Pietro S C Danna
- Department of Radiology, "Maggiore della Carità" Hospital, University of Piemonte Orientale, Via Solaroli 17, 28100 Novara, Italy
| | - Sushant Agarwal
- Advanced Knowledge Engineering Centre, GBTI, Roseville, CA 95661, USA
| | - Lopamudra Mohanty
- ABES Engineering College, Ghaziabad 201009, India
- Department of Computer Science Engineering, Bennett University, Greater Noida 201310, India
| | - Neeraj Sharma
- School of Biomedical Engineering, Indian Institute of Technology (BHU), Varanasi 221005, India
| | - Sarita Yadav
- Bharati Vidyapeeth's College of Engineering, New Delhi 110063, India
| | - Achin Jain
- Bharati Vidyapeeth's College of Engineering, New Delhi 110063, India
| | - Ashish Kumar
- Department of Computer Science Engineering, Bennett University, Greater Noida 201310, India
| | - Mannudeep K Kalra
- Department of Radiology, Massachusetts General Hospital, Boston, MA 02115, USA
| | - David W Sobel
- Men's Health Centre, Miriam Hospital Providence, Providence, RI 02906, USA
| | - John R Laird
- Heart and Vascular Institute, Adventist Health St. Helena, St. Helena, CA 94574, USA
| | - Inder M Singh
- Stroke Monitoring and Diagnostic Division, AtheroPoint™, Roseville, CA 95661, USA
| | - Narpinder Singh
- Department of Food Science and Technology, Graphic Era, Deemed to be University, Dehradun 248002, India
| | - George Tsoulfas
- Department of Surgery, Aristoteleion University of Thessaloniki, 54124 Thessaloniki, Greece
| | - Mostafa M Fouda
- Department of Electrical and Computer Engineering, Idaho State University, Pocatello, ID 83209, USA
| | - Azra Alizad
- Department of Physiology & Biomedical Engineering, Mayo Clinic College of Medicine and Science, Rochester, MN 55905, USA
| | - George D Kitas
- Academic Affairs, Dudley Group NHS Foundation Trust, Dudley DY1 2HQ, UK
| | - Narendra N Khanna
- Department of Cardiology, Indraprastha APOLLO Hospitals, New Delhi 110001, India
| | - Klaudija Viskovic
- Department of Radiology and Ultrasound, University Hospital for Infectious Diseases, 10000 Zagreb, Croatia
| | - Melita Kukuljan
- Department of Interventional and Diagnostic Radiology, Clinical Hospital Center Rijeka, 51000 Rijeka, Croatia
| | - Mustafa Al-Maini
- Allergy, Clinical Immunology & Rheumatology Institute, Toronto, ON L4Z 4C4, Canada
| | - Ayman El-Baz
- Biomedical Engineering Department, University of Louisville, Louisville, KY 40292, USA
| | - Luca Saba
- Department of Radiology, Azienda Ospedaliero Universitaria (A.O.U.), 09123 Cagliari, Italy
| | - Jasjit S Suri
- Stroke Monitoring and Diagnostic Division, AtheroPoint™, Roseville, CA 95661, USA
| |
Collapse
|
14
|
Del Amor R, Silva-Rodríguez J, Naranjo V. Labeling confidence for uncertainty-aware histology image classification. Comput Med Imaging Graph 2023; 107:102231. [PMID: 37087899 DOI: 10.1016/j.compmedimag.2023.102231] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2022] [Revised: 02/23/2023] [Accepted: 03/27/2023] [Indexed: 04/25/2023]
Abstract
Deep learning-based models applied to digital pathology require large, curated datasets with high-quality (HQ) annotations to perform correctly. In many cases, recruiting expert pathologists to annotate large databases is not feasible, and it is necessary to collect additional labeled data with varying label qualities, e.g., pathologists-in-training (henceforth, non-expert annotators). Learning from datasets with noisy labels is more challenging in medical applications since medical imaging datasets tend to have instance-dependent noise and suffer from high inter/intra-observer variability. In this paper, we design an uncertainty-driven labeling strategy with which we generate soft labels from 10 non-expert annotators for multi-class skin cancer classification. Based on this soft annotation, we propose an uncertainty estimation-based framework to handle these noisy labels. This framework is based on a novel formulation using a dual-branch min-max entropy calibration to penalize inexact labels during the training. Comprehensive experiments demonstrate the promising performance of our labeling strategy. Results show a consistent improvement by using soft labels with standard cross-entropy loss during training (∼4.0% F1-score) and increases when calibrating the model with the proposed min-max entropy calibration (∼6.6% F1-score). These improvements are produced at negligible cost, both in terms of annotation and calculation.
Collapse
Affiliation(s)
- Rocío Del Amor
- Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, Universitat Politècnica de València, Valencia, Spain.
| | | | - Valery Naranjo
- Instituto Universitario de Investigación en Tecnología Centrada en el Ser Humano, Universitat Politècnica de València, Valencia, Spain.
| |
Collapse
|
15
|
Xiao Z, Su Y, Deng Z, Zhang W. Efficient Combination of CNN and Transformer for Dual-Teacher Uncertainty-guided Semi-supervised Medical Image Segmentation. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2022; 226:107099. [PMID: 36116398 DOI: 10.1016/j.cmpb.2022.107099] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/12/2022] [Revised: 08/25/2022] [Accepted: 08/28/2022] [Indexed: 06/15/2023]
Abstract
BACKGROUND AND OBJECTIVE Deep learning-based methods for fast target segmentation of magnetic resonance imaging (MRI) have become increasingly popular in recent years. Generally, the success of deep learning methods in medical image segmentation tasks relies on a large amount of labeled data. The time-consuming and labor-intensive problem of data annotation is a major challenge in medical image segmentation tasks. The aim of this work is to enhance the segmentation of MR images using a semi-supervised learning-based method using a small amount of labeled data and a large amount of unlabeled data. METHODS To utilize the effective information of the unlabeled data, we designed the method of guiding the Student segmentation model simultaneously by the Dual-Teacher structure of CNN and transformer forming the subject network. Both Teacher A and Student models are CNNs, and the TA-S module they form is a mean teacher structure with added data noise. In the TB-S module formed by the combination of Student and Teacher B models, their backbone networks CNN and transformer capture the local and global information of the image at the same time, respectively, to create pseudo labels for each other and perform cross-supervision. The Dual-Teacher guides the Student through synchronous training and performs knowledge rectification and communication with each other through consistent regular constraints, which better utilizes the valid information in the unlabeled data. In addition, the segmentation predictions of Teacher A and Student and Teacher A and Teacher B are screened for uncertainty assessment during the training process to enhance the prediction accuracy and generalization of the model. This method uses the mechanism of simultaneous training of the synthetic structure composed of TA-S and TB-S modules to jointly guide the optimization of the Student model to obtain better segmentation ability. RESULTS We evaluated the proposed method on a publicly available MRI dataset from a cardiac segmentation competition organized by MICCAI in 2017. Compared with several existing state-of-the-art semi-supervised segmentation methods, the method achieves better segmentation results in terms of Dice coefficient and HD distance evaluation metrics of 0.878 and 4.9 mm and 0.886 and 5.0 mm, respectively, using a training set containing only 10% and 20% of labeled data. CONCLUSION This method fuses CNN and transformer to design a new Teacher-Student semi-supervised learning optimization strategy, which greatly improves the utilization of a large number of unlabeled medical images and the effectiveness of model segmentation results.
Collapse
Affiliation(s)
- Zhiyong Xiao
- School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi214122, China
| | - Yixin Su
- School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi214122, China
| | - Zhaohong Deng
- School of Artificial Intelligence and Computer Science, Jiangnan University, Wuxi214122, China
| | - Weidong Zhang
- Department of Automation, Shanghai JiaoTong University, Shanghai200240, China.
| |
Collapse
|
16
|
Ghesu FC, Georgescu B, Mansoor A, Yoo Y, Neumann D, Patel P, Vishwanath RS, Balter JM, Cao Y, Grbic S, Comaniciu D. Contrastive self-supervised learning from 100 million medical images with optional supervision. J Med Imaging (Bellingham) 2022; 9:064503. [PMID: 36466078 PMCID: PMC9710476 DOI: 10.1117/1.jmi.9.6.064503] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2022] [Accepted: 11/14/2022] [Indexed: 12/05/2022] Open
Abstract
Purpose Building accurate and robust artificial intelligence systems for medical image assessment requires the creation of large sets of annotated training examples. However, constructing such datasets is very costly due to the complex nature of annotation tasks, which often require expert knowledge (e.g., a radiologist). To counter this limitation, we propose a method to learn from medical images at scale in a self-supervised way. Approach Our approach, based on contrastive learning and online feature clustering, leverages training datasets of over 100,000,000 medical images of various modalities, including radiography, computed tomography (CT), magnetic resonance (MR) imaging, and ultrasonography (US). We propose to use the learned features to guide model training in supervised and hybrid self-supervised/supervised regime on various downstream tasks. Results We highlight a number of advantages of this strategy on challenging image assessment problems in radiography, CT, and MR: (1) significant increase in accuracy compared to the state-of-the-art (e.g., area under the curve boost of 3% to 7% for detection of abnormalities from chest radiography scans and hemorrhage detection on brain CT); (2) acceleration of model convergence during training by up to 85% compared with using no pretraining (e.g., 83% when training a model for detection of brain metastases in MR scans); and (3) increase in robustness to various image augmentations, such as intensity variations, rotations or scaling reflective of data variation seen in the field. Conclusions The proposed approach enables large gains in accuracy and robustness on challenging image assessment problems. The improvement is significant compared with other state-of-the-art approaches trained on medical or vision images (e.g., ImageNet).
Collapse
Affiliation(s)
- Florin C. Ghesu
- Siemens Healthineers, Digital Technology and Innovation, Princeton, New Jersey, United States
| | - Bogdan Georgescu
- Siemens Healthineers, Digital Technology and Innovation, Princeton, New Jersey, United States
| | - Awais Mansoor
- Siemens Healthineers, Digital Technology and Innovation, Princeton, New Jersey, United States
| | - Youngjin Yoo
- Siemens Healthineers, Digital Technology and Innovation, Princeton, New Jersey, United States
| | - Dominik Neumann
- Siemens Healthineers, Digital Technology and Innovation, Erlangen, Germany
| | - Pragneshkumar Patel
- Siemens Healthineers, Digital Technology and Innovation, Princeton, New Jersey, United States
| | | | - James M. Balter
- University of Michigan, Department of Radiation Oncology, Ann Arbor, Michigan, United States
| | - Yue Cao
- University of Michigan, Department of Radiation Oncology, Ann Arbor, Michigan, United States
| | - Sasa Grbic
- Siemens Healthineers, Digital Technology and Innovation, Princeton, New Jersey, United States
| | - Dorin Comaniciu
- Siemens Healthineers, Digital Technology and Innovation, Princeton, New Jersey, United States
| |
Collapse
|
17
|
Wu Z, Xu H. Trustworthy machine reading comprehension with conditional adversarial calibration. APPL INTELL 2022. [DOI: 10.1007/s10489-022-04235-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022]
|
18
|
Kurz A, Hauser K, Mehrtens HA, Krieghoff-Henning E, Hekler A, Kather JN, Fröhling S, von Kalle C, Brinker TJ. Uncertainty Estimation in Medical Image Classification: Systematic Review. JMIR Med Inform 2022; 10:e36427. [PMID: 35916701 PMCID: PMC9382553 DOI: 10.2196/36427] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/14/2022] [Revised: 04/11/2022] [Accepted: 06/04/2022] [Indexed: 01/26/2023] Open
Abstract
Background Deep neural networks are showing impressive results in different medical image classification tasks. However, for real-world applications, there is a need to estimate the network’s uncertainty together with its prediction. Objective In this review, we investigate in what form uncertainty estimation has been applied to the task of medical image classification. We also investigate which metrics are used to describe the effectiveness of the applied uncertainty estimation Methods Google Scholar, PubMed, IEEE Xplore, and ScienceDirect were screened for peer-reviewed studies, published between 2016 and 2021, that deal with uncertainty estimation in medical image classification. The search terms “uncertainty,” “uncertainty estimation,” “network calibration,” and “out-of-distribution detection” were used in combination with the terms “medical images,” “medical image analysis,” and “medical image classification.” Results A total of 22 papers were chosen for detailed analysis through the systematic review process. This paper provides a table for a systematic comparison of the included works with respect to the applied method for estimating the uncertainty. Conclusions The applied methods for estimating uncertainties are diverse, but the sampling-based methods Monte-Carlo Dropout and Deep Ensembles are used most frequently. We concluded that future works can investigate the benefits of uncertainty estimation in collaborative settings of artificial intelligence systems and human experts. International Registered Report Identifier (IRRID) RR2-10.2196/11936
Collapse
Affiliation(s)
- Alexander Kurz
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Katja Hauser
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | | | - Eva Krieghoff-Henning
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Achim Hekler
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | | | - Stefan Fröhling
- Department of Translational Medical Oncology, National Center for Tumor Diseases (NCT), German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Christof von Kalle
- Department of Clinical-Translational Sciences, Berlin Institute of Health (BIH), Berlin, Germany
| | - Titus Josef Brinker
- Digital Biomarkers for Oncology Group, German Cancer Research Center (DKFZ), Heidelberg, Germany
| |
Collapse
|
19
|
Huang L, Ruan S, Decazes P, Denœux T. Lymphoma segmentation from 3D PET-CT images using a deep evidential network. Int J Approx Reason 2022. [DOI: 10.1016/j.ijar.2022.06.007] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
|
20
|
Ju L, Wang X, Wang L, Mahapatra D, Zhao X, Zhou Q, Liu T, Ge Z. Improving Medical Images Classification With Label Noise Using Dual-Uncertainty Estimation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2022; 41:1533-1546. [PMID: 34995185 DOI: 10.1109/tmi.2022.3141425] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Deep neural networks are known to be data-driven and label noise can have a marked impact on model performance. Recent studies have shown great robustness to classic image recognition even under a high noisy rate. In medical applications, learning from datasets with label noise is more challenging since medical imaging datasets tend to have instance-dependent noise (IDN) and suffer from high observer variability. In this paper, we systematically discuss the two common types of label noise in medical images - disagreement label noise from inconsistency expert opinions and single-target label noise from biased aggregation of individual annotations. We then propose an uncertainty estimation-based framework to handle these two label noise amid the medical image classification task. We design a dual-uncertainty estimation approach to measure the disagreement label noise and single-target label noise via improved Direct Uncertainty Prediction and Monte-Carlo-Dropout. A boosting-based curriculum training procedure is later introduced for robust learning. We demonstrate the effectiveness of our method by conducting extensive experiments on three different diseases with synthesized and real-world label noise: skin lesions, prostate cancer, and retinal diseases. We also release a large re-engineered database that consists of annotations from more than ten ophthalmologists with an unbiased golden standard dataset for evaluation and benchmarking. The dataset is available at https://mmai.group/peoples/julie/.
Collapse
|
21
|
Tang P, Yang P, Nie D, Wu X, Zhou J, Wang Y. Unified medical image segmentation by learning from uncertainty in an end-to-end manner. Knowl Based Syst 2022. [DOI: 10.1016/j.knosys.2022.108215] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/27/2022]
|
22
|
Diao Z, Jiang H, Shi T. A unified uncertainty network for tumor segmentation using uncertainty cross entropy loss and prototype similarity. Knowl Based Syst 2022. [DOI: 10.1016/j.knosys.2022.108739] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
|
23
|
Molnár V, Molnár A, Lakner Z, Tárnoki DL, Tárnoki ÁD, Jokkel Z, Szabó H, Dienes A, Angyal E, Németh F, Kunos L, Tamás L. Examination of the diaphragm in obstructive sleep apnea using ultrasound imaging. Sleep Breath 2021; 26:1333-1339. [PMID: 34478056 PMCID: PMC9418095 DOI: 10.1007/s11325-021-02472-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2021] [Revised: 07/31/2021] [Accepted: 08/09/2021] [Indexed: 11/02/2022]
Abstract
PURPOSE The aim of this study was to analyze the effect of obstructive sleep apnea (OSA) on the ultrasound (US) features of the diaphragm and to determine if diaphragmatic US may be a useful screening tool for patients with possible OSA. METHODS Patients complaining of snoring were prospectively enrolled for overnight polygraphy using the ApneaLink Air device. Thickness and motion of the diaphragm during tidal and deep inspiration were measured. Logistic regression was used to assess parameters of the diaphragm associated with OSA. RESULTS Of 100 patients, 64 were defined as having OSA. Thicknesses of the left and right hemidiaphragms were significantly different between OSA and control groups. Using a combination of diaphragmatic dimensions, diaphragm dilation, age, sex, and BMI, we developed an algorithm that predicted the presence of OSA with 91% sensitivity and 81% specificity. CONCLUSION A combination of anthropometric measurements, demographic factors, and US imaging may be useful for screening patients for possible OSA. These findings need to be confirmed in larger sample sizes in different clinical settings.
Collapse
Affiliation(s)
- Viktória Molnár
- Department of Otolaryngology and Head and Neck Surgery, Semmelweis University, Szigony u. 36, H-1083, Budapest, Hungary
| | - András Molnár
- Department of Otolaryngology and Head and Neck Surgery, Semmelweis University, Szigony u. 36, H-1083, Budapest, Hungary.
| | - Zoltán Lakner
- Faculty of Food Science, Szent István University, Budapest, Hungary
| | | | | | - Zsófia Jokkel
- Medical Imaging Centre, Semmelweis University, Budapest, Hungary
| | - Helga Szabó
- Medical Imaging Centre, Semmelweis University, Budapest, Hungary
| | - András Dienes
- Medical Imaging Centre, Semmelweis University, Budapest, Hungary
| | - Emese Angyal
- Department of Otolaryngology and Head and Neck Surgery, Semmelweis University, Szigony u. 36, H-1083, Budapest, Hungary
| | - Fruzsina Németh
- Department of Otolaryngology and Head and Neck Surgery, Semmelweis University, Szigony u. 36, H-1083, Budapest, Hungary
| | | | - László Tamás
- Department of Otolaryngology and Head and Neck Surgery, Semmelweis University, Szigony u. 36, H-1083, Budapest, Hungary
| |
Collapse
|
24
|
Gündel S, Setio AAA, Ghesu FC, Grbic S, Georgescu B, Maier A, Comaniciu D. Robust classification from noisy labels: Integrating additional knowledge for chest radiography abnormality assessment. Med Image Anal 2021; 72:102087. [PMID: 34015595 DOI: 10.1016/j.media.2021.102087] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2020] [Revised: 02/24/2021] [Accepted: 04/16/2021] [Indexed: 12/29/2022]
Abstract
Chest radiography is the most common radiographic examination performed in daily clinical practice for the detection of various heart and lung abnormalities. The large amount of data to be read and reported, with more than 100 studies per day for a single radiologist, poses a challenge in consistently maintaining high interpretation accuracy. The introduction of large-scale public datasets has led to a series of novel systems for automated abnormality classification. However, the labels of these datasets were obtained using natural language processed medical reports, yielding a large degree of label noise that can impact the performance. In this study, we propose novel training strategies that handle label noise from such suboptimal data. Prior label probabilities were measured on a subset of training data re-read by 4 board-certified radiologists and were used during training to increase the robustness of the training model to the label noise. Furthermore, we exploit the high comorbidity of abnormalities observed in chest radiography and incorporate this information to further reduce the impact of label noise. Additionally, anatomical knowledge is incorporated by training the system to predict lung and heart segmentation, as well as spatial knowledge labels. To deal with multiple datasets and images derived from various scanners that apply different post-processing techniques, we introduce a novel image normalization strategy. Experiments were performed on an extensive collection of 297,541 chest radiographs from 86,876 patients, leading to a state-of-the-art performance level for 17 abnormalities from 2 datasets. With an average AUC score of 0.880 across all abnormalities, our proposed training strategies can be used to significantly improve performance scores.
Collapse
Affiliation(s)
- Sebastian Gündel
- Digital Technology and Inovation, Siemens Healthineers, Erlangen 91052, Germany; Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen 91058, Germany.
| | - Arnaud A A Setio
- Digital Technology and Inovation, Siemens Healthineers, Erlangen 91052, Germany
| | - Florin C Ghesu
- Digital Technology and Inovation, Siemens Healthineers, Princeton, NJ 08540, USA
| | - Sasa Grbic
- Digital Technology and Inovation, Siemens Healthineers, Princeton, NJ 08540, USA
| | - Bogdan Georgescu
- Digital Technology and Inovation, Siemens Healthineers, Princeton, NJ 08540, USA
| | - Andreas Maier
- Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen 91058, Germany
| | - Dorin Comaniciu
- Digital Technology and Inovation, Siemens Healthineers, Princeton, NJ 08540, USA
| |
Collapse
|