1
|
Holland R, Kaye R, Hagag AM, Leingang O, Taylor TR, Bogunović H, Schmidt-Erfurth U, Scholl HP, Rueckert D, Lotery AJ, Sivaprasad S, Menten MJ. Deep Learning-Based Clustering of OCT Images for Biomarker Discovery in Age-Related Macular Degeneration (PINNACLE Study Report 4). OPHTHALMOLOGY SCIENCE 2024; 4:100543. [PMID: 39139544 PMCID: PMC11321288 DOI: 10.1016/j.xops.2024.100543] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/06/2023] [Revised: 04/26/2024] [Accepted: 04/26/2024] [Indexed: 08/15/2024]
Abstract
Purpose We introduce a deep learning-based biomarker proposal system for the purpose of accelerating biomarker discovery in age-related macular degeneration (AMD). Design Retrospective analysis of a large data set of retinal OCT images. Participants A total of 3456 adults aged between 51 and 102 years whose OCT images were collected under the PINNACLE project. Methods Our system proposes candidates for novel AMD imaging biomarkers in OCT. It works by first training a neural network using self-supervised contrastive learning to discover, without any clinical annotations, features relating to both known and unknown AMD biomarkers present in 46 496 retinal OCT images. To interpret the learned biomarkers, we partition the images into 30 subsets, termed clusters, that contain similar features. We conduct 2 parallel 1.5-hour semistructured interviews with 2 independent teams of retinal specialists to assign descriptions in clinical language to each cluster. Descriptions of clusters achieving consensus can potentially inform new biomarker candidates. Main Outcome Measures We checked if each cluster showed clear features comprehensible to retinal specialists, if they related to AMD, and how many described established biomarkers used in grading systems as opposed to recently proposed or potentially new biomarkers. We also compared their prognostic value for late-stage wet and dry AMD against an established clinical grading system and a demographic baseline model. Results Overall, both teams independently identified clearly distinct characteristics in 27 of 30 clusters, of which 23 were related to AMD. Seven were recognized as known biomarkers used in established grading systems, and 16 depicted biomarker combinations or subtypes that are either not yet used in grading systems, were only recently proposed, or were unknown. Clusters separated incomplete from complete retinal atrophy, intraretinal from subretinal fluid, and thick from thin choroids, and, in simulation, outperformed clinically used grading systems in prognostic value. Conclusions Using self-supervised deep learning, we were able to automatically propose AMD biomarkers going beyond the set used in clinically established grading systems. Without any clinical annotations, contrastive learning discovered subtle differences between fine-grained biomarkers. Ultimately, we envision that equipping clinicians with discovery-oriented deep learning tools can accelerate the discovery of novel prognostic biomarkers. Financial Disclosures Proprietary or commercial disclosure may be found in the Footnotes and Disclosures at the end of this article.
Collapse
Affiliation(s)
- Robbie Holland
- BioMedIA, Department of Computing, Imperial College London, London, United Kingdom
| | - Rebecca Kaye
- Clinical and Experimental Sciences, Faculty of Medicine, University of Southampton, Southampton, United Kingdom
| | - Ahmed M. Hagag
- Institute of Ophthalmology, University College London, London, United Kingdom
- Moorfields Eye Unit, National Institute for Health Research, London, United Kingdom
| | - Oliver Leingang
- Laboratory for Ophthalmic Image Analysis, Department of Ophthalmology and Optometry, Medical University of Vienna, Vienna, Austria
| | - Thomas R.P. Taylor
- Clinical and Experimental Sciences, Faculty of Medicine, University of Southampton, Southampton, United Kingdom
| | - Hrvoje Bogunović
- Laboratory for Ophthalmic Image Analysis, Department of Ophthalmology and Optometry, Medical University of Vienna, Vienna, Austria
- Christian Doppler Laboratory for Artificial Intelligence in Retina, Christian Doppler Forschungsgesellschaft, Vienna, Austria
| | - Ursula Schmidt-Erfurth
- Laboratory for Ophthalmic Image Analysis, Department of Ophthalmology and Optometry, Medical University of Vienna, Vienna, Austria
| | - Hendrik P.N. Scholl
- Institute of Molecular and Clinical Ophthalmology Basel, Basel, Switzerland
- Department of Ophthalmology, University of Basel, Basel, Switzerland
| | - Daniel Rueckert
- BioMedIA, Department of Computing, Imperial College London, London, United Kingdom
- Institute for AI and Informatics in Medicine, School of Computation, Information and Technology, School of Medicine and Health, Technical University Munich, Munich, Germany
| | - Andrew J. Lotery
- Clinical and Experimental Sciences, Faculty of Medicine, University of Southampton, Southampton, United Kingdom
| | - Sobha Sivaprasad
- Institute of Ophthalmology, University College London, London, United Kingdom
- Moorfields Eye Unit, National Institute for Health Research, London, United Kingdom
| | - Martin J. Menten
- BioMedIA, Department of Computing, Imperial College London, London, United Kingdom
- Institute for AI and Informatics in Medicine, School of Computation, Information and Technology, School of Medicine and Health, Technical University Munich, Munich, Germany
| |
Collapse
|
2
|
Park Y, Kang S, Kim MJ, Lee Y, Kim HS, Yi J. Visual defect obfuscation based self-supervised anomaly detection. Sci Rep 2024; 14:18872. [PMID: 39143358 PMCID: PMC11325017 DOI: 10.1038/s41598-024-69698-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2024] [Accepted: 08/07/2024] [Indexed: 08/16/2024] Open
Abstract
Due to scarcity of anomaly situations in the early manufacturing stage, an unsupervised anomaly detection (UAD) approach is widely adopted which only uses normal samples for training. This approach is based on the assumption that the trained UAD model will accurately reconstruct normal patterns but struggles with unseen anomalies. To enhance the UAD performance, reconstruction-by-inpainting based methods have recently been investigated, especially on the masking strategy of suspected defective regions. However, there are still issues to overcome: (1) time-consuming inference due to multiple masking, (2) output inconsistency by random masking, and (3) inaccurate reconstruction of normal patterns for large masked areas. Motivated by this, this study proposes a novel reconstruction-by-inpainting method, dubbed Excision And Recovery (EAR), that features single deterministic masking based on the ImageNet pre-trained DINO-ViT and visual obfuscation for hint-providing. Experimental results on the MVTec AD dataset show that deterministic masking by pre-trained attention effectively cuts out suspected defective regions and resolves the aforementioned issues 1 and 2. Also, hint-providing by mosaicing proves to enhance the performance than emptying those regions by binary masking, thereby overcomes issue 3. The proposed approach achieves a high performance without any change of the model structure. Promising results are shown through laboratory tests with public industrial datasets. To suggest EAR be possibly adopted in various industries as a practically deployable solution, future steps include evaluating its applicability in relevant manufacturing environments.
Collapse
Affiliation(s)
- YeongHyeon Park
- Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, 16419, Republic of Korea
- SK Planet Co., Ltd., Seongnam, 13487, Republic of Korea
| | - Sungho Kang
- Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, 16419, Republic of Korea
| | - Myung Jin Kim
- SK Planet Co., Ltd., Seongnam, 13487, Republic of Korea
| | - Yeonho Lee
- Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, 16419, Republic of Korea
| | | | - Juneho Yi
- Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, 16419, Republic of Korea.
| |
Collapse
|
3
|
Jebril H, Esengönül M, Bogunović H. Anomaly Detection in Optical Coherence Tomography Angiography (OCTA) with a Vector-Quantized Variational Auto-Encoder (VQ-VAE). Bioengineering (Basel) 2024; 11:682. [PMID: 39061764 PMCID: PMC11273395 DOI: 10.3390/bioengineering11070682] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2024] [Revised: 06/29/2024] [Accepted: 07/02/2024] [Indexed: 07/28/2024] Open
Abstract
Optical coherence tomography angiography (OCTA) provides detailed information on retinal blood flow and perfusion. Abnormal retinal perfusion indicates possible ocular or systemic disease. We propose a deep learning-based anomaly detection model to identify such anomalies in OCTA. It utilizes two deep learning approaches. First, a representation learning with a Vector-Quantized Variational Auto-Encoder (VQ-VAE) followed by Auto-Regressive (AR) modeling. Second, it exploits epistemic uncertainty estimates from Bayesian U-Net employed to segment the vasculature on OCTA en face images. Evaluation on two large public datasets, DRAC and OCTA-500, demonstrates effective anomaly detection (an AUROC of 0.92 for the DRAC and an AUROC of 0.75 for the OCTA-500) and localization (a mean Dice score of 0.61 for the DRAC) on this challenging task. To our knowledge, this is the first work that addresses anomaly detection in OCTA.
Collapse
Affiliation(s)
- Hana Jebril
- Lab for Ophthalmic Image Analysis, Department of Ophthalmology and Optometry, Medical University of Vienna, 1090 Vienna, Austria; (H.J.); (M.E.)
| | - Meltem Esengönül
- Lab for Ophthalmic Image Analysis, Department of Ophthalmology and Optometry, Medical University of Vienna, 1090 Vienna, Austria; (H.J.); (M.E.)
| | - Hrvoje Bogunović
- Lab for Ophthalmic Image Analysis, Department of Ophthalmology and Optometry, Medical University of Vienna, 1090 Vienna, Austria; (H.J.); (M.E.)
- Christian Doppler Lab for Artificial Intelligence in Retina, Department of Ophthalmology and Optometry, Medical University of Vienna, 1090 Vienna, Austria
| |
Collapse
|
4
|
Li X, Zhou Y, Yin H, Zhao P, Tang R, Lv H, Qin Y, Zhuo L, Wang Z. Sub-features orthogonal decoupling: Detecting bone wall absence via a small number of abnormal examples for temporal CT images. Comput Med Imaging Graph 2024; 115:102380. [PMID: 38626631 DOI: 10.1016/j.compmedimag.2024.102380] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2023] [Revised: 02/08/2024] [Accepted: 04/09/2024] [Indexed: 04/18/2024]
Abstract
The absence of bone wall located in the jugular bulb and sigmoid sinus of the temporal bone is one of the important reasons for pulsatile tinnitus. Automatic and accurate detection of these abnormal singes in CT slices has important theoretical significance and clinical value. Due to the shortage of abnormal samples, imbalanced samples, small inter-class differences, and low interpretability, existing deep-learning methods are greatly challenged. In this paper, we proposed a sub-features orthogonal decoupling model, which can effectively disentangle the representation features into class-specific sub-features and class-independent sub-features in a latent space. The former contains the discriminative information, while, the latter preserves information for image reconstruction. In addition, the proposed method can generate image samples using category conversion by combining the different class-specific sub-features and the class-independent sub-features, achieving corresponding mapping between deep features and images of specific classes. The proposed model improves the interpretability of the deep model and provides image synthesis methods for downstream tasks. The effectiveness of the method was verified in the detection of bone wall absence in the temporal bone jugular bulb and sigmoid sinus.
Collapse
Affiliation(s)
- Xiaoguang Li
- Faculty of Information Technology, Beijing University of Technology, No. 100, Pingleyuan, Chaoyang, Beijing, 100124, China
| | - Yichao Zhou
- Faculty of Information Technology, Beijing University of Technology, No. 100, Pingleyuan, Chaoyang, Beijing, 100124, China
| | - Hongxia Yin
- Department of Radiology, Beijing Friendship Hospital, Capital Medical University, No. 95 , Yong'an Road, Xicheng, Beijing, 100050, China.
| | - Pengfei Zhao
- Department of Radiology, Beijing Friendship Hospital, Capital Medical University, No. 95 , Yong'an Road, Xicheng, Beijing, 100050, China
| | - Ruowei Tang
- Department of Radiology, Beijing Friendship Hospital, Capital Medical University, No. 95 , Yong'an Road, Xicheng, Beijing, 100050, China
| | - Han Lv
- Department of Radiology, Beijing Friendship Hospital, Capital Medical University, No. 95 , Yong'an Road, Xicheng, Beijing, 100050, China
| | - Yating Qin
- Faculty of Information Technology, Beijing University of Technology, No. 100, Pingleyuan, Chaoyang, Beijing, 100124, China
| | - Li Zhuo
- Faculty of Information Technology, Beijing University of Technology, No. 100, Pingleyuan, Chaoyang, Beijing, 100124, China
| | - Zhenchang Wang
- Department of Radiology, Beijing Friendship Hospital, Capital Medical University, No. 95 , Yong'an Road, Xicheng, Beijing, 100050, China
| |
Collapse
|
5
|
Yang X, Liu L, Yan Z, Yu J, Hu X, Yu X, Dong C, Chen J, Liu H, Yu Z, Deng X, Ni D, Gou Z, Huang X. Hierarchical online contrastive anomaly detection for fetal arrhythmia diagnosis in ultrasound. Med Image Anal 2024; 97:103229. [PMID: 38897033 DOI: 10.1016/j.media.2024.103229] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/19/2023] [Revised: 03/10/2024] [Accepted: 06/03/2024] [Indexed: 06/21/2024]
Abstract
Arrhythmia is a major cardiac abnormality in fetuses. Therefore, early diagnosis of arrhythmia is clinically crucial. Pulsed-wave Doppler ultrasound is a commonly used diagnostic tool for fetal arrhythmia. Its key step for diagnosis involves identifying adjacent measurable cardiac cycles (MCCs). As cardiac activity is complex and the experience of sonographers is often varied, automation can improve user-independence and diagnostic-validity. However, arrhythmias pose several challenges for automation because of complex waveform variations, which can cause major localization bias and missed or false detection of MCCs. Filtering out non-MCC anomalies is difficult because of large intra-class and small inter-class variations between MCCs and non-MCCs caused by agnostic morphological waveform variations. Moreover, rare arrhythmia cases are insufficient for classification algorithms to adequately learn discriminative features. Using only normal cases for training, we propose a novel hierarchical online contrastive anomaly detection (HOCAD) framework for arrhythmia diagnosis during test time. The contribution of this study is three-fold. First, we develop a coarse-to-fine framework inspired by hierarchical diagnostic logic, which can refine localization and avoid missed detection of MCCs. Second, we propose an online learning-based contrastive anomaly detection with two new anomaly scores, which can adaptively filter out non-MCC anomalies on a single image during testing. With these complementary efforts, we precisely determine MCCs for correct measurements and diagnosis. Third, to the best of our knowledge, this is the first reported study investigating intelligent diagnosis of fetal arrhythmia on a large-scale and multi-center ultrasound dataset. Extensive experiments on 3850 cases, including 266 cases covering three typical types of arrhythmias, demonstrate the effectiveness of the proposed framework.
Collapse
Affiliation(s)
- Xin Yang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China
| | - Lian Liu
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China
| | - Zhongnuo Yan
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China
| | - Junxuan Yu
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China
| | - Xindi Hu
- Shenzhen RayShape Medical Technology Co., Ltd, Shenzhen, Guangdong, China
| | - Xuejuan Yu
- Department of Ultrasonography, Suzhou Xiangcheng People's Hospital, Suzhou, Jiangsu, China
| | - Caixia Dong
- Department of Ultrasonography, Wulin Hospital, Hangzhou, Zhejiang, China
| | - Ju Chen
- Department of Ultrasonography, Taicang First People's Hospital, Suzhou, Jiangsu, China
| | - Hongmei Liu
- Department of Ultrasonography, Panzhou Emerging Hospital, Panzhou, Guizhou, China
| | - Zhuan Yu
- Department of Ultrasonography, The Affiliated Hospital of Jiangsu University, Zhenjiang, Jiangsu, China
| | - Xuedong Deng
- Center for Medical Ultrasound, The Affiliated Suzhou Hospital of Nanjing Medical University, Suzhou Municipal Hospital, Gusu School, Nanjing Medical University, Suzhou, Jiangsu, China
| | - Dong Ni
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China
| | - Zhongshan Gou
- Center for Cardiovascular Disease, The Affiliated Suzhou Hospital of Nanjing Medical University, Suzhou Municipal Hospital, Gusu School, Nanjing Medical University, Suzhou, Jiangsu, China.
| | - Xiaoqiong Huang
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen, China, Shenzhen, China; Medical Ultrasound Image Computing (MUSIC) Laboratory, Shenzhen University, Shenzhen, China.
| |
Collapse
|
6
|
Luo Y, Ma Y, Yang Z. Multi-resolution auto-encoder for anomaly detection of retinal imaging. Phys Eng Sci Med 2024; 47:517-529. [PMID: 38285270 DOI: 10.1007/s13246-023-01381-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/12/2023] [Accepted: 12/27/2023] [Indexed: 01/30/2024]
Abstract
Identifying unknown types of diseases is a crucial step in preceding retinal imaging classification for the sake of safety, which is known as anomaly detection of retinal imaging. However, the widely-used supervised learning algorithms are not suitable for this problem, since the data of the unknown category is unobtainable. Moreover, for retinal imaging with different types of anomalous regions, using a single-resolution input causes information loss. Therefore, we propose an unsupervised auto-encoder model with multi-resolution inputs and outputs. We provide a theoretical understanding of the effectiveness of reconstruction error and the improvement of self-supervised learning for anomaly detection. Our experiments on two widely-used retinal imaging datasets show that the proposed methods are superior to other methods, and further experiments verify the validity of each part of the proposed method.
Collapse
Affiliation(s)
- Yixin Luo
- School of Mathematical Sciences, University of Science and Technology of China, No. 96 Jinzhai Road, Hefei, 230026, Anhui, China
| | - Yangling Ma
- School of Mathematical Sciences, Suzhou University of Science and Technology, No. 99 Xuefu Road, Suzhou, 215009, Jiangsu, China
| | - Zhouwang Yang
- School of Mathematical Sciences, University of Science and Technology of China, No. 96 Jinzhai Road, Hefei, 230026, Anhui, China.
| |
Collapse
|
7
|
Lu S, Zhang W, Guo J, Liu H, Li H, Wang N. PatchCL-AE: Anomaly detection for medical images using patch-wise contrastive learning-based auto-encoder. Comput Med Imaging Graph 2024; 114:102366. [PMID: 38471329 DOI: 10.1016/j.compmedimag.2024.102366] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/11/2023] [Revised: 03/04/2024] [Accepted: 03/04/2024] [Indexed: 03/14/2024]
Abstract
Anomaly detection is an important yet challenging task in medical image analysis. Most anomaly detection methods are based on reconstruction, but the performance of reconstruction-based methods is limited due to over-reliance on pixel-level losses. To address the limitation, we propose a patch-wise contrastive learning-based auto-encoder for medical anomaly detection. The key contribution is the patch-wise contrastive learning loss that provides supervision on local semantics to enforce semantic consistency between corresponding input-output patches. Contrastive learning pulls corresponding patch pairs closer while pushing non-corresponding ones apart between input and output, enabling the model to learn local normal features better and improve discriminability on anomalous regions. Additionally, we design an anomaly score based on local semantic discrepancies to pinpoint abnormalities by comparing feature difference rather than pixel variations. Extensive experiments on three public datasets (i.e., brain MRI, retinal OCT, and chest X-ray) achieve state-of-the-art performance, with our method achieving over 99% AUC on retinal and brain images. Both the contrastive patch-wise supervision and patch-discrepancy score provide targeted advancements to overcome the weaknesses in existing approaches.
Collapse
Affiliation(s)
- Shuai Lu
- Beijing Institute of Technology, Beijing, 100081, China
| | - Weihang Zhang
- Beijing Institute of Technology, Beijing, 100081, China
| | - Jia Guo
- Beijing Institute of Technology, Beijing, 100081, China
| | - Hanruo Liu
- Beijing Institute of Technology, Beijing, 100081, China; Beijing Institute of Ophthalmology, Beijing Tongren Hospital, Capital Medical University, Beijing Ophthalmology & Visual Science Key Lab, Beijing, 100005, China.
| | - Huiqi Li
- Beijing Institute of Technology, Beijing, 100081, China.
| | - Ningli Wang
- Beijing Institute of Ophthalmology, Beijing Tongren Hospital, Capital Medical University, Beijing Ophthalmology & Visual Science Key Lab, Beijing, 100005, China
| |
Collapse
|
8
|
Truong AM, Luong HQ. A non-destructive, autoencoder-based approach to detecting defects and contamination in reusable food packaging. Curr Res Food Sci 2024; 8:100758. [PMID: 38779346 PMCID: PMC11109354 DOI: 10.1016/j.crfs.2024.100758] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2024] [Revised: 04/22/2024] [Accepted: 05/02/2024] [Indexed: 05/25/2024] Open
Abstract
Today, environmental sustainability is one of the most critical issue. Hence, the food service industry is actively seeking ways to minimize its ecological footprint. One solution to address this issue is the adoption of reusable foodware in the food service industry. This approach requires a careful process for the collection and thorough cleaning of the foodware, ensuring it can be safely reused. However, reusable foodware might be damaged during the collection process, which can pose food safety hazards for customers. Additionally, there are cases where the cleaning process might not effectively remove all contaminants and therefore cannot be reused after the washing process. To ensure consumer safety, a manual inspection is typically conducted after the cleaning process. However, this step is labor-intensive and prone to human error, particularly as workers' attention may decrease over extended periods. Consequently, the adoption of precise and automated methods for detecting defects and contaminants is becoming crucial, not only to ensure safety but also to achieve scalability and enhance cost-efficiency in the pursuit of environmental sustainability. In our research, we explore various data augmentation strategies and the application of knowledge transfer from various samples of reusable food containers. This method only requires few images from a clean sample to teach the network about normal patterns, and to detect defects by identifying irregular details that do not exist in normal samples. This allows us to rapidly deploy the detection system even with a limited number of collected samples. Experimental results demonstrate the effectiveness of our approach in detecting both contamination and cracks on food containers.
Collapse
Affiliation(s)
- Anh Minh Truong
- IPI-TELIN, Ghent University-IMEC, Sint-Pietersnieuwstraat 41, Ghent, 9000, East Flanders, Belgium
| | - Hiep Quang Luong
- IPI-TELIN, Ghent University-IMEC, Sint-Pietersnieuwstraat 41, Ghent, 9000, East Flanders, Belgium
| |
Collapse
|
9
|
Getachew Shiferaw T, Yao L. Autoencoder-Based Unsupervised Surface Defect Detection Using Two-Stage Training. J Imaging 2024; 10:111. [PMID: 38786565 PMCID: PMC11121878 DOI: 10.3390/jimaging10050111] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/19/2024] [Revised: 04/22/2024] [Accepted: 05/01/2024] [Indexed: 05/25/2024] Open
Abstract
Accurately detecting defects while reconstructing a high-quality normal background in surface defect detection using unsupervised methods remains a significant challenge. This study proposes an unsupervised method that effectively addresses this challenge by achieving both accurate defect detection and a high-quality normal background reconstruction without noise. We propose an adaptive weighted structural similarity (AW-SSIM) loss for focused feature learning. AW-SSIM improves structural similarity (SSIM) loss by assigning different weights to its sub-functions of luminance, contrast, and structure based on their relative importance for a specific training sample. Moreover, it dynamically adjusts the Gaussian window's standard deviation (σ) during loss calculation to balance noise reduction and detail preservation. An artificial defect generation algorithm (ADGA) is proposed to generate an artificial defect closely resembling real ones. We use a two-stage training strategy. In the first stage, the model trains only on normal samples using AW-SSIM loss, allowing it to learn robust representations of normal features. In the second stage of training, the weights obtained from the first stage are used to train the model on both normal and artificially defective training samples. Additionally, the second stage employs a combined learned Perceptual Image Patch Similarity (LPIPS) and AW-SSIM loss. The combined loss helps the model in achieving high-quality normal background reconstruction while maintaining accurate defect detection. Extensive experimental results demonstrate that our proposed method achieves a state-of-the-art defect detection accuracy. The proposed method achieved an average area under the receiver operating characteristic curve (AuROC) of 97.69% on six samples from the MVTec anomaly detection dataset.
Collapse
Affiliation(s)
| | - Li Yao
- School of Computer Science and Engineering, Southeast University, Nanjing 211189, China;
- Key Laboratory of Computer Network and Information Integration, Southeast University, Ministry of Education, Nanjing 211189, China
| |
Collapse
|
10
|
Yang J, Mehta N, Demirci G, Hu X, Ramakrishnan MS, Naguib M, Chen C, Tsai CL. Anomaly-guided weakly supervised lesion segmentation on retinal OCT images. Med Image Anal 2024; 94:103139. [PMID: 38493532 PMCID: PMC11016376 DOI: 10.1016/j.media.2024.103139] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2023] [Revised: 01/23/2024] [Accepted: 03/05/2024] [Indexed: 03/19/2024]
Abstract
The availability of big data can transform the studies in biomedical research to generate greater scientific insights if expert labeling is available to facilitate supervised learning. However, data annotation can be labor-intensive and cost-prohibitive if pixel-level precision is required. Weakly supervised semantic segmentation (WSSS) with image-level labeling has emerged as a promising solution in medical imaging. However, most existing WSSS methods in the medical domain are designed for single-class segmentation per image, overlooking the complexities arising from the co-existence of multiple classes in a single image. Additionally, the multi-class WSSS methods from the natural image domain cannot produce comparable accuracy for medical images, given the challenge of substantial variation in lesion scales and occurrences. To address this issue, we propose a novel anomaly-guided mechanism (AGM) for multi-class segmentation in a single image on retinal optical coherence tomography (OCT) using only image-level labels. AGM leverages the anomaly detection and self-attention approach to integrate weak abnormal signals with global contextual information into the training process. Furthermore, we include an iterative refinement stage to guide the model to focus more on the potential lesions while suppressing less relevant regions. We validate the performance of our model with two public datasets and one challenging private dataset. Experimental results show that our approach achieves a new state-of-the-art performance in WSSS for lesion segmentation on OCT images.
Collapse
Affiliation(s)
- Jiaqi Yang
- Graduate Center CUNY, 365 5th Ave, NY 10016, USA.
| | - Nitish Mehta
- New York University Department of Ophthalmology, NYU Langone Health, 222 E. 41st St., 3rd Floor, NY 10017, USA
| | | | - Xiaoling Hu
- Stony Brook University, 100 Nicolls Rd, Stony Brook 11794, USA
| | - Meera S Ramakrishnan
- New York University Department of Ophthalmology, NYU Langone Health, 222 E. 41st St., 3rd Floor, NY 10017, USA
| | - Mina Naguib
- New York University Department of Ophthalmology, NYU Langone Health, 222 E. 41st St., 3rd Floor, NY 10017, USA
| | - Chao Chen
- Stony Brook University, 100 Nicolls Rd, Stony Brook 11794, USA
| | | |
Collapse
|
11
|
Tu J, Liu H, Li C. Ordinal Regression for Direction-Related Anomaly Detection. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024; 35:6821-6834. [PMID: 36269929 DOI: 10.1109/tnnls.2022.3212991] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
Anomaly detection is widely used in many fields to reveal the abnormal process of a system. Typical model-based anomaly detection methods work well in general anomaly detection problems. However, in some application-specific scenarios, the anomalies of interest are "direction-related," that is, only deviation in certain directions of the data space is abnormal. Most existing anomaly detection methods do not work well in these scenarios, especially when there is no abnormal data information during training. Considering that in many real anomaly detection applications such as medical disease detection and industrial device faults diagnosis, the normal data have several ordinal levels, and the anomalies can be regarded as an unseen level distributed roughly along the ordinal direction outside the normal levels. Notice that the ordinal information is inherently "direction-related," and we can use the ordinal information to assist in finding a "direction-related" boundary for the normal data to detect anomalies of interest. A typical type of methods utilizing the ordinal information is ordinal regression. However, to the best of our knowledge, the existing ordinal regression methods are unable to be directly applied to anomaly detection. In this article, to detect the aforementioned "direction-related" anomalies, we propose an ordinal regression algorithm for direction-related anomaly detection (ORAD). Specifically, we first formulate ORAD as an optimization problem. Then, we apply the difference of convex functions (DC) programming to solve the problem to obtain a "direction-related" boundary. After that, we calculate the outlier scores based on the deviation from the boundary. Theoretically, we analyze the ordinal properties and the convergence of ORAD. We carry out experiments on both synthetic data and real datasets to demonstrate the effectiveness of the proposed ORAD.
Collapse
|
12
|
Lu S, Zhang W, Zhao H, Liu H, Wang N, Li H. Anomaly Detection for Medical Images Using Heterogeneous Auto-Encoder. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2024; 33:2770-2782. [PMID: 38551828 DOI: 10.1109/tip.2024.3381435] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/04/2024]
Abstract
Anomaly detection is an important task for medical image analysis, which can alleviate the reliance of supervised methods on large labelled datasets. Most existing methods use a pixel-wise self-reconstruction framework for anomaly detection. However, there are two challenges of these studies: 1) they tend to overfit learning an identity mapping between the input and output, which leads to failure in detecting abnormal samples; 2) the reconstruction considers the pixel-wise differences which may lead to an undesirable result. To mitigate the above problems, we propose a novel heterogeneous Auto-Encoder (Hetero-AE) for medical anomaly detection. Our model utilizes a convolutional neural network (CNN) as the encoder and a hybrid CNN-Transformer network as the decoder. The heterogeneous structure enables the model to learn the intrinsic information of normal data and enlarge the difference on abnormal samples. To fully exploit the effectiveness of Transformer in the hybrid network, a multi-scale sparse Transformer block is proposed to trade off modelling long-range feature dependencies and high computational costs. Moreover, the multi-stage feature comparison is introduced to reduce the noise of pixel-wise comparison. Extensive experiments on four public datasets (i.e., retinal OCT, chest X-ray, brain MRI, and COVID-19) verify the effectiveness of our method on different imaging modalities for anomaly detection. Additionally, our method can accurately detect tumors in brain MRI and lesions in retinal OCT with interpretable heatmaps to locate lesion areas, assisting clinicians in diagnosing abnormalities efficiently.
Collapse
|
13
|
Sheng S, Jing J, Wang Z, Zhang H. Cosine similarity knowledge distillation for surface anomaly detection. Sci Rep 2024; 14:8150. [PMID: 38589492 PMCID: PMC11001943 DOI: 10.1038/s41598-024-58409-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2024] [Accepted: 03/28/2024] [Indexed: 04/10/2024] Open
Abstract
The current state-of-the-art anomaly detection methods based on knowledge distillation (KD) typically depend on smaller student networks or reverse distillation to address vanishing representations discrepancy on anomalies. These methods often struggle to achieve precise detection when dealing with complex texture backgrounds containing anomalies due to the similarity between anomalous and non-anomalous regions. Therefore, we propose a new paradigm-Cosine Similarity Knowledge Distillation (CSKD), for surface anomaly detection and localization. We focus on the superior performance of the same deeper teacher and student encoders by the distillation loss in traditional knowledge distillation-based methods. Essentially, we introduce the Attention One-Class Embedding (AOCE) in the student network to enhance learning capabilities and reduce the effect of the teacher-student (T-S) model on response similarity in anomalous regions. Furthermore, we find the optimal models by different classes' hard-coded epochs, and an adaptive optimal model selection method is designed. Extensive experiments on the MVTec dataset with 99.2% image-level AUROC and 98.2%/94.7% pixel-level AUROC/PRO demonstrate that our method outperforms existing unsupervised anomaly detection algorithms. Additional experiments on DAGM dataset, and one-class anomaly detection benchmarks further show the superiority of the proposed method.
Collapse
Affiliation(s)
- Siyu Sheng
- College of Electrical and Information, Xi'an Polytechnic University, Xi'an, 710048, China
| | - Junfeng Jing
- College of Electrical and Information, Xi'an Polytechnic University, Xi'an, 710048, China.
- Xi'an Polytechnic University Branch of Shaanxi Artificial Intelligence Joint Laboratory, Xi'an, 710048, China.
| | - Zhen Wang
- Defense Innovation Institute, Beijing, 100071, China
| | - Huanhuan Zhang
- Xi'an Polytechnic University Branch of Shaanxi Artificial Intelligence Joint Laboratory, Xi'an, 710048, China
| |
Collapse
|
14
|
Mares V, Nehemy MB, Bogunovic H, Frank S, Reiter GS, Schmidt-Erfurth U. AI-based support for optical coherence tomography in age-related macular degeneration. Int J Retina Vitreous 2024; 10:31. [PMID: 38589936 PMCID: PMC11000391 DOI: 10.1186/s40942-024-00549-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2024] [Accepted: 03/16/2024] [Indexed: 04/10/2024] Open
Abstract
Artificial intelligence (AI) has emerged as a transformative technology across various fields, and its applications in the medical domain, particularly in ophthalmology, has gained significant attention. The vast amount of high-resolution image data, such as optical coherence tomography (OCT) images, has been a driving force behind AI growth in this field. Age-related macular degeneration (AMD) is one of the leading causes for blindness in the world, affecting approximately 196 million people worldwide in 2020. Multimodal imaging has been for a long time the gold standard for diagnosing patients with AMD, however, currently treatment and follow-up in routine disease management are mainly driven by OCT imaging. AI-based algorithms have by their precision, reproducibility and speed, the potential to reliably quantify biomarkers, predict disease progression and assist treatment decisions in clinical routine as well as academic studies. This review paper aims to provide a summary of the current state of AI in AMD, focusing on its applications, challenges, and prospects.
Collapse
Affiliation(s)
- Virginia Mares
- Laboratory for Ophthalmic Image Analysis, Department of Ophthalmology and Optometry, Medical University of Vienna, Währinger Gürtel 18-20, 1090, Vienna, Austria
- Department of Ophthalmology, Federal University of Minas Gerais, Belo Horizonte, Brazil
| | - Marcio B Nehemy
- Department of Ophthalmology, Federal University of Minas Gerais, Belo Horizonte, Brazil
| | - Hrvoje Bogunovic
- Laboratory for Ophthalmic Image Analysis, Department of Ophthalmology and Optometry, Medical University of Vienna, Währinger Gürtel 18-20, 1090, Vienna, Austria
| | - Sophie Frank
- Laboratory for Ophthalmic Image Analysis, Department of Ophthalmology and Optometry, Medical University of Vienna, Währinger Gürtel 18-20, 1090, Vienna, Austria
| | - Gregor S Reiter
- Laboratory for Ophthalmic Image Analysis, Department of Ophthalmology and Optometry, Medical University of Vienna, Währinger Gürtel 18-20, 1090, Vienna, Austria
| | - Ursula Schmidt-Erfurth
- Laboratory for Ophthalmic Image Analysis, Department of Ophthalmology and Optometry, Medical University of Vienna, Währinger Gürtel 18-20, 1090, Vienna, Austria.
| |
Collapse
|
15
|
Linmans J, Raya G, van der Laak J, Litjens G. Diffusion models for out-of-distribution detection in digital pathology. Med Image Anal 2024; 93:103088. [PMID: 38228075 DOI: 10.1016/j.media.2024.103088] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/04/2023] [Revised: 12/29/2023] [Accepted: 01/10/2024] [Indexed: 01/18/2024]
Abstract
The ability to detect anomalies, i.e. anything not seen during training or out-of-distribution (OOD), in medical imaging applications is essential for successfully deploying machine learning systems. Filtering out OOD data using unsupervised learning is especially promising because it does not require costly annotations. A new class of models called AnoDDPMs, based on denoising diffusion probabilistic models (DDPMs), has recently achieved significant progress in unsupervised OOD detection. This work provides a benchmark for unsupervised OOD detection methods in digital pathology. By leveraging fast sampling techniques, we apply AnoDDPM on a large enough scale for whole-slide image analysis on the complete test set of the Camelyon16 challenge. Based on ROC analysis, we show that AnoDDPMs can detect OOD data with an AUC of up to 94.13 and 86.93 on two patch-level OOD detection tasks, outperforming the other unsupervised methods. We observe that AnoDDPMs alter the semantic properties of inputs, replacing anomalous data with more benign-looking tissue. Furthermore, we highlight the flexibility of AnoDDPM towards different information bottlenecks by evaluating reconstruction errors for inputs with different signal-to-noise ratios. While there is still a significant performance gap with fully supervised learning, AnoDDPMs show considerable promise in the field of OOD detection in digital pathology.
Collapse
Affiliation(s)
- Jasper Linmans
- Department of Pathology, RadboudUMC Graduate School, Radboud University Medical Center, Nijmegen, The Netherlands.
| | - Gabriel Raya
- Jheronimus Academy of Data Science, 's-Hertogenbosch, The Netherlands
| | - Jeroen van der Laak
- Department of Pathology, RadboudUMC Graduate School, Radboud University Medical Center, Nijmegen, The Netherlands; Center for Medical Image Science and Visualization, Linköping University, Linköping, Sweden
| | - Geert Litjens
- Department of Pathology, RadboudUMC Graduate School, Radboud University Medical Center, Nijmegen, The Netherlands
| |
Collapse
|
16
|
Hojjati H, Ho TKK, Armanfard N. Self-supervised anomaly detection in computer vision and beyond: A survey and outlook. Neural Netw 2024; 172:106106. [PMID: 38232432 DOI: 10.1016/j.neunet.2024.106106] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2023] [Revised: 12/31/2023] [Accepted: 01/05/2024] [Indexed: 01/19/2024]
Abstract
Anomaly detection (AD) plays a crucial role in various domains, including cybersecurity, finance, and healthcare, by identifying patterns or events that deviate from normal behavior. In recent years, significant progress has been made in this field due to the remarkable growth of deep learning models. Notably, the advent of self-supervised learning has sparked the development of novel AD algorithms that outperform the existing state-of-the-art approaches by a considerable margin. This paper aims to provide a comprehensive review of the current methodologies in self-supervised anomaly detection. We present technical details of the standard methods and discuss their strengths and drawbacks. We also compare the performance of these models against each other and other state-of-the-art anomaly detection models. Finally, the paper concludes with a discussion of future directions for self-supervised anomaly detection, including the development of more effective and efficient algorithms and the integration of these techniques with other related fields, such as multi-modal learning.
Collapse
Affiliation(s)
- Hadi Hojjati
- Department of Electrical and Computer Engineering, McGill University, Montreal, QC, Canada; Mila - Quebec AI Institute, Montreal, QC, Canada.
| | - Thi Kieu Khanh Ho
- Department of Electrical and Computer Engineering, McGill University, Montreal, QC, Canada; Mila - Quebec AI Institute, Montreal, QC, Canada
| | - Narges Armanfard
- Department of Electrical and Computer Engineering, McGill University, Montreal, QC, Canada; Mila - Quebec AI Institute, Montreal, QC, Canada
| |
Collapse
|
17
|
López Diez P, Sundgaard JV, Margeta J, Diab K, Patou F, Paulsen RR. Deep reinforcement learning and convolutional autoencoders for anomaly detection of congenital inner ear malformations in clinical CT images. Comput Med Imaging Graph 2024; 113:102343. [PMID: 38325245 DOI: 10.1016/j.compmedimag.2024.102343] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2023] [Revised: 01/25/2024] [Accepted: 01/28/2024] [Indexed: 02/09/2024]
Abstract
Detection of abnormalities within the inner ear is a challenging task even for experienced clinicians. In this study, we propose an automated method for automatic abnormality detection to provide support for the diagnosis and clinical management of various otological disorders. We propose a framework for inner ear abnormality detection based on deep reinforcement learning for landmark detection which is trained uniquely in normative data. In our approach, we derive two abnormality measurements: Dimage and Uimage. The first measurement, Dimage, is based on the variability of the predicted configuration of a well-defined set of landmarks in a subspace formed by the point distribution model of the location of those landmarks in normative data. We create this subspace using Procrustes shape alignment and Principal Component Analysis projection. The second measurement, Uimage, represents the degree of hesitation of the agents when approaching the final location of the landmarks and is based on the distribution of the predicted Q-values of the model for the last ten states. Finally, we unify these measurements in a combined anomaly measurement called Cimage. We compare our method's performance with a 3D convolutional autoencoder technique for abnormality detection using the patch-based mean squared error between the original and the generated image as a basis for classifying abnormal versus normal anatomies. We compare both approaches and show that our method, based on deep reinforcement learning, shows better detection performance for abnormal anatomies on both an artificial and a real clinical CT dataset of various inner ear malformations with an increase of 11.2% of the area under the ROC curve. Our method also shows more robustness against the heterogeneous quality of the images in our dataset.
Collapse
Affiliation(s)
- Paula López Diez
- Department of Applied Mathematics and Computer Science, Technical University of Denmark, Lyngby, Denmark.
| | - Josefine Vilsbøll Sundgaard
- Department of Applied Mathematics and Computer Science, Technical University of Denmark, Lyngby, Denmark; Novo Nordisk A/S, Denmark
| | - Jan Margeta
- KardioMe, Research & Development, Nova Dubnica, Slovakia; Oticon Medical, Research & Technology, Vallauris, France
| | - Khassan Diab
- Tashkent International Clinic, Tashkent, Uzbekistan
| | - François Patou
- Oticon Medical, Research & Technology group, Smørum, Denmark
| | - Rasmus R Paulsen
- Department of Applied Mathematics and Computer Science, Technical University of Denmark, Lyngby, Denmark
| |
Collapse
|
18
|
Seeböck P, Orlando JI, Michl M, Mai J, Schmidt-Erfurth U, Bogunović H. Anomaly guided segmentation: Introducing semantic context for lesion segmentation in retinal OCT using weak context supervision from anomaly detection. Med Image Anal 2024; 93:103104. [PMID: 38350222 DOI: 10.1016/j.media.2024.103104] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2022] [Revised: 12/01/2023] [Accepted: 02/05/2024] [Indexed: 02/15/2024]
Abstract
Automated lesion detection in retinal optical coherence tomography (OCT) scans has shown promise for several clinical applications, including diagnosis, monitoring and guidance of treatment decisions. However, segmentation models still struggle to achieve the desired results for some complex lesions or datasets that commonly occur in real-world, e.g. due to variability of lesion phenotypes, image quality or disease appearance. While several techniques have been proposed to improve them, one line of research that has not yet been investigated is the incorporation of additional semantic context through the application of anomaly detection models. In this study we experimentally show that incorporating weak anomaly labels to standard segmentation models consistently improves lesion segmentation results. This can be done relatively easy by detecting anomalies with a separate model and then adding these output masks as an extra class for training the segmentation model. This provides additional semantic context without requiring extra manual labels. We empirically validated this strategy using two in-house and two publicly available retinal OCT datasets for multiple lesion targets, demonstrating the potential of this generic anomaly guided segmentation approach to be used as an extra tool for improving lesion detection models.
Collapse
Affiliation(s)
- Philipp Seeböck
- Lab for Ophthalmic Image Analysis, Department of Ophthalmology and Optometry, Medical University of Vienna, Austria; Computational Imaging Research Lab, Department of Biomedical Imaging and Image-Guided Therapy, Medical University of Vienna, Austria.
| | - José Ignacio Orlando
- Lab for Ophthalmic Image Analysis, Department of Ophthalmology and Optometry, Medical University of Vienna, Austria; Yatiris Group at PLADEMA Institute, CONICET, Universidad Nacional del Centro de la Provincia de Buenos Aires, Gral. Pinto 399, Tandil, Buenos Aires, Argentina
| | - Martin Michl
- Lab for Ophthalmic Image Analysis, Department of Ophthalmology and Optometry, Medical University of Vienna, Austria
| | - Julia Mai
- Lab for Ophthalmic Image Analysis, Department of Ophthalmology and Optometry, Medical University of Vienna, Austria
| | - Ursula Schmidt-Erfurth
- Lab for Ophthalmic Image Analysis, Department of Ophthalmology and Optometry, Medical University of Vienna, Austria
| | - Hrvoje Bogunović
- Lab for Ophthalmic Image Analysis, Department of Ophthalmology and Optometry, Medical University of Vienna, Austria.
| |
Collapse
|
19
|
Gao Z, Su Y, Xia J, Cao RF, Ding Y, Zheng CH, Wei PJ. DeepFGRN: inference of gene regulatory network with regulation type based on directed graph embedding. Brief Bioinform 2024; 25:bbae143. [PMID: 38581416 PMCID: PMC10998536 DOI: 10.1093/bib/bbae143] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/17/2024] [Revised: 03/02/2024] [Accepted: 03/15/2024] [Indexed: 04/08/2024] Open
Abstract
The inference of gene regulatory networks (GRNs) from gene expression profiles has been a key issue in systems biology, prompting many researchers to develop diverse computational methods. However, most of these methods do not reconstruct directed GRNs with regulatory types because of the lack of benchmark datasets or defects in the computational methods. Here, we collect benchmark datasets and propose a deep learning-based model, DeepFGRN, for reconstructing fine gene regulatory networks (FGRNs) with both regulation types and directions. In addition, the GRNs of real species are always large graphs with direction and high sparsity, which impede the advancement of GRN inference. Therefore, DeepFGRN builds a node bidirectional representation module to capture the directed graph embedding representation of the GRN. Specifically, the source and target generators are designed to learn the low-dimensional dense embedding of the source and target neighbors of a gene, respectively. An adversarial learning strategy is applied to iteratively learn the real neighbors of each gene. In addition, because the expression profiles of genes with regulatory associations are correlative, a correlation analysis module is designed. Specifically, this module not only fully extracts gene expression features, but also captures the correlation between regulators and target genes. Experimental results show that DeepFGRN has a competitive capability for both GRN and FGRN inference. Potential biomarkers and therapeutic drugs for breast cancer, liver cancer, lung cancer and coronavirus disease 2019 are identified based on the candidate FGRNs, providing a possible opportunity to advance our knowledge of disease treatments.
Collapse
Affiliation(s)
- Zhen Gao
- The Key Laboratory of Intelligent Computing and Signal Processing of Ministry of Education, School of Computer Science and Technology, Anhui University, 111 Jiulong Road, Hefei, 230601, Anhui, China
| | - Yansen Su
- The Key Laboratory of Intelligent Computing and Signal Processing of Ministry of Education, School of Artificial Intelligence, Anhui University, 111 Jiulong Road, Hefei, 230601, Anhui, China
| | - Junfeng Xia
- Information Materials and Intelligent Sensing Laboratory of Anhui Province, Institute of Physical Science and Information Technology, Anhui University, 111 Jiulong Road, Hefei, 230601, Anhui, China
| | - Rui-Fen Cao
- The Key Laboratory of Intelligent Computing and Signal Processing of Ministry of Education, School of Computer Science and Technology, Anhui University, 111 Jiulong Road, Hefei, 230601, Anhui, China
| | - Yun Ding
- The Key Laboratory of Intelligent Computing and Signal Processing of Ministry of Education, School of Artificial Intelligence, Anhui University, 111 Jiulong Road, Hefei, 230601, Anhui, China
| | - Chun-Hou Zheng
- The Key Laboratory of Intelligent Computing and Signal Processing of Ministry of Education, School of Artificial Intelligence, Anhui University, 111 Jiulong Road, Hefei, 230601, Anhui, China
| | - Pi-Jing Wei
- Information Materials and Intelligent Sensing Laboratory of Anhui Province, Institute of Physical Science and Information Technology, Anhui University, 111 Jiulong Road, Hefei, 230601, Anhui, China
| |
Collapse
|
20
|
Su J, Shen H, Peng L, Hu D. Few-Shot Domain-Adaptive Anomaly Detection for Cross-Site Brain Images. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2024; 46:1819-1835. [PMID: 34748478 DOI: 10.1109/tpami.2021.3125686] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Early screening is essential for effective intervention and treatment of individuals with mental disorders. Functional magnetic resonance imaging (fMRI) is a noninvasive tool for depicting neural activity and has demonstrated strong potential as a technique for identifying mental disorders. Due to the difficulty in data collection and diagnosis, imaging data from patients are rare at a single site, whereas abundant healthy control data are available from public datasets. However, joint use of these data from multiple sites for classification model training is hindered by cross-domain distribution discrepancy and diverse label spaces. Herein, we propose few-shot domain-adaptive anomaly detection (FAAD) to achieve cross-site anomaly detection of brain images based on only a few labeled samples. We introduce domain adaptation to mitigate cross-domain distribution discrepancy and jointly align the general and conditional feature distributions of imaging data across multiple sites. We utilize fMRI data of healthy subjects in the Human Connectome Project (HCP) as the source domain and fMRI images from six independent sites, including patients with mental disorders and demographically matched healthy controls, as target domains. Experiments showed the superiority of the proposed method compared with binary classification, traditional anomaly detection methods, and several recognized domain adaptation methods.
Collapse
|
21
|
Meissen F, Breuer S, Knolle M, Buyx A, Müller R, Kaissis G, Wiestler B, Rückert D. (Predictable) performance bias in unsupervised anomaly detection. EBioMedicine 2024; 101:105002. [PMID: 38335791 PMCID: PMC10873649 DOI: 10.1016/j.ebiom.2024.105002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/21/2023] [Revised: 01/23/2024] [Accepted: 01/24/2024] [Indexed: 02/12/2024] Open
Abstract
BACKGROUND With the ever-increasing amount of medical imaging data, the demand for algorithms to assist clinicians has amplified. Unsupervised anomaly detection (UAD) models promise to aid in the crucial first step of disease detection. While previous studies have thoroughly explored fairness in supervised models in healthcare, for UAD, this has so far been unexplored. METHODS In this study, we evaluated how dataset composition regarding subgroups manifests in disparate performance of UAD models along multiple protected variables on three large-scale publicly available chest X-ray datasets. Our experiments were validated using two state-of-the-art UAD models for medical images. Finally, we introduced subgroup-AUROC (sAUROC), which aids in quantifying fairness in machine learning. FINDINGS Our experiments revealed empirical "fairness laws" (similar to "scaling laws" for Transformers) for training-dataset composition: Linear relationships between anomaly detection performance within a subpopulation and its representation in the training data. Our study further revealed performance disparities, even in the case of balanced training data, and compound effects that exacerbate the drop in performance for subjects associated with multiple adversely affected groups. INTERPRETATION Our study quantified the disparate performance of UAD models against certain demographic subgroups. Importantly, we showed that this unfairness cannot be mitigated by balanced representation alone. Instead, the representation of some subgroups seems harder to learn by UAD models than that of others. The empirical "fairness laws" discovered in our study make disparate performance in UAD models easier to estimate and aid in determining the most desirable dataset composition. FUNDING European Research Council Deep4MI.
Collapse
Affiliation(s)
- Felix Meissen
- Chair for AI in Healthcare and Medicine, Klinikum rechts der Isar der Technischen Universität München, Einsteinstr. 25, Munich, 81675, Germany.
| | - Svenja Breuer
- Department of Science, Technology and Society, School of Social Sciences and Technology, and Technical University of Munich, Arcisstr. 21, Munich, 80333, Germany; Department of Economics and Policy, School of Management, Technical University of Munich, Arcisstraße 21, 80333, Munich, Germany
| | - Moritz Knolle
- Chair for AI in Healthcare and Medicine, Klinikum rechts der Isar der Technischen Universität München, Einsteinstr. 25, Munich, 81675, Germany; Konrad Zuse School of Excellence in Reliable AI, Munich Data Science Institute (MDSI), Walther-von-Dyck-Str. 10, Garching, 85748, Germany
| | - Alena Buyx
- Department of Science, Technology and Society, School of Social Sciences and Technology, and Technical University of Munich, Arcisstr. 21, Munich, 80333, Germany; Institute for History and Ethics of Medicine, School of Medicine, Technical University of Munich, Prinzregentenstraße 68, Munich, 81675, Germany
| | - Ruth Müller
- Department of Science, Technology and Society, School of Social Sciences and Technology, and Technical University of Munich, Arcisstr. 21, Munich, 80333, Germany; Department of Economics and Policy, School of Management, Technical University of Munich, Arcisstraße 21, 80333, Munich, Germany
| | - Georgios Kaissis
- Chair for AI in Healthcare and Medicine, Klinikum rechts der Isar der Technischen Universität München, Einsteinstr. 25, Munich, 81675, Germany; Institute for Machine Learning in Biomedical Imaging, Helmholtz Munich, Ingolstädter Landstraße 1, 85764, Neuherberg, Germany; Department of Computing, Imperial College London, London, SW7 2AZ, UK
| | - Benedikt Wiestler
- Department of Diagnostic and Interventional Neuroradiology, Klinikum rechts der Isar, Ismaninger Str. 22, Munich, 81675, Germany; TranslaTUM, Center for Translational Cancer Research, Technical University of Munich, Ismaninger Str. 22, Munich, 81675, Germany
| | - Daniel Rückert
- Chair for AI in Healthcare and Medicine, Klinikum rechts der Isar der Technischen Universität München, Einsteinstr. 25, Munich, 81675, Germany; Department of Computing, Imperial College London, London, SW7 2AZ, UK
| |
Collapse
|
22
|
Liu H, Ni Z, Nie D, Shen D, Wang J, Tang Z. Multimodal Brain Tumor Segmentation Boosted by Monomodal Normal Brain Images. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2024; 33:1199-1210. [PMID: 38315584 DOI: 10.1109/tip.2024.3359815] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/07/2024]
Abstract
Many deep learning based methods have been proposed for brain tumor segmentation. Most studies focus on deep network internal structure to improve the segmentation accuracy, while valuable external information, such as normal brain appearance, is often ignored. Inspired by the fact that radiologists often screen lesion regions with normal appearance as reference in mind, in this paper, we propose a novel deep framework for brain tumor segmentation, where normal brain images are adopted as reference to compare with tumor brain images in a learned feature space. In this way, features at tumor regions, i.e., tumor-related features, can be highlighted and enhanced for accurate tumor segmentation. It is known that routine tumor brain images are multimodal, while normal brain images are often monomodal. This causes the feature comparison a big issue, i.e., multimodal vs. monomodal. To this end, we present a new feature alignment module (FAM) to make the feature distribution of monomodal normal brain images consistent/inconsistent with multimodal tumor brain images at normal/tumor regions, making the feature comparison effective. Both public (BraTS2022) and in-house tumor brain image datasets are used to evaluate our framework. Experimental results demonstrate that for both datasets, our framework can effectively improve the segmentation accuracy and outperforms the state-of-the-art segmentation methods. Codes are available at https://github.com/hb-liu/Normal-Brain-Boost-Tumor-Segmentation.
Collapse
|
23
|
Zingman I, Stierstorfer B, Lempp C, Heinemann F. Learning image representations for anomaly detection: Application to discovery of histological alterations in drug development. Med Image Anal 2024; 92:103067. [PMID: 38141454 DOI: 10.1016/j.media.2023.103067] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2022] [Revised: 12/01/2023] [Accepted: 12/19/2023] [Indexed: 12/25/2023]
Abstract
We present a system for anomaly detection in histopathological images. In histology, normal samples are usually abundant, whereas anomalous (pathological) cases are scarce or not available. Under such settings, one-class classifiers trained on healthy data can detect out-of-distribution anomalous samples. Such approaches combined with pre-trained Convolutional Neural Network (CNN) representations of images were previously employed for anomaly detection (AD). However, pre-trained off-the-shelf CNN representations may not be sensitive to abnormal conditions in tissues, while natural variations of healthy tissue may result in distant representations. To adapt representations to relevant details in healthy tissue we propose training a CNN on an auxiliary task that discriminates healthy tissue of different species, organs, and staining reagents. Almost no additional labeling workload is required, since healthy samples come automatically with aforementioned labels. During training we enforce compact image representations with a center-loss term, which further improves representations for AD. The proposed system outperforms established AD methods on a published dataset of liver anomalies. Moreover, it provided comparable results to conventional methods specifically tailored for quantification of liver anomalies. We show that our approach can be used for toxicity assessment of candidate drugs at early development stages and thereby may reduce expensive late-stage drug attrition.
Collapse
Affiliation(s)
- Igor Zingman
- Drug Discovery Sciences, Boehringer Ingelheim Pharma GmbH and Co., Biberach an der Riß, Germany.
| | - Birgit Stierstorfer
- Non-Clinical Drug Safety, Boehringer Ingelheim Pharma GmbH and Co., Biberach an der Riß, Germany
| | - Charlotte Lempp
- Drug Discovery Sciences, Boehringer Ingelheim Pharma GmbH and Co., Biberach an der Riß, Germany
| | - Fabian Heinemann
- Drug Discovery Sciences, Boehringer Ingelheim Pharma GmbH and Co., Biberach an der Riß, Germany.
| |
Collapse
|
24
|
Qu X, Liu Z, Wu CQ, Hou A, Yin X, Chen Z. MFGAN: Multimodal Fusion for Industrial Anomaly Detection Using Attention-Based Autoencoder and Generative Adversarial Network. SENSORS (BASEL, SWITZERLAND) 2024; 24:637. [PMID: 38276328 PMCID: PMC10818673 DOI: 10.3390/s24020637] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/30/2023] [Revised: 01/10/2024] [Accepted: 01/17/2024] [Indexed: 01/27/2024]
Abstract
Anomaly detection plays a critical role in ensuring safe, smooth, and efficient operation of machinery and equipment in industrial environments. With the wide deployment of multimodal sensors and the rapid development of Internet of Things (IoT), the data generated in modern industrial production has become increasingly diverse and complex. However, traditional methods for anomaly detection based on a single data source cannot fully utilize multimodal data to capture anomalies in industrial systems. To address this challenge, we propose a new model for anomaly detection in industrial environments using multimodal temporal data. This model integrates an attention-based autoencoder (AAE) and a generative adversarial network (GAN) to capture and fuse rich information from different data sources. Specifically, the AAE captures time-series dependencies and relevant features in each modality, and the GAN introduces adversarial regularization to enhance the model's ability to reconstruct normal time-series data. We conduct extensive experiments on real industrial data containing both measurements from a distributed control system (DCS) and acoustic signals, and the results demonstrate the performance superiority of the proposed model over the state-of-the-art TimesNet for anomaly detection, with an improvement of 5.6% in F1 score.
Collapse
Affiliation(s)
- Xinji Qu
- School of Information Science and Technology, Northwest University, Xi’an 710127, China; (X.Q.); (Z.L.); (A.H.); (X.Y.); (Z.C.)
| | - Zhuo Liu
- School of Information Science and Technology, Northwest University, Xi’an 710127, China; (X.Q.); (Z.L.); (A.H.); (X.Y.); (Z.C.)
| | - Chase Q. Wu
- Department of Data Science, New Jersey Institute of Technology, Newark, NJ 07102, USA
| | - Aiqin Hou
- School of Information Science and Technology, Northwest University, Xi’an 710127, China; (X.Q.); (Z.L.); (A.H.); (X.Y.); (Z.C.)
| | - Xiaoyan Yin
- School of Information Science and Technology, Northwest University, Xi’an 710127, China; (X.Q.); (Z.L.); (A.H.); (X.Y.); (Z.C.)
| | - Zhulian Chen
- School of Information Science and Technology, Northwest University, Xi’an 710127, China; (X.Q.); (Z.L.); (A.H.); (X.Y.); (Z.C.)
| |
Collapse
|
25
|
Siddiquee MMR, Shah J, Wu T, Chong C, Schwedt TJ, Dumkrieger G, Nikolova S, Li B. Brainomaly: Unsupervised Neurologic Disease Detection Utilizing Unannotated T1-weighted Brain MR Images. IEEE WINTER CONFERENCE ON APPLICATIONS OF COMPUTER VISION. IEEE WINTER CONFERENCE ON APPLICATIONS OF COMPUTER VISION 2024; 2024:7558-7567. [PMID: 38720667 PMCID: PMC11078334 DOI: 10.1109/wacv57701.2024.00740] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/12/2024]
Abstract
Harnessing the power of deep neural networks in the medical imaging domain is challenging due to the difficulties in acquiring large annotated datasets, especially for rare diseases, which involve high costs, time, and effort for annotation. Unsupervised disease detection methods, such as anomaly detection, can significantly reduce human effort in these scenarios. While anomaly detection typically focuses on learning from images of healthy subjects only, real-world situations often present unannotated datasets with a mixture of healthy and diseased subjects. Recent studies have demonstrated that utilizing such unannotated images can improve unsupervised disease and anomaly detection. However, these methods do not utilize knowledge specific to registered neuroimages, resulting in a subpar performance in neurologic disease detection. To address this limitation, we propose Brainomaly, a GAN-based image-to-image translation method specifically designed for neurologic disease detection. Brainomaly not only offers tailored image-to-image translation suitable for neuroimages but also leverages unannotated mixed images to achieve superior neurologic disease detection. Additionally, we address the issue of model selection for inference without annotated samples by proposing a pseudo-AUC metric, further enhancing Brainomaly's detection performance. Extensive experiments and ablation studies demonstrate that Brainomaly outperforms existing state-of-the-art unsupervised disease and anomaly detection methods by significant margins in Alzheimer's disease detection using a publicly available dataset and headache detection using an institutional dataset. The code is available from https://github.com/mahfuzmohammad/Brainomaly.
Collapse
Affiliation(s)
| | - Jay Shah
- Arizona State University
- ASU-Mayo Center for Innovative Imaging
| | - Teresa Wu
- Arizona State University
- ASU-Mayo Center for Innovative Imaging
| | | | | | | | | | - Baoxin Li
- Arizona State University
- ASU-Mayo Center for Innovative Imaging
| |
Collapse
|
26
|
Ravi D, Barkhof F, Alexander DC, Puglisi L, Parker GJM, Eshaghi A. An efficient semi-supervised quality control system trained using physics-based MRI-artefact generators and adversarial training. Med Image Anal 2024; 91:103033. [PMID: 38000256 DOI: 10.1016/j.media.2023.103033] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/10/2022] [Revised: 10/04/2023] [Accepted: 11/13/2023] [Indexed: 11/26/2023]
Abstract
Large medical imaging data sets are becoming increasingly available. A common challenge in these data sets is to ensure that each sample meets minimum quality requirements devoid of significant artefacts. Despite a wide range of existing automatic methods having been developed to identify imperfections and artefacts in medical imaging, they mostly rely on data-hungry methods. In particular, the scarcity of artefact-containing scans available for training has been a major obstacle in the development and implementation of machine learning in clinical research. To tackle this problem, we propose a novel framework having four main components: (1) a set of artefact generators inspired by magnetic resonance physics to corrupt brain MRI scans and augment a training dataset, (2) a set of abstract and engineered features to represent images compactly, (3) a feature selection process that depends on the class of artefact to improve classification performance, and (4) a set of Support Vector Machine (SVM) classifiers trained to identify artefacts. Our novel contributions are threefold: first, we use the novel physics-based artefact generators to generate synthetic brain MRI scans with controlled artefacts as a data augmentation technique. This will avoid the labour-intensive collection and labelling process of scans with rare artefacts. Second, we propose a large pool of abstract and engineered image features developed to identify 9 different artefacts for structural MRI. Finally, we use an artefact-based feature selection block that, for each class of artefacts, finds the set of features that provide the best classification performance. We performed validation experiments on a large data set of scans with artificially-generated artefacts, and in a multiple sclerosis clinical trial where real artefacts were identified by experts, showing that the proposed pipeline outperforms traditional methods. In particular, our data augmentation increases performance by up to 12.5 percentage points on the accuracy, F1, F2, precision and recall. At the same time, the computation cost of our pipeline remains low - less than a second to process a single scan - with the potential for real-time deployment. Our artefact simulators obtained using adversarial learning enable the training of a quality control system for brain MRI that otherwise would have required a much larger number of scans in both supervised and unsupervised settings. We believe that systems for quality control will enable a wide range of high-throughput clinical applications based on the use of automatic image-processing pipelines.
Collapse
Affiliation(s)
- Daniele Ravi
- Centre for Medical Image Computing (CMIC), Department of Computer Science, University College London, UK; Queen Square Analytics, London, UK; School of Physics, Engineering and Computer Science, University of Hertfordshire, Hatfield, UK.
| | - Frederik Barkhof
- Department of Medical Physics and Biomedical Engineering, University College London, UK; Department of Radiology and Nuclear Medicine, Neuroscience Campus Amsterdam, VU University Medical Center, Amsterdam, The Netherlands; Queen Square Analytics, London, UK; NMR Unit, Queen Square Multiple Sclerosis Centre, Department of Neuroinflammation, Queen Square Institutes of Neurology, Faculty of Brain Sciences, University College London, London, UK; Department of Brain Repair and Rehabilitation, Queen Square Institute of Neurology, University College London, London, UK
| | - Daniel C Alexander
- Centre for Medical Image Computing (CMIC), Department of Computer Science, University College London, UK; Queen Square Analytics, London, UK
| | | | - Geoffrey J M Parker
- Department of Medical Physics and Biomedical Engineering, University College London, UK; Queen Square Analytics, London, UK; NMR Unit, Queen Square Multiple Sclerosis Centre, Department of Neuroinflammation, Queen Square Institutes of Neurology, Faculty of Brain Sciences, University College London, London, UK
| | - Arman Eshaghi
- Centre for Medical Image Computing (CMIC), Department of Computer Science, University College London, UK; Queen Square Analytics, London, UK; NMR Unit, Queen Square Multiple Sclerosis Centre, Department of Neuroinflammation, Queen Square Institutes of Neurology, Faculty of Brain Sciences, University College London, London, UK
| |
Collapse
|
27
|
Heger KA, Waldstein SM. Artificial intelligence in retinal imaging: current status and future prospects. Expert Rev Med Devices 2024; 21:73-89. [PMID: 38088362 DOI: 10.1080/17434440.2023.2294364] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/29/2023] [Accepted: 12/09/2023] [Indexed: 12/19/2023]
Abstract
INTRODUCTION The steadily growing and aging world population, in conjunction with continuously increasing prevalences of vision-threatening retinal diseases, is placing an increasing burden on the global healthcare system. The main challenges within retinology involve identifying the comparatively few patients requiring therapy within the large mass, the assurance of comprehensive screening for retinal disease and individualized therapy planning. In order to sustain high-quality ophthalmic care in the future, the incorporation of artificial intelligence (AI) technologies into our clinical practice represents a potential solution. AREAS COVERED This review sheds light onto already realized and promising future applications of AI techniques in retinal imaging. The main attention is directed at the application in diabetic retinopathy and age-related macular degeneration. The principles of use in disease screening, grading, therapeutic planning and prediction of future developments are explained based on the currently available literature. EXPERT OPINION The recent accomplishments of AI in retinal imaging indicate that its implementation into our daily practice is likely to fundamentally change the ophthalmic healthcare system and bring us one step closer to the goal of individualized treatment. However, it must be emphasized that the aim is to optimally support clinicians by gradually incorporating AI approaches, rather than replacing ophthalmologists.
Collapse
Affiliation(s)
- Katharina A Heger
- Department of Ophthalmology, Landesklinikum Mistelbach-Gaenserndorf, Mistelbach, Austria
| | - Sebastian M Waldstein
- Department of Ophthalmology, Landesklinikum Mistelbach-Gaenserndorf, Mistelbach, Austria
| |
Collapse
|
28
|
Lee SM, Han Y, Cho KH. Deep learning untangles the resistance mechanism of p53 reactivator in lung cancer cells. iScience 2023; 26:108377. [PMID: 38034356 PMCID: PMC10682260 DOI: 10.1016/j.isci.2023.108377] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/01/2023] [Revised: 09/12/2023] [Accepted: 10/27/2023] [Indexed: 12/02/2023] Open
Abstract
Tumor suppressor p53 plays a pivotal role in suppressing cancer, so various drugs has been suggested to upregulate its function. However, drug resistance is still the biggest hurdle to be overcome. To address this, we developed a deep learning model called AnoDAN (anomalous gene detection using generative adversarial networks and graph neural networks for overcoming drug resistance) that unravels the hidden resistance mechanisms and identifies a combinatorial target to overcome the resistance. Our findings reveal that the TGF-β signaling pathway, alongside the p53 signaling pathway, mediates the resistance, with THBS1 serving as a core regulatory target in both pathways. Experimental validation in lung cancer cells confirms the effects of THBS1 on responsiveness to a p53 reactivator. We further discovered the positive feedback loop between THBS1 and the TGF-β pathway as the main source of resistance. This study enhances our understanding of p53 regulation and offers insights into overcoming drug resistance.
Collapse
Affiliation(s)
- Soo Min Lee
- Laboratory for Systems Biology and Bio-inspired Engineering, Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea
| | - Younghyun Han
- Laboratory for Systems Biology and Bio-inspired Engineering, Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea
| | - Kwang-Hyun Cho
- Laboratory for Systems Biology and Bio-inspired Engineering, Department of Bio and Brain Engineering, Korea Advanced Institute of Science and Technology (KAIST), Daejeon 34141, Republic of Korea
| |
Collapse
|
29
|
Kascenas A, Sanchez P, Schrempf P, Wang C, Clackett W, Mikhael SS, Voisey JP, Goatman K, Weir A, Pugeault N, Tsaftaris SA, O'Neil AQ. The role of noise in denoising models for anomaly detection in medical images. Med Image Anal 2023; 90:102963. [PMID: 37769551 DOI: 10.1016/j.media.2023.102963] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/15/2022] [Revised: 08/22/2023] [Accepted: 09/07/2023] [Indexed: 10/03/2023]
Abstract
Pathological brain lesions exhibit diverse appearance in brain images, in terms of intensity, texture, shape, size, and location. Comprehensive sets of data and annotations are difficult to acquire. Therefore, unsupervised anomaly detection approaches have been proposed using only normal data for training, with the aim of detecting outlier anomalous voxels at test time. Denoising methods, for instance classical denoising autoencoders (DAEs) and more recently emerging diffusion models, are a promising approach, however naive application of pixelwise noise leads to poor anomaly detection performance. We show that optimization of the spatial resolution and magnitude of the noise improves the performance of different model training regimes, with similar noise parameter adjustments giving good performance for both DAEs and diffusion models. Visual inspection of the reconstructions suggests that the training noise influences the trade-off between the extent of the detail that is reconstructed and the extent of erasure of anomalies, both of which contribute to better anomaly detection performance. We validate our findings on two real-world datasets (tumor detection in brain MRI and hemorrhage/ischemia/tumor detection in brain CT), showing good detection on diverse anomaly appearances. Overall, we find that a DAE trained with coarse noise is a fast and simple method that gives state-of-the-art accuracy. Diffusion models applied to anomaly detection are as yet in their infancy and provide a promising avenue for further research. Code for our DAE model and coarse noise is provided at: https://github.com/AntanasKascenas/DenoisingAE.
Collapse
Affiliation(s)
- Antanas Kascenas
- Canon Medical Research Europe, Bonnington Bond, 2 Anderson Pl, Edinburgh EH6 5NP, United Kingdom; University of Glasgow, Glasgow G12 8QQ, United Kingdom.
| | - Pedro Sanchez
- University of Edinburgh, Kings Buildings, Edinburgh EH9 3FG, United Kingdom
| | - Patrick Schrempf
- Canon Medical Research Europe, Bonnington Bond, 2 Anderson Pl, Edinburgh EH6 5NP, United Kingdom
| | - Chaoyang Wang
- Canon Medical Research Europe, Bonnington Bond, 2 Anderson Pl, Edinburgh EH6 5NP, United Kingdom
| | - William Clackett
- Canon Medical Research Europe, Bonnington Bond, 2 Anderson Pl, Edinburgh EH6 5NP, United Kingdom
| | - Shadia S Mikhael
- Canon Medical Research Europe, Bonnington Bond, 2 Anderson Pl, Edinburgh EH6 5NP, United Kingdom
| | - Jeremy P Voisey
- Canon Medical Research Europe, Bonnington Bond, 2 Anderson Pl, Edinburgh EH6 5NP, United Kingdom
| | - Keith Goatman
- Canon Medical Research Europe, Bonnington Bond, 2 Anderson Pl, Edinburgh EH6 5NP, United Kingdom
| | - Alexander Weir
- Canon Medical Research Europe, Bonnington Bond, 2 Anderson Pl, Edinburgh EH6 5NP, United Kingdom
| | | | - Sotirios A Tsaftaris
- University of Edinburgh, Kings Buildings, Edinburgh EH9 3FG, United Kingdom; The Alan Turing Institute, London, United Kingdom
| | - Alison Q O'Neil
- Canon Medical Research Europe, Bonnington Bond, 2 Anderson Pl, Edinburgh EH6 5NP, United Kingdom; University of Edinburgh, Kings Buildings, Edinburgh EH9 3FG, United Kingdom
| |
Collapse
|
30
|
Dong H, Zhang Y, Gu H, Konz N, Zhang Y, Mazurowski MA. SWSSL: Sliding Window-Based Self-Supervised Learning for Anomaly Detection in High-Resolution Images. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:3860-3870. [PMID: 37695965 PMCID: PMC10766076 DOI: 10.1109/tmi.2023.3314318] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 09/13/2023]
Abstract
Anomaly detection (AD) aims to determine if an instance has properties different from those seen in normal cases. The success of this technique depends on how well a neural network learns from normal instances. We observe that the learning difficulty scales exponentially with the input resolution, making it infeasible to apply AD to high-resolution images. Resizing them to a lower resolution is a compromising solution and does not align with clinical practice where the diagnosis could depend on image details. In this work, we propose to train the network and perform inference at the patch level, through the sliding window algorithm. This simple operation allows the network to receive high-resolution images but introduces additional training difficulties, including inconsistent image structure and higher variance. We address these concerns by setting the network's objective to learn augmentation-invariant features. We further study the augmentation function in the context of medical imaging. In particular, we observe that the resizing operation, a key augmentation in general computer vision literature, is detrimental to detection accuracy, and the inverting operation can be beneficial. We also propose a new module that encourages the network to learn from adjacent patches to boost detection performance. Extensive experiments are conducted on breast tomosynthesis and chest X-ray datasets and our method improves 8.03% and 5.66% AUC on image-level classification respectively over the current leading techniques. The experimental results demonstrate the effectiveness of our approach.
Collapse
|
31
|
Chen RQ, Joffe B, Casteleiro Costa P, Filan C, Wang B, Balakirsky S, Robles F, Roy K, Li J. Real-time semantic segmentation and anomaly detection of functional images for cell therapy manufacturing. Cytotherapy 2023; 25:1361-1369. [PMID: 37725031 PMCID: PMC10719834 DOI: 10.1016/j.jcyt.2023.08.011] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2023] [Revised: 08/21/2023] [Accepted: 08/24/2023] [Indexed: 09/21/2023]
Abstract
BACKGROUND AIMS Cell therapy is a promising treatment method that uses living cells to address a variety of diseases and conditions, including cardiovascular diseases, neurologic disorders and certain cancers. As interest in cell therapy grows, there is a need to shift to a more efficient, scalable and automated manufacturing process that can produce high-quality products at a lower cost. METHODS One way to achieve this is using non-invasive imaging and real-time image analysis techniques to monitor and control the manufacturing process. This work presents a machine learning-based image analysis pipeline that includes semantic segmentation and anomaly detection capabilities. RESULTS/CONCLUSIONS This method can be easily implemented even when given a limited dataset of annotated images, is able to segment cells and debris and can identify anomalies such as contamination or hardware failure.
Collapse
Affiliation(s)
- Rui Qi Chen
- H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia, USA
| | - Benjamin Joffe
- Georgia Tech Research Institute, Georgia Institute of Technology, Atlanta, Georgia, USA
| | - Paloma Casteleiro Costa
- Wallace H. Coulter Department of Biomedical Engineering at Georgia Tech and Emory University, Atlanta, Georgia, USA
| | - Caroline Filan
- Wallace H. Coulter Department of Biomedical Engineering at Georgia Tech and Emory University, Atlanta, Georgia, USA
| | - Bryan Wang
- Wallace H. Coulter Department of Biomedical Engineering at Georgia Tech and Emory University, Atlanta, Georgia, USA
| | - Stephen Balakirsky
- Georgia Tech Research Institute, Georgia Institute of Technology, Atlanta, Georgia, USA
| | - Francisco Robles
- Wallace H. Coulter Department of Biomedical Engineering at Georgia Tech and Emory University, Atlanta, Georgia, USA
| | - Krishnendu Roy
- Wallace H. Coulter Department of Biomedical Engineering at Georgia Tech and Emory University, Atlanta, Georgia, USA
| | - Jing Li
- H. Milton Stewart School of Industrial and Systems Engineering, Georgia Institute of Technology, Atlanta, Georgia, USA.
| |
Collapse
|
32
|
Tian Y, Liu F, Pang G, Chen Y, Liu Y, Verjans JW, Singh R, Carneiro G. Self-supervised pseudo multi-class pre-training for unsupervised anomaly detection and segmentation in medical images. Med Image Anal 2023; 90:102930. [PMID: 37657364 DOI: 10.1016/j.media.2023.102930] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/16/2022] [Revised: 07/12/2023] [Accepted: 08/07/2023] [Indexed: 09/03/2023]
Abstract
Unsupervised anomaly detection (UAD) methods are trained with normal (or healthy) images only, but during testing, they are able to classify normal and abnormal (or disease) images. UAD is an important medical image analysis (MIA) method to be applied in disease screening problems because the training sets available for those problems usually contain only normal images. However, the exclusive reliance on normal images may result in the learning of ineffective low-dimensional image representations that are not sensitive enough to detect and segment unseen abnormal lesions of varying size, appearance, and shape. Pre-training UAD methods with self-supervised learning, based on computer vision techniques, can mitigate this challenge, but they are sub-optimal because they do not explore domain knowledge for designing the pretext tasks, and their contrastive learning losses do not try to cluster the normal training images, which may result in a sparse distribution of normal images that is ineffective for anomaly detection. In this paper, we propose a new self-supervised pre-training method for MIA UAD applications, named Pseudo Multi-class Strong Augmentation via Contrastive Learning (PMSACL). PMSACL consists of a novel optimisation method that contrasts a normal image class from multiple pseudo classes of synthesised abnormal images, with each class enforced to form a dense cluster in the feature space. In the experiments, we show that our PMSACL pre-training improves the accuracy of SOTA UAD methods on many MIA benchmarks using colonoscopy, fundus screening and Covid-19 Chest X-ray datasets.
Collapse
Affiliation(s)
- Yu Tian
- Harvard Ophthalmology AI Lab, Harvard Medical School, United States of America.
| | - Fengbei Liu
- Australian Institute for Machine Learning, University of Adelaide, Australia
| | - Guansong Pang
- School of Computing and Information Systems, Singapore Management University, Singapore
| | - Yuanhong Chen
- Australian Institute for Machine Learning, University of Adelaide, Australia
| | - Yuyuan Liu
- Australian Institute for Machine Learning, University of Adelaide, Australia
| | - Johan W Verjans
- Australian Institute for Machine Learning, University of Adelaide, Australia; South Australian Health and Medical Research Institute, Australia; Faculty of Health and Medical Sciences, University of Adelaide, Australia
| | - Rajvinder Singh
- Faculty of Health and Medical Sciences, University of Adelaide, Australia
| | - Gustavo Carneiro
- Centre for Vision, Speech and Signal Processing, University of Surrey, United Kingdom
| |
Collapse
|
33
|
Luo J, Zhang J. A Method for Image Anomaly Detection Based on Distillation and Reconstruction. SENSORS (BASEL, SWITZERLAND) 2023; 23:9281. [PMID: 38005667 PMCID: PMC10674649 DOI: 10.3390/s23229281] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/28/2023] [Revised: 10/31/2023] [Accepted: 11/18/2023] [Indexed: 11/26/2023]
Abstract
Image anomaly detection is a trending research topic in computer vision. The objective is to build models using available normal samples to detect various abnormal images without depending on real abnormal samples. It has high research significance and value for applications in the detection of defects in product appearance, medical image analysis, hyperspectral image processing, and other fields. This paper proposes an image anomaly detection algorithm based on feature distillation and an autoencoder structure, which uses the feature distillation structure of a dual-teacher network to train the encoder, thus suppressing the reconstruction of abnormal regions. This system also introduces an attention mechanism to highlight the detection objects, achieving effective detection of different defects in product appearance. In addition, this paper proposes a method for anomaly evaluation based on patch similarity that calculates the difference between the reconstructed image and the input image according to different regions of the image, thus improving the sensitivity and accuracy of the anomaly score. This paper conducts experiments on several datasets, and the results show that the proposed algorithm has superior performance in image anomaly detection. It achieves 98.8% average AUC on the SMDC-DET dataset and 98.9% average AUC on the MVTec-AD dataset.
Collapse
Affiliation(s)
- Jiaxiang Luo
- School of Automation Science and Engineering, South China University of Technology, Guangzhou 510641, China;
| | | |
Collapse
|
34
|
Sato Y, Sato J, Tomiyama N, Kido S. High-quality semi-supervised anomaly detection with generative adversarial networks. Int J Comput Assist Radiol Surg 2023:10.1007/s11548-023-03031-9. [PMID: 37943467 DOI: 10.1007/s11548-023-03031-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/14/2023] [Accepted: 10/16/2023] [Indexed: 11/10/2023]
Abstract
PURPOSE The visualization of an anomaly area is easier in anomaly detection methods that use generative models rather than classification models. However, achieving both anomaly detection accuracy and a clear visualization of anomalous areas is challenging. This study aimed to establish a method that combines both detection accuracy and clear visualization of anomalous areas using a generative adversarial network (GAN). METHODS In this study, StyleGAN2 with adaptive discriminator augmentation (StyleGAN2-ADA), which can generate high-resolution and high-quality images with limited number of datasets, was used as the image generation model, and pixel-to-style-to-pixel (pSp) encoder was used to convert images into intermediate latent variables. We combined existing methods for training and proposed a method for calculating anomaly scores using intermediate latent variables. The proposed method, which combines these two methods, is called high-quality anomaly GAN (HQ-AnoGAN). RESULTS The experimental results obtained using three datasets demonstrated that HQ-AnoGAN has equal or better detection accuracy than the existing methods. The results of the visualization of abnormal areas using the generated images showed that HQ-AnoGAN could generate more natural images than the existing methods and was qualitatively more accurate in the visualization of abnormal areas. CONCLUSION In this study, HQ-AnoGAN comprising StyleGAN2-ADA and pSp encoder was proposed with an optimal anomaly score calculation method. The experimental results show that HQ-AnoGAN can achieve both high abnormality detection accuracy and clear visualization of abnormal areas; thus, HQ-AnoGAN demonstrates significant potential for application in medical imaging diagnosis cases where an explanation of diagnosis is required.
Collapse
Affiliation(s)
- Yuki Sato
- Systems and Information Engineering Master's Program in Computer Science, University of Tsukuba, 1-1-1 Tenoudai, Tsukuba City, Ibaraki, 305-0821, Japan.
| | - Junya Sato
- Department of Artificial Intelligence Diagnostic Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka, 565-0871, Japan
- Department of Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka, 565-0871, Japan
| | - Noriyuki Tomiyama
- Department of Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka, 565-0871, Japan
| | - Shoji Kido
- Department of Artificial Intelligence Diagnostic Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka, 565-0871, Japan
| |
Collapse
|
35
|
Wargnier-Dauchelle V, Grenier T, Durand-Dubief F, Cotton F, Sdika M. A Weakly Supervised Gradient Attribution Constraint for Interpretable Classification and Anomaly Detection. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:3336-3347. [PMID: 37276116 DOI: 10.1109/tmi.2023.3282789] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/07/2023]
Abstract
The lack of interpretability of deep learning reduces understanding of what happens when a network does not work as expected and hinders its use in critical fields like medicine, which require transparency of decisions. For example, a healthy vs pathological classification model should rely on radiological signs and not on some training dataset biases. Several post-hoc models have been proposed to explain the decision of a trained network. However, they are very seldom used to enforce interpretability during training and none in accordance with the classification. In this paper, we propose a new weakly supervised method for both interpretable healthy vs pathological classification and anomaly detection. A new loss function is added to a standard classification model to constrain each voxel of healthy images to drive the network decision towards the healthy class according to gradient-based attributions. This constraint reveals pathological structures for patient images, allowing their unsupervised segmentation. Moreover, we advocate both theoretically and experimentally, that constrained training with the simple Gradient attribution is similar to constraints with the heavier Expected Gradient, consequently reducing the computational cost. We also propose a combination of attributions during the constrained training making the model robust to the attribution choice at inference. Our proposition was evaluated on two brain pathologies: tumors and multiple sclerosis. This new constraint provides a more relevant classification, with a more pathology-driven decision. For anomaly detection, the proposed method outperforms state-of-the-art especially on difficult multiple sclerosis lesions segmentation task with a 15 points Dice improvement.
Collapse
|
36
|
Schmetterer L, Scholl H, Garhöfer G, Janeschitz-Kriegl L, Corvi F, Sadda SR, Medeiros FA. Endpoints for clinical trials in ophthalmology. Prog Retin Eye Res 2023; 97:101160. [PMID: 36599784 DOI: 10.1016/j.preteyeres.2022.101160] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2022] [Revised: 12/22/2022] [Accepted: 12/28/2022] [Indexed: 01/03/2023]
Abstract
With the identification of novel targets, the number of interventional clinical trials in ophthalmology has increased. Visual acuity has for a long time been considered the gold standard endpoint for clinical trials, but in the recent years it became evident that other endpoints are required for many indications including geographic atrophy and inherited retinal disease. In glaucoma the currently available drugs were approved based on their IOP lowering capacity. Some recent findings do, however, indicate that at the same level of IOP reduction, not all drugs have the same effect on visual field progression. For neuroprotection trials in glaucoma, novel surrogate endpoints are required, which may either include functional or structural parameters or a combination of both. A number of potential surrogate endpoints for ophthalmology clinical trials have been identified, but their validation is complicated and requires solid scientific evidence. In this article we summarize candidates for clinical endpoints in ophthalmology with a focus on retinal disease and glaucoma. Functional and structural biomarkers, as well as quality of life measures are discussed, and their potential to serve as endpoints in pivotal trials is critically evaluated.
Collapse
Affiliation(s)
- Leopold Schmetterer
- Singapore Eye Research Institute, Singapore; SERI-NTU Advanced Ocular Engineering (STANCE), Singapore; Academic Clinical Program, Duke-NUS Medical School, Singapore; School of Chemistry, Chemical Engineering and Biotechnology, Nanyang Technological University, Singapore; Department of Clinical Pharmacology, Medical University Vienna, Vienna, Austria; Center for Medical Physics and Biomedical Engineering, Medical University Vienna, Vienna, Austria; Institute of Molecular and Clinical Ophthalmology, Basel, Switzerland.
| | - Hendrik Scholl
- Institute of Molecular and Clinical Ophthalmology, Basel, Switzerland; Department of Ophthalmology, University of Basel, Basel, Switzerland
| | - Gerhard Garhöfer
- Department of Clinical Pharmacology, Medical University Vienna, Vienna, Austria
| | - Lucas Janeschitz-Kriegl
- Institute of Molecular and Clinical Ophthalmology, Basel, Switzerland; Department of Ophthalmology, University of Basel, Basel, Switzerland
| | - Federico Corvi
- Eye Clinic, Department of Biomedical and Clinical Sciences "Luigi Sacco", University of Milan, Italy
| | - SriniVas R Sadda
- Doheny Eye Institute, Los Angeles, CA, USA; Department of Ophthalmology, David Geffen School of Medicine at University of California, Los Angeles, CA, USA
| | - Felipe A Medeiros
- Vision, Imaging and Performance Laboratory, Department of Ophthalmology, Duke Eye Center, Duke University, Durham, NC, USA
| |
Collapse
|
37
|
Xu K, Li T, Khan MS, Gao R, Antic SL, Huo Y, Sandler KL, Maldonado F, Landman BA. Body composition assessment with limited field-of-view computed tomography: A semantic image extension perspective. Med Image Anal 2023; 88:102852. [PMID: 37276799 PMCID: PMC10527087 DOI: 10.1016/j.media.2023.102852] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2022] [Revised: 01/30/2023] [Accepted: 05/23/2023] [Indexed: 06/07/2023]
Abstract
Field-of-view (FOV) tissue truncation beyond the lungs is common in routine lung screening computed tomography (CT). This poses limitations for opportunistic CT-based body composition (BC) assessment as key anatomical structures are missing. Traditionally, extending the FOV of CT is considered as a CT reconstruction problem using limited data. However, this approach relies on the projection domain data which might not be available in application. In this work, we formulate the problem from the semantic image extension perspective which only requires image data as inputs. The proposed two-stage method identifies a new FOV border based on the estimated extent of the complete body and imputes missing tissues in the truncated region. The training samples are simulated using CT slices with complete body in FOV, making the model development self-supervised. We evaluate the validity of the proposed method in automatic BC assessment using lung screening CT with limited FOV. The proposed method effectively restores the missing tissues and reduces BC assessment error introduced by FOV tissue truncation. In the BC assessment for large-scale lung screening CT datasets, this correction improves both the intra-subject consistency and the correlation with anthropometric approximations. The developed method is available at https://github.com/MASILab/S-EFOV.
Collapse
Affiliation(s)
- Kaiwen Xu
- Vanderbilt University, 2301 Vanderbilt Place, Nashville, 37235, United States.
| | - Thomas Li
- Vanderbilt University, 2301 Vanderbilt Place, Nashville, 37235, United States
| | - Mirza S Khan
- Vanderbilt University Medical Center, 1211 Medical Center Drive, Nashville, 37232, United States
| | - Riqiang Gao
- Vanderbilt University, 2301 Vanderbilt Place, Nashville, 37235, United States
| | - Sanja L Antic
- Vanderbilt University Medical Center, 1211 Medical Center Drive, Nashville, 37232, United States
| | - Yuankai Huo
- Vanderbilt University, 2301 Vanderbilt Place, Nashville, 37235, United States
| | - Kim L Sandler
- Vanderbilt University Medical Center, 1211 Medical Center Drive, Nashville, 37232, United States
| | - Fabien Maldonado
- Vanderbilt University Medical Center, 1211 Medical Center Drive, Nashville, 37232, United States
| | - Bennett A Landman
- Vanderbilt University, 2301 Vanderbilt Place, Nashville, 37235, United States; Vanderbilt University Medical Center, 1211 Medical Center Drive, Nashville, 37232, United States
| |
Collapse
|
38
|
Shin K, Lee JS, Lee JY, Lee H, Kim J, Byeon JS, Jung HY, Kim DH, Kim N. An Image Turing Test on Realistic Gastroscopy Images Generated by Using the Progressive Growing of Generative Adversarial Networks. J Digit Imaging 2023; 36:1760-1769. [PMID: 36914855 PMCID: PMC10406771 DOI: 10.1007/s10278-023-00803-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2022] [Revised: 02/21/2023] [Accepted: 02/23/2023] [Indexed: 03/16/2023] Open
Abstract
Generative adversarial networks (GAN) in medicine are valuable techniques for augmenting unbalanced rare data, anomaly detection, and avoiding patient privacy issues. However, there were limits to generating high-quality endoscopic images with various characteristics, such as peristalsis, viewpoints, light sources, and mucous patterns. This study used the progressive growing of GAN (PGGAN) within the normal distribution dataset to confirm the ability to generate high-quality gastrointestinal images and investigated what barriers PGGAN has to generate endoscopic images. We trained the PGGAN with 107,060 gastroscopy images from 4165 normal patients to generate highly realistic 5122 pixel-sized images. For the evaluation, visual Turing tests were conducted on 100 real and 100 synthetic images to distinguish the authenticity of images by 19 endoscopists. The endoscopists were divided into three groups based on their years of clinical experience for subgroup analysis. The overall accuracy, sensitivity, and specificity of the 19 endoscopist groups were 61.3%, 70.3%, and 52.4%, respectively. The mean accuracy of the three endoscopist groups was 62.4 [Group I], 59.8 [Group II], and 59.1% [Group III], which was not considered a significant difference. There were no statistically significant differences in the location of the stomach. However, the real images with the anatomical landmark pylorus had higher detection sensitivity. The images generated by PGGAN showed highly realistic depictions that were difficult to distinguish, regardless of their expertise as endoscopists. However, it was necessary to establish GANs that could better represent the rugal folds and mucous membrane texture.
Collapse
Affiliation(s)
- Keewon Shin
- Biomedical Engineering Research Center, Asan Medical Center, Seoul, Republic of Korea
| | - Jung Su Lee
- Department of Gastroenterology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea
- Seoul Samsung Internal Medicine Clinic, Seoul, Republic of Korea
| | - Ji Young Lee
- Department of Health Screening and Promotion Center, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea
| | - Hyunsu Lee
- Department of Medical Informatics, Keimyung University School of Medicine, Daegu, Republic of Korea
| | - Jeongseok Kim
- Department of Internal Medicine, Keimyung University School of Medicine, Daegu, Republic of Korea
| | - Jeong-Sik Byeon
- Department of Gastroenterology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea
| | - Hwoon-Yong Jung
- Department of Gastroenterology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea
| | - Do Hoon Kim
- Department of Gastroenterology, University of Ulsan College of Medicine, Asan Medical Center, Seoul, Republic of Korea.
| | - Namkug Kim
- Biomedical Engineering Research Center, Asan Medical Center, Seoul, Republic of Korea.
- Department of Convergence Medicine, University of Ulsan College of Medicine & Asan Medical Center, Seoul, Republic of Korea.
| |
Collapse
|
39
|
Tripathi A, Kumar P, Mayya V, Tulsani A. Generating OCT B-Scan DME images using optimized Generative Adversarial Networks (GANs). Heliyon 2023; 9:e18773. [PMID: 37609420 PMCID: PMC10440457 DOI: 10.1016/j.heliyon.2023.e18773] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/03/2023] [Revised: 07/26/2023] [Accepted: 07/27/2023] [Indexed: 08/24/2023] Open
Abstract
Diabetic Macular Edema (DME) represents a significant visual impairment among individuals with diabetes, leading to a dramatic reduction in visual acuity and potentially resulting in irreversible vision loss. Optical Coherence Tomography (OCT), a technique that produces high-resolution retinal images, plays a vital role in the clinical assessment of this condition. Physicians typically rely on OCT B-Scan images to evaluate DME severity. However, manual interpretation of these images is susceptible to errors, which can lead to detrimental consequences, such as misdiagnosis and improper treatment strategies. Hence, there is a critical need for more reliable diagnostic methods. This study aims to address this gap by proposing an automated model based on Generative Adversarial Networks (GANs) to generate OCT B-Scan images of DME. The model synthesizes images from patients' baseline OCT B-Scan images, which could potentially enhance the robustness of DME detection systems. We employ five distinct GANs in this study: Deep Convolutional GAN, Conditional GAN, CycleGAN, StyleGAN2, and StyleGAN3, drawing comparisons across their performance. Subsequently, the hyperparameters of the best-performing GAN are fine-tuned using Particle Swarm Optimization (PSO) to produce more realistic OCT images. This comparative analysis not only serves to improve the detection of DME severity using OCT images but also provides insights into the appropriate choice of GANs for the effective generation of realistic OCT images from the baseline OCT datasets.
Collapse
Affiliation(s)
- Aditya Tripathi
- Department of Information & Communication Technology, Manipal Institute of Technology, Manipal, Manipal Academy of Higher Education, Manipal, Karnataka, 576104, India
| | - Preetham Kumar
- Department of Information & Communication Technology, Manipal Institute of Technology, Manipal, Manipal Academy of Higher Education, Manipal, Karnataka, 576104, India
| | - Veena Mayya
- Department of Information & Communication Technology, Manipal Institute of Technology, Manipal, Manipal Academy of Higher Education, Manipal, Karnataka, 576104, India
| | - Akshat Tulsani
- Department of Information & Communication Technology, Manipal Institute of Technology, Manipal, Manipal Academy of Higher Education, Manipal, Karnataka, 576104, India
| |
Collapse
|
40
|
Guo X, Gichoya JW, Trivedi H, Purkayastha S, Banerjee I. MedShift: Automated Identification of Shift Data for Medical Image Dataset Curation. IEEE J Biomed Health Inform 2023; 27:3936-3947. [PMID: 37167055 PMCID: PMC10513895 DOI: 10.1109/jbhi.2023.3275104] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/13/2023]
Abstract
Automated curation of noisy external data in the medical domain has long been in high demand, as AI technologies need to be validated using various sources with clean, annotated data. Identifying the variance between internal and external sources is a fundamental step in curating a high-quality dataset, as the data distributions from different sources can vary significantly and subsequently affect the performance of AI models. The primary challenges for detecting data shifts are - (1) accessing private data across healthcare institutions for manual detection and (2) the lack of automated approaches to learn efficient shift-data representation without training samples. To overcome these problems, we propose an automated pipeline called MedShift to detect top-level shift samples and evaluate the significance of shift data without sharing data between internal and external organizations. MedShift employs unsupervised anomaly detectors to learn the internal distribution and identify samples showing significant shiftness for external datasets, and then compares their performance. To quantify the effects of detected shift data, we train a multi-class classifier that learns internal domain knowledge and evaluates the classification performance for each class in external domains after dropping the shift data. We also propose a data quality metric to quantify the dissimilarity between internal and external datasets. We verify the efficacy of MedShift using musculoskeletal radiographs (MURA) and chest X-ray datasets from multiple external sources. Our experiments show that our proposed shift data detection pipeline can be beneficial for medical centers to curate high-quality datasets more efficiently.
Collapse
|
41
|
Sato J, Suzuki Y, Wataya T, Nishigaki D, Kita K, Yamagata K, Tomiyama N, Kido S. Anatomy-aware self-supervised learning for anomaly detection in chest radiographs. iScience 2023; 26:107086. [PMID: 37434699 PMCID: PMC10331430 DOI: 10.1016/j.isci.2023.107086] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2023] [Revised: 04/17/2023] [Accepted: 06/06/2023] [Indexed: 07/13/2023] Open
Abstract
In this study, we present a self-supervised learning (SSL)-based model that enables anatomical structure-based unsupervised anomaly detection (UAD). The model employs an anatomy-aware pasting (AnatPaste) augmentation tool that uses a threshold-based lung segmentation pretext task to create anomalies in normal chest radiographs used for model pretraining. These anomalies are similar to real anomalies and help the model recognize them. We evaluate our model using three open-source chest radiograph datasets. Our model exhibits area under curves of 92.1%, 78.7%, and 81.9%, which are the highest among those of existing UAD models. To the best of our knowledge, this is the first SSL model to employ anatomical information from segmentation as a pretext task. The performance of AnatPaste shows that incorporating anatomical information into SSL can effectively improve accuracy.
Collapse
Affiliation(s)
- Junya Sato
- Department of Artificial Intelligence Diagnostic Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka 565-0871, Japan
- Department of Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka 565-0871, Japan
- Graduate School of Information Science and Technology, Osaka University, Yamadaoka, 1-5 Suita, Osaka 565-0871, Japan
| | - Yuki Suzuki
- Department of Artificial Intelligence Diagnostic Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka 565-0871, Japan
| | - Tomohiro Wataya
- Department of Artificial Intelligence Diagnostic Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka 565-0871, Japan
- Department of Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka 565-0871, Japan
| | - Daiki Nishigaki
- Department of Artificial Intelligence Diagnostic Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka 565-0871, Japan
- Department of Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka 565-0871, Japan
| | - Kosuke Kita
- Department of Artificial Intelligence Diagnostic Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka 565-0871, Japan
| | - Kazuki Yamagata
- Department of Artificial Intelligence Diagnostic Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka 565-0871, Japan
- Department of Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka 565-0871, Japan
| | - Noriyuki Tomiyama
- Department of Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka 565-0871, Japan
| | - Shoji Kido
- Department of Artificial Intelligence Diagnostic Radiology, Osaka University Graduate School of Medicine, 2-2, Yamadaoka, Suita, Osaka 565-0871, Japan
| |
Collapse
|
42
|
Chen Y, Yan Y, Wang X, Zheng Y. IoT-Enabled Few-Shot Image Generation for Power Scene Defect Detection Based on Self-Attention and Global-Local Fusion. SENSORS (BASEL, SWITZERLAND) 2023; 23:6531. [PMID: 37514825 PMCID: PMC10383857 DOI: 10.3390/s23146531] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/22/2023] [Revised: 06/13/2023] [Accepted: 07/14/2023] [Indexed: 07/30/2023]
Abstract
Defect detection in power scenarios is a critical task that plays a significant role in ensuring the safety, reliability, and efficiency of power systems. The existing technology requires enhancement in its learning ability from large volumes of data to achieve ideal detection effect results. Power scene data involve privacy and security issues, and there is an imbalance in the number of samples across different defect categories, all of which will affect the performance of defect detection models. With the emergence of the Internet of Things (IoT), the integration of IoT with machine learning offers a new direction for defect detection in power equipment. Meanwhile, a generative adversarial network based on multi-view fusion and self-attention is proposed for few-shot image generation, named MVSA-GAN. The IoT devices capture real-time data from the power scene, which are then used to train the MVSA-GAN model, enabling it to generate realistic and diverse defect data. The designed self-attention encoder focuses on the relevant features of different parts of the image to capture the contextual information of the input image and improve the authenticity and coherence of the image. A multi-view feature fusion module is proposed to capture the complex structure and texture of the power scene through the selective fusion of global and local features, and improve the authenticity and diversity of generated images. Experiments show that the few-shot image generation method proposed in this paper can generate real and diverse defect data for power scene defects. The proposed method achieved FID and LPIPS scores of 67.87 and 0.179, surpassing SOTA methods, such as FIGR and DAWSON.
Collapse
Affiliation(s)
- Yi Chen
- College of Electrical Engineering, Zhejiang University, Hangzhou 310058, China
| | - Yunfeng Yan
- School of Mechanical Engineering, Zhejiang University, Hangzhou 310027, China
- Hainan Institute of Zhejiang University, Sanya 572025, China
| | - Xianbo Wang
- Hainan Institute of Zhejiang University, Sanya 572025, China
| | - Yi Zheng
- College of Electrical Engineering, Zhejiang University, Hangzhou 310058, China
| |
Collapse
|
43
|
Konz N, Dong H, Mazurowski MA. Unsupervised anomaly localization in high-resolution breast scans using deep pluralistic image completion. Med Image Anal 2023; 87:102836. [PMID: 37201220 PMCID: PMC10247574 DOI: 10.1016/j.media.2023.102836] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/02/2022] [Revised: 04/19/2023] [Accepted: 05/03/2023] [Indexed: 05/20/2023]
Abstract
Automated tumor detection in Digital Breast Tomosynthesis (DBT) is a difficult task due to natural tumor rarity, breast tissue variability, and high resolution. Given the scarcity of abnormal images and the abundance of normal images for this problem, an anomaly detection/localization approach could be well-suited. However, most anomaly localization research in machine learning focuses on non-medical datasets, and we find that these methods fall short when adapted to medical imaging datasets. The problem is alleviated when we solve the task from the image completion perspective, in which the presence of anomalies can be indicated by a discrepancy between the original appearance and its auto-completion conditioned on the surroundings. However, there are often many valid normal completions given the same surroundings, especially in the DBT dataset, making this evaluation criterion less precise. To address such an issue, we consider pluralistic image completion by exploring the distribution of possible completions instead of generating fixed predictions. This is achieved through our novel application of spatial dropout on the completion network during inference time only, which requires no additional training cost and is effective at generating diverse completions. We further propose minimum completion distance (MCD), a new metric for detecting anomalies, thanks to these stochastic completions. We provide theoretical as well as empirical support for the superiority over existing methods of using the proposed method for anomaly localization. On the DBT dataset, our model outperforms other state-of-the-art methods by at least 10% AUROC for pixel-level detection.
Collapse
Affiliation(s)
- Nicholas Konz
- Department of Electrical and Computer Engineering, Duke University, Durham, NC, 27708, USA.
| | - Haoyu Dong
- Department of Electrical and Computer Engineering, Duke University, Durham, NC, 27708, USA
| | - Maciej A Mazurowski
- Department of Electrical and Computer Engineering, Duke University, Durham, NC, 27708, USA; Department of Computer Science, Duke University, Durham, NC, 27708, USA; Department of Radiology, Duke University, Durham, NC, 27708, USA; Department of Biostatistics & Bioinformatics, Duke University, Durham, NC, 27708, USA
| |
Collapse
|
44
|
Burger B, Bernathova M, Seeböck P, Singer CF, Helbich TH, Langs G. Deep learning for predicting future lesion emergence in high-risk breast MRI screening: a feasibility study. Eur Radiol Exp 2023; 7:32. [PMID: 37280478 DOI: 10.1186/s41747-023-00343-y] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/25/2023] [Accepted: 04/04/2023] [Indexed: 06/08/2023] Open
Abstract
BACKGROUND International societies have issued guidelines for high-risk breast cancer (BC) screening, recommending contrast-enhanced magnetic resonance imaging (CE-MRI) of the breast as a supplemental diagnostic tool. In our study, we tested the applicability of deep learning-based anomaly detection to identify anomalous changes in negative breast CE-MRI screens associated with future lesion emergence. METHODS In this prospective study, we trained a generative adversarial network on dynamic CE-MRI of 33 high-risk women who participated in a screening program but did not develop BC. We defined an anomaly score as the deviation of an observed CE-MRI scan from the model of normal breast tissue variability. We evaluated the anomaly score's association with future lesion emergence on the level of local image patches (104,531 normal patches, 455 patches of future lesion location) and entire CE-MRI exams (21 normal, 20 with future lesion). Associations were analyzed by receiver operating characteristic (ROC) curves on the patch level and logistic regression on the examination level. RESULTS The local anomaly score on image patches was a good predictor for future lesion emergence (area under the ROC curve 0.804). An exam-level summary score was significantly associated with the emergence of lesions at any location at a later time point (p = 0.045). CONCLUSIONS Breast cancer lesions are associated with anomalous appearance changes in breast CE-MRI occurring before the lesion emerges in high-risk women. These early image signatures are detectable and may be a basis for adjusting individual BC risk and personalized screening. RELEVANCE STATEMENT Anomalies in screening MRI preceding lesion emergence in women at high-risk of breast cancer may inform individualized screening and intervention strategies. KEY POINTS • Breast lesions are associated with preceding anomalies in CE-MRI of high-risk women. • Deep learning-based anomaly detection can help to adjust risk assessment for future lesions. • An appearance anomaly score may be used for adjusting screening interval times.
Collapse
Affiliation(s)
- Bianca Burger
- Department of Biomedical Imaging and Image-Guided Therapy, Division of Computational Imaging Research (CIR), Medical University of Vienna, Währinger Gürtel 18-20, 1090, Vienna, Austria
| | - Maria Bernathova
- Department of Biomedical Imaging and Image-Guided Therapy, Division of General and Pediatric Radiology, Medical University of Vienna, Vienna, Austria
| | - Philipp Seeböck
- Department of Biomedical Imaging and Image-Guided Therapy, Division of Computational Imaging Research (CIR), Medical University of Vienna, Währinger Gürtel 18-20, 1090, Vienna, Austria
| | - Christian F Singer
- Department of Obstetrics and Gynecology, Division of Special Gynecology, Medical University of Vienna, Vienna, Austria
- Comprehensive Cancer Center, Medical University of Vienna, Vienna, Austria
| | - Thomas H Helbich
- Department of Biomedical Imaging and Image-Guided Therapy, Division of General and Pediatric Radiology, Medical University of Vienna, Vienna, Austria
| | - Georg Langs
- Department of Biomedical Imaging and Image-Guided Therapy, Division of Computational Imaging Research (CIR), Medical University of Vienna, Währinger Gürtel 18-20, 1090, Vienna, Austria.
- Computer Science and Artificial Intelligence Laboratory, Massachusetts Institute of Technology, Cambridge, MA, USA.
| |
Collapse
|
45
|
Jha SB, Babiceanu RF. Deep CNN-based visual defect detection: Survey of current literature. COMPUT IND 2023. [DOI: 10.1016/j.compind.2023.103911] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/05/2023]
|
46
|
Rajabi MM, Komeilian P, Wan X, Farmani R. Leak detection and localization in water distribution networks using conditional deep convolutional generative adversarial networks. WATER RESEARCH 2023; 238:120012. [PMID: 37150062 DOI: 10.1016/j.watres.2023.120012] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/15/2023] [Revised: 04/06/2023] [Accepted: 04/26/2023] [Indexed: 05/09/2023]
Abstract
This paper explores the use of 'conditional convolutional generative adversarial networks' (CDCGAN) for image-based leak detection and localization (LD&L) in water distribution networks (WDNs). The method employs pressure measurements and is based on four pillars: (1) hydraulic model-based generation of leak-free training data by taking into account the demand uncertainty, (2) conversion of hydraulic model input demand-output pressure pairs into images using kriging interpolation, (3) training of a CDCGAN model for image-to-image translation, and (4) using the structural similarity (SSIM) index for LD&L. SSIM, computed over the entire pressure distribution image is used for leak detection, and a local estimate of SSIM is employed for leak localization. The CDCGAN model employed in this paper is based on the pix2pix architecture. The effectiveness of the proposed methodology is demonstrated on leakage datasets under various scenarios. Results show that the method has an accuracy of approximately 70% for real-time leak detection. The proposed method is well-suited for real-time applications due to the low computational cost of CDCGAN predictions compared to WDN hydraulic models, is robust in presence of uncertainty due to the nature of generative adversarial networks, and scales well to large and variable-sized monitoring data due to the use of an image-based approach.
Collapse
Affiliation(s)
- Mohammad Mahdi Rajabi
- Civil and Environmental Engineering Faculty, Tarbiat Modares University, PO Box 14115-397, Tehran, Iran.
| | - Pooya Komeilian
- Department of Civil Engineering, Sharif University of Technology, Tehran, Iran
| | - Xi Wan
- Centre for Water Systems, Department of Engineering, University of Exeter, Exeter, Devon EX4 4QF, UK
| | - Raziyeh Farmani
- Centre for Water Systems, Department of Engineering, University of Exeter, Exeter, Devon EX4 4QF, UK
| |
Collapse
|
47
|
Fountoukidou T, Sznitman R. A reinforcement learning approach for VQA validation: An application to diabetic macular edema grading. Med Image Anal 2023; 87:102822. [PMID: 37182321 DOI: 10.1016/j.media.2023.102822] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2021] [Revised: 08/12/2022] [Accepted: 04/12/2023] [Indexed: 05/16/2023]
Abstract
Recent advances in machine learning models have greatly increased the performance of automated methods in medical image analysis. However, the internal functioning of such models is largely hidden, which hinders their integration in clinical practice. Explainability and trust are viewed as important aspects of modern methods, for the latter's widespread use in clinical communities. As such, validation of machine learning models represents an important aspect and yet, most methods are only validated in a limited way. In this work, we focus on providing a richer and more appropriate validation approach for highly powerful Visual Question Answering (VQA) algorithms. To better understand the performance of these methods, which answer arbitrary questions related to images, this work focuses on an automatic visual Turing test (VTT). That is, we propose an automatic adaptive questioning method, that aims to expose the reasoning behavior of a VQA algorithm. Specifically, we introduce a reinforcement learning (RL) agent that observes the history of previously asked questions, and uses it to select the next question to pose. We demonstrate our approach in the context of evaluating algorithms that automatically answer questions related to diabetic macular edema (DME) grading. The experiments show that such an agent has similar behavior to a clinician, whereby asking questions that are relevant to key clinical concepts.
Collapse
Affiliation(s)
- Tatiana Fountoukidou
- Artificial Intelligence in Medical Imaging, ARTORG Center, University of Bern, Murtenstrasse 50, 3008 Bern, Switzerland.
| | - Raphael Sznitman
- Artificial Intelligence in Medical Imaging, ARTORG Center, University of Bern, Murtenstrasse 50, 3008 Bern, Switzerland
| |
Collapse
|
48
|
Anaam A, Al-Antari MA, Hussain J, Abdel Samee N, Alabdulhafith M, Gofuku A. Deep Active Learning for Automatic Mitotic Cell Detection on HEp-2 Specimen Medical Images. Diagnostics (Basel) 2023; 13:diagnostics13081416. [PMID: 37189517 DOI: 10.3390/diagnostics13081416] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2023] [Revised: 04/09/2023] [Accepted: 04/12/2023] [Indexed: 05/17/2023] Open
Abstract
Identifying Human Epithelial Type 2 (HEp-2) mitotic cells is a crucial procedure in anti-nuclear antibodies (ANAs) testing, which is the standard protocol for detecting connective tissue diseases (CTD). Due to the low throughput and labor-subjectivity of the ANAs' manual screening test, there is a need to develop a reliable HEp-2 computer-aided diagnosis (CAD) system. The automatic detection of mitotic cells from the microscopic HEp-2 specimen images is an essential step to support the diagnosis process and enhance the throughput of this test. This work proposes a deep active learning (DAL) approach to overcoming the cell labeling challenge. Moreover, deep learning detectors are tailored to automatically identify the mitotic cells directly in the entire microscopic HEp-2 specimen images, avoiding the segmentation step. The proposed framework is validated using the I3A Task-2 dataset over 5-fold cross-validation trials. Using the YOLO predictor, promising mitotic cell prediction results are achieved with an average of 90.011% recall, 88.307% precision, and 81.531% mAP. Whereas, average scores of 86.986% recall, 85.282% precision, and 78.506% mAP are obtained using the Faster R-CNN predictor. Employing the DAL method over four labeling rounds effectively enhances the accuracy of the data annotation, and hence, improves the prediction performance. The proposed framework could be practically applicable to support medical personnel in making rapid and accurate decisions about the mitotic cells' existence.
Collapse
Affiliation(s)
- Asaad Anaam
- Graduate School of Interdisciplinary Science and Engineering in Health Systems, Okayama University, Okayama 700-8530, Japan
| | - Mugahed A Al-Antari
- Department of Artificial Intelligence, College of Software & Convergence Technology, Daeyang AI Center, Sejong University, Seoul 05006, Republic of Korea
| | - Jamil Hussain
- Department of Data Science, College of Software & Convergence Technology, Daeyang AI Center, Sejong University, Seoul 05006, Republic of Korea
| | - Nagwan Abdel Samee
- Department of Information Technology, College of Computer and Information Sciences, Princess Nourah bint Abdulrahman University, P.O. Box 84428, Riyadh 11671, Saudi Arabia
| | - Maali Alabdulhafith
- Department of Information Technology, College of Computer and Information Sciences, Princess Nourah bint Abdulrahman University, P.O. Box 84428, Riyadh 11671, Saudi Arabia
| | - Akio Gofuku
- Graduate School of Interdisciplinary Science and Engineering in Health Systems, Okayama University, Okayama 700-8530, Japan
| |
Collapse
|
49
|
Li Y, Lao Q, Kang Q, Jiang Z, Du S, Zhang S, Li K. Self-supervised anomaly detection, staging and segmentation for retinal images. Med Image Anal 2023; 87:102805. [PMID: 37104995 DOI: 10.1016/j.media.2023.102805] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/05/2022] [Revised: 11/14/2022] [Accepted: 03/30/2023] [Indexed: 04/29/2023]
Abstract
Unsupervised anomaly detection (UAD) is to detect anomalies through learning the distribution of normal data without labels and therefore has a wide application in medical images by alleviating the burden of collecting annotated medical data. Current UAD methods mostly learn the normal data by the reconstruction of the original input, but often lack the consideration of any prior information that has semantic meanings. In this paper, we first propose a universal unsupervised anomaly detection framework SSL-AnoVAE, which utilizes a self-supervised learning (SSL) module for providing more fine-grained semantics depending on the to-be detected anomalies in the retinal images. We also explore the relationship between the data transformation adopted in the SSL module and the quality of anomaly detection for retinal images. Moreover, to take full advantage of the proposed SSL-AnoVAE and apply towards clinical usages for computer-aided diagnosis of retinal-related diseases, we further propose to stage and segment the anomalies in retinal images detected by SSL-AnoVAE in an unsupervised manner. Experimental results demonstrate the effectiveness of our proposed method for unsupervised anomaly detection, staging and segmentation on both retinal optical coherence tomography images and color fundus photograph images.
Collapse
Affiliation(s)
- Yiyue Li
- Department of Ophthalmology and West China Biomedical Big Data Center, West China Hospital, Sichuan University, Chengdu, Sichuan, 610041, China; West China Biomedical Big Data Center, Med-X Center for Informatics, Sichuan University, Chengdu, Sichuan, 610041, China
| | - Qicheng Lao
- School of Artificial Intelligence, Beijing University of Posts and Telecommunications, Beijing, 100876, China; Shanghai Artificial Intelligence Laboratory, Shanghai, 200030, China.
| | - Qingbo Kang
- West China Biomedical Big Data Center, Med-X Center for Informatics, Sichuan University, Chengdu, Sichuan, 610041, China; Shanghai Artificial Intelligence Laboratory, Shanghai, 200030, China
| | - Zekun Jiang
- West China Biomedical Big Data Center, Med-X Center for Informatics, Sichuan University, Chengdu, Sichuan, 610041, China
| | - Shiyi Du
- West China Biomedical Big Data Center, Med-X Center for Informatics, Sichuan University, Chengdu, Sichuan, 610041, China
| | - Shaoting Zhang
- Shanghai Artificial Intelligence Laboratory, Shanghai, 200030, China
| | - Kang Li
- West China Biomedical Big Data Center, Med-X Center for Informatics, Sichuan University, Chengdu, Sichuan, 610041, China; Sichuan University Pittsburgh Institute, Chengdu, Sichuan, 610065, China; Shanghai Artificial Intelligence Laboratory, Shanghai, 200030, China.
| |
Collapse
|
50
|
Yun TJ, Choi JW, Han M, Jung WS, Choi SH, Yoo RE, Hwang IP. Deep learning based automatic detection algorithm for acute intracranial haemorrhage: a pivotal randomized clinical trial. NPJ Digit Med 2023; 6:61. [PMID: 37029272 PMCID: PMC10082037 DOI: 10.1038/s41746-023-00798-8] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/10/2022] [Accepted: 03/10/2023] [Indexed: 04/09/2023] Open
Abstract
Acute intracranial haemorrhage (AIH) is a potentially life-threatening emergency that requires prompt and accurate assessment and management. This study aims to develop and validate an artificial intelligence (AI) algorithm for diagnosing AIH using brain-computed tomography (CT) images. A retrospective, multi-reader, pivotal, crossover, randomised study was performed to validate the performance of an AI algorithm was trained using 104,666 slices from 3010 patients. Brain CT images (12,663 slices from 296 patients) were evaluated by nine reviewers belonging to one of the three subgroups (non-radiologist physicians, n = 3; board-certified radiologists, n = 3; and neuroradiologists, n = 3) with and without the aid of our AI algorithm. Sensitivity, specificity, and accuracy were compared between AI-unassisted and AI-assisted interpretations using the chi-square test. Brain CT interpretation with AI assistance results in significantly higher diagnostic accuracy than that without AI assistance (0.9703 vs. 0.9471, p < 0.0001, patient-wise). Among the three subgroups of reviewers, non-radiologist physicians demonstrate the greatest improvement in diagnostic accuracy for brain CT interpretation with AI assistance compared to that without AI assistance. For board-certified radiologists, the diagnostic accuracy for brain CT interpretation is significantly higher with AI assistance than without AI assistance. For neuroradiologists, although brain CT interpretation with AI assistance results in a trend for higher diagnostic accuracy compared to that without AI assistance, the difference does not reach statistical significance. For the detection of AIH, brain CT interpretation with AI assistance results in better diagnostic performance than that without AI assistance, with the most significant improvement observed for non-radiologist physicians.
Collapse
Affiliation(s)
- Tae Jin Yun
- Institute of Radiation Medicine, Seoul National University Medical Research Center, Seoul, Republic of Korea
- Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea
| | - Jin Wook Choi
- Department of Radiology, Ajou University School of Medicine, Suwon, Republic of Korea.
| | - Miran Han
- Department of Radiology, Ajou University School of Medicine, Suwon, Republic of Korea
| | - Woo Sang Jung
- Department of Radiology, Ajou University School of Medicine, Suwon, Republic of Korea
| | - Seung Hong Choi
- Institute of Radiation Medicine, Seoul National University Medical Research Center, Seoul, Republic of Korea
- Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea
| | - Roh-Eul Yoo
- Institute of Radiation Medicine, Seoul National University Medical Research Center, Seoul, Republic of Korea
- Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea
| | - In Pyeong Hwang
- Institute of Radiation Medicine, Seoul National University Medical Research Center, Seoul, Republic of Korea
- Department of Radiology, Seoul National University Hospital, Seoul, Republic of Korea
| |
Collapse
|