1
|
Chen T, Bai Y, Mao H, Liu S, Xu K, Xiong Z, Ma S, Yang F, Zhao Y. Cross-modality transfer learning with knowledge infusion for diabetic retinopathy grading. Front Med (Lausanne) 2024; 11:1400137. [PMID: 38808141 PMCID: PMC11130363 DOI: 10.3389/fmed.2024.1400137] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2024] [Accepted: 04/15/2024] [Indexed: 05/30/2024] Open
Abstract
Background Ultra-wide-field (UWF) fundus photography represents an emerging retinal imaging technique offering a broader field of view, thus enhancing its utility in screening and diagnosing various eye diseases, notably diabetic retinopathy (DR). However, the application of computer-aided diagnosis for DR using UWF images confronts two major challenges. The first challenge arises from the limited availability of labeled UWF data, making it daunting to train diagnostic models due to the high cost associated with manual annotation of medical images. Secondly, existing models' performance requires enhancement due to the absence of prior knowledge to guide the learning process. Purpose By leveraging extensively annotated datasets within the field, which encompass large-scale, high-quality color fundus image datasets annotated at either image-level or pixel-level, our objective is to transfer knowledge from these datasets to our target domain through unsupervised domain adaptation. Methods Our approach presents a robust model for assessing the severity of diabetic retinopathy (DR) by leveraging unsupervised lesion-aware domain adaptation in ultra-wide-field (UWF) images. Furthermore, to harness the wealth of detailed annotations in publicly available color fundus image datasets, we integrate an adversarial lesion map generator. This generator supplements the grading model by incorporating auxiliary lesion information, drawing inspiration from the clinical methodology of evaluating DR severity by identifying and quantifying associated lesions. Results We conducted both quantitative and qualitative evaluations of our proposed method. In particular, among the six representative DR grading methods, our approach achieved an accuracy (ACC) of 68.18% and a precision (pre) of 67.43%. Additionally, we conducted extensive experiments in ablation studies to validate the effectiveness of each component of our proposed method. Conclusion In conclusion, our method not only improves the accuracy of DR grading, but also enhances the interpretability of the results, providing clinicians with a reliable DR grading scheme.
Collapse
Affiliation(s)
- Tao Chen
- Cixi Biomedical Research Institute, Wenzhou Medical University, Ningbo, China
- Institute of Biomedical Engineering, Ningbo Institute of Materials Technology and Engineering, Chinese Academy of SciencesNingbo, China
| | - Yanmiao Bai
- Institute of Biomedical Engineering, Ningbo Institute of Materials Technology and Engineering, Chinese Academy of SciencesNingbo, China
| | - Haiting Mao
- Cixi Biomedical Research Institute, Wenzhou Medical University, Ningbo, China
- Institute of Biomedical Engineering, Ningbo Institute of Materials Technology and Engineering, Chinese Academy of SciencesNingbo, China
| | - Shouyue Liu
- Cixi Biomedical Research Institute, Wenzhou Medical University, Ningbo, China
- Institute of Biomedical Engineering, Ningbo Institute of Materials Technology and Engineering, Chinese Academy of SciencesNingbo, China
| | - Keyi Xu
- Cixi Biomedical Research Institute, Wenzhou Medical University, Ningbo, China
- Institute of Biomedical Engineering, Ningbo Institute of Materials Technology and Engineering, Chinese Academy of SciencesNingbo, China
| | - Zhouwei Xiong
- Cixi Biomedical Research Institute, Wenzhou Medical University, Ningbo, China
- Institute of Biomedical Engineering, Ningbo Institute of Materials Technology and Engineering, Chinese Academy of SciencesNingbo, China
| | - Shaodong Ma
- Institute of Biomedical Engineering, Ningbo Institute of Materials Technology and Engineering, Chinese Academy of SciencesNingbo, China
| | - Fang Yang
- Cixi Biomedical Research Institute, Wenzhou Medical University, Ningbo, China
- Institute of Biomedical Engineering, Ningbo Institute of Materials Technology and Engineering, Chinese Academy of SciencesNingbo, China
| | - Yitian Zhao
- Cixi Biomedical Research Institute, Wenzhou Medical University, Ningbo, China
- Institute of Biomedical Engineering, Ningbo Institute of Materials Technology and Engineering, Chinese Academy of SciencesNingbo, China
| |
Collapse
|
2
|
Ktena I, Wiles O, Albuquerque I, Rebuffi SA, Tanno R, Roy AG, Azizi S, Belgrave D, Kohli P, Cemgil T, Karthikesalingam A, Gowal S. Generative models improve fairness of medical classifiers under distribution shifts. Nat Med 2024; 30:1166-1173. [PMID: 38600282 PMCID: PMC11031395 DOI: 10.1038/s41591-024-02838-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2023] [Accepted: 01/26/2024] [Indexed: 04/12/2024]
Abstract
Domain generalization is a ubiquitous challenge for machine learning in healthcare. Model performance in real-world conditions might be lower than expected because of discrepancies between the data encountered during deployment and development. Underrepresentation of some groups or conditions during model development is a common cause of this phenomenon. This challenge is often not readily addressed by targeted data acquisition and 'labeling' by expert clinicians, which can be prohibitively expensive or practically impossible because of the rarity of conditions or the available clinical expertise. We hypothesize that advances in generative artificial intelligence can help mitigate this unmet need in a steerable fashion, enriching our training dataset with synthetic examples that address shortfalls of underrepresented conditions or subgroups. We show that diffusion models can automatically learn realistic augmentations from data in a label-efficient manner. We demonstrate that learned augmentations make models more robust and statistically fair in-distribution and out of distribution. To evaluate the generality of our approach, we studied three distinct medical imaging contexts of varying difficulty: (1) histopathology, (2) chest X-ray and (3) dermatology images. Complementing real samples with synthetic ones improved the robustness of models in all three medical tasks and increased fairness by improving the accuracy of clinical diagnosis within underrepresented groups, especially out of distribution.
Collapse
|
3
|
Wan C, Fang J, Li K, Zhang Q, Zhang S, Yang W. A new segmentation algorithm for peripapillary atrophy and optic disk from ultra-widefield Photographs 1. Comput Biol Med 2024; 172:108281. [PMID: 38503096 DOI: 10.1016/j.compbiomed.2024.108281] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/28/2023] [Revised: 03/09/2024] [Accepted: 03/12/2024] [Indexed: 03/21/2024]
Abstract
BACKGROUND AND OBJECTIVE The prevalence of myopia and high myopia is increasing globally, underscoring the growing importance of diagnosing high myopia-related pathologies. While existing image segmentation models, such as U-Net, UNet++, ResU-Net, and TransUNet, have achieved significant success in medical image segmentation, they still face challenges when dealing with ultra-widefield (UWF) fundus images. This study introduces a novel automatic segmentation algorithm for the optic disc and peripapillary atrophy (PPA) based on UWF fundus images, aimed at assisting ophthalmologists in more accurately diagnosing high myopia-related diseases. METHODS In this study, we developed a segmentation model leveraging a Transformer-based network structure, complemented by atrous convolution and selective boundary aggregation modules, to elevate the accuracy of segmenting the optic disc and PPA in UWF photography. The atrous convolution module adeptly manages multi-scale features, catering to the variances in target sizes and expanding the deep network's receptive field. Concurrently, the incorporation of the selective boundary aggregation module in the skip connections of the model significantly improves the differentiation of boundary information between segmentation targets. Moreover, the comparison of our proposed algorithm with classical segmentation models like U-Net, UNet++, ResU-Net, and TransUNet highlights its considerable advantages in processing UWF photographs. RESULTS The experimental results show that, compared to the other four models, our algorithm demonstrates substantial improvements in segmenting the optic disc and PPA in UWF photographs. In PPA segmentation, our algorithm improves by 0.8% in Dice, 1.8% in sensitivity, and 1.3% in intersection over union (IOU). In optic disc segmentation, our algorithm improves by 0.3% in Dice, 0.6% in precision, and 0.4% in IOU. CONCLUSION Our proposed method improves the segmentation accuracy of PPA and optic disks based on UWF photographs, which is valuable for diagnosing high myopia-related diseases in ophthalmology clinics.
Collapse
Affiliation(s)
- Cheng Wan
- College of Electronic Information Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, 211106, China
| | - Jiyi Fang
- College of Electronic Information Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, 211106, China
| | - Kunke Li
- Shenzhen Eye Institute, Shenzhen Eye Hospital, Jinan University, Shenzhen, 518040, China
| | - Qing Zhang
- The Third Affiliated Hospital of Xinxiang Medical University, Xinxiang Medical University, Xinxiang, 453000, China
| | - Shaochong Zhang
- Shenzhen Eye Institute, Shenzhen Eye Hospital, Jinan University, Shenzhen, 518040, China.
| | - Weihua Yang
- Shenzhen Eye Institute, Shenzhen Eye Hospital, Jinan University, Shenzhen, 518040, China.
| |
Collapse
|
4
|
Nouri H, Nasri R, Abtahi SH. Addressing inter-device variations in optical coherence tomography angiography: will image-to-image translation systems help? Int J Retina Vitreous 2023; 9:51. [PMID: 37644613 PMCID: PMC10466880 DOI: 10.1186/s40942-023-00491-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/08/2023] [Accepted: 08/17/2023] [Indexed: 08/31/2023] Open
Abstract
BACKGROUND Optical coherence tomography angiography (OCTA) is an innovative technology providing visual and quantitative data on retinal microvasculature in a non-invasive manner. MAIN BODY Due to variations in the technical specifications of different OCTA devices, there are significant inter-device differences in OCTA data, which can limit their comparability and generalizability. These variations can also result in a domain shift problem that may interfere with applicability of machine learning models on data obtained from different OCTA machines. One possible approach to address this issue may be unsupervised deep image-to-image translation leveraging systems such as Cycle-Consistent Generative Adversarial Networks (Cycle-GANs) and Denoising Diffusion Probabilistic Models (DDPMs). Through training on unpaired images from different device domains, Cycle-GANs and DDPMs may enable cross-domain translation of images. They have been successfully applied in various medical imaging tasks, including segmentation, denoising, and cross-modality image-to-image translation. In this commentary, we briefly describe how Cycle-GANs and DDPMs operate, and review the recent experiments with these models on medical and ocular imaging data. We then discuss the benefits of applying such techniques for inter-device translation of OCTA data and the potential challenges ahead. CONCLUSION Retinal imaging technologies and deep learning-based domain adaptation techniques are rapidly evolving. We suggest exploring the potential of image-to-image translation methods in improving the comparability of OCTA data from different centers or devices. This may facilitate more efficient analysis of heterogeneous data and broader applicability of machine learning models trained on limited datasets in this field.
Collapse
Affiliation(s)
- Hosein Nouri
- Ophthalmic Research Center, Research Institute for Ophthalmology and Vision Science, Shahid Beheshti University of Medical Sciences, Tehran, Iran.
- School of Medicine, Isfahan University of Medical Sciences, Isfahan, Iran.
| | - Reza Nasri
- School of Engineering, University of Isfahan, Isfahan, Iran
| | - Seyed-Hossein Abtahi
- Ophthalmic Research Center, Research Institute for Ophthalmology and Vision Science, Shahid Beheshti University of Medical Sciences, Tehran, Iran.
- Department of Ophthalmology, Torfe Medical Center, Shahid Beheshti University of Medical Sciences, Tehran, Iran.
| |
Collapse
|
5
|
van Tulder G, de Bruijne M. Unpaired, unsupervised domain adaptation assumes your domains are already similar. Med Image Anal 2023; 87:102825. [PMID: 37116296 DOI: 10.1016/j.media.2023.102825] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2022] [Revised: 03/30/2023] [Accepted: 04/17/2023] [Indexed: 04/30/2023]
Abstract
Unsupervised domain adaptation is a popular method in medical image analysis, but it can be tricky to make it work: without labels to link the domains, domains must be matched using feature distributions. If there is no additional information, this often leaves a choice between multiple possibilities to map the data that may be equally likely but not equally correct. In this paper we explore the fundamental problems that may arise in unsupervised domain adaptation, and discuss conditions that might still make it work. Focusing on medical image analysis, we argue that images from different domains may have similar class balance, similar intensities, similar spatial structure, or similar textures. We demonstrate how these implicit conditions can affect domain adaptation performance in experiments with synthetic data, MNIST digits, and medical images. We observe that practical success of unsupervised domain adaptation relies on existing similarities in the data, and is anything but guaranteed in the general case. Understanding these implicit assumptions is a key step in identifying potential problems in domain adaptation and improving the reliability of the results.
Collapse
Affiliation(s)
- Gijs van Tulder
- Data Science group, Faculty of Science, Radboud University, Postbus 9010, 6500 GL Nijmegen, The Netherlands; Biomedical Imaging Group, Erasmus MC, Postbus 2040, 3000 CA Rotterdam, The Netherlands.
| | - Marleen de Bruijne
- Biomedical Imaging Group, Erasmus MC, Postbus 2040, 3000 CA Rotterdam, The Netherlands; Department of Computer Science, University of Copenhagen, Universitetsparken 1, 2100 Copenhagen, Denmark.
| |
Collapse
|
6
|
Anaam A, Al-Antari MA, Hussain J, Abdel Samee N, Alabdulhafith M, Gofuku A. Deep Active Learning for Automatic Mitotic Cell Detection on HEp-2 Specimen Medical Images. Diagnostics (Basel) 2023; 13:diagnostics13081416. [PMID: 37189517 DOI: 10.3390/diagnostics13081416] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2023] [Revised: 04/09/2023] [Accepted: 04/12/2023] [Indexed: 05/17/2023] Open
Abstract
Identifying Human Epithelial Type 2 (HEp-2) mitotic cells is a crucial procedure in anti-nuclear antibodies (ANAs) testing, which is the standard protocol for detecting connective tissue diseases (CTD). Due to the low throughput and labor-subjectivity of the ANAs' manual screening test, there is a need to develop a reliable HEp-2 computer-aided diagnosis (CAD) system. The automatic detection of mitotic cells from the microscopic HEp-2 specimen images is an essential step to support the diagnosis process and enhance the throughput of this test. This work proposes a deep active learning (DAL) approach to overcoming the cell labeling challenge. Moreover, deep learning detectors are tailored to automatically identify the mitotic cells directly in the entire microscopic HEp-2 specimen images, avoiding the segmentation step. The proposed framework is validated using the I3A Task-2 dataset over 5-fold cross-validation trials. Using the YOLO predictor, promising mitotic cell prediction results are achieved with an average of 90.011% recall, 88.307% precision, and 81.531% mAP. Whereas, average scores of 86.986% recall, 85.282% precision, and 78.506% mAP are obtained using the Faster R-CNN predictor. Employing the DAL method over four labeling rounds effectively enhances the accuracy of the data annotation, and hence, improves the prediction performance. The proposed framework could be practically applicable to support medical personnel in making rapid and accurate decisions about the mitotic cells' existence.
Collapse
Affiliation(s)
- Asaad Anaam
- Graduate School of Interdisciplinary Science and Engineering in Health Systems, Okayama University, Okayama 700-8530, Japan
| | - Mugahed A Al-Antari
- Department of Artificial Intelligence, College of Software & Convergence Technology, Daeyang AI Center, Sejong University, Seoul 05006, Republic of Korea
| | - Jamil Hussain
- Department of Data Science, College of Software & Convergence Technology, Daeyang AI Center, Sejong University, Seoul 05006, Republic of Korea
| | - Nagwan Abdel Samee
- Department of Information Technology, College of Computer and Information Sciences, Princess Nourah bint Abdulrahman University, P.O. Box 84428, Riyadh 11671, Saudi Arabia
| | - Maali Alabdulhafith
- Department of Information Technology, College of Computer and Information Sciences, Princess Nourah bint Abdulrahman University, P.O. Box 84428, Riyadh 11671, Saudi Arabia
| | - Akio Gofuku
- Graduate School of Interdisciplinary Science and Engineering in Health Systems, Okayama University, Okayama 700-8530, Japan
| |
Collapse
|
7
|
Qin X, Bui F, Han Z. Semantically preserving adversarial unsupervised domain adaptation network for improving disease recognition from chest x-rays. Comput Med Imaging Graph 2023; 107:102232. [PMID: 37062171 DOI: 10.1016/j.compmedimag.2023.102232] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2023] [Revised: 03/18/2023] [Accepted: 03/27/2023] [Indexed: 04/18/2023]
Abstract
Supervised deep learning methods have been successfully applied in medical imaging. However, training deep learning systems often requires ample annotated data. Due to cost and time restrictions, not all collected medical images, e.g., chest x-rays (CXRs), can be labeled in practice. To classify these unlabeled images, a solution may involve adopting a model trained with sufficient labeled data in relevant domains (with both source and target being CXRs). However, domain shift may cause the trained model not able to generalize well on unlabeled target datasets. This work aims to develop a novel unsupervised domain adaptation (UDA) framework to improve recognition performance on unlabeled target data. We present a semantically preserving adversarial UDA network, i.e., SPA-UDA net, with the potential to bridge the domain gap, by reconstructing the images in the target domain via an adversarial encode-and-reconstruct translation architecture. To preserve the class-specific semantic information (i.e., with or without disease) of the original images when translating, a semantically consistent framework is embedded. This framework is designed to guarantee that fine-grained disease-related information on the original images can be safely transferred. Furthermore, the proposed SPA-UDA net does not require paired images from source and target domains when training, which reduces the cost of arranging data significantly and is ideal for UDA. We evaluate the proposed SPA-UDA net on two public CXR datasets for lung disease recognition. The experimental results show that the proposed framework achieves significant performance improvements compared to other state-of-the-art UDA methods.
Collapse
Affiliation(s)
- Xiaoli Qin
- University of Saskatchewan, Department of Elec. and Comp. Engineering, Saskatoon, S7N 5A9, SK, Canada.
| | - Francis Bui
- University of Saskatchewan, Department of Elec. and Comp. Engineering, Saskatoon, S7N 5A9, SK, Canada.
| | - Zhu Han
- University of Houston, Department of Elec. and Comp. Engineering, Houston, 77004, TX, USA; Kyung Hee University, Department of Comp. Sci. Engineering, Seoul, 446-701, South Korea.
| |
Collapse
|
8
|
Goceri E. Medical image data augmentation: techniques, comparisons and interpretations. Artif Intell Rev 2023; 56:1-45. [PMID: 37362888 PMCID: PMC10027281 DOI: 10.1007/s10462-023-10453-z] [Citation(s) in RCA: 8] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/27/2023] [Indexed: 03/29/2023]
Abstract
Designing deep learning based methods with medical images has always been an attractive area of research to assist clinicians in rapid examination and accurate diagnosis. Those methods need a large number of datasets including all variations in their training stages. On the other hand, medical images are always scarce due to several reasons, such as not enough patients for some diseases, patients do not want to allow their images to be used, lack of medical equipment or equipment, inability to obtain images that meet the desired criteria. This issue leads to bias in datasets, overfitting, and inaccurate results. Data augmentation is a common solution to overcome this issue and various augmentation techniques have been applied to different types of images in the literature. However, it is not clear which data augmentation technique provides more efficient results for which image type since different diseases are handled, different network architectures are used, and these architectures are trained and tested with different numbers of data sets in the literature. Therefore, in this work, the augmentation techniques used to improve performances of deep learning based diagnosis of the diseases in different organs (brain, lung, breast, and eye) from different imaging modalities (MR, CT, mammography, and fundoscopy) have been examined. Also, the most commonly used augmentation methods have been implemented, and their effectiveness in classifications with a deep network has been discussed based on quantitative performance evaluations. Experiments indicated that augmentation techniques should be chosen carefully according to image types.
Collapse
Affiliation(s)
- Evgin Goceri
- Department of Biomedical Engineering, Engineering Faculty, Akdeniz University, Antalya, Turkey
| |
Collapse
|
9
|
Atasever S, Azginoglu N, Terzi DS, Terzi R. A comprehensive survey of deep learning research on medical image analysis with focus on transfer learning. Clin Imaging 2023; 94:18-41. [PMID: 36462229 DOI: 10.1016/j.clinimag.2022.11.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2022] [Revised: 10/17/2022] [Accepted: 11/01/2022] [Indexed: 11/13/2022]
Abstract
This survey aims to identify commonly used methods, datasets, future trends, knowledge gaps, constraints, and limitations in the field to provide an overview of current solutions used in medical image analysis in parallel with the rapid developments in transfer learning (TL). Unlike previous studies, this survey grouped the last five years of current studies for the period between January 2017 and February 2021 according to different anatomical regions and detailed the modality, medical task, TL method, source data, target data, and public or private datasets used in medical imaging. Also, it provides readers with detailed information on technical challenges, opportunities, and future research trends. In this way, an overview of recent developments is provided to help researchers to select the most effective and efficient methods and access widely used and publicly available medical datasets, research gaps, and limitations of the available literature.
Collapse
Affiliation(s)
- Sema Atasever
- Computer Engineering Department, Nevsehir Hacı Bektas Veli University, Nevsehir, Turkey.
| | - Nuh Azginoglu
- Computer Engineering Department, Kayseri University, Kayseri, Turkey.
| | | | - Ramazan Terzi
- Computer Engineering Department, Amasya University, Amasya, Turkey.
| |
Collapse
|
10
|
Wang C, Zhang R, Wei X, Wang L, Wu P, Yao Q. Deep learning and sub-band fluorescence imaging-based method for caries and calculus diagnosis embeddable on different smartphones. BIOMEDICAL OPTICS EXPRESS 2023; 14:866-882. [PMID: 36874478 PMCID: PMC9979668 DOI: 10.1364/boe.479818] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/04/2022] [Revised: 12/28/2022] [Accepted: 01/13/2023] [Indexed: 06/18/2023]
Abstract
Popularizing community and home early caries screening is essential for caries prevention and treatment. However, a high-precision, low-cost, and portable automated screening tool is currently lacking. This study constructed an automated diagnosis model for dental caries and calculus using fluorescence sub-band imaging combined with deep learning. The proposed method is divided into two stages: the first stage collects imaging information of dental caries in different fluorescence spectral bands and obtains six-channel fluorescence images. The second stage employs a 2-D-3-D hybrid convolutional neural network combined with the attention mechanism for classification and diagnosis. The experiments demonstrate that the method has competitive performance compared to existing methods. In addition, the feasibility of transferring this approach to different smartphones is discussed. This highly accurate, low-cost, portable method has potential applications in community and at-home caries detection.
Collapse
Affiliation(s)
- Cheng Wang
- Department of Optical Science and Engineering, Academy for Engineering and Technology, Fudan University, Shanghai 200433, China
- State Key Laboratory of Applied Optics, Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, China
| | - Rongjun Zhang
- Department of Optical Science and Engineering, Academy for Engineering and Technology, Fudan University, Shanghai 200433, China
| | - Xiaoling Wei
- Department of Endodontics, Shanghai Stomatological Hospital, Shanghai Key Laboratory of Craniomaxillofacial Development and Diseases, Fudan University, Shanghai 200001, China
| | - Le Wang
- Department of Optical Science and Engineering, Academy for Engineering and Technology, Fudan University, Shanghai 200433, China
| | - Peiyu Wu
- Department of Optical Science and Engineering, Academy for Engineering and Technology, Fudan University, Shanghai 200433, China
- State Key Laboratory of Applied Optics, Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, China
| | - Qi Yao
- Department of Optical Science and Engineering, Academy for Engineering and Technology, Fudan University, Shanghai 200433, China
- State Key Laboratory of Applied Optics, Changchun Institute of Optics, Fine Mechanics and Physics, Chinese Academy of Sciences, Changchun 130033, China
| |
Collapse
|
11
|
LAC-GAN: Lesion attention conditional GAN for Ultra-widefield image synthesis. Neural Netw 2023; 158:89-98. [PMID: 36446158 DOI: 10.1016/j.neunet.2022.11.005] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2022] [Revised: 08/30/2022] [Accepted: 11/03/2022] [Indexed: 11/13/2022]
Abstract
Automatic detection of retinal diseases based on deep learning technology and Ultra-widefield (UWF) images plays an important role in clinical practices in recent years. However, due to small lesions and limited data samples, it is not easy to train a detection-accurate model with strong generalization ability. In this paper, we propose a lesion attention conditional generative adversarial network (LAC-GAN) to synthesize retinal images with realistic lesion details to improve the training of the disease detection model. Specifically, the generator takes the vessel mask and class label as the conditional inputs, and processes the random Gaussian noise by a series of residual block to generate the synthetic images. To focus on pathological information, we propose a lesion feature attention mechanism based on random forest (RF) method, which constructs its reverse activation network to activate the lesion features. For discriminator, a weight-sharing multi-discriminator is designed to improve the performance of model by affine transformations. Experimental results on multi-center UWF image datasets demonstrate that the proposed method can generate retinal images with reasonable details, which helps to enhance the performance of the disease detection model.
Collapse
|
12
|
Bhambra N, Antaki F, Malt FE, Xu A, Duval R. Deep learning for ultra-widefield imaging: a scoping review. Graefes Arch Clin Exp Ophthalmol 2022; 260:3737-3778. [PMID: 35857087 DOI: 10.1007/s00417-022-05741-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2021] [Revised: 05/16/2022] [Accepted: 06/22/2022] [Indexed: 11/04/2022] Open
Abstract
PURPOSE This article is a scoping review of published and peer-reviewed articles using deep-learning (DL) applied to ultra-widefield (UWF) imaging. This study provides an overview of the published uses of DL and UWF imaging for the detection of ophthalmic and systemic diseases, generative image synthesis, quality assessment of images, and segmentation and localization of ophthalmic image features. METHODS A literature search was performed up to August 31st, 2021 using PubMed, Embase, Cochrane Library, and Google Scholar. The inclusion criteria were as follows: (1) deep learning, (2) ultra-widefield imaging. The exclusion criteria were as follows: (1) articles published in any language other than English, (2) articles not peer-reviewed (usually preprints), (3) no full-text availability, (4) articles using machine learning algorithms other than deep learning. No study design was excluded from consideration. RESULTS A total of 36 studies were included. Twenty-three studies discussed ophthalmic disease detection and classification, 5 discussed segmentation and localization of ultra-widefield images (UWFIs), 3 discussed generative image synthesis, 3 discussed ophthalmic image quality assessment, and 2 discussed detecting systemic diseases via UWF imaging. CONCLUSION The application of DL to UWF imaging has demonstrated significant effectiveness in the diagnosis and detection of ophthalmic diseases including diabetic retinopathy, retinal detachment, and glaucoma. DL has also been applied in the generation of synthetic ophthalmic images. This scoping review highlights and discusses the current uses of DL with UWF imaging, and the future of DL applications in this field.
Collapse
Affiliation(s)
- Nishaant Bhambra
- Faculty of Medicine, McGill University, Montréal, Québec, Canada
| | - Fares Antaki
- Department of Ophthalmology, Université de Montréal, Montréal, Québec, Canada.,Centre Universitaire d'Ophtalmologie (CUO), Hôpital Maisonneuve-Rosemont, CIUSSS de L'Est-de-L'Île-de-Montréal, 5415 Assumption Blvd, Montréal, Québec, H1T 2M4, Canada
| | - Farida El Malt
- Faculty of Medicine, McGill University, Montréal, Québec, Canada
| | - AnQi Xu
- Faculty of Medicine, Université de Montréal, Montréal, Québec, Canada
| | - Renaud Duval
- Department of Ophthalmology, Université de Montréal, Montréal, Québec, Canada. .,Centre Universitaire d'Ophtalmologie (CUO), Hôpital Maisonneuve-Rosemont, CIUSSS de L'Est-de-L'Île-de-Montréal, 5415 Assumption Blvd, Montréal, Québec, H1T 2M4, Canada.
| |
Collapse
|
13
|
MinimalGAN: diverse medical image synthesis for data augmentation using minimal training data. APPL INTELL 2022. [DOI: 10.1007/s10489-022-03609-x] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/14/2022]
|
14
|
Hofer D, Schmidt-Erfurth U, Orlando JI, Goldbach F, Gerendas BS, Seeböck P. Improving foveal avascular zone segmentation in fluorescein angiograms by leveraging manual vessel labels from public color fundus pictures. BIOMEDICAL OPTICS EXPRESS 2022; 13:2566-2580. [PMID: 35774310 PMCID: PMC9203117 DOI: 10.1364/boe.452873] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/13/2022] [Revised: 03/11/2022] [Accepted: 03/24/2022] [Indexed: 06/15/2023]
Abstract
In clinical routine, ophthalmologists frequently analyze the shape and size of the foveal avascular zone (FAZ) to detect and monitor retinal diseases. In order to extract those parameters, the contours of the FAZ need to be segmented, which is normally achieved by analyzing the retinal vasculature (RV) around the macula in fluorescein angiograms (FA). Computer-aided segmentation methods based on deep learning (DL) can automate this task. However, current approaches for segmenting the FAZ are often tailored to a specific dataset or require manual initialization. Furthermore, they do not take the variability and challenges of clinical FA into account, which are often of low quality and difficult to analyze. In this paper we propose a DL-based framework to automatically segment the FAZ in challenging FA scans from clinical routine. Our approach mimics the workflow of retinal experts by using additional RV labels as a guidance during training. Hence, our model is able to produce RV segmentations simultaneously. We minimize the annotation work by using a multi-modal approach that leverages already available public datasets of color fundus pictures (CFPs) and their respective manual RV labels. Our experimental evaluation on two datasets with FA from 1) clinical routine and 2) large multicenter clinical trials shows that the addition of weak RV labels as a guidance during training improves the FAZ segmentation significantly with respect to using only manual FAZ annotations.
Collapse
Affiliation(s)
- Dominik Hofer
- Vienna Reading Center, Department of Ophthalmology and Optometry, Medical University of Vienna, Spitalgasse 23, 1090 Vienna, Austria
| | - Ursula Schmidt-Erfurth
- Vienna Reading Center, Department of Ophthalmology and Optometry, Medical University of Vienna, Spitalgasse 23, 1090 Vienna, Austria
| | - José Ignacio Orlando
- Vienna Reading Center, Department of Ophthalmology and Optometry, Medical University of Vienna, Spitalgasse 23, 1090 Vienna, Austria
- Yatiris Group, PLADEMA Institute, CON-ICET, Universidad Nacional del Centro de la Provincia de Buenos Aires, Gral. Pinto 399, Tandil, Buenos Aires, Argentina
| | - Felix Goldbach
- Vienna Reading Center, Department of Ophthalmology and Optometry, Medical University of Vienna, Spitalgasse 23, 1090 Vienna, Austria
| | - Bianca S. Gerendas
- Vienna Reading Center, Department of Ophthalmology and Optometry, Medical University of Vienna, Spitalgasse 23, 1090 Vienna, Austria
| | - Philipp Seeböck
- Vienna Reading Center, Department of Ophthalmology and Optometry, Medical University of Vienna, Spitalgasse 23, 1090 Vienna, Austria
| |
Collapse
|
15
|
Chen Y, Yang XH, Wei Z, Heidari AA, Zheng N, Li Z, Chen H, Hu H, Zhou Q, Guan Q. Generative Adversarial Networks in Medical Image augmentation: A review. Comput Biol Med 2022; 144:105382. [PMID: 35276550 DOI: 10.1016/j.compbiomed.2022.105382] [Citation(s) in RCA: 49] [Impact Index Per Article: 24.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2022] [Revised: 02/25/2022] [Accepted: 03/02/2022] [Indexed: 12/31/2022]
Abstract
OBJECT With the development of deep learning, the number of training samples for medical image-based diagnosis and treatment models is increasing. Generative Adversarial Networks (GANs) have attracted attention in medical image processing due to their excellent image generation capabilities and have been widely used in data augmentation. In this paper, a comprehensive and systematic review and analysis of medical image augmentation work are carried out, and its research status and development prospects are reviewed. METHOD This paper reviews 105 medical image augmentation related papers, which mainly collected by ELSEVIER, IEEE Xplore, and Springer from 2018 to 2021. We counted these papers according to the parts of the organs corresponding to the images, and sorted out the medical image datasets that appeared in them, the loss function in model training, and the quantitative evaluation metrics of image augmentation. At the same time, we briefly introduce the literature collected in three journals and three conferences that have received attention in medical image processing. RESULT First, we summarize the advantages of various augmentation models, loss functions, and evaluation metrics. Researchers can use this information as a reference when designing augmentation tasks. Second, we explore the relationship between augmented models and the amount of the training set, and tease out the role that augmented models may play when the quality of the training set is limited. Third, the statistical number of papers shows that the development momentum of this research field remains strong. Furthermore, we discuss the existing limitations of this type of model and suggest possible research directions. CONCLUSION We discuss GAN-based medical image augmentation work in detail. This method effectively alleviates the challenge of limited training samples for medical image diagnosis and treatment models. It is hoped that this review will benefit researchers interested in this field.
Collapse
Affiliation(s)
- Yizhou Chen
- College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China.
| | - Xu-Hua Yang
- College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China.
| | - Zihan Wei
- College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China.
| | - Ali Asghar Heidari
- School of Surveying and Geospatial Engineering, College of Engineering, University of Tehran, Tehran, Iran; Department of Computer Science, School of Computing, National University of Singapore, Singapore, Singapore.
| | - Nenggan Zheng
- Qiushi Academy for Advanced Studies, Zhejiang University, Hangzhou, Zhejiang, China.
| | - Zhicheng Li
- Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China.
| | - Huiling Chen
- College of Computer Science and Artificial Intelligence, Wenzhou University, Wenzhou, Zhejiang, 325035, China.
| | - Haigen Hu
- College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China.
| | - Qianwei Zhou
- College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China.
| | - Qiu Guan
- College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China.
| |
Collapse
|
16
|
You A, Kim JK, Ryu IH, Yoo TK. Application of generative adversarial networks (GAN) for ophthalmology image domains: a survey. EYE AND VISION (LONDON, ENGLAND) 2022; 9:6. [PMID: 35109930 PMCID: PMC8808986 DOI: 10.1186/s40662-022-00277-3] [Citation(s) in RCA: 48] [Impact Index Per Article: 24.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/15/2021] [Accepted: 01/11/2022] [Indexed: 12/12/2022]
Abstract
BACKGROUND Recent advances in deep learning techniques have led to improved diagnostic abilities in ophthalmology. A generative adversarial network (GAN), which consists of two competing types of deep neural networks, including a generator and a discriminator, has demonstrated remarkable performance in image synthesis and image-to-image translation. The adoption of GAN for medical imaging is increasing for image generation and translation, but it is not familiar to researchers in the field of ophthalmology. In this work, we present a literature review on the application of GAN in ophthalmology image domains to discuss important contributions and to identify potential future research directions. METHODS We performed a survey on studies using GAN published before June 2021 only, and we introduced various applications of GAN in ophthalmology image domains. The search identified 48 peer-reviewed papers in the final review. The type of GAN used in the analysis, task, imaging domain, and the outcome were collected to verify the usefulness of the GAN. RESULTS In ophthalmology image domains, GAN can perform segmentation, data augmentation, denoising, domain transfer, super-resolution, post-intervention prediction, and feature extraction. GAN techniques have established an extension of datasets and modalities in ophthalmology. GAN has several limitations, such as mode collapse, spatial deformities, unintended changes, and the generation of high-frequency noises and artifacts of checkerboard patterns. CONCLUSIONS The use of GAN has benefited the various tasks in ophthalmology image domains. Based on our observations, the adoption of GAN in ophthalmology is still in a very early stage of clinical validation compared with deep learning classification techniques because several problems need to be overcome for practical use. However, the proper selection of the GAN technique and statistical modeling of ocular imaging will greatly improve the performance of each image analysis. Finally, this survey would enable researchers to access the appropriate GAN technique to maximize the potential of ophthalmology datasets for deep learning research.
Collapse
Affiliation(s)
- Aram You
- School of Architecture, Kumoh National Institute of Technology, Gumi, Gyeongbuk, South Korea
| | - Jin Kuk Kim
- B&VIIT Eye Center, Seoul, South Korea
- VISUWORKS, Seoul, South Korea
| | - Ik Hee Ryu
- B&VIIT Eye Center, Seoul, South Korea
- VISUWORKS, Seoul, South Korea
| | - Tae Keun Yoo
- B&VIIT Eye Center, Seoul, South Korea.
- Department of Ophthalmology, Aerospace Medical Center, Republic of Korea Air Force, 635 Danjae-ro, Namil-myeon, Cheongwon-gun, Cheongju, Chungcheongbuk-do, 363-849, South Korea.
| |
Collapse
|
17
|
Lei H, Liu W, Xie H, Zhao B, Yue G, Lei B. Unsupervised Domain Adaptation Based Image Synthesis and Feature Alignment for Joint Optic Disc and Cup Segmentation. IEEE J Biomed Health Inform 2021; 26:90-102. [PMID: 34061755 DOI: 10.1109/jbhi.2021.3085770] [Citation(s) in RCA: 14] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Due to the discrepancy of different devices for fundus image collection, a well-trained neural network is usually unsuitable for another new dataset. To solve this problem, the unsupervised domain adaptation strategy attracts a lot of attentions. In this paper, we propose an unsupervised domain adaptation method based image synthesis and feature alignment (ISFA) method to segment optic disc and cup on the fundus image. The GAN-based image synthesis (IS) mechanism along with the boundary information of optic disc and cup is utilized to generate target-like query images, which serves as the intermediate latent space between source domain and target domain images to alleviate the domain shift problem. Specifically, we use content and style feature alignment (CSFA) to ensure the feature consistency among source domain images, target-like query images and target domain images. The adversarial learning is used to extract domain invariant features for output-level feature alignment (OLFA). To enhance the representation ability of domain-invariant boundary structure information, we introduce the edge attention module (EAM) for low-level feature maps. Eventually, we train our proposed method on the training set of the REFUGE challenge dataset and test it on Drishti-GS and RIM-ONE_r3 datasets. On the Drishti-GS dataset, our method achieves about 3% improvement of Dice on optic cup segmentation over the next best method. We comprehensively discuss the robustness of our method for small dataset domain adaptation. The experimental results also demonstrate the effectiveness of our method. Our code is available at https://github.com/thinkobj/ISFA.
Collapse
|