1
|
Waisberg E, Ong J, Kamran SA, Masalkhi M, Paladugu P, Zaman N, Lee AG, Tavakkoli A. Generative artificial intelligence in ophthalmology. Surv Ophthalmol 2024:S0039-6257(24)00044-4. [PMID: 38762072 DOI: 10.1016/j.survophthal.2024.04.009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/23/2022] [Revised: 04/28/2024] [Accepted: 04/29/2024] [Indexed: 05/20/2024]
Abstract
Generative AI has revolutionized medicine over the past several years. A generative adversarial network (GAN) is a deep learning framework that has become a powerful technique in medicine, particularly in ophthalmology and image analysis. In this paper we review the current ophthalmic literature involving GANs, and highlight key contributions in the field. We briefly touch on ChatGPT, another application of generative AI, and its potential in ophthalmology. We also explore the potential uses for GANs in ocular imaging, with a specific emphasis on 3 primary domains: image enhancement, disease identification, and generating of synthetic data. PubMed, Ovid MEDLINE, Google Scholar were searched from inception to October 30, 2022 to identify applications of GAN in ophthalmology. A total of 40 papers were included in this review. We cover various applications of GANs in ophthalmic-related imaging including optical coherence tomography, orbital magnetic resonance imaging, fundus photography, and ultrasound; however, we also highlight several challenges, that resulted in the generation of inaccurate and atypical results during certain iterations. Finally, we examine future directions and considerations for generative AI in ophthalmology.
Collapse
Affiliation(s)
- Ethan Waisberg
- Department of Ophthalmology, University of Cambridge, Cambridge, United Kingdom.
| | - Joshua Ong
- Michigan Medicine, University of Michigan, Ann Arbor, United States
| | - Sharif Amit Kamran
- School of Medicine, University College Dublin, Belfield, Dublin, Ireland
| | - Mouayad Masalkhi
- School of Medicine, University College Dublin, Belfield, Dublin, Ireland
| | - Phani Paladugu
- Sidney Kimmel Medical College, Thomas Jefferson University, Philadelphia, Pennsylvania, United States; Brigham and Women's Hospital, Harvard Medical School, Boston, Massachusetts, United States
| | - Nasif Zaman
- Human-Machine Perception Laboratory, Department of Computer Science and Engineering, University of Nevada, Reno, Reno, Nevada, United States
| | - Andrew G Lee
- Center for Space Medicine, Baylor College of Medicine, Houston, Texas, United States; Department of Ophthalmology, Blanton Eye Institute, Houston Methodist Hospital, Houston, Texas, United States; The Houston Methodist Research Institute, Houston Methodist Hospital, Houston, Texas, United States; Departments of Ophthalmology, Neurology, and Neurosurgery, Weill Cornell Medicine, New York, New York, United States; Department of Ophthalmology, University of Texas Medical Branch, Galveston, Texas, United States; University of Texas MD Anderson Cancer Center, Houston, Texas, United States; Texas A&M College of Medicine, Texas, United States; Department of Ophthalmology, The University of Iowa Hospitals and Clinics, Iowa City, Iowa, United States
| | - Alireza Tavakkoli
- Human-Machine Perception Laboratory, Department of Computer Science and Engineering, University of Nevada, Reno, Reno, Nevada, United States
| |
Collapse
|
2
|
Lakshmi KS, Sargunam B. Exploration of AI-powered DenseNet121 for effective diabetic retinopathy detection. Int Ophthalmol 2024; 44:90. [PMID: 38367098 DOI: 10.1007/s10792-024-03027-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/23/2023] [Accepted: 01/11/2024] [Indexed: 02/19/2024]
Abstract
OBJECTIVE Diabetic Retinopathy (DR) is a severe complication of diabetes that damages the retina and affects approximately 80% of patients with diabetes for 10 years or more. This condition primarily impacts young and productive individuals, resulting in significant long-term medical complications for patients and society. The early stages of diabetic retinopathy often advance without noticeable symptoms, resulting in delayed identification and intervention. Therefore, develop approaches employing transfer learning methodologies to enhance early detection capabilities, facilitating timely diagnosis and intervention to mitigate the progression of diabetic retinopathy. METHODS This study introduces a transfer learning approach for detecting four stages of DR: No DR, Mild, Moderate, and Severe. The methods AlexNet, VGG16, ResNet50, Inception v3, and DenseNet121 are utilized and trained using the Kaggle DR dataset. RESULTS To assess the efficiency of the suggested improved network, the Kaggle dataset is employed to analyze four performance metrics: Sensitivity, Precision, Accuracy, and F1 score. DenseNet121 demonstrated superior accuracy among the two models, outperforming other models, making it a suitable option for automatic DR sign detection. CONCLUSION The integration of the DenseNet121 model shows great promise in transforming the timely identification and treatment of DR, resulting in enhanced patient results in the long run and alleviating the burden on society.
Collapse
Affiliation(s)
- K Santhiya Lakshmi
- Department of Electronics and Communication Engineering, Avinashilingam Institute for Home Science and Higher Education for Women, Coimbatore, Tamil Nadu, India.
| | - B Sargunam
- Department of Electronics and Communication Engineering, School of Engineering, Avinashilingam Institute for Home Science and Higher Education for Women, Coimbatore, Tamil Nadu, India
| |
Collapse
|
3
|
Manan MA, Jinchao F, Khan TM, Yaqub M, Ahmed S, Chuhan IS. Semantic segmentation of retinal exudates using a residual encoder-decoder architecture in diabetic retinopathy. Microsc Res Tech 2023; 86:1443-1460. [PMID: 37194727 DOI: 10.1002/jemt.24345] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2023] [Revised: 04/21/2023] [Accepted: 05/04/2023] [Indexed: 05/18/2023]
Abstract
Exudates are a common sign of diabetic retinopathy, which is a disease that affects the blood vessels in the retina. Early detection of exudates is critical to avoiding vision problems through continuous screening and treatment. In traditional clinical practice, the involved lesions are manually detected using photographs of the fundus. However, this task is cumbersome and time-consuming and requires intense effort due to the small size of the lesion and the low contrast of the images. Thus, computer-assisted diagnosis of retinal disease based on the detection of red lesions has been actively explored recently. In this paper, we present a comparison of deep convolutional neural network (CNN) architectures and propose a residual CNN with residual skip connections to reduce the parameter for the semantic segmentation of exudates in retinal images. A suitable image augmentation technique is used to improve the performance of network architecture. The proposed network can robustly segment exudates with high accuracy, which makes it suitable for diabetic retinopathy screening. A comparative performance analysis of three benchmark databases: E-ophtha, DIARETDB1, and Hamilton Ophthalmology Institute's Macular Edema, is presented. The proposed method achieves a precision of 0.95, 0.92, 0.97, accuracy of 0.98, 0.98, 0.98, sensitivity of 0.97, 0.95, 0.95, specificity of 0.99, 0.99, 0.99, and area under the curve of 0.97, 0.94, and 0.96, respectively. RESEARCH HIGHLIGHTS: The research focuses on the detection and segmentation of exudates in diabetic retinopathy, a disease affecting the retina. Early detection of exudates is important to avoid vision problems and requires continuous screening and treatment. Currently, manual detection is time-consuming and requires intense effort. The authors compare qualitative results of the state-of-the-art convolutional neural network (CNN) architectures and propose a computer-assisted diagnosis approach based on deep learning, using a residual CNN with residual skip connections to reduce parameters. The proposed method is evaluated on three benchmark databases and demonstrates high accuracy and suitability for diabetic retinopathy screening.
Collapse
Affiliation(s)
- Malik Abdul Manan
- Beijing Key Laboratory of Computational Intelligence and Intelligent System, Faculty of Information Technology, Beijing University of Technology, Beijing, China
| | - Feng Jinchao
- Beijing Key Laboratory of Computational Intelligence and Intelligent System, Faculty of Information Technology, Beijing University of Technology, Beijing, China
| | - Tariq M Khan
- School of IT, Deakin University, Waurn Ponds, Australia
| | - Muhammad Yaqub
- Beijing Key Laboratory of Computational Intelligence and Intelligent System, Faculty of Information Technology, Beijing University of Technology, Beijing, China
| | - Shahzad Ahmed
- Beijing Key Laboratory of Computational Intelligence and Intelligent System, Faculty of Information Technology, Beijing University of Technology, Beijing, China
| | - Imran Shabir Chuhan
- Interdisciplinary Research Institute, Faculty of Science, Beijing University of Technology, Beijing, China
| |
Collapse
|
4
|
Liu YF, Ji YK, Fei FQ, Chen NM, Zhu ZT, Fei XZ. Research progress in artificial intelligence assisted diabetic retinopathy diagnosis. Int J Ophthalmol 2023; 16:1395-1405. [PMID: 37724288 PMCID: PMC10475636 DOI: 10.18240/ijo.2023.09.05] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/28/2023] [Accepted: 06/14/2023] [Indexed: 09/20/2023] Open
Abstract
Diabetic retinopathy (DR) is one of the most common retinal vascular diseases and one of the main causes of blindness worldwide. Early detection and treatment can effectively delay vision decline and even blindness in patients with DR. In recent years, artificial intelligence (AI) models constructed by machine learning and deep learning (DL) algorithms have been widely used in ophthalmology research, especially in diagnosing and treating ophthalmic diseases, particularly DR. Regarding DR, AI has mainly been used in its diagnosis, grading, and lesion recognition and segmentation, and good research and application results have been achieved. This study summarizes the research progress in AI models based on machine learning and DL algorithms for DR diagnosis and discusses some limitations and challenges in AI research.
Collapse
Affiliation(s)
- Yun-Fang Liu
- Department of Ophthalmology, First People's Hospital of Huzhou, Huzhou University, Huzhou 313000, Zhejiang Province, China
| | - Yu-Ke Ji
- Eye Hospital, Nanjing Medical University, Nanjing 210000, Jiangsu Province, China
| | - Fang-Qin Fei
- Department of Endocrinology, First People's Hospital of Huzhou, Huzhou University, Huzhou 313000, Zhejiang Province, China
| | - Nai-Mei Chen
- Department of Ophthalmology, Huai'an Hospital of Huai'an City, Huai'an 223000, Jiangsu Province, China
| | - Zhen-Tao Zhu
- Department of Ophthalmology, Huai'an Hospital of Huai'an City, Huai'an 223000, Jiangsu Province, China
| | - Xing-Zhen Fei
- Department of Endocrinology, First People's Hospital of Huzhou, Huzhou University, Huzhou 313000, Zhejiang Province, China
| |
Collapse
|
5
|
CLC-Net: Contextual and Local Collaborative Network for Lesion Segmentation in Diabetic Retinopathy Images. Neurocomputing 2023. [DOI: 10.1016/j.neucom.2023.01.013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/12/2023]
|
6
|
Sun LC, Pao SI, Huang KH, Wei CY, Lin KF, Chen PN. Generative adversarial network-based deep learning approach in classification of retinal conditions with optical coherence tomography images. Graefes Arch Clin Exp Ophthalmol 2022; 261:1399-1412. [PMID: 36441228 DOI: 10.1007/s00417-022-05919-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2022] [Revised: 09/21/2022] [Accepted: 11/22/2022] [Indexed: 11/29/2022] Open
Abstract
PURPOSE To determine whether a deep learning approach using generative adversarial networks (GANs) is beneficial for the classification of retinal conditions with Optical coherence tomography (OCT) images. METHODS Our study utilized 84,452 retinal OCT images obtained from a publicly available dataset (Kermany Dataset). Employing GAN, synthetic OCT images are produced to balance classes of retinal disorders. A deep learning classification model is constructed using pretrained deep neural networks (DNNs), and outcomes are evaluated using 2082 images collected from patients who visited the Department of Ophthalmology and the Department of Endocrinology and Metabolism at the Tri-service General Hospital in Taipei from January 2017 to December 2021. RESULTS The highest classification accuracies accomplished by deep learning machines trained on the unbalanced dataset for its training set, validation set, fivefold cross validation (CV), Kermany test set, and TSGH test set were 97.73%, 96.51%, 97.14%, 99.59%, and 81.03%, respectively. The highest classification accuracies accomplished by deep learning machines trained on the synthesis-balanced dataset for its training set, validation set, fivefold CV, Kermany test set, and TSGH test set were 98.60%, 98.41%, 98.52%, 99.38%, and 84.92%, respectively. In comparing the highest accuracies, deep learning machines trained on the synthesis-balanced dataset outperformed deep learning machines trained on the unbalanced dataset for the training set, validation set, fivefold CV, and TSGH test set. CONCLUSIONS Overall, deep learning machines on a synthesis-balanced dataset demonstrated to be advantageous over deep learning machines trained on an unbalanced dataset for the classification of retinal conditions.
Collapse
Affiliation(s)
- Ling-Chun Sun
- School of Medicine, National Defense Medical Center, Taipei, Taiwan
| | - Shu-I Pao
- Department of Ophthalmology, Tri-Service General Hospital, National Defense Medical Center, Taipei, Taiwan
| | - Ke-Hao Huang
- Department of Ophthalmology, Song-Shan Branch of Tri-Service General Hospital, National Defense Medical Center, Taipei, Taiwan
| | - Chih-Yuan Wei
- Graduate Institute of Life Sciences, National Defense Medical Center, Taipei, Taiwan
| | - Ke-Feng Lin
- Medical Informatics Office, Tri-Service General Hospital, National Defense Medical Center, Taipei, Taiwan
- School of Public Health, National Defense Medical Center, Taipei, Taiwan
| | - Ping-Nan Chen
- Department of Biomedical Engineering, National Defense Medical Center, No.161, Sec.6, Minchiuan E. Rd., Neihu Dist, Taipei, 11490, Taiwan.
| |
Collapse
|
7
|
Wang W, Li X, Xu Z, Yu W, Zhao J, Ding D, Chen Y. Learning Two-Stream CNN for Multi-Modal Age-related Macular Degeneration Categorization. IEEE J Biomed Health Inform 2022; 26:4111-4122. [PMID: 35503853 DOI: 10.1109/jbhi.2022.3171523] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
This paper tackles automated categorization of Age-related Macular Degeneration (AMD), a common macular disease among people over 50. Previous research efforts mainly focus on AMD categorization with a single-modal input, let it be a color fundus photograph (CFP) or an OCT B-scan image. By contrast, we consider AMD categorization given a multi-modal input, a direction that is clinically meaningful yet mostly unexplored. Contrary to the prior art that takes a traditional approach of feature extraction plus classifier training that cannot be jointly optimized, we opt for end-to-end multi-modal Convolutional Neural Networks (MM-CNN). Our MM-CNN is instantiated by a two-stream CNN, with spatially-invariant fusion to combine information from the CFP and OCT streams. In order to visually interpret the contribution of the individual modalities to the final prediction, we extend the class activation mapping (CAM) technique to the multi-modal scenario. For effective training of MM-CNN, we develop two data augmentation methods. One is GAN-based CFP/OCT image synthesis, with our novel use of CAMs as conditional input of a high-resolution image-to-image translation GAN. The other method is Loose Pairing, which pairs a CFP image and an OCT image on the basis of their classes instead of eye identities. Experiments on a clinical dataset consisting of 1,094 CFP images and 1,289 OCT images acquired from 1,093 distinct eyes show that the proposed solution obtains better F1 and Accuracy than multiple baselines for multi-modal AMD categorization. Code and data are available at https://github.com/li-xirong/mmc-amd.
Collapse
|
8
|
Li M, Wan C. The use of deep learning technology for the detection of optic neuropathy. Quant Imaging Med Surg 2022; 12:2129-2143. [PMID: 35284277 PMCID: PMC8899937 DOI: 10.21037/qims-21-728] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/14/2021] [Accepted: 10/26/2021] [Indexed: 03/14/2024]
Abstract
The emergence of computer graphics processing units (GPUs), improvements in mathematical models, and the availability of big data, has allowed artificial intelligence (AI) to use machine learning and deep learning (DL) technology to achieve robust performance in various fields of medicine. The DL system provides improved capabilities, especially in image recognition and image processing. Recent progress in the sorting of AI data sets has stimulated great interest in the development of DL algorithms. Compared with subjective evaluation and other traditional methods, DL algorithms can identify diseases faster and more accurately in diagnostic tests. Medical imaging is of great significance in the clinical diagnosis and individualized treatment of ophthalmic diseases. Based on the morphological data sets of millions of data points, various image-related diagnostic techniques can now impart high-resolution information on anatomical and functional changes, thereby providing unprecedented insights in ophthalmic clinical practice. As ophthalmology relies heavily on imaging examinations, it is one of the first medical fields to apply DL algorithms in clinical practice. Such algorithms can assist in the analysis of large amounts of data acquired from the examination of auxiliary images. In recent years, rapid advancements in imaging technology have facilitated the application of DL in the automatic identification and classification of pathologies that are characteristic of ophthalmic diseases, thereby providing high quality diagnostic information. This paper reviews the origins, development, and application of DL technology. The technical and clinical problems associated with building DL systems to meet clinical needs and the potential challenges of clinical application are discussed, especially in relation to the field of optic nerve diseases.
Collapse
Affiliation(s)
- Mei Li
- Department of Ophthalmology, Yanan People’s Hospital, Yanan, China
| | - Chao Wan
- Department of Ophthalmology, the First Hospital of China Medical University, Shenyang, China
| |
Collapse
|
9
|
Xu X, Li J, Guan Y, Zhao L, Zhao Q, Zhang L, Li L. GLA-Net: A global-local attention network for automatic cataract classification. J Biomed Inform 2021; 124:103939. [PMID: 34752858 DOI: 10.1016/j.jbi.2021.103939] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/01/2021] [Revised: 10/02/2021] [Accepted: 10/25/2021] [Indexed: 10/19/2022]
Abstract
Cataracts are the most crucial cause of blindness among all ophthalmic diseases. Convenient and cost-effective early cataract screening is urgently needed to reduce the risks of visual loss. To date, many studies have investigated automatic cataract classification based on fundus images. However, existing methods mainly rely on global image information while ignoring various local and subtle features. Notably, these local features are highly helpful for the identification of cataracts with different severities. To avoid this disadvantage, we introduce a deep learning technique to learn multilevel feature representations of the fundus image simultaneously. Specifically, a global-local attention network (GLA-Net) is proposed to handle the cataract classification task, which consists of two levels of subnets: the global-level attention subnet pays attention to the global structure information of the fundus image, while the local-level attention subnet focuses on the local discriminative features of the specific regions. These two types of subnets extract retinal features at different attention levels, which are then combined for final cataract classification. Our GLA-Net achieves the best performance in all metrics (90.65% detection accuracy, 83.47% grading accuracy, and 81.11% classification accuracy of grades 1 and 2). The experimental results on a real clinical dataset show that the combination of global-level and local-level attention models is effective for cataract screening and provides significant potential for other medical tasks.
Collapse
Affiliation(s)
- Xi Xu
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Jianqiang Li
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Yu Guan
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Linna Zhao
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Qing Zhao
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China.
| | - Li Zhang
- Beijing Tongren Eye Center, Beijing Tongren Hospital, Capital Medical University, Beijing, China
| | - Li Li
- National Center for Children's Health, Beijing Children's Hospital, Capital Medical University, Beijing, China
| |
Collapse
|
10
|
EAD-Net: A Novel Lesion Segmentation Method in Diabetic Retinopathy Using Neural Networks. DISEASE MARKERS 2021; 2021:6482665. [PMID: 34512815 PMCID: PMC8429028 DOI: 10.1155/2021/6482665] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/13/2021] [Accepted: 08/19/2021] [Indexed: 02/05/2023]
Abstract
Diabetic retinopathy (DR) is a common chronic fundus disease, which has four different kinds of microvessel structure and microvascular lesions: microaneurysms (MAs), hemorrhages (HEs), hard exudates, and soft exudates. Accurate detection and counting of them are a basic but important work. The manual annotation of these lesions is a labor-intensive task in clinical analysis. To solve the problem, we proposed a novel segmentation method for different lesions in DR. Our method is based on a convolutional neural network and can be divided into encoder module, attention module, and decoder module, so we refer it as EAD-Net. After normalization and augmentation, the fundus images were sent to the EAD-Net for automated feature extraction and pixel-wise label prediction. Given the evaluation metrics based on the matching degree between detected candidates and ground truth lesions, our method achieved sensitivity of 92.77%, specificity of 99.98%, and accuracy of 99.97% on the e_ophtha_EX dataset and comparable AUPR (Area under Precision-Recall curve) scores on IDRiD dataset. Moreover, the results on the local dataset also show that our EAD-Net has better performance than original U-net in most metrics, especially in the sensitivity and F1-score, with nearly ten percent improvement. The proposed EAD-Net is a novel method based on clinical DR diagnosis. It has satisfactory results on the segmentation of four different kinds of lesions. These effective segmentations have important clinical significance in the monitoring and diagnosis of DR.
Collapse
|
11
|
Deep learning for diabetic retinopathy detection and classification based on fundus images: A review. Comput Biol Med 2021; 135:104599. [PMID: 34247130 DOI: 10.1016/j.compbiomed.2021.104599] [Citation(s) in RCA: 51] [Impact Index Per Article: 17.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2021] [Revised: 06/12/2021] [Accepted: 06/18/2021] [Indexed: 02/02/2023]
Abstract
Diabetic Retinopathy is a retina disease caused by diabetes mellitus and it is the leading cause of blindness globally. Early detection and treatment are necessary in order to delay or avoid vision deterioration and vision loss. To that end, many artificial-intelligence-powered methods have been proposed by the research community for the detection and classification of diabetic retinopathy on fundus retina images. This review article provides a thorough analysis of the use of deep learning methods at the various steps of the diabetic retinopathy detection pipeline based on fundus images. We discuss several aspects of that pipeline, ranging from the datasets that are widely used by the research community, the preprocessing techniques employed and how these accelerate and improve the models' performance, to the development of such deep learning models for the diagnosis and grading of the disease as well as the localization of the disease's lesions. We also discuss certain models that have been applied in real clinical settings. Finally, we conclude with some important insights and provide future research directions.
Collapse
|
12
|
privGAN: Protecting GANs from membership inference attacks at low cost to utility. PROCEEDINGS ON PRIVACY ENHANCING TECHNOLOGIES 2021. [DOI: 10.2478/popets-2021-0041] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/21/2022]
Abstract
Abstract
Generative Adversarial Networks (GANs) have made releasing of synthetic images a viable approach to share data without releasing the original dataset. It has been shown that such synthetic data can be used for a variety of downstream tasks such as training classifiers that would otherwise require the original dataset to be shared. However, recent work has shown that the GAN models and their synthetically generated data can be used to infer the training set membership by an adversary who has access to the entire dataset and some auxiliary information. Current approaches to mitigate this problem (such as DPGAN [1]) lead to dramatically poorer generated sample quality than the original non–private GANs. Here we develop a new GAN architecture (privGAN), where the generator is trained not only to cheat the discriminator but also to defend membership inference attacks. The new mechanism is shown to empirically provide protection against this mode of attack while leading to negligible loss in downstream performances. In addition, our algorithm has been shown to explicitly prevent memorization of the training set, which explains why our protection is so effective. The main contributions of this paper are: i) we propose a novel GAN architecture that can generate synthetic data in a privacy preserving manner with minimal hyperparameter tuning and architecture selection, ii) we provide a theoretical understanding of the optimal solution of the privGAN loss function, iii) we empirically demonstrate the effectiveness of our model against several white and black–box attacks on several benchmark datasets, iv) we empirically demonstrate on three common benchmark datasets that synthetic images generated by privGAN lead to negligible loss in downstream performance when compared against non– private GANs. While we have focused on benchmarking privGAN exclusively on image datasets, the architecture of privGAN is not exclusive to image datasets and can be easily extended to other types of datasets. Repository link: https://github.com/microsoft/privGAN.
Collapse
|
13
|
Pradhan P, Meyer T, Vieth M, Stallmach A, Waldner M, Schmitt M, Popp J, Bocklitz T. Computational tissue staining of non-linear multimodal imaging using supervised and unsupervised deep learning. BIOMEDICAL OPTICS EXPRESS 2021; 12:2280-2298. [PMID: 33996229 PMCID: PMC8086483 DOI: 10.1364/boe.415962] [Citation(s) in RCA: 14] [Impact Index Per Article: 4.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/30/2020] [Revised: 01/28/2021] [Accepted: 02/17/2021] [Indexed: 05/24/2023]
Abstract
Hematoxylin and Eosin (H&E) staining is the 'gold-standard' method in histopathology. However, standard H&E staining of high-quality tissue sections requires long sample preparation times including sample embedding, which restricts its application for 'real-time' disease diagnosis. Due to this reason, a label-free alternative technique like non-linear multimodal (NLM) imaging, which is the combination of three non-linear optical modalities including coherent anti-Stokes Raman scattering, two-photon excitation fluorescence and second-harmonic generation, is proposed in this work. To correlate the information of the NLM images with H&E images, this work proposes computational staining of NLM images using deep learning models in a supervised and an unsupervised approach. In the supervised and the unsupervised approach, conditional generative adversarial networks (CGANs) and cycle conditional generative adversarial networks (cycle CGANs) are used, respectively. Both CGAN and cycle CGAN models generate pseudo H&E images, which are quantitatively analyzed based on mean squared error, structure similarity index and color shading similarity index. The mean of the three metrics calculated for the computationally generated H&E images indicate significant performance. Thus, utilizing CGAN and cycle CGAN models for computational staining is beneficial for diagnostic applications without performing a laboratory-based staining procedure. To the author's best knowledge, it is the first time that NLM images are computationally stained to H&E images using GANs in an unsupervised manner.
Collapse
Affiliation(s)
- Pranita Pradhan
- Institute of Physical Chemistry and Abbe Center of Photonics, Friedrich-Schiller-University, Jena, Germany
- Leibniz Institute of Photonic Technology, Member of Leibniz Health Technologies Jena, Germany
| | - Tobias Meyer
- Leibniz Institute of Photonic Technology, Member of Leibniz Health Technologies Jena, Germany
| | - Michael Vieth
- Institute of Pathology, Klinikum Bayreuth, Bayreuth, Germany
| | - Andreas Stallmach
- Department of Internal Medicine IV (Gastroenterology, Hepatology, and Infectious Diseases), Jena University Hospital, Jena, Germany
| | - Maximilian Waldner
- Erlangen Graduate School in Advanced Optical Technologies (SAOT), Friedrich-Alexander University of Erlangen-Nuremberg, 91052 Erlangen, Germany
- Medical Department 1, Friedrich-Alexander University of Erlangen-Nuremberg, Erlangen, Germany
| | - Michael Schmitt
- Institute of Physical Chemistry and Abbe Center of Photonics, Friedrich-Schiller-University, Jena, Germany
| | - Juergen Popp
- Institute of Physical Chemistry and Abbe Center of Photonics, Friedrich-Schiller-University, Jena, Germany
- Leibniz Institute of Photonic Technology, Member of Leibniz Health Technologies Jena, Germany
| | - Thomas Bocklitz
- Institute of Physical Chemistry and Abbe Center of Photonics, Friedrich-Schiller-University, Jena, Germany
- Leibniz Institute of Photonic Technology, Member of Leibniz Health Technologies Jena, Germany
| |
Collapse
|
14
|
|
15
|
Automatic detection of non-perfusion areas in diabetic macular edema from fundus fluorescein angiography for decision making using deep learning. Sci Rep 2020; 10:15138. [PMID: 32934283 PMCID: PMC7492239 DOI: 10.1038/s41598-020-71622-6] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/30/2020] [Accepted: 07/30/2020] [Indexed: 02/05/2023] Open
Abstract
Vision loss caused by diabetic macular edema (DME) can be prevented by early detection and laser photocoagulation. As there is no comprehensive detection technique to recognize NPA, we proposed an automatic detection method of NPA on fundus fluorescein angiography (FFA) in DME. The study included 3,014 FFA images of 221 patients with DME. We use 3 convolutional neural networks (CNNs), including DenseNet, ResNet50, and VGG16, to identify non-perfusion regions (NP), microaneurysms, and leakages in FFA images. The NPA was segmented using attention U-net. To validate its performance, we applied our detection algorithm on 249 FFA images in which the NPA areas were manually delineated by 3 ophthalmologists. For DR lesion classification, area under the curve is 0.8855 for NP regions, 0.9782 for microaneurysms, and 0.9765 for leakage classifier. The average precision of NP region overlap ratio is 0.643. NP regions of DME in FFA images are identified based a new automated deep learning algorithm. This study is an in-depth study from computer-aided diagnosis to treatment, and will be the theoretical basis for the application of intelligent guided laser.
Collapse
|
16
|
Luo Y, Chen K, Liu L, Liu J, Mao J, Ke G, Sun M. Dehaze of Cataractous Retinal Images Using an Unpaired Generative Adversarial Network. IEEE J Biomed Health Inform 2020; 24:3374-3383. [PMID: 32750919 DOI: 10.1109/jbhi.2020.2999077] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Cataracts are the leading cause of visual impairment worldwide. Examination of the retina through cataracts using a fundus camera is challenging and error-prone due to degraded image quality. We sought to develop an algorithm to dehaze such images to support diagnosis by either ophthalmologists or computer-aided diagnosis systems. Based on the generative adversarial network (GAN) concept, we designed two neural networks: CataractSimGAN and CataractDehazeNet. CataractSimGAN was intended for the synthesis of cataract-like images through unpaired clear retinal images and cataract images. CataractDehazeNet was trained using pairs of synthesized cataract-like images and the corresponding clear images through supervised learning. With two networks trained independently, the number of hyper-parameters was reduced, leading to better performance. We collected 400 retinal images without cataracts and 400 hazy images from cataract patients as the training dataset. Fifty cataract images and the corresponding clear images from the same patients after surgery comprised the test dataset. The clear images after surgery were used for reference to evaluate the performance of our method. CataractDehazeNet was able to enhance the degraded image from cataract patients substantially and to visualize blood vessels and the optic disc, while actively suppressing the artifacts common in application of similar methods. Thus, we developed an algorithm to improve the quality of the retinal images acquired from cataract patients. We achieved high structure similarity and fidelity between processed images and images from the same patients after cataract surgery.
Collapse
|
17
|
He M, Li Z, Liu C, Shi D, Tan Z. Deployment of Artificial Intelligence in Real-World Practice: Opportunity and Challenge. Asia Pac J Ophthalmol (Phila) 2020; 9:299-307. [PMID: 32694344 DOI: 10.1097/apo.0000000000000301] [Citation(s) in RCA: 23] [Impact Index Per Article: 5.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022] Open
Abstract
Artificial intelligence has rapidly evolved from the experimental phase to the implementation phase in many image-driven clinical disciplines, including ophthalmology. A combination of the increasing availability of large datasets and computing power with revolutionary progress in deep learning has created unprecedented opportunities for major breakthrough improvements in the performance and accuracy of automated diagnoses that primarily focus on image recognition and feature detection. Such an automated disease classification would significantly improve the accessibility, efficiency, and cost-effectiveness of eye care systems where it is less dependent on human input, potentially enabling diagnosis to be cheaper, quicker, and more consistent. Although this technology will have a profound impact on clinical flow and practice patterns sooner or later, translating such a technology into clinical practice is challenging and requires similar levels of accountability and effectiveness as any new medication or medical device due to the potential problems of bias, and ethical, medical, and legal issues that might arise. The objective of this review is to summarize the opportunities and challenges of this transition and to facilitate the integration of artificial intelligence (AI) into routine clinical practice based on our best understanding and experience in this area.
Collapse
Affiliation(s)
- Mingguang He
- State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangzhou, China
- Centre for Eye Research Australia, Royal Victorian Eye & Ear Hospital, Melbourne, Australia
| | - Zhixi Li
- State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangzhou, China
| | - Chi Liu
- State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangzhou, China
- School of Computer Science, University of Technology Sydney, Ultimo NSW, Australia
| | - Danli Shi
- State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-sen University, Guangzhou, China
| | - Zachary Tan
- Faculty of Medicine, The University of Queensland, Brisbane, Australia
- Schwarzman College, Tsinghua University, Beijing, China
| |
Collapse
|
18
|
Tong Y, Lu W, Yu Y, Shen Y. Application of machine learning in ophthalmic imaging modalities. EYE AND VISION 2020; 7:22. [PMID: 32322599 PMCID: PMC7160952 DOI: 10.1186/s40662-020-00183-6] [Citation(s) in RCA: 36] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 10/16/2019] [Accepted: 03/10/2020] [Indexed: 12/27/2022]
Abstract
In clinical ophthalmology, a variety of image-related diagnostic techniques have begun to offer unprecedented insights into eye diseases based on morphological datasets with millions of data points. Artificial intelligence (AI), inspired by the human multilayered neuronal system, has shown astonishing success within some visual and auditory recognition tasks. In these tasks, AI can analyze digital data in a comprehensive, rapid and non-invasive manner. Bioinformatics has become a focus particularly in the field of medical imaging, where it is driven by enhanced computing power and cloud storage, as well as utilization of novel algorithms and generation of data in massive quantities. Machine learning (ML) is an important branch in the field of AI. The overall potential of ML to automatically pinpoint, identify and grade pathological features in ocular diseases will empower ophthalmologists to provide high-quality diagnosis and facilitate personalized health care in the near future. This review offers perspectives on the origin, development, and applications of ML technology, particularly regarding its applications in ophthalmic imaging modalities.
Collapse
Affiliation(s)
- Yan Tong
- 1Eye Center, Renmin Hospital of Wuhan University, Wuhan, 430060 Hubei China
| | - Wei Lu
- 1Eye Center, Renmin Hospital of Wuhan University, Wuhan, 430060 Hubei China
| | - Yue Yu
- 1Eye Center, Renmin Hospital of Wuhan University, Wuhan, 430060 Hubei China
| | - Yin Shen
- 1Eye Center, Renmin Hospital of Wuhan University, Wuhan, 430060 Hubei China.,2Medical Research Institute, Wuhan University, Wuhan, Hubei China
| |
Collapse
|