1
|
Chew EY, Burns SA, Abraham AG, Bakhoum MF, Beckman JA, Chui TYP, Finger RP, Frangi AF, Gottesman RF, Grant MB, Hanssen H, Lee CS, Meyer ML, Rizzoni D, Rudnicka AR, Schuman JS, Seidelmann SB, Tang WHW, Adhikari BB, Danthi N, Hong Y, Reid D, Shen GL, Oh YS. Standardization and clinical applications of retinal imaging biomarkers for cardiovascular disease: a Roadmap from an NHLBI workshop. Nat Rev Cardiol 2024:10.1038/s41569-024-01060-8. [PMID: 39039178 DOI: 10.1038/s41569-024-01060-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Accepted: 06/21/2024] [Indexed: 07/24/2024]
Abstract
The accessibility of the retina with the use of non-invasive and relatively low-cost ophthalmic imaging techniques and analytics provides a unique opportunity to improve the detection, diagnosis and monitoring of systemic diseases. The National Heart, Lung, and Blood Institute conducted a workshop in October 2022 to examine this concept. On the basis of the discussions at that workshop, this Roadmap describes current knowledge gaps and new research opportunities to evaluate the relationships between the eye (in particular, retinal biomarkers) and the risk of cardiovascular diseases, including coronary artery disease, heart failure, stroke, hypertension and vascular dementia. Identified gaps include the need to simplify and standardize the capture of high-quality images of the eye by non-ophthalmic health workers and to conduct longitudinal studies using multidisciplinary networks of diverse at-risk populations with improved implementation and methods to protect participant and dataset privacy. Other gaps include improving the measurement of structural and functional retinal biomarkers, determining the relationship between microvascular and macrovascular risk factors, improving multimodal imaging 'pipelines', and integrating advanced imaging with 'omics', lifestyle factors, primary care data and radiological reports, by using artificial intelligence technology to improve the identification of individual-level risk. Future research on retinal microvascular disease and retinal biomarkers might additionally provide insights into the temporal development of microvascular disease across other systemic vascular beds.
Collapse
Affiliation(s)
- Emily Y Chew
- Division of Epidemiology and Clinical Applications, National Eye Institute, NIH, Bethesda, MD, USA.
| | - Stephen A Burns
- School of Optometry, Indiana University, Bloomington, IN, USA
| | - Alison G Abraham
- Department of Epidemiology, Colorado School of Public Health, University of Colorado, Aurora, CO, USA
| | - Mathieu F Bakhoum
- Departments of Ophthalmology and Visual Science and Pathology, School of Medicine, Yale University, New Haven, CT, USA
| | - Joshua A Beckman
- Division of Vascular Medicine, University of Southwestern Medical Center, Dallas, TX, USA
| | - Toco Y P Chui
- Department of Ophthalmology, New York Eye and Ear Infirmary of Mount Sinai, New York, NY, USA
| | - Robert P Finger
- Department of Ophthalmology, Mannheim Medical Faculty, University of Heidelberg, Mannheim, Germany
| | - Alejandro F Frangi
- Division of Informatics, Imaging and Data Science (School of Health Sciences), Department of Computer Science (School of Engineering), University of Manchester, Manchester, UK
- Alan Turing Institute, London, UK
| | - Rebecca F Gottesman
- Stroke Branch, National Institute of Neurological Disorders and Stroke, NIH, Bethesda, MD, USA
| | - Maria B Grant
- Department of Ophthalmology and Visual Sciences, School of Medicine, University of Alabama Heersink School of Medicine, Birmingham, AL, USA
| | - Henner Hanssen
- Department of Sport, Exercise and Health, University of Basel, Basel, Switzerland
| | - Cecilia S Lee
- Department of Ophthalmology, University of Washington, Seattle, WA, USA
| | - Michelle L Meyer
- Department of Emergency Medicine, University of North Carolina, Chapel Hill, NC, USA
| | - Damiano Rizzoni
- Department of Clinical and Experimental Sciences, University of Brescia, Brescia, Italy
| | - Alicja R Rudnicka
- Population Health Research Institute, St. George's University of London, London, UK
| | - Joel S Schuman
- Wills Eye Hospital, Sidney Kimmel Medical College at Thomas Jefferson University, Philadelphia, PA, USA
| | - Sara B Seidelmann
- Department of Clinical Medicine, Columbia College of Physicians and Surgeons, Greenwich, CT, USA
| | - W H Wilson Tang
- Lerner Research Institute, Cleveland Clinic, Cleveland, OH, USA
| | - Bishow B Adhikari
- Division of Cardiovascular Sciences, National Heart, Lung, and Blood Institute, NIH, Bethesda, MD, USA
| | - Narasimhan Danthi
- Division of Cardiovascular Sciences, National Heart, Lung, and Blood Institute, NIH, Bethesda, MD, USA
| | - Yuling Hong
- Division of Cardiovascular Sciences, National Heart, Lung, and Blood Institute, NIH, Bethesda, MD, USA
| | - Diane Reid
- Division of Cardiovascular Sciences, National Heart, Lung, and Blood Institute, NIH, Bethesda, MD, USA
| | - Grace L Shen
- Retinal Diseases Program, Division of Extramural Science Programs, National Eye Institute, NIH, Bethesda, MD, USA
| | - Young S Oh
- Division of Cardiovascular Sciences, National Heart, Lung, and Blood Institute, NIH, Bethesda, MD, USA
| |
Collapse
|
2
|
Luo Y, Tian Y, Shi M, Pasquale LR, Shen LQ, Zebardast N, Elze T, Wang M. Harvard Glaucoma Fairness: A Retinal Nerve Disease Dataset for Fairness Learning and Fair Identity Normalization. IEEE TRANSACTIONS ON MEDICAL IMAGING 2024; 43:2623-2633. [PMID: 38478455 PMCID: PMC11251413 DOI: 10.1109/tmi.2024.3377552] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 03/20/2024]
Abstract
Fairness (also known as equity interchangeably) in machine learning is important for societal well-being, but limited public datasets hinder its progress. Currently, no dedicated public medical datasets with imaging data for fairness learning are available, though underrepresented groups suffer from more health issues. To address this gap, we introduce Harvard Glaucoma Fairness (Harvard-GF), a retinal nerve disease dataset including 3,300 subjects with both 2D and 3D imaging data and balanced racial groups for glaucoma detection. Glaucoma is the leading cause of irreversible blindness globally with Blacks having doubled glaucoma prevalence than other races. We also propose a fair identity normalization (FIN) approach to equalize the feature importance between different identity groups. Our FIN approach is compared with various state-of-the-art fairness learning methods with superior performance in the racial, gender, and ethnicity fairness tasks with 2D and 3D imaging data, demonstrating the utilities of our dataset Harvard-GF for fairness learning. To facilitate fairness comparisons between different models, we propose an equity-scaled performance measure, which can be flexibly used to compare all kinds of performance metrics in the context of fairness. The dataset and code are publicly accessible via https://ophai.hms.harvard.edu/datasets/harvard-gf3300/.
Collapse
|
3
|
Perumalraja R, Felcia Logan's Deshna B, Swetha N. Statistical performance review on diagnosis of leukemia, glaucoma and diabetes mellitus using AI. Stat Med 2024; 43:1227-1237. [PMID: 38247116 DOI: 10.1002/sim.10004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2023] [Revised: 12/28/2023] [Accepted: 12/29/2023] [Indexed: 01/23/2024]
Abstract
The growth of artificial intelligence (AI) in the healthcare industry tremendously increases the patient outcomes by reshaping the way we diagnose, treat and monitor patients. AI-based innovation in healthcare include exploration of drugs, personalized medicine, clinical diagnosis investigations, robotic-assisted surgery, verified prescriptions, pregnancy care for women, radiology, and reviewed patient information analytics. However, prediction of AI-based solutions are depends mainly on the implementation of statistical algorithms and input data set. In this article, statistical performance review on various algorithms, Accuracy, Precision, Recall and F1-Score used to predict the diagnosis of leukemia, glaucoma, and diabetes mellitus is presented. Review on statistical algorithms' performance, used for individual disease diagnosis gives a complete picture of various research efforts during the last two decades. At the end of statistical review on each disease diagnosis, we have discussed our inferences that will give future directions for the new researchers on selection of AI statistical algorithm as well as the input data set.
Collapse
Affiliation(s)
- Rengaraju Perumalraja
- Department of Information Technology, Velammal College of Engineering and Technology, Madurai, India
| | - B Felcia Logan's Deshna
- Department of Information Technology, Velammal College of Engineering and Technology, Madurai, India
| | - N Swetha
- Department of Information Technology, Velammal College of Engineering and Technology, Madurai, India
| |
Collapse
|
4
|
Choi JY, Ryu IH, Kim JK, Lee IS, Yoo TK. Development of a generative deep learning model to improve epiretinal membrane detection in fundus photography. BMC Med Inform Decis Mak 2024; 24:25. [PMID: 38273286 PMCID: PMC10811871 DOI: 10.1186/s12911-024-02431-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/29/2023] [Accepted: 01/17/2024] [Indexed: 01/27/2024] Open
Abstract
BACKGROUND The epiretinal membrane (ERM) is a common retinal disorder characterized by abnormal fibrocellular tissue at the vitreomacular interface. Most patients with ERM are asymptomatic at early stages. Therefore, screening for ERM will become increasingly important. Despite the high prevalence of ERM, few deep learning studies have investigated ERM detection in the color fundus photography (CFP) domain. In this study, we built a generative model to enhance ERM detection performance in the CFP. METHODS This deep learning study retrospectively collected 302 ERM and 1,250 healthy CFP data points from a healthcare center. The generative model using StyleGAN2 was trained using single-center data. EfficientNetB0 with StyleGAN2-based augmentation was validated using independent internal single-center data and external datasets. We randomly assigned healthcare center data to the development (80%) and internal validation (20%) datasets. Data from two publicly accessible sources were used as external validation datasets. RESULTS StyleGAN2 facilitated realistic CFP synthesis with the characteristic cellophane reflex features of the ERM. The proposed method with StyleGAN2-based augmentation outperformed the typical transfer learning without a generative adversarial network. The proposed model achieved an area under the receiver operating characteristic (AUC) curve of 0.926 for internal validation. AUCs of 0.951 and 0.914 were obtained for the two external validation datasets. Compared with the deep learning model without augmentation, StyleGAN2-based augmentation improved the detection performance and contributed to the focus on the location of the ERM. CONCLUSIONS We proposed an ERM detection model by synthesizing realistic CFP images with the pathological features of ERM through generative deep learning. We believe that our deep learning framework will help achieve a more accurate detection of ERM in a limited data setting.
Collapse
Affiliation(s)
- Joon Yul Choi
- Department of Biomedical Engineering, Yonsei University, Wonju, South Korea
| | - Ik Hee Ryu
- Department of Refractive Surgery, B&VIIT Eye Center, B2 GT Tower, 1317-23 Seocho-Dong, Seocho-Gu, Seoul, South Korea
- Research and development department, VISUWORKS, Seoul, South Korea
| | - Jin Kuk Kim
- Department of Refractive Surgery, B&VIIT Eye Center, B2 GT Tower, 1317-23 Seocho-Dong, Seocho-Gu, Seoul, South Korea
- Research and development department, VISUWORKS, Seoul, South Korea
| | - In Sik Lee
- Department of Refractive Surgery, B&VIIT Eye Center, B2 GT Tower, 1317-23 Seocho-Dong, Seocho-Gu, Seoul, South Korea
| | - Tae Keun Yoo
- Department of Refractive Surgery, B&VIIT Eye Center, B2 GT Tower, 1317-23 Seocho-Dong, Seocho-Gu, Seoul, South Korea.
- Research and development department, VISUWORKS, Seoul, South Korea.
| |
Collapse
|
5
|
Chen X, Liu X, Wu Y, Wang Z, Wang SH. Research related to the diagnosis of prostate cancer based on machine learning medical images: A review. Int J Med Inform 2024; 181:105279. [PMID: 37977054 DOI: 10.1016/j.ijmedinf.2023.105279] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/21/2023] [Revised: 09/06/2023] [Accepted: 10/29/2023] [Indexed: 11/19/2023]
Abstract
BACKGROUND Prostate cancer is currently the second most prevalent cancer among men. Accurate diagnosis of prostate cancer can provide effective treatment for patients and greatly reduce mortality. The current medical imaging tools for screening prostate cancer are mainly MRI, CT and ultrasound. In the past 20 years, these medical imaging methods have made great progress with machine learning, especially the rise of deep learning has led to a wider application of artificial intelligence in the use of image-assisted diagnosis of prostate cancer. METHOD This review collected medical image processing methods, prostate and prostate cancer on MR images, CT images, and ultrasound images through search engines such as web of science, PubMed, and Google Scholar, including image pre-processing methods, segmentation of prostate gland on medical images, registration between prostate gland on different modal images, detection of prostate cancer lesions on the prostate. CONCLUSION Through these collated papers, it is found that the current research on the diagnosis and staging of prostate cancer using machine learning and deep learning is in its infancy, and most of the existing studies are on the diagnosis of prostate cancer and classification of lesions, and the accuracy is low, with the best results having an accuracy of less than 0.95. There are fewer studies on staging. The research is mainly focused on MR images and much less on CT images, ultrasound images. DISCUSSION Machine learning and deep learning combined with medical imaging have a broad application prospect for the diagnosis and staging of prostate cancer, but the research in this area still has more room for development.
Collapse
Affiliation(s)
- Xinyi Chen
- School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai 201620, China.
| | - Xiang Liu
- School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai 201620, China.
| | - Yuke Wu
- School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai 201620, China.
| | - Zhenglei Wang
- Department of Medical Imaging, Shanghai Electric Power Hospital, Shanghai 201620, China.
| | - Shuo Hong Wang
- Department of Molecular and Cellular Biology and Center for Brain Science, Harvard University, Cambridge, MA 02138, USA.
| |
Collapse
|
6
|
Huang X, Islam MR, Akter S, Ahmed F, Kazami E, Serhan HA, Abd-Alrazaq A, Yousefi S. Artificial intelligence in glaucoma: opportunities, challenges, and future directions. Biomed Eng Online 2023; 22:126. [PMID: 38102597 PMCID: PMC10725017 DOI: 10.1186/s12938-023-01187-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2023] [Accepted: 12/01/2023] [Indexed: 12/17/2023] Open
Abstract
Artificial intelligence (AI) has shown excellent diagnostic performance in detecting various complex problems related to many areas of healthcare including ophthalmology. AI diagnostic systems developed from fundus images have become state-of-the-art tools in diagnosing retinal conditions and glaucoma as well as other ocular diseases. However, designing and implementing AI models using large imaging data is challenging. In this study, we review different machine learning (ML) and deep learning (DL) techniques applied to multiple modalities of retinal data, such as fundus images and visual fields for glaucoma detection, progression assessment, staging and so on. We summarize findings and provide several taxonomies to help the reader understand the evolution of conventional and emerging AI models in glaucoma. We discuss opportunities and challenges facing AI application in glaucoma and highlight some key themes from the existing literature that may help to explore future studies. Our goal in this systematic review is to help readers and researchers to understand critical aspects of AI related to glaucoma as well as determine the necessary steps and requirements for the successful development of AI models in glaucoma.
Collapse
Affiliation(s)
- Xiaoqin Huang
- Department of Ophthalmology, University of Tennessee Health Science Center, Memphis, USA
| | - Md Rafiqul Islam
- Business Information Systems, Australian Institute of Higher Education, Sydney, Australia
| | - Shanjita Akter
- School of Computer Science, Taylors University, Subang Jaya, Malaysia
| | - Fuad Ahmed
- Department of Computer Science & Engineering, Islamic University of Technology (IUT), Gazipur, Bangladesh
| | - Ehsan Kazami
- Ophthalmology, General Hospital of Mahabad, Urmia University of Medical Sciences, Urmia, Iran
| | - Hashem Abu Serhan
- Department of Ophthalmology, Hamad Medical Corporations, Doha, Qatar
| | - Alaa Abd-Alrazaq
- AI Center for Precision Health, Weill Cornell Medicine-Qatar, Doha, Qatar
| | - Siamak Yousefi
- Department of Ophthalmology, University of Tennessee Health Science Center, Memphis, USA.
- Department of Genetics, Genomics, and Informatics, University of Tennessee Health Science Center, Memphis, USA.
| |
Collapse
|
7
|
Fan R, Bowd C, Brye N, Christopher M, Weinreb RN, Kriegman DJ, Zangwill LM. One-Vote Veto: Semi-Supervised Learning for Low-Shot Glaucoma Diagnosis. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:3764-3778. [PMID: 37610903 PMCID: PMC11214580 DOI: 10.1109/tmi.2023.3307689] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/25/2023]
Abstract
Convolutional neural networks (CNNs) are a promising technique for automated glaucoma diagnosis from images of the fundus, and these images are routinely acquired as part of an ophthalmic exam. Nevertheless, CNNs typically require a large amount of well-labeled data for training, which may not be available in many biomedical image classification applications, especially when diseases are rare and where labeling by experts is costly. This article makes two contributions to address this issue: 1) It extends the conventional Siamese network and introduces a training method for low-shot learning when labeled data are limited and imbalanced, and 2) it introduces a novel semi-supervised learning strategy that uses additional unlabeled training data to achieve greater accuracy. Our proposed multi-task Siamese network (MTSN) can employ any backbone CNN, and we demonstrate with four backbone CNNs that its accuracy with limited training data approaches the accuracy of backbone CNNs trained with a dataset that is 50 times larger. We also introduce One-Vote Veto (OVV) self-training, a semi-supervised learning strategy that is designed specifically for MTSNs. By taking both self-predictions and contrastive predictions of the unlabeled training data into account, OVV self-training provides additional pseudo labels for fine-tuning a pre-trained MTSN. Using a large (imbalanced) dataset with 66,715 fundus photographs acquired over 15 years, extensive experimental results demonstrate the effectiveness of low-shot learning with MTSN and semi-supervised learning with OVV self-training. Three additional, smaller clinical datasets of fundus images acquired under different conditions (cameras, instruments, locations, populations) are used to demonstrate the generalizability of the proposed methods.
Collapse
|
8
|
Rezaei M, Näppi JJ, Bischl B, Yoshida H. Bayesian uncertainty estimation for detection of long-tailed and unseen conditions in medical images. J Med Imaging (Bellingham) 2023; 10:054501. [PMID: 37818179 PMCID: PMC10560997 DOI: 10.1117/1.jmi.10.5.054501] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2023] [Revised: 09/19/2023] [Accepted: 09/20/2023] [Indexed: 10/12/2023] Open
Abstract
Purpose Deep supervised learning provides an effective approach for developing robust models for various computer-aided diagnosis tasks. However, there is often an underlying assumption that the frequencies of the samples between the different classes of the training dataset are either similar or balanced. In real-world medical data, the samples of positive classes often occur too infrequently to satisfy this assumption. Thus, there is an unmet need for deep-learning systems that can automatically identify and adapt to the real-world conditions of imbalanced data. Approach We propose a deep Bayesian ensemble learning framework to address the representation learning problem of long-tailed and out-of-distribution (OOD) samples when training from medical images. By estimating the relative uncertainties of the input data, our framework can adapt to imbalanced data for learning generalizable classifiers. We trained and tested our framework on four public medical imaging datasets with various imbalance ratios and imaging modalities across three different learning tasks: semantic medical image segmentation, OOD detection, and in-domain generalization. We compared the performance of our framework with those of state-of-the-art comparator methods. Results Our proposed framework outperformed the comparator models significantly across all performance metrics (pairwise t -test: p < 0.01 ) in the semantic segmentation of high-resolution CT and MR images as well as in the detection of OOD samples (p < 0.01 ), thereby showing significant improvement in handling the associated long-tailed data distribution. The results of the in-domain generalization also indicated that our framework can enhance the prediction of retinal glaucoma, contributing to clinical decision-making processes. Conclusions Training of the proposed deep Bayesian ensemble learning framework with dynamic Monte-Carlo dropout and a combination of losses yielded the best generalization to unseen samples from imbalanced medical imaging datasets across different learning tasks.
Collapse
Affiliation(s)
- Mina Rezaei
- LMU Munich, Department of Statistics, Munich, Germany
- Munich Center for Machine Learning, Munich, Germany
| | - Janne J. Näppi
- Massachusetts General Hospital, Harvard Medical School, 3D Imaging Research, Department of Radiology, Boston, Massachusetts, United States
| | - Bernd Bischl
- LMU Munich, Department of Statistics, Munich, Germany
- Munich Center for Machine Learning, Munich, Germany
| | - Hiroyuki Yoshida
- Massachusetts General Hospital, Harvard Medical School, 3D Imaging Research, Department of Radiology, Boston, Massachusetts, United States
| |
Collapse
|
9
|
Xie Y, Wan Q, Xie H, Xu Y, Wang T, Wang S, Lei B. Fundus Image-Label Pairs Synthesis and Retinopathy Screening via GANs With Class-Imbalanced Semi-Supervised Learning. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:2714-2725. [PMID: 37030825 DOI: 10.1109/tmi.2023.3263216] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
Retinopathy is the primary cause of irreversible yet preventable blindness. Numerous deep-learning algorithms have been developed for automatic retinal fundus image analysis. However, existing methods are usually data-driven, which rarely consider the costs associated with fundus image collection and annotation, along with the class-imbalanced distribution that arises from the relative scarcity of disease-positive individuals in the population. Semi-supervised learning on class-imbalanced data, despite a realistic problem, has been relatively little studied. To fill the existing research gap, we explore generative adversarial networks (GANs) as a potential answer to that problem. Specifically, we present a novel framework, named CISSL-GANs, for class-imbalanced semi-supervised learning (CISSL) by leveraging a dynamic class-rebalancing (DCR) sampler, which exploits the property that the classifier trained on class-imbalanced data produces high-precision pseudo-labels on minority classes to leverage the bias inherent in pseudo-labels. Also, given the well-known difficulty of training GANs on complex data, we investigate three practical techniques to improve the training dynamics without altering the global equilibrium. Experimental results demonstrate that our CISSL-GANs are capable of simultaneously improving fundus image class-conditional generation and classification performance under a typical label insufficient and imbalanced scenario. Our code is available at: https://github.com/Xyporz/CISSL-GANs.
Collapse
|
10
|
FixMatch-LS: Semi-supervised skin lesion classification with label smoothing. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2023.104709] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/06/2023]
|
11
|
Wang Z, Lim G, Ng WY, Tan TE, Lim J, Lim SH, Foo V, Lim J, Sinisterra LG, Zheng F, Liu N, Tan GSW, Cheng CY, Cheung GCM, Wong TY, Ting DSW. Synthetic artificial intelligence using generative adversarial network for retinal imaging in detection of age-related macular degeneration. Front Med (Lausanne) 2023; 10:1184892. [PMID: 37425325 PMCID: PMC10324667 DOI: 10.3389/fmed.2023.1184892] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/12/2023] [Accepted: 05/30/2023] [Indexed: 07/11/2023] Open
Abstract
Introduction Age-related macular degeneration (AMD) is one of the leading causes of vision impairment globally and early detection is crucial to prevent vision loss. However, the screening of AMD is resource dependent and demands experienced healthcare providers. Recently, deep learning (DL) systems have shown the potential for effective detection of various eye diseases from retinal fundus images, but the development of such robust systems requires a large amount of datasets, which could be limited by prevalence of the disease and privacy of patient. As in the case of AMD, the advanced phenotype is often scarce for conducting DL analysis, which may be tackled via generating synthetic images using Generative Adversarial Networks (GANs). This study aims to develop GAN-synthesized fundus photos with AMD lesions, and to assess the realness of these images with an objective scale. Methods To build our GAN models, a total of 125,012 fundus photos were used from a real-world non-AMD phenotypical dataset. StyleGAN2 and human-in-the-loop (HITL) method were then applied to synthesize fundus images with AMD features. To objectively assess the quality of the synthesized images, we proposed a novel realness scale based on the frequency of the broken vessels observed in the fundus photos. Four residents conducted two rounds of gradings on 300 images to distinguish real from synthetic images, based on their subjective impression and the objective scale respectively. Results and discussion The introduction of HITL training increased the percentage of synthetic images with AMD lesions, despite the limited number of AMD images in the initial training dataset. Qualitatively, the synthesized images have been proven to be robust in that our residents had limited ability to distinguish real from synthetic ones, as evidenced by an overall accuracy of 0.66 (95% CI: 0.61-0.66) and Cohen's kappa of 0.320. For the non-referable AMD classes (no or early AMD), the accuracy was only 0.51. With the objective scale, the overall accuracy improved to 0.72. In conclusion, GAN models built with HITL training are capable of producing realistic-looking fundus images that could fool human experts, while our objective realness scale based on broken vessels can help identifying the synthetic fundus photos.
Collapse
Affiliation(s)
- Zhaoran Wang
- Duke-NUS Medical School, National University of Singapore, Singapore, Singapore
| | - Gilbert Lim
- Duke-NUS Medical School, National University of Singapore, Singapore, Singapore
- Singapore Eye Research Institute, Singapore, Singapore
| | - Wei Yan Ng
- Singapore Eye Research Institute, Singapore, Singapore
- Singapore National Eye Centre, Singapore, Singapore
| | - Tien-En Tan
- Singapore Eye Research Institute, Singapore, Singapore
- Singapore National Eye Centre, Singapore, Singapore
| | - Jane Lim
- Singapore Eye Research Institute, Singapore, Singapore
- Singapore National Eye Centre, Singapore, Singapore
| | - Sing Hui Lim
- Singapore Eye Research Institute, Singapore, Singapore
- Singapore National Eye Centre, Singapore, Singapore
| | - Valencia Foo
- Singapore Eye Research Institute, Singapore, Singapore
- Singapore National Eye Centre, Singapore, Singapore
| | - Joshua Lim
- Singapore Eye Research Institute, Singapore, Singapore
- Singapore National Eye Centre, Singapore, Singapore
| | | | - Feihui Zheng
- Singapore Eye Research Institute, Singapore, Singapore
| | - Nan Liu
- Duke-NUS Medical School, National University of Singapore, Singapore, Singapore
- Singapore Eye Research Institute, Singapore, Singapore
| | - Gavin Siew Wei Tan
- Duke-NUS Medical School, National University of Singapore, Singapore, Singapore
- Singapore Eye Research Institute, Singapore, Singapore
- Singapore National Eye Centre, Singapore, Singapore
| | - Ching-Yu Cheng
- Duke-NUS Medical School, National University of Singapore, Singapore, Singapore
- Singapore Eye Research Institute, Singapore, Singapore
- Singapore National Eye Centre, Singapore, Singapore
| | - Gemmy Chui Ming Cheung
- Duke-NUS Medical School, National University of Singapore, Singapore, Singapore
- Singapore Eye Research Institute, Singapore, Singapore
- Singapore National Eye Centre, Singapore, Singapore
| | - Tien Yin Wong
- Singapore National Eye Centre, Singapore, Singapore
- School of Medicine, Tsinghua University, Beijing, China
| | - Daniel Shu Wei Ting
- Duke-NUS Medical School, National University of Singapore, Singapore, Singapore
- Singapore Eye Research Institute, Singapore, Singapore
- Singapore National Eye Centre, Singapore, Singapore
| |
Collapse
|
12
|
Velpula VK, Sharma LD. Multi-stage glaucoma classification using pre-trained convolutional neural networks and voting-based classifier fusion. Front Physiol 2023; 14:1175881. [PMID: 37383146 PMCID: PMC10293617 DOI: 10.3389/fphys.2023.1175881] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2023] [Accepted: 05/19/2023] [Indexed: 06/30/2023] Open
Abstract
Aim: To design an automated glaucoma detection system for early detection of glaucoma using fundus images. Background: Glaucoma is a serious eye problem that can cause vision loss and even permanent blindness. Early detection and prevention are crucial for effective treatment. Traditional diagnostic approaches are time consuming, manual, and often inaccurate, thus making automated glaucoma diagnosis necessary. Objective: To propose an automated glaucoma stage classification model using pre-trained deep convolutional neural network (CNN) models and classifier fusion. Methods: The proposed model utilized five pre-trained CNN models: ResNet50, AlexNet, VGG19, DenseNet-201, and Inception-ResNet-v2. The model was tested using four public datasets: ACRIMA, RIM-ONE, Harvard Dataverse (HVD), and Drishti. Classifier fusion was created to merge the decisions of all CNN models using the maximum voting-based approach. Results: The proposed model achieved an area under the curve of 1 and an accuracy of 99.57% for the ACRIMA dataset. The HVD dataset had an area under the curve of 0.97 and an accuracy of 85.43%. The accuracy rates for Drishti and RIM-ONE were 90.55 and 94.95%, respectively. The experimental results showed that the proposed model performed better than the state-of-the-art methods in classifying glaucoma in its early stages. Understanding the model output includes both attribution-based methods such as activations and gradient class activation map and perturbation-based methods such as locally interpretable model-agnostic explanations and occlusion sensitivity, which generate heatmaps of various sections of an image for model prediction. Conclusion: The proposed automated glaucoma stage classification model using pre-trained CNN models and classifier fusion is an effective method for the early detection of glaucoma. The results indicate high accuracy rates and superior performance compared to the existing methods.
Collapse
|
13
|
Tadisetty S, Chodavarapu R, Jin R, Clements RJ, Yu M. Identifying the Edges of the Optic Cup and the Optic Disc in Glaucoma Patients by Segmentation. SENSORS (BASEL, SWITZERLAND) 2023; 23:4668. [PMID: 37430580 PMCID: PMC10221430 DOI: 10.3390/s23104668] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/13/2023] [Revised: 05/08/2023] [Accepted: 05/10/2023] [Indexed: 07/12/2023]
Abstract
With recent advancements in artificial intelligence, fundus diseases can be classified automatically for early diagnosis, and this is an interest of many researchers. The study aims to detect the edges of the optic cup and the optic disc of fundus images taken from glaucoma patients, which has further applications in the analysis of the cup-to-disc ratio (CDR). We apply a modified U-Net model architecture on various fundus datasets and use segmentation metrics to evaluate the model. We apply edge detection and dilation to post-process the segmentation and better visualize the optic cup and optic disc. Our model results are based on ORIGA, RIM-ONE v3, REFUGE, and Drishti-GS datasets. Our results show that our methodology obtains promising segmentation efficiency for CDR analysis.
Collapse
Affiliation(s)
- Srikanth Tadisetty
- Department of Computer Science, Kent State University, Kent, OH 44242, USA; (S.T.); (R.C.)
| | - Ranjith Chodavarapu
- Department of Computer Science, Kent State University, Kent, OH 44242, USA; (S.T.); (R.C.)
| | - Ruoming Jin
- Department of Computer Science, Kent State University, Kent, OH 44242, USA; (S.T.); (R.C.)
| | - Robert J. Clements
- Department of Biological Sciences, Kent State University, Kent, OH 44242, USA;
| | - Minzhong Yu
- Department of Ophthalmology, University Hospitals, Case Western Reserve University, Cleveland, OH 44106, USA
| |
Collapse
|
14
|
Gutierrez A, Chen TC. Artificial intelligence in glaucoma: posterior segment optical coherence tomography. Curr Opin Ophthalmol 2023; 34:245-254. [PMID: 36728784 PMCID: PMC10090343 DOI: 10.1097/icu.0000000000000934] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/03/2023]
Abstract
PURPOSE OF REVIEW To summarize the recent literature on deep learning (DL) model applications in glaucoma detection and surveillance using posterior segment optical coherence tomography (OCT) imaging. RECENT FINDINGS DL models use OCT derived parameters including retinal nerve fiber layer (RNFL) scans, macular scans, and optic nerve head (ONH) scans, as well as a combination of these parameters, to achieve high diagnostic accuracy in detecting glaucomatous optic neuropathy (GON). Although RNFL segmentation is the most widely used OCT parameter for glaucoma detection by ophthalmologists, newer DL models most commonly use a combination of parameters, which provide a more comprehensive approach. Compared to DL models for diagnosing glaucoma, DL models predicting glaucoma progression are less commonly studied but have also been developed. SUMMARY DL models offer time-efficient, objective, and potential options in the management of glaucoma. Although artificial intelligence models have already been commercially accepted as diagnostic tools for other ophthalmic diseases, there is no commercially approved DL tool for the diagnosis of glaucoma, most likely in part due to the lack of a universal definition of glaucoma defined by OCT derived parameters alone (see Supplemental Digital Content 1 for video abstract, http://links.lww.com/COOP/A54 ).
Collapse
Affiliation(s)
- Alfredo Gutierrez
- Tufts School of Medicine
- Department of Ophthalmology, Massachusetts Eye and Ear Infirmary, Glaucoma Service
| | - Teresa C. Chen
- Department of Ophthalmology, Massachusetts Eye and Ear Infirmary, Glaucoma Service
- Harvard Medical School, Boston, Massachusetts, USA
| |
Collapse
|
15
|
Li Y, Yip MYT, Ting DSW, Ang M. Artificial intelligence and digital solutions for myopia. Taiwan J Ophthalmol 2023; 13:142-150. [PMID: 37484621 PMCID: PMC10361438 DOI: 10.4103/tjo.tjo-d-23-00032] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/12/2023] [Accepted: 03/16/2023] [Indexed: 07/25/2023] Open
Abstract
Myopia as an uncorrected visual impairment is recognized as a global public health issue with an increasing burden on health-care systems. Moreover, high myopia increases one's risk of developing pathologic myopia, which can lead to irreversible visual impairment. Thus, increased resources are needed for the early identification of complications, timely intervention to prevent myopia progression, and treatment of complications. Emerging artificial intelligence (AI) and digital technologies may have the potential to tackle these unmet needs through automated detection for screening and risk stratification, individualized prediction, and prognostication of myopia progression. AI applications in myopia for children and adults have been developed for the detection, diagnosis, and prediction of progression. Novel AI technologies, including multimodal AI, explainable AI, federated learning, automated machine learning, and blockchain, may further improve prediction performance, safety, accessibility, and also circumvent concerns of explainability. Digital technology advancements include digital therapeutics, self-monitoring devices, virtual reality or augmented reality technology, and wearable devices - which provide possible avenues for monitoring myopia progression and control. However, there are challenges in the implementation of these technologies, which include requirements for specific infrastructure and resources, demonstrating clinically acceptable performance and safety of data management. Nonetheless, this remains an evolving field with the potential to address the growing global burden of myopia.
Collapse
Affiliation(s)
- Yong Li
- Singapore Eye Research Institute, Singapore National Eye Centre, Singapore
- Department of Ophthalmology and Visual Sciences, Duke-NUS Medical School, National University of Singapore, Singapore
| | - Michelle Y. T. Yip
- Singapore Eye Research Institute, Singapore National Eye Centre, Singapore
| | - Daniel S. W. Ting
- Singapore Eye Research Institute, Singapore National Eye Centre, Singapore
- Department of Ophthalmology and Visual Sciences, Duke-NUS Medical School, National University of Singapore, Singapore
| | - Marcus Ang
- Singapore Eye Research Institute, Singapore National Eye Centre, Singapore
- Department of Ophthalmology and Visual Sciences, Duke-NUS Medical School, National University of Singapore, Singapore
| |
Collapse
|
16
|
Ataş İ. Comparison of deep convolution and least squares GANs for diabetic retinopathy image synthesis. Neural Comput Appl 2023. [DOI: 10.1007/s00521-023-08482-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/29/2023]
|
17
|
Deng C, Deng Z, Lu S, He M, Miao J, Peng Y. Fault Diagnosis Method for Imbalanced Data Based on Multi-Signal Fusion and Improved Deep Convolution Generative Adversarial Network. SENSORS (BASEL, SWITZERLAND) 2023; 23:s23052542. [PMID: 36904745 PMCID: PMC10007067 DOI: 10.3390/s23052542] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/20/2023] [Revised: 02/11/2023] [Accepted: 02/20/2023] [Indexed: 05/27/2023]
Abstract
The realization of accurate fault diagnosis is crucial to ensure the normal operation of machines. At present, an intelligent fault diagnosis method based on deep learning has been widely applied in mechanical areas due to its strong ability of feature extraction and accurate identification. However, it often depends on enough training samples. Generally, the model performance depends on sufficient training samples. However, the fault data are always insufficient in practical engineering as the mechanical equipment often works under normal conditions, resulting in imbalanced data. Deep learning-based models trained directly with the imbalanced data will greatly reduce the diagnosis accuracy. In this paper, a diagnosis method is proposed to address the imbalanced data problem and enhance the diagnosis accuracy. Firstly, signals from multiple sensors are processed by the wavelet transform to enhance data features, which are then squeezed and fused through pooling and splicing operations. Subsequently, improved adversarial networks are constructed to generate new samples for data augmentation. Finally, an improved residual network is constructed by introducing the convolutional block attention module for enhancing the diagnosis performance. The experiments containing two different types of bearing datasets are adopted to validate the effectiveness and superiority of the proposed method in single-class and multi-class data imbalance cases. The results show that the proposed method can generate high-quality synthetic samples and improve the diagnosis accuracy presenting great potential in imbalanced fault diagnosis.
Collapse
Affiliation(s)
- Congying Deng
- School of Advanced Manufacturing Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
| | - Zihao Deng
- School of Advanced Manufacturing Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
| | - Sheng Lu
- School of Advanced Manufacturing Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
| | - Mingge He
- CNPC Chuanqing Drilling Engineering Co., Ltd., Chengdu 610051, China
| | - Jianguo Miao
- College of Automation, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
| | - Ying Peng
- School of Advanced Manufacturing Engineering, Chongqing University of Posts and Telecommunications, Chongqing 400065, China
| |
Collapse
|
18
|
Peng Z, Zhang D, Tian S, Wu W, Yu L, Zhou S, Huang S. FaxMatch: Multi-Curriculum Pseudo-Labeling for semi-supervised medical image classification. Med Phys 2023; 50:3210-3222. [PMID: 36779849 DOI: 10.1002/mp.16312] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2022] [Revised: 12/28/2022] [Accepted: 02/05/2023] [Indexed: 02/14/2023] Open
Abstract
BACKGROUND Semi-supervised learning (SSL) can effectively use information from unlabeled data to improve model performance, which has great significance in medical imaging tasks. Pseudo-labeling is a classical SSL method that uses a model to predict unlabeled samples and selects the prediction with the highest confidence level as the pseudo-labels and then uses the generated pseudo-labels to train the model. Most of the current pseudo-label-based SSL algorithms use predefined fixed thresholds for all classes to select unlabeled data. PURPOSE However, data imbalance is a common problem in medical image tasks, where the use of fixed threshold to generate pseudo-labels ignores different classes of learning status and learning difficulties. The aim of this study is to develop an algorithm to solve this problem. METHODS In this work, we propose Multi-Curriculum Pseudo-Labeling (MCPL), which evaluates the learning status of the model for each class at each epoch and automatically adjusts the thresholds for each class. We apply MCPL to FixMatch and propose a new SSL framework for medical image classification, which we call the improved algorithm FaxMatch. To mitigate the impact of incorrect pseudo-labels on the model, we use label smoothing (LS) strategy to generate soft labels (SL) for pseudo-labels. RESULTS We have conducted extensive experiments to evaluate our method on two public benchmark medical image classification datasets: the ISIC 2018 skin lesion analysis and COVID-CT datasets. Experimental results show that our method outperforms fully supervised baseline, which uses only labeled data to train the model. Moreover, our method also outperforms other state-of-the-art methods. CONCLUSIONS We propose MCPL and construct a semi-supervised medical image classification framework to reduce the reliance of the model on a large number of labeled images and reduce the manual workload of labeling medical image data.
Collapse
Affiliation(s)
- Zhen Peng
- College of Software, Xinjiang University, Urumqi, China.,Key Laboratory of Software Engineering Technology, College of Software, Xin Jiang University, Urumqi, China
| | - Dezhi Zhang
- People's Hospital of Xinjiang Uygur Autonomous Region, Urumqi, China
| | - Shengwei Tian
- College of Software, Xinjiang University, Urumqi, China.,Key Laboratory of Software Engineering Technology, College of Software, Xin Jiang University, Urumqi, China
| | - Weidong Wu
- People's Hospital of Xinjiang Uygur Autonomous Region, Urumqi, China
| | - Long Yu
- College of Information Science and Engineering, Xinjiang University, Urumqi, China.,College of Network Center, Xinjiang University, Urumqi, China
| | - Shaofeng Zhou
- College of Software, Xinjiang University, Urumqi, China.,Key Laboratory of Software Engineering Technology, College of Software, Xin Jiang University, Urumqi, China
| | - Shanhang Huang
- College of Software, Xinjiang University, Urumqi, China.,Key Laboratory of Software Engineering Technology, College of Software, Xin Jiang University, Urumqi, China
| |
Collapse
|
19
|
LAC-GAN: Lesion attention conditional GAN for Ultra-widefield image synthesis. Neural Netw 2023; 158:89-98. [PMID: 36446158 DOI: 10.1016/j.neunet.2022.11.005] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2022] [Revised: 08/30/2022] [Accepted: 11/03/2022] [Indexed: 11/13/2022]
Abstract
Automatic detection of retinal diseases based on deep learning technology and Ultra-widefield (UWF) images plays an important role in clinical practices in recent years. However, due to small lesions and limited data samples, it is not easy to train a detection-accurate model with strong generalization ability. In this paper, we propose a lesion attention conditional generative adversarial network (LAC-GAN) to synthesize retinal images with realistic lesion details to improve the training of the disease detection model. Specifically, the generator takes the vessel mask and class label as the conditional inputs, and processes the random Gaussian noise by a series of residual block to generate the synthetic images. To focus on pathological information, we propose a lesion feature attention mechanism based on random forest (RF) method, which constructs its reverse activation network to activate the lesion features. For discriminator, a weight-sharing multi-discriminator is designed to improve the performance of model by affine transformations. Experimental results on multi-center UWF image datasets demonstrate that the proposed method can generate retinal images with reasonable details, which helps to enhance the performance of the disease detection model.
Collapse
|
20
|
Semi-supervised medical image classification with adaptive threshold pseudo-labeling and unreliable sample contrastive loss. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2022.104142] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022]
|
21
|
Yi Y, Guo C, Hu Y, Zhou W, Wang W. BCR-UNet: Bi-directional ConvLSTM residual U-Net for retinal blood vessel segmentation. Front Public Health 2022; 10:1056226. [PMID: 36483248 PMCID: PMC9722738 DOI: 10.3389/fpubh.2022.1056226] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2022] [Accepted: 11/04/2022] [Indexed: 11/23/2022] Open
Abstract
Background High precision segmentation of retinal blood vessels from retinal images is a significant step for doctors to diagnose many diseases such as glaucoma and cardiovascular diseases. However, at the peripheral region of vessels, previous U-Net-based segmentation methods failed to significantly preserve the low-contrast tiny vessels. Methods For solving this challenge, we propose a novel network model called Bi-directional ConvLSTM Residual U-Net (BCR-UNet), which takes full advantage of U-Net, Dropblock, Residual convolution and Bi-directional ConvLSTM (BConvLSTM). In this proposed BCR-UNet model, we propose a novel Structured Dropout Residual Block (SDRB) instead of using the original U-Net convolutional block, to construct our network skeleton for improving the robustness of the network. Furthermore, to improve the discriminative ability of the network and preserve more original semantic information of tiny vessels, we adopt BConvLSTM to integrate the feature maps captured from the first residual block and the last up-convolutional layer in a nonlinear manner. Results and discussion We conduct experiments on four public retinal blood vessel datasets, and the results show that the proposed BCR-UNet can preserve more tiny blood vessels at the low-contrast peripheral regions, even outperforming previous state-of-the-art methods.
Collapse
Affiliation(s)
- Yugen Yi
- School of Software, Jiangxi Normal University, Nanchang, China
| | - Changlu Guo
- Yichun Economic and Technological Development Zone, Yichun, China,*Correspondence: Changlu Guo
| | - Yangtao Hu
- The 908th Hospital of Chinese People's Liberation Army Joint Logistic Support Force, Nanchang, China,Yangtao Hu
| | - Wei Zhou
- College of Computer Science, Shenyang Aerospace University, Shenyang, China
| | - Wenle Wang
- School of Software, Jiangxi Normal University, Nanchang, China
| |
Collapse
|
22
|
Fu J, Cao L, Wei S, Xu M, Song Y, Li H, You Y. A GAN-based deep enhancer for quality enhancement of retinal images photographed by a handheld fundus camera. ADVANCES IN OPHTHALMOLOGY PRACTICE AND RESEARCH 2022; 2:100077. [PMID: 37846289 PMCID: PMC10577846 DOI: 10.1016/j.aopr.2022.100077] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/28/2022] [Revised: 08/05/2022] [Accepted: 08/12/2022] [Indexed: 10/18/2023]
Abstract
Objective Due to limited imaging conditions, the quality of fundus images is often unsatisfactory, especially for images photographed by handheld fundus cameras. Here, we have developed an automated method based on combining two mirror-symmetric generative adversarial networks (GANs) for image enhancement. Methods A total of 1047 retinal images were included. The raw images were enhanced by a GAN-based deep enhancer and another methods based on luminosity and contrast adjustment. All raw images and enhanced images were anonymously assessed and classified into 6 levels of quality classification by three experienced ophthalmologists. The quality classification and quality change of images were compared. In addition, image-detailed reading results for the number of dubiously pathological fundi were also compared. Results After GAN enhancement, 42.9% of images increased their quality, 37.5% remained stable, and 19.6% decreased. After excluding the images at the highest level (level 0) before enhancement, a large number (75.6%) of images showed an increase in quality classification, and only a minority (9.3%) showed a decrease. The GAN-enhanced method was superior for quality improvement over a luminosity and contrast adjustment method (P<0.001). In terms of image reading results, the consistency rate fluctuated from 86.6% to 95.6%, and for the specific disease subtypes, both discrepancy number and discrepancy rate were less than 15 and 15%, for two ophthalmologists. Conclusions Learning the style of high-quality retinal images based on the proposed deep enhancer may be an effective way to improve the quality of retinal images photographed by handheld fundus cameras.
Collapse
Affiliation(s)
- Junxia Fu
- Beijing Aier Intech Eye Hospital, Beijing, China
- Aier Eye Hospital Group, Hunan, China
- Department of Ophthalmology, The Chinese People's Liberation Army General Hospital, Beijing, China
| | - Lvchen Cao
- School of Artificial Intelligence, Henan University, Zhengzhou, China
| | - Shihui Wei
- Department of Ophthalmology, The Chinese People's Liberation Army General Hospital, Beijing, China
| | - Ming Xu
- Aier Eye Hospital Group, Hunan, China
| | - Yali Song
- Aier Eye Hospital Group, Hunan, China
| | - Huiqi Li
- School of Information and Electronics, Beijing Institute of Technology, Beijing, China
| | - Yuxia You
- Beijing Aier Intech Eye Hospital, Beijing, China
- Aier Eye Hospital Group, Hunan, China
| |
Collapse
|
23
|
Jin K, Ye J. Artificial intelligence and deep learning in ophthalmology: Current status and future perspectives. ADVANCES IN OPHTHALMOLOGY PRACTICE AND RESEARCH 2022; 2:100078. [PMID: 37846285 PMCID: PMC10577833 DOI: 10.1016/j.aopr.2022.100078] [Citation(s) in RCA: 24] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Subscribe] [Scholar Register] [Received: 06/21/2022] [Revised: 08/01/2022] [Accepted: 08/18/2022] [Indexed: 10/18/2023]
Abstract
Background The ophthalmology field was among the first to adopt artificial intelligence (AI) in medicine. The availability of digitized ocular images and substantial data have made deep learning (DL) a popular topic. Main text At the moment, AI in ophthalmology is mostly used to improve disease diagnosis and assist decision-making aiming at ophthalmic diseases like diabetic retinopathy (DR), glaucoma, age-related macular degeneration (AMD), cataract and other anterior segment diseases. However, most of the AI systems developed to date are still in the experimental stages, with only a few having achieved clinical applications. There are a number of reasons for this phenomenon, including security, privacy, poor pervasiveness, trust and explainability concerns. Conclusions This review summarizes AI applications in ophthalmology, highlighting significant clinical considerations for adopting AI techniques and discussing the potential challenges and future directions.
Collapse
Affiliation(s)
- Kai Jin
- Department of Ophthalmology, The Second Affiliated Hospital of Zhejiang University School of Medicine, Hangzhou, China
| | - Juan Ye
- Department of Ophthalmology, The Second Affiliated Hospital of Zhejiang University School of Medicine, Hangzhou, China
| |
Collapse
|
24
|
Zhao M, Lu Z, Zhu S, Wang X, Feng J. Automatic generation of retinal optical coherence tomography images based on generative adversarial networks. Med Phys 2022; 49:7357-7367. [PMID: 36122302 DOI: 10.1002/mp.15988] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/14/2022] [Revised: 07/13/2022] [Accepted: 08/28/2022] [Indexed: 12/13/2022] Open
Abstract
SIGNIFICANCE The automatic generation algorithm of optical coherence tomography (OCT) images based on generative adversarial networks (GAN) can generate a large number of simulation images by a relatively small number of real images, which can effectively improve the classification performance. AIM We proposed an automatic generation algorithm for retinal OCT images based on GAN to alleviate the problem of insufficient images with high quality in deep learning, and put the diagnosis algorithm toward clinical application. APPROACH We designed a generation network based on GAN and trained the network with a data set constructed by 2014_BOE_Srinivasan and OCT2017 to acquire three models. Then, we generated a large number of images by the three models to augment age-related macular degeneration (AMD), diabetic macular edema (DME), and normal images. We evaluated the generated images by subjective visual observation, Fréchet inception distance (FID) scores, and a classification experiment. RESULTS Visual observation shows that the generated images have clear and similar features compared with the real images. Also, the lesion regions containing similar features in the real image and the generated image are randomly distributed in the image field of view. When the FID scores of the three types of generated images are lowest, three local optimal models are obtained for AMD, DME, and normal images, indicating the generated images have high quality and diversity. Moreover, the classification experiment results show that the model performance trained with the mixed images is better than that of the model trained with real images, in which the accuracy, sensitivity, and specificity are improved by 5.56%, 8.89%, and 2.22%. In addition, compared with the generation method based on variational auto-encoder (VAE), the method improved the accuracy, sensitivity, and specificity by 1.97%, 2.97%, and 0.99%, for the same test set. CONCLUSIONS The results show that our method can augment the three kinds of OCT images, not only effectively alleviating the problem of insufficient images with high quality but also improving the diagnosis performance.
Collapse
Affiliation(s)
- Mengmeng Zhao
- Beijing University of Technology, Intelligent Physiological Measurement and Clinical Translation, International Base for Science and Technology Cooperation, Department of Biomedical Engineering, Beijing University of Technology, Beijing, China
| | - Zhenzhen Lu
- Beijing University of Technology, Intelligent Physiological Measurement and Clinical Translation, International Base for Science and Technology Cooperation, Department of Biomedical Engineering, Beijing University of Technology, Beijing, China
| | - Shuyuan Zhu
- Beijing University of Technology, Intelligent Physiological Measurement and Clinical Translation, International Base for Science and Technology Cooperation, Department of Biomedical Engineering, Beijing University of Technology, Beijing, China
| | - Xiaobing Wang
- Capital University of Physical Education and Sports, Sports and Medicine Integrative Innovation Center, Capital University of Physical Education and Sports, Beijing, China
| | - Jihong Feng
- Beijing University of Technology, Intelligent Physiological Measurement and Clinical Translation, International Base for Science and Technology Cooperation, Department of Biomedical Engineering, Beijing University of Technology, Beijing, China
| |
Collapse
|
25
|
Kugelman J, Alonso-Caneiro D, Read SA, Collins MJ. A review of generative adversarial network applications in optical coherence tomography image analysis. JOURNAL OF OPTOMETRY 2022; 15 Suppl 1:S1-S11. [PMID: 36241526 PMCID: PMC9732473 DOI: 10.1016/j.optom.2022.09.004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/31/2022] [Revised: 08/19/2022] [Accepted: 09/20/2022] [Indexed: 06/16/2023]
Abstract
Optical coherence tomography (OCT) has revolutionized ophthalmic clinical practice and research, as a result of the high-resolution images that the method is able to capture in a fast, non-invasive manner. Although clinicians can interpret OCT images qualitatively, the ability to quantitatively and automatically analyse these images represents a key goal for eye care by providing clinicians with immediate and relevant metrics to inform best clinical practice. The range of applications and methods to analyse OCT images is rich and rapidly expanding. With the advent of deep learning methods, the field has experienced significant progress with state-of-the-art-performance for several OCT image analysis tasks. Generative adversarial networks (GANs) represent a subfield of deep learning that allows for a range of novel applications not possible in most other deep learning methods, with the potential to provide more accurate and robust analyses. In this review, the progress in this field and clinical impact are reviewed and the potential future development of applications of GANs to OCT image processing are discussed.
Collapse
Affiliation(s)
- Jason Kugelman
- Queensland University of Technology (QUT), Contact Lens and Visual Optics Laboratory, Centre for Vision and Eye Research, School of Optometry and Vision Science, Kelvin Grove, QLD 4059, Australia.
| | - David Alonso-Caneiro
- Queensland University of Technology (QUT), Contact Lens and Visual Optics Laboratory, Centre for Vision and Eye Research, School of Optometry and Vision Science, Kelvin Grove, QLD 4059, Australia
| | - Scott A Read
- Queensland University of Technology (QUT), Contact Lens and Visual Optics Laboratory, Centre for Vision and Eye Research, School of Optometry and Vision Science, Kelvin Grove, QLD 4059, Australia
| | - Michael J Collins
- Queensland University of Technology (QUT), Contact Lens and Visual Optics Laboratory, Centre for Vision and Eye Research, School of Optometry and Vision Science, Kelvin Grove, QLD 4059, Australia
| |
Collapse
|
26
|
Parashar D, Agrawal DK. Classification of Glaucoma Stages Using Image Empirical Mode Decomposition from Fundus Images. J Digit Imaging 2022; 35:1283-1292. [PMID: 35581407 PMCID: PMC9582090 DOI: 10.1007/s10278-022-00648-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/18/2020] [Revised: 03/18/2022] [Accepted: 04/03/2022] [Indexed: 11/29/2022] Open
Abstract
One of the most prevalent causes of visual loss and blindness is glaucoma. Conventionally, instrument-based tools are employed for glaucoma screening. However, they are inefficient, time-consuming, and manual. Hence, computerized methodologies are needed for fast and accurate diagnosis of glaucoma. Therefore, we proposed a Computer-Aided Diagnosis (CAD) method for the classification of glaucoma stages using Image Empirical Mode decomposition (IEMD). In this study, IEMD is applied to decompose the preprocessed fundus photographs into different Intrinsic Mode Functions (IMFs) to capture the pixel variations. Then, the significant texture-based descriptors have been computed from the IMFs. A dimensionality reduction approach called Principal Component Analysis (PCA) has been employed to pick the robust descriptors from the retrieved feature set. We used the Analysis of Variance (ANOVA) test for feature ranking. Finally, the LS-SVM classifier has been employed to classify glaucoma stages. The proposed CAD system achieved a classification accuracy of 94.45% for the binary classification on the RIM-ONE r12 database. Our approach demonstrated better glaucoma classification performance than the existing automated systems.
Collapse
Affiliation(s)
- Deepak Parashar
- Department of Electronics and Communication Engineering, IES College of Technology, Bhopal, 462044, MP, India.
- Department of Electronics and Communication Engineering, Maulana Azad National Institute of Technology, Bhopal, 462003, MP, India.
| | - Dheraj Kumar Agrawal
- Department of Electronics and Communication Engineering, Maulana Azad National Institute of Technology, Bhopal, 462003, MP, India
| |
Collapse
|
27
|
Guo X, Lu X, Lin Q, Zhang J, Hu X, Che S. A novel retinal image generation model with the preservation of structural similarity and high resolution. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2022.104004] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022]
|
28
|
Huang W, Luo M, Li J, Zhang P, Zha Y. A novel locally-constrained GAN-based ensemble to synthesize arterial spin labeling images. Inf Sci (N Y) 2022. [DOI: 10.1016/j.ins.2022.07.091] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
|
29
|
Semi-Supervised Medical Image Classification Based on Attention and Intrinsic Features of Samples. APPLIED SCIENCES-BASEL 2022. [DOI: 10.3390/app12136726] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/04/2023]
Abstract
The training of deep neural networks usually requires a lot of high-quality data with good annotations to obtain good performance. However, in clinical medicine, obtaining high-quality marker data is laborious and expensive because it requires the professional skill of clinicians. In this paper, based on the consistency strategy, we propose a new semi-supervised model for medical image classification which introduces a self-attention mechanism into the backbone network to learn more meaningful features in image classification tasks and uses the improved version of focal loss at the supervision loss to reduce the misclassification of samples. Finally, we add a consistency loss similar to the unsupervised consistency loss to encourage the model to learn more about the internal features of unlabeled samples. Our method achieved 94.02% AUC and 72.03% Sensitivity on the ISIC 2018 dataset and 79.74% AUC on the ChestX-ray14 dataset. These results show the effectiveness of our method in single-label and multi-label classification.
Collapse
|
30
|
SemiSANet: A Semi-Supervised High-Resolution Remote Sensing Image Change Detection Model Using Siamese Networks with Graph Attention. REMOTE SENSING 2022. [DOI: 10.3390/rs14122801] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/01/2023]
Abstract
Change detection (CD) is one of the important applications of remote sensing and plays an important role in disaster assessment, land use detection, and urban sprawl tracking. High-accuracy fully supervised methods are the main methods for CD tasks at present. However, these methods require a large amount of labeled data consisting of bi-temporal images and their change maps. Moreover, creating change maps takes a lot of labor and time. To address this limitation, a simple semi-supervised change detection method based on consistency regularization and strong augmentation is proposed in this paper. First, we construct a Siamese nested UNet with graph attention mechanism (SANet) and pre-train it with a small amount of labeled data. Then, we feed the unlabeled data into the pre-trained SANet and confidence threshold filter to obtain pseudo-labels with high confidence. At the same time, we produce distorted images by performing strong augmentation on unlabeled data. The model is trained to make the CD results of the distorted images consistent with the corresponding pseudo-label. Extensive experiments are conducted on two high-resolution remote sensing datasets. The results demonstrate that our method can effectively improve the performance of change detection under insufficient labels. Our methods can increase the IoU by more than 25% compared to the state-of-the-art methods.
Collapse
|
31
|
Sánchez-Morales A, Morales-Sánchez J, Kovalyk O, Verdú-Monedero R, Sancho-Gómez JL. Improving Glaucoma Diagnosis Assembling Deep Networks and Voting Schemes. Diagnostics (Basel) 2022; 12:diagnostics12061382. [PMID: 35741192 PMCID: PMC9222078 DOI: 10.3390/diagnostics12061382] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/13/2022] [Revised: 05/30/2022] [Accepted: 05/31/2022] [Indexed: 02/05/2023] Open
Abstract
Glaucoma is a group of eye conditions that damage the optic nerve, the health of which is vital for good eyesight. This damage is often caused by higher-than-normal pressure in the eye. In the past few years, the applications of artificial intelligence and data science have increased rapidly in medicine especially in imaging applications. In particular, deep learning tools have been successfully applied obtaining, in some cases, results superior to those obtained by humans. In this article, we present a soft novel ensemble model based on the K-NN algorithm, that combines the probability of class membership obtained by several deep learning models. In this research, three models of different nature (CNN, CapsNets and Convolutional Autoencoders) have been selected searching for diversity. The latent space of these models are combined using the local information provided by the true sample labels and the K-NN algorithm is applied to determine the final decision. The results obtained on two different datasets of retinal images show that the proposed ensemble model improves the diagnosis capabilities for both the individual models and the state-of-the-art results.
Collapse
|
32
|
An intelligent location method of key boundary points for assisting the diameter measurement of central serous chorioretinopathy lesion area. Comput Biol Med 2022; 147:105730. [PMID: 35792473 DOI: 10.1016/j.compbiomed.2022.105730] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/20/2022] [Revised: 06/06/2022] [Accepted: 06/11/2022] [Indexed: 11/24/2022]
|
33
|
Lim JS, Hong M, Lam WST, Zhang Z, Teo ZL, Liu Y, Ng WY, Foo LL, Ting DSW. Novel technical and privacy-preserving technology for artificial intelligence in ophthalmology. Curr Opin Ophthalmol 2022; 33:174-187. [PMID: 35266894 DOI: 10.1097/icu.0000000000000846] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
PURPOSE OF REVIEW The application of artificial intelligence (AI) in medicine and ophthalmology has experienced exponential breakthroughs in recent years in diagnosis, prognosis, and aiding clinical decision-making. The use of digital data has also heralded the need for privacy-preserving technology to protect patient confidentiality and to guard against threats such as adversarial attacks. Hence, this review aims to outline novel AI-based systems for ophthalmology use, privacy-preserving measures, potential challenges, and future directions of each. RECENT FINDINGS Several key AI algorithms used to improve disease detection and outcomes include: Data-driven, imagedriven, natural language processing (NLP)-driven, genomics-driven, and multimodality algorithms. However, deep learning systems are susceptible to adversarial attacks, and use of data for training models is associated with privacy concerns. Several data protection methods address these concerns in the form of blockchain technology, federated learning, and generative adversarial networks. SUMMARY AI-applications have vast potential to meet many eyecare needs, consequently reducing burden on scarce healthcare resources. A pertinent challenge would be to maintain data privacy and confidentiality while supporting AI endeavors, where data protection methods would need to rapidly evolve with AI technology needs. Ultimately, for AI to succeed in medicine and ophthalmology, a balance would need to be found between innovation and privacy.
Collapse
Affiliation(s)
- Jane S Lim
- Singapore National Eye Centre, Singapore Eye Research Institute
| | | | - Walter S T Lam
- Yong Loo Lin School of Medicine, National University of Singapore
| | - Zheting Zhang
- Lee Kong Chian School of Medicine, Nanyang Technological University
| | - Zhen Ling Teo
- Singapore National Eye Centre, Singapore Eye Research Institute
| | - Yong Liu
- National University of Singapore, DukeNUS Medical School, Singapore
| | - Wei Yan Ng
- Singapore National Eye Centre, Singapore Eye Research Institute
| | - Li Lian Foo
- Singapore National Eye Centre, Singapore Eye Research Institute
| | - Daniel S W Ting
- Singapore National Eye Centre, Singapore Eye Research Institute
| |
Collapse
|
34
|
Liu P, Zheng G. Handling Imbalanced Data: Uncertainty-guided Virtual Adversarial Training with Batch Nuclear-norm Optimization for Semi-supervised Medical Image Classification. IEEE J Biomed Health Inform 2022; 26:2983-2994. [PMID: 35344500 DOI: 10.1109/jbhi.2022.3162748] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
In many clinical settings, a lot of medical image datasets suffer from imbalance problems, which makes predictions of trained models to be biased toward majority classes. Semi-supervised Learning (SSL) algorithms trained with such imbalanced datasets become more problematic since pseudo-supervision of unlabeled data are generated from the model's biased predictions. To address these issues, in this work, we propose a novel semi-supervised deep learning method, i.e., uncertainty-guided virtual adversarial training (VAT) with batch nuclear-norm (BNN) optimization, for large-scale medical image classification. To effectively exploit useful information from both labeled and unlabeled data, we leverage VAT and BNN optimization to harness the underlying knowledge, which helps to improve discriminability, diversity and generalization of the trained models. More concretely, our network is trained by minimizing a combination of four types of losses, including a supervised cross-entropy loss, a BNN loss defined on the output matrix of labeled data batch (lBNN loss), a negative BNN loss defined on the output matrix of unlabeled data batch (uBNN loss), and a VAT loss on both labeled and unlabeled data. We additionally propose to use uncertainty estimation to filter out unlabeled samples near the decision boundary when computing the VAT loss. We conduct comprehensive experiments to evaluate the performance of our method on two publicly available datasets and one in-house collected dataset. The experimental results demonstrated that our method achieved better results than state-of-the-art SSL methods.
Collapse
|
35
|
DSLN: Dual-tutor student learning network for multiracial glaucoma detection. Neural Comput Appl 2022. [DOI: 10.1007/s00521-022-07078-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
|
36
|
Chen Y, Yang XH, Wei Z, Heidari AA, Zheng N, Li Z, Chen H, Hu H, Zhou Q, Guan Q. Generative Adversarial Networks in Medical Image augmentation: A review. Comput Biol Med 2022; 144:105382. [PMID: 35276550 DOI: 10.1016/j.compbiomed.2022.105382] [Citation(s) in RCA: 49] [Impact Index Per Article: 24.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2022] [Revised: 02/25/2022] [Accepted: 03/02/2022] [Indexed: 12/31/2022]
Abstract
OBJECT With the development of deep learning, the number of training samples for medical image-based diagnosis and treatment models is increasing. Generative Adversarial Networks (GANs) have attracted attention in medical image processing due to their excellent image generation capabilities and have been widely used in data augmentation. In this paper, a comprehensive and systematic review and analysis of medical image augmentation work are carried out, and its research status and development prospects are reviewed. METHOD This paper reviews 105 medical image augmentation related papers, which mainly collected by ELSEVIER, IEEE Xplore, and Springer from 2018 to 2021. We counted these papers according to the parts of the organs corresponding to the images, and sorted out the medical image datasets that appeared in them, the loss function in model training, and the quantitative evaluation metrics of image augmentation. At the same time, we briefly introduce the literature collected in three journals and three conferences that have received attention in medical image processing. RESULT First, we summarize the advantages of various augmentation models, loss functions, and evaluation metrics. Researchers can use this information as a reference when designing augmentation tasks. Second, we explore the relationship between augmented models and the amount of the training set, and tease out the role that augmented models may play when the quality of the training set is limited. Third, the statistical number of papers shows that the development momentum of this research field remains strong. Furthermore, we discuss the existing limitations of this type of model and suggest possible research directions. CONCLUSION We discuss GAN-based medical image augmentation work in detail. This method effectively alleviates the challenge of limited training samples for medical image diagnosis and treatment models. It is hoped that this review will benefit researchers interested in this field.
Collapse
Affiliation(s)
- Yizhou Chen
- College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China.
| | - Xu-Hua Yang
- College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China.
| | - Zihan Wei
- College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China.
| | - Ali Asghar Heidari
- School of Surveying and Geospatial Engineering, College of Engineering, University of Tehran, Tehran, Iran; Department of Computer Science, School of Computing, National University of Singapore, Singapore, Singapore.
| | - Nenggan Zheng
- Qiushi Academy for Advanced Studies, Zhejiang University, Hangzhou, Zhejiang, China.
| | - Zhicheng Li
- Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China.
| | - Huiling Chen
- College of Computer Science and Artificial Intelligence, Wenzhou University, Wenzhou, Zhejiang, 325035, China.
| | - Haigen Hu
- College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China.
| | - Qianwei Zhou
- College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China.
| | - Qiu Guan
- College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, China.
| |
Collapse
|
37
|
You A, Kim JK, Ryu IH, Yoo TK. Application of generative adversarial networks (GAN) for ophthalmology image domains: a survey. EYE AND VISION (LONDON, ENGLAND) 2022; 9:6. [PMID: 35109930 PMCID: PMC8808986 DOI: 10.1186/s40662-022-00277-3] [Citation(s) in RCA: 48] [Impact Index Per Article: 24.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/15/2021] [Accepted: 01/11/2022] [Indexed: 12/12/2022]
Abstract
BACKGROUND Recent advances in deep learning techniques have led to improved diagnostic abilities in ophthalmology. A generative adversarial network (GAN), which consists of two competing types of deep neural networks, including a generator and a discriminator, has demonstrated remarkable performance in image synthesis and image-to-image translation. The adoption of GAN for medical imaging is increasing for image generation and translation, but it is not familiar to researchers in the field of ophthalmology. In this work, we present a literature review on the application of GAN in ophthalmology image domains to discuss important contributions and to identify potential future research directions. METHODS We performed a survey on studies using GAN published before June 2021 only, and we introduced various applications of GAN in ophthalmology image domains. The search identified 48 peer-reviewed papers in the final review. The type of GAN used in the analysis, task, imaging domain, and the outcome were collected to verify the usefulness of the GAN. RESULTS In ophthalmology image domains, GAN can perform segmentation, data augmentation, denoising, domain transfer, super-resolution, post-intervention prediction, and feature extraction. GAN techniques have established an extension of datasets and modalities in ophthalmology. GAN has several limitations, such as mode collapse, spatial deformities, unintended changes, and the generation of high-frequency noises and artifacts of checkerboard patterns. CONCLUSIONS The use of GAN has benefited the various tasks in ophthalmology image domains. Based on our observations, the adoption of GAN in ophthalmology is still in a very early stage of clinical validation compared with deep learning classification techniques because several problems need to be overcome for practical use. However, the proper selection of the GAN technique and statistical modeling of ocular imaging will greatly improve the performance of each image analysis. Finally, this survey would enable researchers to access the appropriate GAN technique to maximize the potential of ophthalmology datasets for deep learning research.
Collapse
Affiliation(s)
- Aram You
- School of Architecture, Kumoh National Institute of Technology, Gumi, Gyeongbuk, South Korea
| | - Jin Kuk Kim
- B&VIIT Eye Center, Seoul, South Korea
- VISUWORKS, Seoul, South Korea
| | - Ik Hee Ryu
- B&VIIT Eye Center, Seoul, South Korea
- VISUWORKS, Seoul, South Korea
| | - Tae Keun Yoo
- B&VIIT Eye Center, Seoul, South Korea.
- Department of Ophthalmology, Aerospace Medical Center, Republic of Korea Air Force, 635 Danjae-ro, Namil-myeon, Cheongwon-gun, Cheongju, Chungcheongbuk-do, 363-849, South Korea.
| |
Collapse
|
38
|
Patwary MJ, Cao W, Wang XZ, Haque MA. Fuzziness based semi-supervised multimodal learning for patient’s activity recognition using RGBDT videos. Appl Soft Comput 2022. [DOI: 10.1016/j.asoc.2022.108655] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022]
|
39
|
Wan C, Zhou X, You Q, Sun J, Shen J, Zhu S, Jiang Q, Yang W. Retinal Image Enhancement Using Cycle-Constraint Adversarial Network. Front Med (Lausanne) 2022; 8:793726. [PMID: 35096883 PMCID: PMC8789669 DOI: 10.3389/fmed.2021.793726] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2021] [Accepted: 12/14/2021] [Indexed: 11/25/2022] Open
Abstract
Retinal images are the most intuitive medical images for the diagnosis of fundus diseases. Low-quality retinal images cause difficulties in computer-aided diagnosis systems and the clinical diagnosis of ophthalmologists. The high quality of retinal images is an important basis of precision medicine in ophthalmology. In this study, we propose a retinal image enhancement method based on deep learning to enhance multiple low-quality retinal images. A generative adversarial network is employed to build a symmetrical network, and a convolutional block attention module is introduced to improve the feature extraction capability. The retinal images in our dataset are sorted into two sets according to their quality: low and high quality. Generators and discriminators alternately learn the features of low/high-quality retinal images without the need for paired images. We analyze the proposed method both qualitatively and quantitatively on public datasets and a private dataset. The study results demonstrate that the proposed method is superior to other advanced algorithms, especially in enhancing color-distorted retinal images. It also performs well in the task of retinal vessel segmentation. The proposed network effectively enhances low-quality retinal images, aiding ophthalmologists and enabling computer-aided diagnosis in pathological analysis. Our method enhances multiple types of low-quality retinal images using a deep learning network.
Collapse
Affiliation(s)
- Cheng Wan
- College of Electronic Information Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, China
| | - Xueting Zhou
- College of Electronic Information Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, China
| | - Qijing You
- College of Electronic Information Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, China
| | - Jing Sun
- College of Electronic Information Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, China
| | - Jianxin Shen
- College of Mechanical and Electrical Engineering, Nanjing University of Aeronautics and Astronautics, Nanjing, China
| | - Shaojun Zhu
- School of Information Engineering, Huzhou University, Huzhou, China
| | - Qin Jiang
- The Laboratory of Artificial Intelligence and Bigdata in Ophthalmology, The Affiliated Eye Hospital of Nanjing Medical University, Nanjing, China
| | - Weihua Yang
- The Laboratory of Artificial Intelligence and Bigdata in Ophthalmology, The Affiliated Eye Hospital of Nanjing Medical University, Nanjing, China
| |
Collapse
|
40
|
Camara J, Neto A, Pires IM, Villasana MV, Zdravevski E, Cunha A. Literature Review on Artificial Intelligence Methods for Glaucoma Screening, Segmentation, and Classification. J Imaging 2022; 8:jimaging8020019. [PMID: 35200722 PMCID: PMC8878383 DOI: 10.3390/jimaging8020019] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/17/2021] [Revised: 01/11/2022] [Accepted: 01/17/2022] [Indexed: 12/20/2022] Open
Abstract
Artificial intelligence techniques are now being applied in different medical solutions ranging from disease screening to activity recognition and computer-aided diagnosis. The combination of computer science methods and medical knowledge facilitates and improves the accuracy of the different processes and tools. Inspired by these advances, this paper performs a literature review focused on state-of-the-art glaucoma screening, segmentation, and classification based on images of the papilla and excavation using deep learning techniques. These techniques have been shown to have high sensitivity and specificity in glaucoma screening based on papilla and excavation images. The automatic segmentation of the contours of the optic disc and the excavation then allows the identification and assessment of the glaucomatous disease’s progression. As a result, we verified whether deep learning techniques may be helpful in performing accurate and low-cost measurements related to glaucoma, which may promote patient empowerment and help medical doctors better monitor patients.
Collapse
Affiliation(s)
- José Camara
- R. Escola Politécnica, Universidade Aberta, 1250-100 Lisboa, Portugal;
- Instituto de Engenharia de Sistemas e Computadores, Tecnologia e Ciência, 3200-465 Porto, Portugal;
| | - Alexandre Neto
- Instituto de Engenharia de Sistemas e Computadores, Tecnologia e Ciência, 3200-465 Porto, Portugal;
- Escola de Ciências e Tecnologia, University of Trás-os-Montes e Alto Douro, Quinta de Prados, 5001-801 Vila Real, Portugal;
| | - Ivan Miguel Pires
- Escola de Ciências e Tecnologia, University of Trás-os-Montes e Alto Douro, Quinta de Prados, 5001-801 Vila Real, Portugal;
- Instituto de Telecomunicações, Universidade da Beira Interior, 6200-001 Covilhã, Portugal
| | - María Vanessa Villasana
- Centro Hospitalar Universitário Cova da Beira, 6200-251 Covilhã, Portugal;
- UICISA:E Research Centre, School of Health, Polytechnic Institute of Viseu, 3504-510 Viseu, Portugal
| | - Eftim Zdravevski
- Faculty of Computer Science and Engineering, University Ss Cyril and Methodius, 1000 Skopje, North Macedonia;
| | - António Cunha
- Instituto de Engenharia de Sistemas e Computadores, Tecnologia e Ciência, 3200-465 Porto, Portugal;
- Escola de Ciências e Tecnologia, University of Trás-os-Montes e Alto Douro, Quinta de Prados, 5001-801 Vila Real, Portugal;
- Correspondence: ; Tel.: +351-931-636-373
| |
Collapse
|
41
|
Sunija A, Gopi VP, Palanisamy P. Redundancy reduced depthwise separable convolution for glaucoma classification using OCT images. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2021.103192] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022]
|
42
|
Li X, Jiang Y, Rodriguez-Andina JJ, Luo H, Yin S, Kaynak O. When medical images meet generative adversarial network: recent development and research opportunities. DISCOVER ARTIFICIAL INTELLIGENCE 2021. [DOI: 10.1007/s44163-021-00006-0] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/27/2022]
Abstract
AbstractDeep learning techniques have promoted the rise of artificial intelligence (AI) and performed well in computer vision. Medical image analysis is an important application of deep learning, which is expected to greatly reduce the workload of doctors, contributing to more sustainable health systems. However, most current AI methods for medical image analysis are based on supervised learning, which requires a lot of annotated data. The number of medical images available is usually small and the acquisition of medical image annotations is an expensive process. Generative adversarial network (GAN), an unsupervised method that has become very popular in recent years, can simulate the distribution of real data and reconstruct approximate real data. GAN opens some exciting new ways for medical image generation, expanding the number of medical images available for deep learning methods. Generated data can solve the problem of insufficient data or imbalanced data categories. Adversarial training is another contribution of GAN to medical imaging that has been applied to many tasks, such as classification, segmentation, or detection. This paper investigates the research status of GAN in medical images and analyzes several GAN methods commonly applied in this area. The study addresses GAN application for both medical image synthesis and adversarial learning for other medical image tasks. The open challenges and future research directions are also discussed.
Collapse
|
43
|
Chen Y, Long J, Guo J. RF-GANs: A Method to Synthesize Retinal Fundus Images Based on Generative Adversarial Network. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2021; 2021:3812865. [PMID: 34804140 PMCID: PMC8598326 DOI: 10.1155/2021/3812865] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 06/15/2021] [Revised: 10/03/2021] [Accepted: 10/23/2021] [Indexed: 11/17/2022]
Abstract
Diabetic retinopathy (DR) is a diabetic complication affecting the eyes, which is the main cause of blindness in young and middle-aged people. In order to speed up the diagnosis of DR, a mass of deep learning methods have been used for the detection of this disease, but they failed to attain excellent results due to unbalanced training data, i.e., the lack of DR fundus images. To address the problem of data imbalance, this paper proposes a method dubbed retinal fundus images generative adversarial networks (RF-GANs), which is based on generative adversarial network, to synthesize retinal fundus images. RF-GANs is composed of two generation models, RF-GAN1 and RF-GAN2. Firstly, RF-GAN1 is employed to translate retinal fundus images from source domain (the domain of semantic segmentation datasets) to target domain (the domain of EyePACS dataset connected to Kaggle (EyePACS)). Then, we train the semantic segmentation models with the translated images, and employ the trained models to extract the structural and lesion masks (hereafter, we refer to it as Masks) of EyePACS. Finally, we employ RF-GAN2 to synthesize retinal fundus images using the Masks and DR grading labels. This paper verifies the effectiveness of the method: RF-GAN1 can narrow down the domain gap between different datasets to improve the performance of the segmentation models. RF-GAN2 can synthesize realistic retinal fundus images. Adopting the synthesized images for data augmentation, the accuracy and quadratic weighted kappa of the state-of-the-art DR grading model on the testing set of EyePACS increase by 1.53% and 1.70%, respectively.
Collapse
Affiliation(s)
- Yu Chen
- Information and Computer Engineering College, Northeast Forestry University, Harbin, China
| | - Jun Long
- Information and Computer Engineering College, Northeast Forestry University, Harbin, China
| | - Jifeng Guo
- Information and Computer Engineering College, Northeast Forestry University, Harbin, China
| |
Collapse
|
44
|
Self-supervised driven consistency training for annotation efficient histopathology image analysis. Med Image Anal 2021; 75:102256. [PMID: 34717189 DOI: 10.1016/j.media.2021.102256] [Citation(s) in RCA: 36] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/05/2021] [Revised: 08/27/2021] [Accepted: 09/27/2021] [Indexed: 01/18/2023]
Abstract
Training a neural network with a large labeled dataset is still a dominant paradigm in computational histopathology. However, obtaining such exhaustive manual annotations is often expensive, laborious, and prone to inter and intra-observer variability. While recent self-supervised and semi-supervised methods can alleviate this need by learning unsupervised feature representations, they still struggle to generalize well to downstream tasks when the number of labeled instances is small. In this work, we overcome this challenge by leveraging both task-agnostic and task-specific unlabeled data based on two novel strategies: (i) a self-supervised pretext task that harnesses the underlying multi-resolution contextual cues in histology whole-slide images to learn a powerful supervisory signal for unsupervised representation learning; (ii) a new teacher-student semi-supervised consistency paradigm that learns to effectively transfer the pretrained representations to downstream tasks based on prediction consistency with the task-specific unlabeled data. We carry out extensive validation experiments on three histopathology benchmark datasets across two classification and one regression based tasks, i.e., tumor metastasis detection, tissue type classification, and tumor cellularity quantification. Under limited-label data, the proposed method yields tangible improvements, which is close to or even outperforming other state-of-the-art self-supervised and supervised baselines. Furthermore, we empirically show that the idea of bootstrapping the self-supervised pretrained features is an effective way to improve the task-specific semi-supervised learning on standard benchmarks. Code and pretrained models are made available at: https://github.com/srinidhiPY/SSL_CR_Histo.
Collapse
|
45
|
Zhang Y, Li M, Ji Z, Fan W, Yuan S, Liu Q, Chen Q. Twin self-supervision based semi-supervised learning (TS-SSL): Retinal anomaly classification in SD-OCT images. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2021.08.051] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/29/2023]
|
46
|
Song D, Fu B, Li F, Xiong J, He J, Zhang X, Qiao Y. Deep Relation Transformer for Diagnosing Glaucoma With Optical Coherence Tomography and Visual Field Function. IEEE TRANSACTIONS ON MEDICAL IMAGING 2021; 40:2392-2402. [PMID: 33945474 DOI: 10.1109/tmi.2021.3077484] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Glaucoma is the leading reason for irreversible blindness. Early detection and timely treatment of glaucoma are essential for preventing visual field loss or even blindness. In clinical practice, Optical Coherence Tomography (OCT) and Visual Field (VF) exams are two widely-used and complementary techniques for diagnosing glaucoma. OCT provides quantitative measurements of the optic nerve head (ONH) structure, while VF test is the functional assessment of peripheral vision. In this paper, we propose a Deep Relation Transformer (DRT) to perform glaucoma diagnosis with OCT and VF information combined. A novel deep reasoning mechanism is proposed to explore implicit pairwise relations between OCT and VF information in global and regional manners. With the pairwise relations, a carefully-designed deep transformer mechanism is developed to enhance the representation with complementary information for each modal. Based on reasoning and transformer mechanisms, three successive modules are designed to extract and collect valuable information for glaucoma diagnosis, the global relation module, the guided regional relation module, and the interaction transformer module, namely. Moreover, we build a large dataset, namely ZOC-OCT&VF dataset, which includes 1395 OCT-VF pairs for developing and evaluating our DRT. We conduct extensive experiments to validate the effectiveness of the proposed method. Experimental results show that our method achieves 88.3% accuracy and outperforms the existing single-modal approaches with a large margin. The codes and dataset will be publicly available in the future.
Collapse
|
47
|
Wang Z, Lim G, Ng WY, Keane PA, Campbell JP, Tan GSW, Schmetterer L, Wong TY, Liu Y, Ting DSW. Generative adversarial networks in ophthalmology: what are these and how can they be used? Curr Opin Ophthalmol 2021; 32:459-467. [PMID: 34324454 PMCID: PMC10276657 DOI: 10.1097/icu.0000000000000794] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/18/2022]
Abstract
PURPOSE OF REVIEW The development of deep learning (DL) systems requires a large amount of data, which may be limited by costs, protection of patient information and low prevalence of some conditions. Recent developments in artificial intelligence techniques have provided an innovative alternative to this challenge via the synthesis of biomedical images within a DL framework known as generative adversarial networks (GANs). This paper aims to introduce how GANs can be deployed for image synthesis in ophthalmology and to discuss the potential applications of GANs-produced images. RECENT FINDINGS Image synthesis is the most relevant function of GANs to the medical field, and it has been widely used for generating 'new' medical images of various modalities. In ophthalmology, GANs have mainly been utilized for augmenting classification and predictive tasks, by synthesizing fundus images and optical coherence tomography images with and without pathologies such as age-related macular degeneration and diabetic retinopathy. Despite their ability to generate high-resolution images, the development of GANs remains data intensive, and there is a lack of consensus on how best to evaluate the outputs produced by GANs. SUMMARY Although the problem of artificial biomedical data generation is of great interest, image synthesis by GANs represents an innovation with yet unclear relevance for ophthalmology.
Collapse
Affiliation(s)
- Zhaoran Wang
- Duke-NUS Medical School, National University of Singapore
| | - Gilbert Lim
- Duke-NUS Medical School, National University of Singapore
- Singapore Eye Research Institute, Singapore, Singapore National Eye Centre, Singapore
| | - Wei Yan Ng
- Duke-NUS Medical School, National University of Singapore
- Singapore Eye Research Institute, Singapore, Singapore National Eye Centre, Singapore
| | - Pearse A. Keane
- Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR), Singapore
| | - J. Peter Campbell
- Department of Ophthalmology, Casey Eye Institute, Oregon Health & Science University, Portland, Oregon, USA
| | - Gavin Siew Wei Tan
- Duke-NUS Medical School, National University of Singapore
- Singapore Eye Research Institute, Singapore, Singapore National Eye Centre, Singapore
| | - Leopold Schmetterer
- Duke-NUS Medical School, National University of Singapore
- Singapore Eye Research Institute, Singapore, Singapore National Eye Centre, Singapore
- SERI-NTU Advanced Ocular Engineering (STANCE)
- School of Chemical and Biomedical Engineering, Nanyang Technological University, Singapore
- Institute of Molecular and Clinical Ophthalmology Basel, Basel, Switzerland
- Department of Clinical Pharmacology
- Center for Medical Physics and Biomedical Engineering, Medical University of Vienna, Vienna, Austria
| | - Tien Yin Wong
- Duke-NUS Medical School, National University of Singapore
- Singapore Eye Research Institute, Singapore, Singapore National Eye Centre, Singapore
| | - Yong Liu
- Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR), Singapore
| | - Daniel Shu Wei Ting
- Duke-NUS Medical School, National University of Singapore
- Singapore Eye Research Institute, Singapore, Singapore National Eye Centre, Singapore
| |
Collapse
|
48
|
Saeed AQ, Sheikh Abdullah SNH, Che-Hamzah J, Abdul Ghani AT. Accuracy of Using Generative Adversarial Networks for Glaucoma Detection During the COVID-19 Pandemic: A Systematic Review and Bibliometric Analysis. J Med Internet Res 2021; 23:e27414. [PMID: 34236992 PMCID: PMC8493455 DOI: 10.2196/27414] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2021] [Revised: 05/11/2021] [Accepted: 07/05/2021] [Indexed: 01/19/2023] Open
Abstract
Background Glaucoma leads to irreversible blindness. Globally, it is the second most common retinal disease that leads to blindness, slightly less common than cataracts. Therefore, there is a great need to avoid the silent growth of this disease using recently developed generative adversarial networks (GANs). Objective This paper aims to introduce a GAN technology for the diagnosis of eye disorders, particularly glaucoma. This paper illustrates deep adversarial learning as a potential diagnostic tool and the challenges involved in its implementation. This study describes and analyzes many of the pitfalls and problems that researchers will need to overcome to implement this kind of technology. Methods To organize this review comprehensively, articles and reviews were collected using the following keywords: (“Glaucoma,” “optic disc,” “blood vessels”) and (“receptive field,” “loss function,” “GAN,” “Generative Adversarial Network,” “Deep learning,” “CNN,” “convolutional neural network” OR encoder). The records were identified from 5 highly reputed databases: IEEE Xplore, Web of Science, Scopus, ScienceDirect, and PubMed. These libraries broadly cover the technical and medical literature. Publications within the last 5 years, specifically 2015-2020, were included because the target GAN technique was invented only in 2014 and the publishing date of the collected papers was not earlier than 2016. Duplicate records were removed, and irrelevant titles and abstracts were excluded. In addition, we excluded papers that used optical coherence tomography and visual field images, except for those with 2D images. A large-scale systematic analysis was performed, and then a summarized taxonomy was generated. Furthermore, the results of the collected articles were summarized and a visual representation of the results was presented on a T-shaped matrix diagram. This study was conducted between March 2020 and November 2020. Results We found 59 articles after conducting a comprehensive survey of the literature. Among the 59 articles, 30 present actual attempts to synthesize images and provide accurate segmentation/classification using single/multiple landmarks or share certain experiences. The other 29 articles discuss the recent advances in GANs, do practical experiments, and contain analytical studies of retinal disease. Conclusions Recent deep learning techniques, namely GANs, have shown encouraging performance in retinal disease detection. Although this methodology involves an extensive computing budget and optimization process, it saturates the greedy nature of deep learning techniques by synthesizing images and solves major medical issues. This paper contributes to this research field by offering a thorough analysis of existing works, highlighting current limitations, and suggesting alternatives to support other researchers and participants in further improving and strengthening future work. Finally, new directions for this research have been identified.
Collapse
Affiliation(s)
- Ali Q Saeed
- Faculty of Information Science & Technology (FTSM), Universiti Kebangsaan Malaysia (UKM), UKM, 43600 Bangi, Selangor, Malaysia, Selangor, MY.,Computer Center, Northern Technical University, Ninevah, IQ
| | - Siti Norul Huda Sheikh Abdullah
- Faculty of Information Science & Technology (FTSM), Universiti Kebangsaan Malaysia (UKM), UKM, 43600 Bangi, Selangor, Malaysia, Selangor, MY
| | - Jemaima Che-Hamzah
- Department of Ophthalmology, Faculty of Medicine, Universiti Kebangsaan Malaysia (UKM), Cheras, Kuala Lumpur, MY
| | - Ahmad Tarmizi Abdul Ghani
- Faculty of Information Science & Technology (FTSM), Universiti Kebangsaan Malaysia (UKM), UKM, 43600 Bangi, Selangor, Malaysia, Selangor, MY
| |
Collapse
|
49
|
Zheng C, Koh V, Bian F, Li L, Xie X, Wang Z, Yang J, Chew PTK, Zhang M. Semi-supervised generative adversarial networks for closed-angle detection on anterior segment optical coherence tomography images: an empirical study with a small training dataset. ANNALS OF TRANSLATIONAL MEDICINE 2021; 9:1073. [PMID: 34422985 PMCID: PMC8339863 DOI: 10.21037/atm-20-7436] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/12/2020] [Accepted: 03/17/2021] [Indexed: 02/05/2023]
Abstract
BACKGROUND Semi-supervised learning algorithms can leverage an unlabeled dataset when labeling is limited or expensive to obtain. In the current study, we developed and evaluated a semi-supervised generative adversarial networks (GANs) model that detects closed-angle on anterior segment optical coherence tomography (AS-OCT) images using a small labeled dataset. METHODS In this cross-sectional study, a semi-supervised GANs model was developed for automatic closed-angle detection training on a small labeled and large unsupervised training dataset collected from the Joint Shantou International Eye Center of Shantou University and the Chinese University of Hong Kong (JSIEC). The closed-angle was defined as iris-trabecular contact beyond the scleral spur in AS-OCT images. We further developed two supervised deep learning (DL) models training on the same supervised dataset and the whole dataset separately. The semi-supervised GANs model and supervised DL models' performance were compared on two independent testing datasets from JSIEC (515 images) and the Department of Ophthalmology (84 images), National University Health System, respectively. The diagnostic performance was assessed by evaluation matrices, including the accuracy, sensitivity, specificity, and area under the receiver operating characteristic curve (AUC). RESULTS For closed-angle detection using clinician grading of AS-OCT imaging as the reference standard, the semi-supervised GANs model showed comparable performance, with AUCs of 0.97 (95% CI, 0.96-0.99) and 0.98 (95% CI, 0.94-1.00), compared with the supervised DL model (using the whole dataset) [AUCs of 0.97 (95% CI, 0.96-0.99), and 0.97 (95% CI, 0.94-1.00)]. When training on the same small supervised dataset, the semi-supervised GANs achieved performance at least as well as, if not better than, the supervised DL model [AUCs of 0.90 (95% CI: 0.84-0.96), and 0.92 (95% CI, 0.86-0.97)]. CONCLUSIONS The semi-supervised GANs method achieves diagnostic performance at least as good as a supervised DL model when trained on small labeled datasets. Further development of semi-supervised learning methods could be useful within clinical and research settings. TRIAL REGISTRATION NUMBER ChiCTR2000037892.
Collapse
Affiliation(s)
- Ce Zheng
- Department of Ophthalmology, Xinhua Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, China, Shanghai, China
| | - Victor Koh
- Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore
| | - Fang Bian
- Department of Ophthalmology, Deyang People’s Hospital, Deyang, China
| | - Luo Li
- Joint Shantou International Eye Center of Shantou University and the Chinese University of Hong Kong, Shantou University Medical College, Shantou, China
| | - Xiaolin Xie
- Joint Shantou International Eye Center of Shantou University and the Chinese University of Hong Kong, Shantou University Medical College, Shantou, China
| | - Zilei Wang
- Shanghai Children’s Hospital, Shanghai, China
| | - Jianlong Yang
- Ningbo Institute of Industrial Technology, Chinese Academy of Sciences, Ningbo, China
| | - Paul Tec Kuan Chew
- Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore, Singapore
| | - Mingzhi Zhang
- Joint Shantou International Eye Center of Shantou University and the Chinese University of Hong Kong, Shantou University Medical College, Shantou, China
| |
Collapse
|
50
|
García G, Colomer A, Naranjo V. Glaucoma Detection from Raw SD-OCT Volumes: A Novel Approach Focused on Spatial Dependencies. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2021; 200:105855. [PMID: 33303289 DOI: 10.1016/j.cmpb.2020.105855] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/31/2020] [Accepted: 11/16/2020] [Indexed: 06/12/2023]
Abstract
BACKGROUND AND OBJECTIVE Glaucoma is the leading cause of blindness worldwide. Many studies based on fundus image and optical coherence tomography (OCT) imaging have been developed in the literature to help ophthalmologists through artificial-intelligence techniques. Currently, 3D spectral-domain optical coherence tomography (SD-OCT) samples have become more important since they could enclose promising information for glaucoma detection. To analyse the hidden knowledge of the 3D scans for glaucoma detection, we have proposed, for the first time, a deep-learning methodology based on leveraging the spatial dependencies of the features extracted from the B-scans. METHODS The experiments were performed on a database composed of 176 healthy and 144 glaucomatous SD-OCT volumes centred on the optic nerve head (ONH). The proposed methodology consists of two well-differentiated training stages: a slide-level feature extractor and a volume-based predictive model. The slide-level discriminator is characterised by two new, residual and attention, convolutional modules which are combined via skip-connections with other fine-tuned architectures. Regarding the second stage, we first carried out a data-volume conditioning before extracting the features from the slides of the SD-OCT volumes. Then, Long Short-Term Memory (LSTM) networks were used to combine the recurrent dependencies embedded in the latent space to provide a holistic feature vector, which was generated by the proposed sequential-weighting module (SWM). RESULTS The feature extractor reports AUC values higher than 0.93 both in the primary and external test sets. Otherwise, the proposed end-to-end system based on a combination of CNN and LSTM networks achieves an AUC of 0.8847 in the prediction stage, which outperforms other state-of-the-art approaches intended for glaucoma detection. Additionally, Class Activation Maps (CAMs) were computed to highlight the most interesting regions per B-scan when discerning between healthy and glaucomatous eyes from raw SD-OCT volumes. CONCLUSIONS The proposed model is able to extract the features from the B-scans of the volumes and combine the information of the latent space to perform a volume-level glaucoma prediction. Our model, which combines residual and attention blocks with a sequential weighting module to refine the LSTM outputs, surpass the results achieved from current state-of-the-art methods focused on 3D deep-learning architectures.
Collapse
Affiliation(s)
- Gabriel García
- Instituto de Investigación e Innovación en Bioingeniería (I3B), Universitat Politécnica de Valéncia (UPV), Valencia 46022, Spain.
| | - Adrián Colomer
- Instituto de Investigación e Innovación en Bioingeniería (I3B), Universitat Politécnica de Valéncia (UPV), Valencia 46022, Spain
| | - Valery Naranjo
- Instituto de Investigación e Innovación en Bioingeniería (I3B), Universitat Politécnica de Valéncia (UPV), Valencia 46022, Spain
| |
Collapse
|