1
|
Holste G, Lin M, Zhou R, Wang F, Liu L, Yan Q, Van Tassel SH, Kovacs K, Chew EY, Lu Z, Wang Z, Peng Y. Harnessing the power of longitudinal medical imaging for eye disease prognosis using Transformer-based sequence modeling. NPJ Digit Med 2024; 7:216. [PMID: 39152209 PMCID: PMC11329720 DOI: 10.1038/s41746-024-01207-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2024] [Accepted: 07/29/2024] [Indexed: 08/19/2024] Open
Abstract
Deep learning has enabled breakthroughs in automated diagnosis from medical imaging, with many successful applications in ophthalmology. However, standard medical image classification approaches only assess disease presence at the time of acquisition, neglecting the common clinical setting of longitudinal imaging. For slow, progressive eye diseases like age-related macular degeneration (AMD) and primary open-angle glaucoma (POAG), patients undergo repeated imaging over time to track disease progression and forecasting the future risk of developing a disease is critical to properly plan treatment. Our proposed Longitudinal Transformer for Survival Analysis (LTSA) enables dynamic disease prognosis from longitudinal medical imaging, modeling the time to disease from sequences of fundus photography images captured over long, irregular time periods. Using longitudinal imaging data from the Age-Related Eye Disease Study (AREDS) and Ocular Hypertension Treatment Study (OHTS), LTSA significantly outperformed a single-image baseline in 19/20 head-to-head comparisons on late AMD prognosis and 18/20 comparisons on POAG prognosis. A temporal attention analysis also suggested that, while the most recent image is typically the most influential, prior imaging still provides additional prognostic value.
Collapse
Affiliation(s)
- Gregory Holste
- Department of Population Health Sciences, Weill Cornell Medicine, New York, NY, USA
- Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX, USA
| | - Mingquan Lin
- Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX, USA
- Department of Surgery, University of Minnesota, Minneapolis, MN, USA
| | - Ruiwen Zhou
- Center for Biostatistics and Data Science, Washington University School of Medicine, St. Louis, MO, USA
| | - Fei Wang
- Department of Population Health Sciences, Weill Cornell Medicine, New York, NY, USA
| | - Lei Liu
- Center for Biostatistics and Data Science, Washington University School of Medicine, St. Louis, MO, USA
| | - Qi Yan
- Department of Obstetrics & Gynecology, Columbia University Irving Medical Center, New York, NY, USA
| | - Sarah H Van Tassel
- Israel Englander Department of Ophthalmology, Weill Cornell Medicine, New York, NY, USA
| | - Kyle Kovacs
- Israel Englander Department of Ophthalmology, Weill Cornell Medicine, New York, NY, USA
| | - Emily Y Chew
- Division of Epidemiology and Clinical Applications, National Eye Institute, National Institutes of Health (NIH), Bethesda, MD, USA
| | - Zhiyong Lu
- National Center for Biotechnology Information (NCBI), National Library of Medicine (NLM), National Institutes of Health (NIH), Bethesda, MD, USA
| | - Zhangyang Wang
- Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX, USA.
| | - Yifan Peng
- Department of Population Health Sciences, Weill Cornell Medicine, New York, NY, USA.
| |
Collapse
|
2
|
Abd El-Khalek AA, Balaha HM, Sewelam A, Ghazal M, Khalil AT, Abo-Elsoud MEA, El-Baz A. A Comprehensive Review of AI Diagnosis Strategies for Age-Related Macular Degeneration (AMD). Bioengineering (Basel) 2024; 11:711. [PMID: 39061793 PMCID: PMC11273790 DOI: 10.3390/bioengineering11070711] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/12/2024] [Revised: 07/02/2024] [Accepted: 07/09/2024] [Indexed: 07/28/2024] Open
Abstract
The rapid advancement of computational infrastructure has led to unprecedented growth in machine learning, deep learning, and computer vision, fundamentally transforming the analysis of retinal images. By utilizing a wide array of visual cues extracted from retinal fundus images, sophisticated artificial intelligence models have been developed to diagnose various retinal disorders. This paper concentrates on the detection of Age-Related Macular Degeneration (AMD), a significant retinal condition, by offering an exhaustive examination of recent machine learning and deep learning methodologies. Additionally, it discusses potential obstacles and constraints associated with implementing this technology in the field of ophthalmology. Through a systematic review, this research aims to assess the efficacy of machine learning and deep learning techniques in discerning AMD from different modalities as they have shown promise in the field of AMD and retinal disorders diagnosis. Organized around prevalent datasets and imaging techniques, the paper initially outlines assessment criteria, image preprocessing methodologies, and learning frameworks before conducting a thorough investigation of diverse approaches for AMD detection. Drawing insights from the analysis of more than 30 selected studies, the conclusion underscores current research trajectories, major challenges, and future prospects in AMD diagnosis, providing a valuable resource for both scholars and practitioners in the domain.
Collapse
Affiliation(s)
- Aya A. Abd El-Khalek
- Communications and Electronics Engineering Department, Nile Higher Institute for Engineering and Technology, Mansoura 35511, Egypt;
| | - Hossam Magdy Balaha
- Department of Bioengineering, J.B. Speed School of Engineering, University of Louisville, Louisville, KY 40292, USA
| | - Ashraf Sewelam
- Ophthalmology Department, Faculty of Medicine, Mansoura University, Mansoura 35511, Egypt;
| | - Mohammed Ghazal
- Electrical, Computer, and Biomedical Engineering Department, Abu Dhabi University, Abu Dhabi 59911, United Arab Emirates;
| | - Abeer T. Khalil
- Communications and Electronics Engineering Department, Faculty of Engineering, Mansoura University, Mansoura 35511, Egypt; (A.T.K.); (M.E.A.A.-E.)
| | - Mohy Eldin A. Abo-Elsoud
- Communications and Electronics Engineering Department, Faculty of Engineering, Mansoura University, Mansoura 35511, Egypt; (A.T.K.); (M.E.A.A.-E.)
| | - Ayman El-Baz
- Department of Bioengineering, J.B. Speed School of Engineering, University of Louisville, Louisville, KY 40292, USA
| |
Collapse
|
3
|
Mishra Z, Wang Z, Xu E, Xu S, Majid I, Sadda SR, Hu ZJ. Recurrent and Concurrent Prediction of Longitudinal Progression of Stargardt Atrophy and Geographic Atrophy. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2024.02.11.24302670. [PMID: 38405807 PMCID: PMC10888984 DOI: 10.1101/2024.02.11.24302670] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/27/2024]
Abstract
Stargardt disease and age-related macular degeneration are the leading causes of blindness in the juvenile and geriatric populations, respectively. The formation of atrophic regions of the macula is a hallmark of the end-stages of both diseases. The progression of these diseases is tracked using various imaging modalities, two of the most common being fundus autofluorescence (FAF) imaging and spectral-domain optical coherence tomography (SD-OCT). This study seeks to investigate the use of longitudinal FAF and SD-OCT imaging (month 0, month 6, month 12, and month 18) data for the predictive modelling of future atrophy in Stargardt and geographic atrophy. To achieve such an objective, we develop a set of novel deep convolutional neural networks enhanced with recurrent network units for longitudinal prediction and concurrent learning of ensemble network units (termed ReConNet) which take advantage of improved retinal layer features beyond the mean intensity features. Using FAF images, the neural network presented in this paper achieved mean (± standard deviation, SD) and median Dice coefficients of 0.895 (± 0.086) and 0.922 for Stargardt atrophy, and 0.864 (± 0.113) and 0.893 for geographic atrophy. Using SD-OCT images for Stargardt atrophy, the neural network achieved mean and median Dice coefficients of 0.882 (± 0.101) and 0.906, respectively. When predicting only the interval growth of the atrophic lesions with FAF images, mean (± SD) and median Dice coefficients of 0.557 (± 0.094) and 0.559 were achieved for Stargardt atrophy, and 0.612 (± 0.089) and 0.601 for geographic atrophy. The prediction performance in OCT images is comparably good to that using FAF which opens a new, more efficient, and practical door in the assessment of atrophy progression for clinical trials and retina clinics, beyond widely used FAF. These results are highly encouraging for a high-performance interval growth prediction when more frequent or longer-term longitudinal data are available in our clinics. This is a pressing task for our next step in ongoing research.
Collapse
Affiliation(s)
- Zubin Mishra
- Doheny Image Analysis Laboratory, Doheny Eye Institute, Pasadena, CA, 91103, USA
- Case Western Reserve University School of Medicine, Cleveland, OH, 44106, USA
| | - Ziyuan Wang
- Doheny Image Analysis Laboratory, Doheny Eye Institute, Pasadena, CA, 91103, USA
- The University of California, Los Angeles, CA, 90095, USA
| | - Emily Xu
- Doheny Image Analysis Laboratory, Doheny Eye Institute, Pasadena, CA, 91103, USA
| | - Sophia Xu
- Doheny Image Analysis Laboratory, Doheny Eye Institute, Pasadena, CA, 91103, USA
| | - Iyad Majid
- Doheny Image Analysis Laboratory, Doheny Eye Institute, Pasadena, CA, 91103, USA
| | - SriniVas R. Sadda
- Doheny Image Analysis Laboratory, Doheny Eye Institute, Pasadena, CA, 91103, USA
- The University of California, Los Angeles, CA, 90095, USA
| | - Zhihong Jewel Hu
- Doheny Image Analysis Laboratory, Doheny Eye Institute, Pasadena, CA, 91103, USA
| |
Collapse
|
4
|
Abd El-Khalek AA, Balaha HM, Alghamdi NS, Ghazal M, Khalil AT, Abo-Elsoud MEA, El-Baz A. A concentrated machine learning-based classification system for age-related macular degeneration (AMD) diagnosis using fundus images. Sci Rep 2024; 14:2434. [PMID: 38287062 PMCID: PMC10825213 DOI: 10.1038/s41598-024-52131-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/30/2023] [Accepted: 01/14/2024] [Indexed: 01/31/2024] Open
Abstract
The increase in eye disorders among older individuals has raised concerns, necessitating early detection through regular eye examinations. Age-related macular degeneration (AMD), a prevalent condition in individuals over 45, is a leading cause of vision impairment in the elderly. This paper presents a comprehensive computer-aided diagnosis (CAD) framework to categorize fundus images into geographic atrophy (GA), intermediate AMD, normal, and wet AMD categories. This is crucial for early detection and precise diagnosis of age-related macular degeneration (AMD), enabling timely intervention and personalized treatment strategies. We have developed a novel system that extracts both local and global appearance markers from fundus images. These markers are obtained from the entire retina and iso-regions aligned with the optical disc. Applying weighted majority voting on the best classifiers improves performance, resulting in an accuracy of 96.85%, sensitivity of 93.72%, specificity of 97.89%, precision of 93.86%, F1 of 93.72%, ROC of 95.85%, balanced accuracy of 95.81%, and weighted sum of 95.38%. This system not only achieves high accuracy but also provides a detailed assessment of the severity of each retinal region. This approach ensures that the final diagnosis aligns with the physician's understanding of AMD, aiding them in ongoing treatment and follow-up for AMD patients.
Collapse
Affiliation(s)
- Aya A Abd El-Khalek
- Communications and Electronics Engineering Department, Nile Higher Institute for Engineering and Technology, Mansoura, Egypt
| | - Hossam Magdy Balaha
- BioImaging Lab, Department of Bioengineering, J.B. Speed School of Engineering, University of Louisville, Louisville, KY, USA
| | - Norah Saleh Alghamdi
- Department of Computer Sciences, College of Computer and Information Sciences, Princess Nourah bint Abdulrahman University, Riyadh, Saudi Arabia
| | - Mohammed Ghazal
- Electrical, Computer, and Biomedical Engineering Depatrment, Abu Dhabi University, Abu Dhabi, UAE
| | - Abeer T Khalil
- Communications and Electronics Engineering Department, Faculty of Engineering, Mansoura University, Mansoura, Egypt
| | - Mohy Eldin A Abo-Elsoud
- Communications and Electronics Engineering Department, Faculty of Engineering, Mansoura University, Mansoura, Egypt
| | - Ayman El-Baz
- BioImaging Lab, Department of Bioengineering, J.B. Speed School of Engineering, University of Louisville, Louisville, KY, USA.
| |
Collapse
|
5
|
Khosravi P, Huck NA, Shahraki K, Hunter SC, Danza CN, Kim SY, Forbes BJ, Dai S, Levin AV, Binenbaum G, Chang PD, Suh DW. Deep Learning Approach for Differentiating Etiologies of Pediatric Retinal Hemorrhages: A Multicenter Study. Int J Mol Sci 2023; 24:15105. [PMID: 37894785 PMCID: PMC10606803 DOI: 10.3390/ijms242015105] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2023] [Revised: 09/29/2023] [Accepted: 10/10/2023] [Indexed: 10/29/2023] Open
Abstract
Retinal hemorrhages in pediatric patients can be a diagnostic challenge for ophthalmologists. These hemorrhages can occur due to various underlying etiologies, including abusive head trauma, accidental trauma, and medical conditions. Accurate identification of the etiology is crucial for appropriate management and legal considerations. In recent years, deep learning techniques have shown promise in assisting healthcare professionals in making more accurate and timely diagnosis of a variety of disorders. We explore the potential of deep learning approaches for differentiating etiologies of pediatric retinal hemorrhages. Our study, which spanned multiple centers, analyzed 898 images, resulting in a final dataset of 597 retinal hemorrhage fundus photos categorized into medical (49.9%) and trauma (50.1%) etiologies. Deep learning models, specifically those based on ResNet and transformer architectures, were applied; FastViT-SA12, a hybrid transformer model, achieved the highest accuracy (90.55%) and area under the receiver operating characteristic curve (AUC) of 90.55%, while ResNet18 secured the highest sensitivity value (96.77%) on an independent test dataset. The study highlighted areas for optimization in artificial intelligence (AI) models specifically for pediatric retinal hemorrhages. While AI proves valuable in diagnosing these hemorrhages, the expertise of medical professionals remains irreplaceable. Collaborative efforts between AI specialists and pediatric ophthalmologists are crucial to fully harness AI's potential in diagnosing etiologies of pediatric retinal hemorrhages.
Collapse
Affiliation(s)
- Pooya Khosravi
- Department of Ophthalmology, School of Medicine, University of California, Irvine, CA 92697, USA; (P.K.); (N.A.H.); (K.S.); (C.N.D.)
- Gavin Herbert Eye Institute, University of California, Irvine, CA 92697, USA
- Donald Bren School of Information and Computer Sciences, University of California, Irvine, CA 92697, USA;
| | - Nolan A. Huck
- Department of Ophthalmology, School of Medicine, University of California, Irvine, CA 92697, USA; (P.K.); (N.A.H.); (K.S.); (C.N.D.)
- Gavin Herbert Eye Institute, University of California, Irvine, CA 92697, USA
| | - Kourosh Shahraki
- Department of Ophthalmology, School of Medicine, University of California, Irvine, CA 92697, USA; (P.K.); (N.A.H.); (K.S.); (C.N.D.)
- Gavin Herbert Eye Institute, University of California, Irvine, CA 92697, USA
| | - Stephen C. Hunter
- School of Medicine, University of California, 900 University Ave, Riverside, CA 92521, USA;
| | - Clifford Neil Danza
- Department of Ophthalmology, School of Medicine, University of California, Irvine, CA 92697, USA; (P.K.); (N.A.H.); (K.S.); (C.N.D.)
- Gavin Herbert Eye Institute, University of California, Irvine, CA 92697, USA
| | - So Young Kim
- Department of Ophthalmology, College of Medicine, Soonchunhyang University, Cheonan 31151, Chungcheongnam-do, Republic of Korea;
| | - Brian J. Forbes
- Division of Ophthalmology, Children’s Hospital of Philadelphia, Philadelphia, PA 19104, USA; (B.J.F.); (G.B.)
| | - Shuan Dai
- Department of Ophthalmology, Queensland Children’s Hospital, South Brisbane, QLD 4101, Australia;
| | - Alex V. Levin
- Department of Ophthalmology, Flaum Eye Institute, Golisano Children’s Hospital, Rochester, NY 14642, USA;
| | - Gil Binenbaum
- Division of Ophthalmology, Children’s Hospital of Philadelphia, Philadelphia, PA 19104, USA; (B.J.F.); (G.B.)
| | - Peter D. Chang
- Donald Bren School of Information and Computer Sciences, University of California, Irvine, CA 92697, USA;
- Department of Radiological Sciences, School of Medicine, University of California, Irvine, CA 92697, USA
| | - Donny W. Suh
- Department of Ophthalmology, School of Medicine, University of California, Irvine, CA 92697, USA; (P.K.); (N.A.H.); (K.S.); (C.N.D.)
- Gavin Herbert Eye Institute, University of California, Irvine, CA 92697, USA
| |
Collapse
|