1
|
Wang S, He X, Jian Z, Li J, Xu C, Chen Y, Liu Y, Chen H, Huang C, Hu J, Liu Z. Advances and prospects of multi-modal ophthalmic artificial intelligence based on deep learning: a review. EYE AND VISION (LONDON, ENGLAND) 2024; 11:38. [PMID: 39350240 PMCID: PMC11443922 DOI: 10.1186/s40662-024-00405-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/21/2024] [Accepted: 09/02/2024] [Indexed: 10/04/2024]
Abstract
BACKGROUND In recent years, ophthalmology has emerged as a new frontier in medical artificial intelligence (AI) with multi-modal AI in ophthalmology garnering significant attention across interdisciplinary research. This integration of various types and data models holds paramount importance as it enables the provision of detailed and precise information for diagnosing eye and vision diseases. By leveraging multi-modal ophthalmology AI techniques, clinicians can enhance the accuracy and efficiency of diagnoses, and thus reduce the risks associated with misdiagnosis and oversight while also enabling more precise management of eye and vision health. However, the widespread adoption of multi-modal ophthalmology poses significant challenges. MAIN TEXT In this review, we first summarize comprehensively the concept of modalities in the field of ophthalmology, the forms of fusion between modalities, and the progress of multi-modal ophthalmic AI technology. Finally, we discuss the challenges of current multi-modal AI technology applications in ophthalmology and future feasible research directions. CONCLUSION In the field of ophthalmic AI, evidence suggests that when utilizing multi-modal data, deep learning-based multi-modal AI technology exhibits excellent diagnostic efficacy in assisting the diagnosis of various ophthalmic diseases. Particularly, in the current era marked by the proliferation of large-scale models, multi-modal techniques represent the most promising and advantageous solution for addressing the diagnosis of various ophthalmic diseases from a comprehensive perspective. However, it must be acknowledged that there are still numerous challenges associated with the application of multi-modal techniques in ophthalmic AI before they can be effectively employed in the clinical setting.
Collapse
Affiliation(s)
- Shaopan Wang
- Institute of Artificial Intelligence, Xiamen University, Xiamen, Fujian, China
- School of Informatics, Xiamen University, Xiamen, Fujian, China
- Xiamen University Affiliated Xiamen Eye Center, Fujian Provincial Key Laboratory of Ophthalmology and Visual Science, Fujian Engineering and Research Center of Eye Regenerative Medicine, Eye Institute of Xiamen University, School of Medicine, Xiamen University, Chengyi Building, 4Th Floor, 4221-122, South Xiang'an Rd, Xiamen, 361005, Fujian, China
| | - Xin He
- Xiamen University Affiliated Xiamen Eye Center, Fujian Provincial Key Laboratory of Ophthalmology and Visual Science, Fujian Engineering and Research Center of Eye Regenerative Medicine, Eye Institute of Xiamen University, School of Medicine, Xiamen University, Chengyi Building, 4Th Floor, 4221-122, South Xiang'an Rd, Xiamen, 361005, Fujian, China
- Department of Ophthalmology, the First Affiliated Hospital of Xiamen University, Xiamen University, Xiamen, Fujian, China
| | - Zhongquan Jian
- Institute of Artificial Intelligence, Xiamen University, Xiamen, Fujian, China
- School of Informatics, Xiamen University, Xiamen, Fujian, China
| | - Jie Li
- National Engineering Research Center of Ophthalmology and Optometry, Eye Hospital, Wenzhou Medical University, Wenzhou, China
| | - Changsheng Xu
- Institute of Artificial Intelligence, Xiamen University, Xiamen, Fujian, China
- School of Informatics, Xiamen University, Xiamen, Fujian, China
- Xiamen University Affiliated Xiamen Eye Center, Fujian Provincial Key Laboratory of Ophthalmology and Visual Science, Fujian Engineering and Research Center of Eye Regenerative Medicine, Eye Institute of Xiamen University, School of Medicine, Xiamen University, Chengyi Building, 4Th Floor, 4221-122, South Xiang'an Rd, Xiamen, 361005, Fujian, China
| | - Yuguang Chen
- Institute of Artificial Intelligence, Xiamen University, Xiamen, Fujian, China
- School of Informatics, Xiamen University, Xiamen, Fujian, China
- Xiamen University Affiliated Xiamen Eye Center, Fujian Provincial Key Laboratory of Ophthalmology and Visual Science, Fujian Engineering and Research Center of Eye Regenerative Medicine, Eye Institute of Xiamen University, School of Medicine, Xiamen University, Chengyi Building, 4Th Floor, 4221-122, South Xiang'an Rd, Xiamen, 361005, Fujian, China
| | - Yuwen Liu
- Xiamen University Affiliated Xiamen Eye Center, Fujian Provincial Key Laboratory of Ophthalmology and Visual Science, Fujian Engineering and Research Center of Eye Regenerative Medicine, Eye Institute of Xiamen University, School of Medicine, Xiamen University, Chengyi Building, 4Th Floor, 4221-122, South Xiang'an Rd, Xiamen, 361005, Fujian, China
| | - Han Chen
- Department of Ophthalmology, the First Affiliated Hospital of Xiamen University, Xiamen University, Xiamen, Fujian, China
| | - Caihong Huang
- Xiamen University Affiliated Xiamen Eye Center, Fujian Provincial Key Laboratory of Ophthalmology and Visual Science, Fujian Engineering and Research Center of Eye Regenerative Medicine, Eye Institute of Xiamen University, School of Medicine, Xiamen University, Chengyi Building, 4Th Floor, 4221-122, South Xiang'an Rd, Xiamen, 361005, Fujian, China
| | - Jiaoyue Hu
- Xiamen University Affiliated Xiamen Eye Center, Fujian Provincial Key Laboratory of Ophthalmology and Visual Science, Fujian Engineering and Research Center of Eye Regenerative Medicine, Eye Institute of Xiamen University, School of Medicine, Xiamen University, Chengyi Building, 4Th Floor, 4221-122, South Xiang'an Rd, Xiamen, 361005, Fujian, China.
- Department of Ophthalmology, Xiang'an Hospital of Xiamen University, Xiamen, Fujian, China.
| | - Zuguo Liu
- Institute of Artificial Intelligence, Xiamen University, Xiamen, Fujian, China.
- Xiamen University Affiliated Xiamen Eye Center, Fujian Provincial Key Laboratory of Ophthalmology and Visual Science, Fujian Engineering and Research Center of Eye Regenerative Medicine, Eye Institute of Xiamen University, School of Medicine, Xiamen University, Chengyi Building, 4Th Floor, 4221-122, South Xiang'an Rd, Xiamen, 361005, Fujian, China.
- Department of Ophthalmology, Xiang'an Hospital of Xiamen University, Xiamen, Fujian, China.
| |
Collapse
|
2
|
Liu Z, Hu Y, Qiu Z, Niu Y, Zhou D, Li X, Shen J, Jiang H, Li H, Liu J. Cross-modal attention network for retinal disease classification based on multi-modal images. BIOMEDICAL OPTICS EXPRESS 2024; 15:3699-3714. [PMID: 38867787 PMCID: PMC11166426 DOI: 10.1364/boe.516764] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 01/03/2024] [Revised: 04/15/2024] [Accepted: 05/02/2024] [Indexed: 06/14/2024]
Abstract
Multi-modal eye disease screening improves diagnostic accuracy by providing lesion information from different sources. However, existing multi-modal automatic diagnosis methods tend to focus on the specificity of modalities and ignore the spatial correlation of images. This paper proposes a novel cross-modal retinal disease diagnosis network (CRD-Net) that digs out the relevant features from modal images aided for multiple retinal disease diagnosis. Specifically, our model introduces a cross-modal attention (CMA) module to query and adaptively pay attention to the relevant features of the lesion in the different modal images. In addition, we also propose multiple loss functions to fuse features with modality correlation and train a multi-modal retinal image classification network to achieve a more accurate diagnosis. Experimental evaluation on three publicly available datasets shows that our CRD-Net outperforms existing single-modal and multi-modal methods, demonstrating its superior performance.
Collapse
Affiliation(s)
- Zirong Liu
- School of Ophthalmology and Optometry and Eye Hospital, Wenzhou Medical University, Wenzhou 325027, China
| | - Yan Hu
- Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China
| | - Zhongxi Qiu
- Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China
| | - Yanyan Niu
- School of Ophthalmology and Optometry and Eye Hospital, Wenzhou Medical University, Wenzhou 325027, China
| | - Dan Zhou
- School of Ophthalmology and Optometry and Eye Hospital, Wenzhou Medical University, Wenzhou 325027, China
| | - Xiaoling Li
- School of Ophthalmology and Optometry and Eye Hospital, Wenzhou Medical University, Wenzhou 325027, China
| | - Junyong Shen
- Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China
| | - Hongyang Jiang
- Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China
| | - Heng Li
- Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China
| | - Jiang Liu
- School of Ophthalmology and Optometry and Eye Hospital, Wenzhou Medical University, Wenzhou 325027, China
- Research Institute of Trustworthy Autonomous Systems and Department of Computer Science and Engineering, Southern University of Science and Technology, Shenzhen 518055, China
| |
Collapse
|
3
|
Peng Z, Ma R, Zhang Y, Yan M, Lu J, Cheng Q, Liao J, Zhang Y, Wang J, Zhao Y, Zhu J, Qin B, Jiang Q, Shi F, Qian J, Chen X, Zhao C. Development and evaluation of multimodal AI for diagnosis and triage of ophthalmic diseases using ChatGPT and anterior segment images: protocol for a two-stage cross-sectional study. Front Artif Intell 2023; 6:1323924. [PMID: 38145231 PMCID: PMC10748413 DOI: 10.3389/frai.2023.1323924] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2023] [Accepted: 11/22/2023] [Indexed: 12/26/2023] Open
Abstract
Introduction Artificial intelligence (AI) technology has made rapid progress for disease diagnosis and triage. In the field of ophthalmic diseases, image-based diagnosis has achieved high accuracy but still encounters limitations due to the lack of medical history. The emergence of ChatGPT enables human-computer interaction, allowing for the development of a multimodal AI system that integrates interactive text and image information. Objective To develop a multimodal AI system using ChatGPT and anterior segment images for diagnosing and triaging ophthalmic diseases. To assess the AI system's performance through a two-stage cross-sectional study, starting with silent evaluation and followed by early clinical evaluation in outpatient clinics. Methods and analysis Our study will be conducted across three distinct centers in Shanghai, Nanjing, and Suqian. The development of the smartphone-based multimodal AI system will take place in Shanghai with the goal of achieving ≥90% sensitivity and ≥95% specificity for diagnosing and triaging ophthalmic diseases. The first stage of the cross-sectional study will explore the system's performance in Shanghai's outpatient clinics. Medical histories will be collected without patient interaction, and anterior segment images will be captured using slit lamp equipment. This stage aims for ≥85% sensitivity and ≥95% specificity with a sample size of 100 patients. The second stage will take place at three locations, with Shanghai serving as the internal validation dataset, and Nanjing and Suqian as the external validation dataset. Medical history will be collected through patient interviews, and anterior segment images will be captured via smartphone devices. An expert panel will establish reference standards and assess AI accuracy for diagnosis and triage throughout all stages. A one-vs.-rest strategy will be used for data analysis, and a post-hoc power calculation will be performed to evaluate the impact of disease types on AI performance. Discussion Our study may provide a user-friendly smartphone-based multimodal AI system for diagnosis and triage of ophthalmic diseases. This innovative system may support early detection of ocular abnormalities, facilitate establishment of a tiered healthcare system, and reduce the burdens on tertiary facilities. Trial registration The study was registered in ClinicalTrials.gov on June 25th, 2023 (NCT05930444).
Collapse
Affiliation(s)
- Zhiyu Peng
- Department of Ophthalmology, Fudan Eye & ENT Hospital, Shanghai, China
- Department of Ophthalmology, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, Zhejiang, China
- Laboratory of Myopia, Chinese Academy of Medical Sciences, Shanghai, China
- NHC Key Laboratory of Myopia, Fudan University, Shanghai, China
| | - Ruiqi Ma
- Department of Ophthalmology, Fudan Eye & ENT Hospital, Shanghai, China
- Laboratory of Myopia, Chinese Academy of Medical Sciences, Shanghai, China
- NHC Key Laboratory of Myopia, Fudan University, Shanghai, China
| | - Yihan Zhang
- Department of Ophthalmology, Fudan Eye & ENT Hospital, Shanghai, China
- Laboratory of Myopia, Chinese Academy of Medical Sciences, Shanghai, China
- NHC Key Laboratory of Myopia, Fudan University, Shanghai, China
| | - Mingxu Yan
- Department of Ophthalmology, Fudan Eye & ENT Hospital, Shanghai, China
- Laboratory of Myopia, Chinese Academy of Medical Sciences, Shanghai, China
- NHC Key Laboratory of Myopia, Fudan University, Shanghai, China
- School of Basic Medical Sciences, Fudan University, Shanghai, China
| | - Jie Lu
- Department of Ophthalmology, Fudan Eye & ENT Hospital, Shanghai, China
- Laboratory of Myopia, Chinese Academy of Medical Sciences, Shanghai, China
- NHC Key Laboratory of Myopia, Fudan University, Shanghai, China
- School of Public Health, Fudan University, Shanghai, China
| | - Qian Cheng
- Medical Image Processing, Analysis, and Visualization (MIVAP) Lab, School of Electronics and Information Engineering, Soochow University, Suzhou, China
| | - Jingjing Liao
- Medical Image Processing, Analysis, and Visualization (MIVAP) Lab, School of Electronics and Information Engineering, Soochow University, Suzhou, China
| | - Yunqiu Zhang
- School of Public Health, Fudan University, Shanghai, China
| | - Jinghan Wang
- Department of Ophthalmology, Fudan Eye & ENT Hospital, Shanghai, China
- Laboratory of Myopia, Chinese Academy of Medical Sciences, Shanghai, China
- NHC Key Laboratory of Myopia, Fudan University, Shanghai, China
| | - Yue Zhao
- The Affiliated Eye Hospital, Nanjing Medical University, Nanjing, China
| | - Jiang Zhu
- Department of Ophthalmology, Suqian First Hospital, Suqian, China
| | - Bing Qin
- Department of Ophthalmology, Suqian First Hospital, Suqian, China
| | - Qin Jiang
- The Affiliated Eye Hospital, Nanjing Medical University, Nanjing, China
- The Fourth School of Clinical Medicine, Nanjing Medical University, Nanjing, China
| | - Fei Shi
- Medical Image Processing, Analysis, and Visualization (MIVAP) Lab, School of Electronics and Information Engineering, Soochow University, Suzhou, China
| | - Jiang Qian
- Department of Ophthalmology, Fudan Eye & ENT Hospital, Shanghai, China
- Laboratory of Myopia, Chinese Academy of Medical Sciences, Shanghai, China
- NHC Key Laboratory of Myopia, Fudan University, Shanghai, China
| | - Xinjian Chen
- Medical Image Processing, Analysis, and Visualization (MIVAP) Lab, School of Electronics and Information Engineering, Soochow University, Suzhou, China
- State Key Laboratory of Radiation Medicine and Protection, Soochow University, Suzhou, China
| | - Chen Zhao
- Department of Ophthalmology, Fudan Eye & ENT Hospital, Shanghai, China
- Laboratory of Myopia, Chinese Academy of Medical Sciences, Shanghai, China
- NHC Key Laboratory of Myopia, Fudan University, Shanghai, China
| |
Collapse
|
4
|
Koseoglu ND, Grzybowski A, Liu TYA. Deep Learning Applications to Classification and Detection of Age-Related Macular Degeneration on Optical Coherence Tomography Imaging: A Review. Ophthalmol Ther 2023; 12:2347-2359. [PMID: 37493854 PMCID: PMC10441995 DOI: 10.1007/s40123-023-00775-0] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2023] [Accepted: 07/14/2023] [Indexed: 07/27/2023] Open
Abstract
Age-related macular degeneration (AMD) is one of the leading causes of blindness in the elderly, more commonly in developed countries. Optical coherence tomography (OCT) is a non-invasive imaging device widely used for the diagnosis and management of AMD. Deep learning (DL) uses multilayered artificial neural networks (NN) for feature extraction, and is the cutting-edge technique for medical image analysis for diagnostic and prognostication purposes. Application of DL models to OCT image analysis has garnered significant interest in recent years. In this review, we aimed to summarize studies focusing on DL models used in classification and detection of AMD. Additionally, we provide a brief introduction to other DL applications in AMD, such as segmentation, prediction/prognostication, and models trained on multimodal imaging.
Collapse
Affiliation(s)
- Neslihan Dilruba Koseoglu
- Wilmer Eye Institute, Johns Hopkins University, 600 N. Wolfe St., Maumenee 726, Baltimore, MD, 21287, USA
| | - Andrzej Grzybowski
- Institute for Research in Ophthalmology, Foundation for Ophthalmology Development, Poznan, Poland
| | - T Y Alvin Liu
- Wilmer Eye Institute, Johns Hopkins University, 600 N. Wolfe St., Maumenee 726, Baltimore, MD, 21287, USA.
| |
Collapse
|