1
|
Christopher M, Hallaj S, Jiravarnsirikul A, Baxter SL, Zangwill LM. Novel Technologies in Artificial Intelligence and Telemedicine for Glaucoma Screening. J Glaucoma 2024; 33:S26-S32. [PMID: 38506792 DOI: 10.1097/ijg.0000000000002367] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/11/2024] [Accepted: 01/22/2024] [Indexed: 03/21/2024]
Abstract
PURPOSE To provide an overview of novel technologies in telemedicine and artificial intelligence (AI) approaches for cost-effective glaucoma screening. METHODS/RESULTS A narrative review was performed by summarizing research results, recent developments in glaucoma detection and care, and considerations related to telemedicine and AI in glaucoma screening. Telemedicine and AI approaches provide the opportunity for novel glaucoma screening programs in primary care, optometry, portable, and home-based settings. These approaches offer several advantages for glaucoma screening, including increasing access to care, lowering costs, identifying patients in need of urgent treatment, and enabling timely diagnosis and early intervention. However, challenges remain in implementing these systems, including integration into existing clinical workflows, ensuring equity for patients, and meeting ethical and regulatory requirements. Leveraging recent work towards standardized data acquisition as well as tools and techniques developed for automated diabetic retinopathy screening programs may provide a model for a cost-effective approach to glaucoma screening. CONCLUSION Leveraging novel technologies and advances in telemedicine and AI-based approaches to glaucoma detection show promise for improving our ability to detect moderate and advanced glaucoma in primary care settings and target higher individuals at high risk for having the disease.
Collapse
Affiliation(s)
- Mark Christopher
- Viterbi Family Department of Ophthalmology, Hamilton Glaucoma Center
- Viterbi Family Department of Ophthalmology, Division of Ophthalmology Informatics and Data Science, Shiley Eye Institute
| | - Shahin Hallaj
- Viterbi Family Department of Ophthalmology, Hamilton Glaucoma Center
- Viterbi Family Department of Ophthalmology, Division of Ophthalmology Informatics and Data Science, Shiley Eye Institute
| | - Anuwat Jiravarnsirikul
- Viterbi Family Department of Ophthalmology, Hamilton Glaucoma Center
- Department of Medicine, Division of Biomedical Informatics, University of California San Diego, La Jolla, CA
| | - Sally L Baxter
- Viterbi Family Department of Ophthalmology, Hamilton Glaucoma Center
- Viterbi Family Department of Ophthalmology, Division of Ophthalmology Informatics and Data Science, Shiley Eye Institute
- Department of Medicine, Division of Biomedical Informatics, University of California San Diego, La Jolla, CA
| | - Linda M Zangwill
- Viterbi Family Department of Ophthalmology, Hamilton Glaucoma Center
- Viterbi Family Department of Ophthalmology, Division of Ophthalmology Informatics and Data Science, Shiley Eye Institute
| |
Collapse
|
2
|
Salaheldin AM, Abdel Wahed M, Saleh N. A hybrid model for the detection of retinal disorders using artificial intelligence techniques. Biomed Phys Eng Express 2024; 10:055005. [PMID: 38955139 DOI: 10.1088/2057-1976/ad5db2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/17/2024] [Accepted: 07/02/2024] [Indexed: 07/04/2024]
Abstract
The prevalence of vision impairment is increasing at an alarming rate. The goal of the study was to create an automated method that uses optical coherence tomography (OCT) to classify retinal disorders into four categories: choroidal neovascularization, diabetic macular edema, drusen, and normal cases. This study proposed a new framework that combines machine learning and deep learning-based techniques. The utilized classifiers were support vector machine (SVM), K-nearest neighbor (K-NN), decision tree (DT), and ensemble model (EM). A feature extractor, the InceptionV3 convolutional neural network, was also employed. The performance of the models was evaluated against nine criteria using a dataset of 18000 OCT images. For the SVM, K-NN, DT, and EM classifiers, the analysis exhibited state-of-the-art performance, with classification accuracies of 99.43%, 99.54%, 97.98%, and 99.31%, respectively. A promising methodology has been introduced for the automatic identification and classification of retinal disorders, leading to reduced human error and saved time.
Collapse
Affiliation(s)
- Ahmed M Salaheldin
- Systems and Biomedical Engineering Department, Faculty of Engineering, Cairo University, Giza, Egypt
- Systems and Biomedical Engineering Department, Higher Institute of Engineering, EL Shorouk Academy, Cairo, Egypt
| | - Manal Abdel Wahed
- Systems and Biomedical Engineering Department, Faculty of Engineering, Cairo University, Giza, Egypt
| | - Neven Saleh
- Systems and Biomedical Engineering Department, Higher Institute of Engineering, EL Shorouk Academy, Cairo, Egypt
- Electrical Communication and Electronic Systems Engineering Department, Engineering Faculty, October University for Modern Sciences and Arts, Giza, Egypt
| |
Collapse
|
3
|
Wang X, Lu Z, Huang S, Ting Y, Ting JSZ, Chen W, Tan CH, Huang W. TransMVAN: Multi-view Aggregation Network with Transformer for Pneumonia Diagnosis. JOURNAL OF IMAGING INFORMATICS IN MEDICINE 2024:10.1007/s10278-024-01169-9. [PMID: 38977615 DOI: 10.1007/s10278-024-01169-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/17/2024] [Revised: 04/30/2024] [Accepted: 05/01/2024] [Indexed: 07/10/2024]
Abstract
Automated and accurate classification of pneumonia plays a crucial role in improving the performance of computer-aided diagnosis systems for chest X-ray images. Nevertheless, it is a challenging task due to the difficulty of learning the complex structure information of lung abnormality from chest X-ray images. In this paper, we propose a multi-view aggregation network with Transformer (TransMVAN) for pneumonia classification in chest X-ray images. Specifically, we propose to incorporate the knowledge from glance and focus views to enrich the feature representation of lung abnormality. Moreover, to capture the complex relationships among different lung regions, we propose a bi-directional multi-scale vision Transformer (biMSVT), with which the informative messages between different lung regions are propagated through two directions. In addition, we also propose a gated multi-view aggregation (GMVA) to adaptively select the feature information from glance and focus views for further performance enhancement of pneumonia diagnosis. Our proposed method achieves AUCs of 0.9645 and 0.9550 for pneumonia classification on two different chest X-ray image datasets. In addition, it achieves an AUC of 0.9761 for evaluating positive and negative polymerase chain reaction (PCR). Furthermore, our proposed method also attains an AUC of 0.9741 for classifying non-COVID-19 pneumonia, COVID-19 pneumonia, and normal cases. Experimental results demonstrate the effectiveness of our method over other methods used for comparison in pneumonia diagnosis from chest X-ray images.
Collapse
Affiliation(s)
- Xiaohong Wang
- Institute for Infocomm Research (I²R), A*STAR, 138632, Singapore, Singapore
| | - Zhongkang Lu
- Institute for Infocomm Research (I²R), A*STAR, 138632, Singapore, Singapore
| | - Su Huang
- Institute for Infocomm Research (I²R), A*STAR, 138632, Singapore, Singapore
| | - Yonghan Ting
- Department of Diagnostic Radiology, Tan Tock Seng Hospital, 308433, Singapore, Singapore
| | - Jordan Sim Zheng Ting
- Department of Diagnostic Radiology, Tan Tock Seng Hospital, 308433, Singapore, Singapore
| | - Wenxiang Chen
- Department of Diagnostic Radiology, Tan Tock Seng Hospital, 308433, Singapore, Singapore
| | - Cher Heng Tan
- Department of Diagnostic Radiology, Tan Tock Seng Hospital, 308433, Singapore, Singapore
- Lee Kong Chian School of Medicine, Nanyang Technological University, 308232, Singapore, Singapore
| | - Weimin Huang
- Institute for Infocomm Research (I²R), A*STAR, 138632, Singapore, Singapore.
| |
Collapse
|
4
|
Bowd C, Belghith A, Rezapour J, Jonas JB, Hyman L, Weinreb RN, Zangwill LM. Wide-Field Optical Coherence Tomography Imaging Improves Rate of Change Detection in Progressing Glaucomatous Eyes Compared With Standard-Field Imaging. Invest Ophthalmol Vis Sci 2024; 65:18. [PMID: 38980269 PMCID: PMC11235143 DOI: 10.1167/iovs.65.8.18] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2023] [Accepted: 06/11/2024] [Indexed: 07/10/2024] Open
Abstract
Purpose To compare rates of retinal nerve fiber layer change over time in healthy, eyes with nonprogressing glaucoma and eyes with progressing glaucoma using single wide-field (SWF) and optic nerve head (ONH) cube scan optical coherence tomography (OCT) images. Methods Forty-five eyes of 25 healthy individuals and 263 eyes of 161 glaucoma patients from the Diagnostic Innovations in Glaucoma Study were included. All eyes underwent 24-2 visual field testing and OCT (Spectralis SD-OCT) ONH and macular imaging. SWF images (up to 43° × 28°) were created by stitching together ONH cube scans centered on the optic disc and macular cube scans centered on the fovea. Visual field progression was defined as guided progression analysis likely progression and/or a significant (P < 0.01) mean deviation slope of less than -1.0 dB/year. Mixed effects models were used to compare rates of change. Highly myopic eyes were included. Results Thirty glaucomatous eyes were classified as progressing. In eyes with glaucoma, mean global rate of change was -1.22 µm/year (P < 0.001) using SWF images and -0.83 µm/year (P = 0.003) using ONH cube scans. Rate of change was significantly greater in eyes with progressing glaucoma compared with eyes with nonprogressing glaucoma (-1.51 µm/year vs. -1.24 µm/year; P = 0.002) using SWF images and was similar using ONH cube scans (P = 0.27). Conclusions In this cohort that includes eyes with and without high axial myopia, the mean rate of retinal nerve fiber layer thinning measured using SWF images was faster in eyes with progressing glaucoma than in eyes with nonprogressing glaucoma. Wide-field OCT images including the ONH and macula can be effective for monitoring glaucomatous progression in patients with and without high myopia.
Collapse
Affiliation(s)
- Christopher Bowd
- Hamilton Glaucoma Center, Shiley Eye Institute, The Viterbi Family Department of Ophthalmology, University of California San Diego, La Jolla, California, United States
| | - Akram Belghith
- Hamilton Glaucoma Center, Shiley Eye Institute, The Viterbi Family Department of Ophthalmology, University of California San Diego, La Jolla, California, United States
| | - Jasmin Rezapour
- Department of Ophthalmology, University Medical Center Mainz, Mainz, Germany
| | - Jost B. Jonas
- Institute of Molecular and Clinical Ophthalmology IOB Basel, Switzerland
| | - Leslie Hyman
- Wills Eye Hospital, Thomas Jefferson University, Philadelphia, Pennsylvania, United States
| | - Robert N. Weinreb
- Hamilton Glaucoma Center, Shiley Eye Institute, The Viterbi Family Department of Ophthalmology, University of California San Diego, La Jolla, California, United States
| | - Linda M. Zangwill
- Hamilton Glaucoma Center, Shiley Eye Institute, The Viterbi Family Department of Ophthalmology, University of California San Diego, La Jolla, California, United States
| |
Collapse
|
5
|
Liu Y, Zhang Y, Wang Y, Hou F, Yuan J, Tian J, Zhang Y, Shi Z, Fan J, He Z. A Survey of Visual Transformers. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024; 35:7478-7498. [PMID: 37015131 DOI: 10.1109/tnnls.2022.3227717] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
Transformer, an attention-based encoder-decoder model, has already revolutionized the field of natural language processing (NLP). Inspired by such significant achievements, some pioneering works have recently been done on employing Transformer-liked architectures in the computer vision (CV) field, which have demonstrated their effectiveness on three fundamental CV tasks (classification, detection, and segmentation) as well as multiple sensory data stream (images, point clouds, and vision-language data). Because of their competitive modeling capabilities, the visual Transformers have achieved impressive performance improvements over multiple benchmarks as compared with modern convolution neural networks (CNNs). In this survey, we have reviewed over 100 of different visual Transformers comprehensively according to three fundamental CV tasks and different data stream types, where taxonomy is proposed to organize the representative methods according to their motivations, structures, and application scenarios. Because of their differences on training settings and dedicated vision tasks, we have also evaluated and compared all these existing visual Transformers under different configurations. Furthermore, we have revealed a series of essential but unexploited aspects that may empower such visual Transformers to stand out from numerous architectures, e.g., slack high-level semantic embeddings to bridge the gap between the visual Transformers and the sequential ones. Finally, two promising research directions are suggested for future investment. We will continue to update the latest articles and their released source codes at https://github.com/liuyang-ict/awesome-visual-transformers.
Collapse
|
6
|
Li-Han LY, Eizenman M, Shi RB, Buys YM, Trope GE, Wong W. Using Fused Data from Perimetry and Optical Coherence Tomography to Improve the Detection of Visual Field Progression in Glaucoma. Bioengineering (Basel) 2024; 11:250. [PMID: 38534524 DOI: 10.3390/bioengineering11030250] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2024] [Revised: 02/16/2024] [Accepted: 02/27/2024] [Indexed: 03/28/2024] Open
Abstract
Perimetry and optical coherence tomography (OCT) are both used to monitor glaucoma progression. However, combining these modalities can be a challenge due to differences in data types. To overcome this, we have developed an autoencoder data fusion (AEDF) model to learn compact encoding (AE-fused data) from both perimetry and OCT. The AEDF model, optimized specifically for visual field (VF) progression detection, incorporates an encoding loss to ensure the interpretation of the AE-fused data is similar to VF data while capturing key features from OCT measurements. For model training and evaluation, our study included 2504 longitudinal VF and OCT tests from 140 glaucoma patients. VF progression was determined from linear regression slopes of longitudinal mean deviations. Progression detection with AE-fused data was compared to VF-only data (standard clinical method) as well as data from a Bayesian linear regression (BLR) model. In the initial 2-year follow-up period, AE-fused data achieved a detection F1 score of 0.60 (95% CI: 0.57 to 0.62), significantly outperforming (p < 0.001) the clinical method (0.45, 95% CI: 0.43 to 0.47) and the BLR model (0.48, 95% CI: 0.45 to 0.51). The capacity of the AEDF model to generate clinically interpretable fused data that improves VF progression detection makes it a promising data integration tool in glaucoma management.
Collapse
Affiliation(s)
- Leo Yan Li-Han
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering, University of Toronto, Toronto, ON M5S 3G4, Canada
| | - Moshe Eizenman
- Department of Ophthalmology & Vision Sciences, University of Toronto, Toronto, ON M5T 3A9, Canada
- Temerty Faculty of Medicine, University of Toronto, Toronto, ON M5S 1A8, Canada
| | - Runjie Bill Shi
- Temerty Faculty of Medicine, University of Toronto, Toronto, ON M5S 1A8, Canada
- Institute of Biomedical Engineering, University of Toronto, Toronto, ON M5S 3E2, Canada
| | - Yvonne M Buys
- Department of Ophthalmology & Vision Sciences, University of Toronto, Toronto, ON M5T 3A9, Canada
| | - Graham E Trope
- Department of Ophthalmology & Vision Sciences, University of Toronto, Toronto, ON M5T 3A9, Canada
| | - Willy Wong
- The Edward S. Rogers Sr. Department of Electrical & Computer Engineering, University of Toronto, Toronto, ON M5S 3G4, Canada
- Institute of Biomedical Engineering, University of Toronto, Toronto, ON M5S 3E2, Canada
| |
Collapse
|
7
|
Liu K, Zhang J. Glaucoma detection model by exploiting multi-region and multi-scan-pattern OCT images with dynamical region score. BIOMEDICAL OPTICS EXPRESS 2024; 15:1370-1392. [PMID: 38495692 PMCID: PMC10942704 DOI: 10.1364/boe.512138] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/13/2023] [Revised: 12/19/2023] [Accepted: 01/12/2024] [Indexed: 03/19/2024]
Abstract
Currently, deep learning-based methods have achieved success in glaucoma detection. However, most models focus on OCT images captured by a single scan pattern within a given region, holding the high risk of the omission of valuable features in the remaining regions or scan patterns. Therefore, we proposed a multi-region and multi-scan-pattern fusion model to address this issue. Our proposed model exploits comprehensive OCT images from three fundus anatomical regions (macular, middle, and optic nerve head regions) being captured by four scan patterns (radial, volume, single-line, and circular scan patterns). Moreover, to enhance the efficacy of integrating features across various scan patterns within a region and multiple regional features, we employed an attention multi-scan fusion module and an attention multi-region fusion module that auto-assign contribution to distinct scan-pattern features and region features adapting to characters of different samples, respectively. To alleviate the absence of available datasets, we have collected a specific dataset (MRMSG-OCT) comprising OCT images captured by four scan patterns from three regions. The experimental results and visualized feature maps both demonstrate that our proposed model achieves superior performance against the single scan-pattern models and single region-based models. Moreover, compared with the average fusion strategy, our proposed fusion modules yield superior performance, particularly reversing the performance degradation observed in some models relying on fixed weights, validating the efficacy of the proposed dynamic region scores adapted to different samples. Moreover, the derived region contribution scores enhance the interpretability of the model and offer an overview of the model's decision-making process, assisting ophthalmologists in prioritizing regions with heightened scores and increasing efficiency in clinical practice.
Collapse
Affiliation(s)
- Kai Liu
- School of Biological Science and Medical Engineering, Beihang University, Beijing, 100083, China
- Beijing Advanced Innovation Centre for Biomedical Engineering, Beihang University, Beijing, 100083, China
- Department of Computer Science, City University of Hong Kong, Hong Kong SAR, 98121, China
| | - Jicong Zhang
- School of Biological Science and Medical Engineering, Beihang University, Beijing, 100083, China
- Beijing Advanced Innovation Centre for Biomedical Engineering, Beihang University, Beijing, 100083, China
- Hefei Innovation Research Institute, Beihang University, Hefei, 230012, China
| |
Collapse
|
8
|
Hwang EE, Chen D, Han Y, Jia L, Shan J. Multi-Dataset Comparison of Vision Transformers and Convolutional Neural Networks for Detecting Glaucomatous Optic Neuropathy from Fundus Photographs. Bioengineering (Basel) 2023; 10:1266. [PMID: 38002390 PMCID: PMC10669064 DOI: 10.3390/bioengineering10111266] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/17/2023] [Revised: 10/26/2023] [Accepted: 10/27/2023] [Indexed: 11/26/2023] Open
Abstract
Glaucomatous optic neuropathy (GON) can be diagnosed and monitored using fundus photography, a widely available and low-cost approach already adopted for automated screening of ophthalmic diseases such as diabetic retinopathy. Despite this, the lack of validated early screening approaches remains a major obstacle in the prevention of glaucoma-related blindness. Deep learning models have gained significant interest as potential solutions, as these models offer objective and high-throughput methods for processing image-based medical data. While convolutional neural networks (CNN) have been widely utilized for these purposes, more recent advances in the application of Transformer architectures have led to new models, including Vision Transformer (ViT,) that have shown promise in many domains of image analysis. However, previous comparisons of these two architectures have not sufficiently compared models side-by-side with more than a single dataset, making it unclear which model is more generalizable or performs better in different clinical contexts. Our purpose is to investigate comparable ViT and CNN models tasked with GON detection from fundus photos and highlight their respective strengths and weaknesses. We train CNN and ViT models on six unrelated, publicly available databases and compare their performance using well-established statistics including AUC, sensitivity, and specificity. Our results indicate that ViT models often show superior performance when compared with a similarly trained CNN model, particularly when non-glaucomatous images are over-represented in a given dataset. We discuss the clinical implications of these findings and suggest that ViT can further the development of accurate and scalable GON detection for this leading cause of irreversible blindness worldwide.
Collapse
Affiliation(s)
- Elizabeth E. Hwang
- Department of Ophthalmology, University of California, San Francisco, San Francisco, CA 94143, USA
- Medical Scientist Training Program, University of California, San Francisco, San Francisco, CA 94143, USA
| | - Dake Chen
- Department of Ophthalmology, University of California, San Francisco, San Francisco, CA 94143, USA
| | - Ying Han
- Department of Ophthalmology, University of California, San Francisco, San Francisco, CA 94143, USA
| | - Lin Jia
- Digillect LLC, San Francisco, CA 94158, USA
| | - Jing Shan
- Department of Ophthalmology, University of California, San Francisco, San Francisco, CA 94143, USA
| |
Collapse
|
9
|
Bowd C, Belghith A, Rezapour J, Christopher M, Jonas JB, Hyman L, Fazio MA, Weinreb RN, Zangwill LM. Multimodal Deep Learning Classifier for Primary Open Angle Glaucoma Diagnosis Using Wide-Field Optic Nerve Head Cube Scans in Eyes With and Without High Myopia. J Glaucoma 2023; 32:841-847. [PMID: 37523623 DOI: 10.1097/ijg.0000000000002267] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/02/2023] [Accepted: 06/18/2023] [Indexed: 08/02/2023]
Abstract
PRCIS An optical coherence tomography (OCT)-based multimodal deep learning (DL) classification model, including texture information, is introduced that outperforms single-modal models and multimodal models without texture information for glaucoma diagnosis in eyes with and without high myopia. BACKGROUND/AIMS To evaluate the diagnostic accuracy of a multimodal DL classifier using wide OCT optic nerve head cube scans in eyes with and without axial high myopia. MATERIALS AND METHODS Three hundred seventy-one primary open angle glaucoma (POAG) eyes and 86 healthy eyes, all without axial high myopia [axial length (AL) ≤ 26 mm] and 92 POAG eyes and 44 healthy eyes, all with axial high myopia (AL > 26 mm) were included. The multimodal DL classifier combined features of 3 individual VGG-16 models: (1) texture-based en face image, (2) retinal nerve fiber layer (RNFL) thickness map image, and (3) confocal scanning laser ophthalmoscope (cSLO) image. Age, AL, and disc area adjusted area under the receiver operating curves were used to compare model accuracy. RESULTS Adjusted area under the receiver operating curve for the multimodal DL model was 0.91 (95% CI = 0.87, 0.95). This value was significantly higher than the values of individual models [0.83 (0.79, 0.86) for texture-based en face image; 0.84 (0.81, 0.87) for RNFL thickness map; and 0.68 (0.61, 0.74) for cSLO image; all P ≤ 0.05]. Using only highly myopic eyes, the multimodal DL model showed significantly higher diagnostic accuracy [0.89 (0.86, 0.92)] compared with texture en face image [0.83 (0.78, 0.85)], RNFL [0.85 (0.81, 0.86)] and cSLO image models [0.69 (0.63, 0.76)] (all P ≤ 0.05). CONCLUSIONS Combining OCT-based RNFL thickness maps with texture-based en face images showed a better ability to discriminate between healthy and POAG than thickness maps alone, particularly in high axial myopic eyes.
Collapse
Affiliation(s)
- Christopher Bowd
- Viterbi Family Department of Ophthalmology, Shiley Eye Institute, Hamilton Glaucoma Center
| | - Akram Belghith
- Viterbi Family Department of Ophthalmology, Shiley Eye Institute, Hamilton Glaucoma Center
| | - Jasmin Rezapour
- Viterbi Family Department of Ophthalmology, Shiley Eye Institute, Hamilton Glaucoma Center
- Department of Ophthalmology, University Medical Center of the Johannes Gutenberg University Mainz
| | - Mark Christopher
- Viterbi Family Department of Ophthalmology, Shiley Eye Institute, Hamilton Glaucoma Center
| | - Jost B Jonas
- Department of Ophthalmology, Heidelberg University, Mannheim, Germany
| | - Leslie Hyman
- Vickie and Jack Farber Vision Research Center, Wills Eye Hospital, Thomas Jefferson University, Philadelphia, PA
| | - Massimo A Fazio
- Department of Ophthalmology and Visual Sciences, The University of Alabama at Birmingham, Birmingham, AL
| | - Robert N Weinreb
- Viterbi Family Department of Ophthalmology, Shiley Eye Institute, Hamilton Glaucoma Center
| | - Linda M Zangwill
- Viterbi Family Department of Ophthalmology, Shiley Eye Institute, Hamilton Glaucoma Center
| |
Collapse
|
10
|
Shi M, Lokhande A, Fazli MS, Sharma V, Tian Y, Luo Y, Pasquale LR, Elze T, Boland MV, Zebardast N, Friedman DS, Shen LQ, Wang M. Artifact-Tolerant Clustering-Guided Contrastive Embedding Learning for Ophthalmic Images in Glaucoma. IEEE J Biomed Health Inform 2023; 27:4329-4340. [PMID: 37347633 PMCID: PMC10560582 DOI: 10.1109/jbhi.2023.3288830] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 06/24/2023]
Abstract
Ophthalmic images, along with their derivatives like retinal nerve fiber layer (RNFL) thickness maps, play a crucial role in detecting and monitoring eye diseases such as glaucoma. For computer-aided diagnosis of eye diseases, the key technique is to automatically extract meaningful features from ophthalmic images that can reveal the biomarkers (e.g., RNFL thinning patterns) associated with functional vision loss. However, representation learning from ophthalmic images that links structural retinal damage with human vision loss is non-trivial mostly due to large anatomical variations between patients. This challenge is further amplified by the presence of image artifacts, commonly resulting from image acquisition and automated segmentation issues. In this paper, we present an artifact-tolerant unsupervised learning framework called EyeLearn for learning ophthalmic image representations in glaucoma cases. EyeLearn includes an artifact correction module to learn representations that optimally predict artifact-free images. In addition, EyeLearn adopts a clustering-guided contrastive learning strategy to explicitly capture the affinities within and between images. During training, images are dynamically organized into clusters to form contrastive samples, which encourage learning similar or dissimilar representations for images in the same or different clusters, respectively. To evaluate EyeLearn, we use the learned representations for visual field prediction and glaucoma detection with a real-world dataset of glaucoma patient ophthalmic images. Extensive experiments and comparisons with state-of-the-art methods confirm the effectiveness of EyeLearn in learning optimal feature representations from ophthalmic images.
Collapse
|
11
|
Khader F, Kather JN, Müller-Franzes G, Wang T, Han T, Tayebi Arasteh S, Hamesch K, Bressem K, Haarburger C, Stegmaier J, Kuhl C, Nebelung S, Truhn D. Medical transformer for multimodal survival prediction in intensive care: integration of imaging and non-imaging data. Sci Rep 2023; 13:10666. [PMID: 37393383 PMCID: PMC10314902 DOI: 10.1038/s41598-023-37835-1] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2023] [Accepted: 06/28/2023] [Indexed: 07/03/2023] Open
Abstract
When clinicians assess the prognosis of patients in intensive care, they take imaging and non-imaging data into account. In contrast, many traditional machine learning models rely on only one of these modalities, limiting their potential in medical applications. This work proposes and evaluates a transformer-based neural network as a novel AI architecture that integrates multimodal patient data, i.e., imaging data (chest radiographs) and non-imaging data (clinical data). We evaluate the performance of our model in a retrospective study with 6,125 patients in intensive care. We show that the combined model (area under the receiver operating characteristic curve [AUROC] of 0.863) is superior to the radiographs-only model (AUROC = 0.811, p < 0.001) and the clinical data-only model (AUROC = 0.785, p < 0.001) when tasked with predicting in-hospital survival per patient. Furthermore, we demonstrate that our proposed model is robust in cases where not all (clinical) data points are available.
Collapse
Affiliation(s)
- Firas Khader
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany.
| | - Jakob Nikolas Kather
- Department of Medicine III, University Hospital Aachen, Aachen, Germany
- Else Kroener Fresenius Center for Digital Health, Medical Faculty Carl Gustav Carus, Technical University Dresden, Dresden, Germany
- Division of Pathology and Data Analytics, Leeds Institute of Medical Research at St James's, University of Leeds, Leeds, UK
- Medical Oncology, National Center for Tumor Diseases (NCT), University Hospital Heidelberg, Heidelberg, Germany
| | - Gustav Müller-Franzes
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
| | - Tianci Wang
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
| | - Tianyu Han
- Physics of Molecular Imaging Systems, Experimental Molecular Imaging, RWTH Aachen University, Aachen, Germany
| | - Soroosh Tayebi Arasteh
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
| | - Karim Hamesch
- Department of Medicine III, University Hospital Aachen, Aachen, Germany
| | - Keno Bressem
- Department of Radiology, Charité-University Medicine Berlin, Berlin, Germany
| | | | - Johannes Stegmaier
- Institute of Imaging and Computer Vision, RWTH Aachen University, Aachen, Germany
| | - Christiane Kuhl
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
| | - Sven Nebelung
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
| | - Daniel Truhn
- Department of Diagnostic and Interventional Radiology, University Hospital Aachen, Aachen, Germany
| |
Collapse
|
12
|
He J, Wang J, Han Z, Ma J, Wang C, Qi M. An interpretable transformer network for the retinal disease classification using optical coherence tomography. Sci Rep 2023; 13:3637. [PMID: 36869160 PMCID: PMC9984386 DOI: 10.1038/s41598-023-30853-z] [Citation(s) in RCA: 12] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2022] [Accepted: 03/02/2023] [Indexed: 03/05/2023] Open
Abstract
Retinal illnesses such as age-related macular degeneration and diabetic macular edema will lead to irreversible blindness. With optical coherence tomography (OCT), doctors are able to see cross-sections of the retinal layers and provide patients with a diagnosis. Manual reading of OCT images is time-consuming, labor-intensive and even error-prone. Computer-aided diagnosis algorithms improve efficiency by automatically analyzing and diagnosing retinal OCT images. However, the accuracy and interpretability of these algorithms can be further improved through effective feature extraction, loss optimization and visualization analysis. In this paper, we propose an interpretable Swin-Poly Transformer network for performing automatically retinal OCT image classification. By shifting the window partition, the Swin-Poly Transformer constructs connections between neighboring non-overlapping windows in the previous layer and thus has the flexibility to model multi-scale features. Besides, the Swin-Poly Transformer modifies the importance of polynomial bases to refine cross entropy for better retinal OCT image classification. In addition, the proposed method also provides confidence score maps, assisting medical practitioners to understand the models' decision-making process. Experiments in OCT2017 and OCT-C8 reveal that the proposed method outperforms both the convolutional neural network approach and ViT, with an accuracy of 99.80% and an AUC of 99.99%.
Collapse
Affiliation(s)
- Jingzhen He
- Department of Radiology, Qilu Hospital of Shandong University, Jinan, 250012, China.
| | - Junxia Wang
- School of Information Science and Engineering, Shandong Normal University, Jinan, 250358, China
| | - Zeyu Han
- School of Mathematics and Statistics, Shandong University, Weihai, 264209, China
| | - Jun Ma
- School of Cyber Science and Engineering, Southeast University, Nanjing, 211189, China
| | - Chongjing Wang
- China Academy of Information and Communications Technology, Beijing, 100191, China
| | - Meng Qi
- School of Information Science and Engineering, Shandong Normal University, Jinan, 250358, China.
| |
Collapse
|
13
|
Zhang Y, Luo L, Dou Q, Heng PA. Triplet attention and dual-pool contrastive learning for clinic-driven multi-label medical image classification. Med Image Anal 2023; 86:102772. [PMID: 36822050 DOI: 10.1016/j.media.2023.102772] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2022] [Revised: 11/21/2022] [Accepted: 02/10/2023] [Indexed: 02/18/2023]
Abstract
Multi-label classification (MLC) can attach multiple labels on single image, and has achieved promising results on medical images. But existing MLC methods still face challenging clinical realities in practical use, such as: (1) medical risks arising from misclassification, (2) sample imbalance problem among different diseases, (3) inability to classify the diseases that are not pre-defined (unseen diseases). Here, we design a hybrid label to improve the flexibility of MLC methods and alleviate the sample imbalance problem. Specifically, in the labeled training set, we remain independent labels for high-frequency diseases with enough samples and use a hybrid label to merge low-frequency diseases with fewer samples. The hybrid label can also be used to put unseen diseases in practical use. In this paper, we propose Triplet Attention and Dual-pool Contrastive Learning (TA-DCL) for multi-label medical image classification based on the aforementioned label representation. TA-DCL architecture is a triplet attention network (TAN), which combines category-attention, self-attention and cross-attention together to learn high-quality label embeddings for all disease labels by mining effective information from medical images. DCL includes dual-pool contrastive training (DCT) and dual-pool contrastive inference (DCI). DCT optimizes the clustering centers of label embeddings belonging to different disease labels to improve the discrimination of label embeddings. DCI relieves the error classification of sick cases for reducing the clinical risk and improving the ability to detect unseen diseases by contrast of differences. TA-DCL is validated on two public medical image datasets, ODIR and NIH-ChestXray14, showing superior performance than other state-of-the-art MLC methods. Code is available at https://github.com/ZhangYH0502/TA-DCL.
Collapse
Affiliation(s)
- Yuhan Zhang
- Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China; Institute of Medical Intelligence and XR, The Chinese University of Hong Kong, Hong Kong, China; Shenzhen Research Institute, The Chinese University of Hong Kong, Hong Kong, China.
| | - Luyang Luo
- Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China.
| | - Qi Dou
- Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China; Institute of Medical Intelligence and XR, The Chinese University of Hong Kong, Hong Kong, China.
| | - Pheng-Ann Heng
- Department of Computer Science and Engineering, The Chinese University of Hong Kong, Hong Kong, China; Institute of Medical Intelligence and XR, The Chinese University of Hong Kong, Hong Kong, China.
| |
Collapse
|
14
|
Song D, Li F, Li C, Xiong J, He J, Zhang X, Qiao Y. Asynchronous feature regularization and cross-modal distillation for OCT based glaucoma diagnosis. Comput Biol Med 2022; 151:106283. [PMID: 36442272 DOI: 10.1016/j.compbiomed.2022.106283] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2022] [Revised: 10/03/2022] [Accepted: 10/30/2022] [Indexed: 11/11/2022]
Abstract
Glaucoma has become a major cause of vision loss. Early-stage diagnosis of glaucoma is critical for treatment planning to avoid irreversible vision damage. Meanwhile, interpreting the rapidly accumulated medical data from ophthalmic exams is cumbersome and resource-intensive. Therefore, automated methods are highly desired to assist ophthalmologists in achieving fast and accurate glaucoma diagnosis. Deep learning has achieved great successes in diagnosing glaucoma by analyzing data from different kinds of tests, such as peripapillary optical coherence tomography (OCT) and visual field (VF) testing. Nevertheless, applying these developed models to clinical practice is still challenging because of various limiting factors. OCT models present worse glaucoma diagnosis performances compared to those achieved by OCT&VF based models, whereas VF is time-consuming and highly variable, which can restrict the wide employment of OCT&VF models. To this end, we develop a novel deep learning framework that leverages the OCT&VF model to enhance the performance of the OCT model. To transfer the complementary knowledge from the structural and functional assessments to the OCT model, a cross-modal knowledge transfer method is designed by integrating a designed distillation loss and a proposed asynchronous feature regularization (AFR) module. We demonstrate the effectiveness of the proposed method for glaucoma diagnosis by utilizing a public OCT&VF dataset and evaluating it on an external OCT dataset. Our final model with only OCT inputs achieves the accuracy of 87.4% (3.1% absolute improvement) and AUC of 92.3%, which are on par with the OCT&VF joint model. Moreover, results on the external dataset sufficiently indicate the effectiveness and generalization capability of our model.
Collapse
Affiliation(s)
- Diping Song
- Guangdong Provincial Key Laboratory of Computer Vision and Virtual Reality Technology, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, 518055, China; University of Chinese Academy of Sciences, Beijing, 100049, China; Shanghai Artificial Intelligence Laboratory, Shanghai, 200232, China.
| | - Fei Li
- State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-Sen University, Guangzhou, 510060, China.
| | - Cheng Li
- Paul C. Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, 518055, China.
| | - Jian Xiong
- Ophthalmic Center, The Second Affiliated Hospital of Nanchang University, Nanchang, 330000, China.
| | - Junjun He
- Shanghai Artificial Intelligence Laboratory, Shanghai, 200232, China.
| | - Xiulan Zhang
- State Key Laboratory of Ophthalmology, Zhongshan Ophthalmic Center, Sun Yat-Sen University, Guangzhou, 510060, China.
| | - Yu Qiao
- Guangdong Provincial Key Laboratory of Computer Vision and Virtual Reality Technology, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, 518055, China; Shanghai Artificial Intelligence Laboratory, Shanghai, 200232, China.
| |
Collapse
|
15
|
Fan R, Alipour K, Bowd C, Christopher M, Brye N, Proudfoot JA, Goldbaum MH, Belghith A, Girkin CA, Fazio MA, Liebmann JM, Weinreb RN, Pazzani M, Kriegman D, Zangwill LM. Detecting Glaucoma from Fundus Photographs Using Deep Learning without Convolutions: Transformer for Improved Generalization. OPHTHALMOLOGY SCIENCE 2022; 3:100233. [PMID: 36545260 PMCID: PMC9762193 DOI: 10.1016/j.xops.2022.100233] [Citation(s) in RCA: 21] [Impact Index Per Article: 10.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 06/17/2022] [Revised: 10/04/2022] [Accepted: 10/12/2022] [Indexed: 12/14/2022]
Abstract
Purpose To compare the diagnostic accuracy and explainability of a Vision Transformer deep learning technique, Data-efficient image Transformer (DeiT), and ResNet-50, trained on fundus photographs from the Ocular Hypertension Treatment Study (OHTS) to detect primary open-angle glaucoma (POAG) and identify the salient areas of the photographs most important for each model's decision-making process. Design Evaluation of a diagnostic technology. Subjects Participants and Controls Overall 66 715 photographs from 1636 OHTS participants and an additional 5 external datasets of 16 137 photographs of healthy and glaucoma eyes. Methods Data-efficient image Transformer models were trained to detect 5 ground-truth OHTS POAG classifications: OHTS end point committee POAG determinations because of disc changes (model 1), visual field (VF) changes (model 2), or either disc or VF changes (model 3) and Reading Center determinations based on disc (model 4) and VFs (model 5). The best-performing DeiT models were compared with ResNet-50 models on OHTS and 5 external datasets. Main Outcome Measures Diagnostic performance was compared using areas under the receiver operating characteristic curve (AUROC) and sensitivities at fixed specificities. The explainability of the DeiT and ResNet-50 models was compared by evaluating the attention maps derived directly from DeiT to 3 gradient-weighted class activation map strategies. Results Compared with our best-performing ResNet-50 models, the DeiT models demonstrated similar performance on the OHTS test sets for all 5 ground-truth POAG labels; AUROC ranged from 0.82 (model 5) to 0.91 (model 1). Data-efficient image Transformer AUROC was consistently higher than ResNet-50 on the 5 external datasets. For example, AUROC for the main OHTS end point (model 3) was between 0.08 and 0.20 higher in the DeiT than ResNet-50 models. The saliency maps from the DeiT highlight localized areas of the neuroretinal rim, suggesting important rim features for classification. The same maps in the ResNet-50 models show a more diffuse, generalized distribution around the optic disc. Conclusions Vision Transformers have the potential to improve generalizability and explainability in deep learning models, detecting eye disease and possibly other medical conditions that rely on imaging for clinical diagnosis and management.
Collapse
Key Words
- AI, artificial intelligence
- AUROC, areas under the receiver operating characteristic curve
- CI, confidence interval
- CNN, convolutional neural network
- DL, deep learning
- Deep learning
- DeiT, Data-efficient image Transformer
- Fundus photographs
- Glaucoma detection
- LAG, Large-Scale Attention-Based Glaucoma
- OHTS, Ocular Hypertension Treatment Study
- POAG, primary open-angle glaucoma
- SoTA, state-of-the-art
- VF, visual field
- ViT, Vision Transformer
- Vision Transformers
Collapse
Affiliation(s)
- Rui Fan
- Hamilton Glaucoma Center, Viterbi Family Department of Ophthalmology and Shiley Eye Institute, University of California San Diego, La Jolla, California,Department of Computer Science and Engineering, University of California San Diego, La Jolla, California,Department of Control Science and Engineering, Tongji University, Shanghai 201804, China
| | - Kamran Alipour
- Department of Computer Science and Engineering, University of California San Diego, La Jolla, California
| | - Christopher Bowd
- Hamilton Glaucoma Center, Viterbi Family Department of Ophthalmology and Shiley Eye Institute, University of California San Diego, La Jolla, California
| | - Mark Christopher
- Hamilton Glaucoma Center, Viterbi Family Department of Ophthalmology and Shiley Eye Institute, University of California San Diego, La Jolla, California
| | - Nicole Brye
- Hamilton Glaucoma Center, Viterbi Family Department of Ophthalmology and Shiley Eye Institute, University of California San Diego, La Jolla, California
| | - James A. Proudfoot
- Hamilton Glaucoma Center, Viterbi Family Department of Ophthalmology and Shiley Eye Institute, University of California San Diego, La Jolla, California
| | - Michael H. Goldbaum
- Hamilton Glaucoma Center, Viterbi Family Department of Ophthalmology and Shiley Eye Institute, University of California San Diego, La Jolla, California
| | - Akram Belghith
- Hamilton Glaucoma Center, Viterbi Family Department of Ophthalmology and Shiley Eye Institute, University of California San Diego, La Jolla, California
| | - Christopher A. Girkin
- Department of Ophthalmology, School of Medicine, The University of Alabama at Birmingham, Birmingham, Alabama
| | - Massimo A. Fazio
- Hamilton Glaucoma Center, Viterbi Family Department of Ophthalmology and Shiley Eye Institute, University of California San Diego, La Jolla, California,Department of Ophthalmology, School of Medicine, The University of Alabama at Birmingham, Birmingham, Alabama,Department of Biomedical Engineering, School of Engineering, The University of Alabama at Birmingham, Birmingham, Alabama
| | - Jeffrey M. Liebmann
- Bernard and Shirlee Brown Glaucoma Research Laboratory, Edward S. Harkness Eye Institute, Columbia University Medical Center, New York, New York
| | - Robert N. Weinreb
- Hamilton Glaucoma Center, Viterbi Family Department of Ophthalmology and Shiley Eye Institute, University of California San Diego, La Jolla, California
| | - Michael Pazzani
- Department of Computer Science and Engineering, University of California San Diego, La Jolla, California
| | - David Kriegman
- Department of Computer Science and Engineering, University of California San Diego, La Jolla, California
| | - Linda M. Zangwill
- Hamilton Glaucoma Center, Viterbi Family Department of Ophthalmology and Shiley Eye Institute, University of California San Diego, La Jolla, California,Correspondence: Linda M. Zangwill, 9500 Gilman Dr., #0946, La Jolla, California 92093-0946.
| |
Collapse
|
16
|
Huang S, Li J, Xiao Y, Shen N, Xu T. RTNet: Relation Transformer Network for Diabetic Retinopathy Multi-Lesion Segmentation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2022; 41:1596-1607. [PMID: 35041595 DOI: 10.1109/tmi.2022.3143833] [Citation(s) in RCA: 25] [Impact Index Per Article: 12.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Automatic diabetic retinopathy (DR) lesions segmentation makes great sense of assisting ophthalmologists in diagnosis. Although many researches have been conducted on this task, most prior works paid too much attention to the designs of networks instead of considering the pathological association for lesions. Through investigating the pathogenic causes of DR lesions in advance, we found that certain lesions are closed to specific vessels and present relative patterns to each other. Motivated by the observation, we propose a relation transformer block (RTB) to incorporate attention mechanisms at two main levels: a self-attention transformer exploits global dependencies among lesion features, while a cross-attention transformer allows interactions between lesion and vessel features by integrating valuable vascular information to alleviate ambiguity in lesion detection caused by complex fundus structures. In addition, to capture the small lesion patterns first, we propose a global transformer block (GTB) which preserves detailed information in deep network. By integrating the above blocks of dual-branches, our network segments the four kinds of lesions simultaneously. Comprehensive experiments on IDRiD and DDR datasets well demonstrate the superiority of our approach, which achieves competitive performance compared to state-of-the-arts.
Collapse
|
17
|
CST: A Multitask Learning Framework for Colorectal Cancer Region Mining Based on Transformer. BIOMED RESEARCH INTERNATIONAL 2021; 2021:6207964. [PMID: 34671677 PMCID: PMC8523251 DOI: 10.1155/2021/6207964] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/30/2021] [Accepted: 08/30/2021] [Indexed: 11/17/2022]
Abstract
Colorectal cancer is a high death rate cancer until now; from the clinical view, the diagnosis of the tumour region is critical for the doctors. But with data accumulation, this task takes lots of time and labor with large variances between different doctors. With the development of computer vision, detection and segmentation of the colorectal cancer region from CT or MRI image series are a great challenge in the past decades, and there still have great demands on automatic diagnosis. In this paper, we proposed a novel transfer learning protocol, called CST, that is, a union framework for colorectal cancer region detection and segmentation task based on the transformer model, which effectively constructs the cancer region detection and its segmentation jointly. To make a higher detection accuracy, we incorporate an autoencoder-based image-level decision approach that leverages the image-level decision of a cancer slice. We also compared our framework with one-stage and two-stage object detection methods; the results show that our proposed method achieves better results on detection and segmentation tasks. And this proposed framework will give another pathway for colorectal cancer screen by way of artificial intelligence.
Collapse
|