1
|
Meng Y, Zhang Y, Xie J, Duan J, Joddrell M, Madhusudhan S, Peto T, Zhao Y, Zheng Y. Multi-granularity learning of explicit geometric constraint and contrast for label-efficient medical image segmentation and differentiable clinical function assessment. Med Image Anal 2024; 95:103183. [PMID: 38692098 DOI: 10.1016/j.media.2024.103183] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/04/2023] [Revised: 01/26/2024] [Accepted: 04/18/2024] [Indexed: 05/03/2024]
Abstract
Automated segmentation is a challenging task in medical image analysis that usually requires a large amount of manually labeled data. However, most current supervised learning based algorithms suffer from insufficient manual annotations, posing a significant difficulty for accurate and robust segmentation. In addition, most current semi-supervised methods lack explicit representations of geometric structure and semantic information, restricting segmentation accuracy. In this work, we propose a hybrid framework to learn polygon vertices, region masks, and their boundaries in a weakly/semi-supervised manner that significantly advances geometric and semantic representations. Firstly, we propose multi-granularity learning of explicit geometric structure constraints via polygon vertices (PolyV) and pixel-wise region (PixelR) segmentation masks in a semi-supervised manner. Secondly, we propose eliminating boundary ambiguity by using an explicit contrastive objective to learn a discriminative feature space of boundary contours at the pixel level with limited annotations. Thirdly, we exploit the task-specific clinical domain knowledge to differentiate the clinical function assessment end-to-end. The ground truth of clinical function assessment, on the other hand, can serve as auxiliary weak supervision for PolyV and PixelR learning. We evaluate the proposed framework on two tasks, including optic disc (OD) and cup (OC) segmentation along with vertical cup-to-disc ratio (vCDR) estimation in fundus images; left ventricle (LV) segmentation at end-diastolic and end-systolic frames along with ejection fraction (LVEF) estimation in two-dimensional echocardiography images. Experiments on nine large-scale datasets of the two tasks under different label settings demonstrate our model's superior performance on segmentation and clinical function assessment.
Collapse
Affiliation(s)
- Yanda Meng
- Department of Eye and Vision Sciences, University of Liverpool, Liverpool, United Kingdom
| | - Yuchen Zhang
- Center for Bioinformatics, Peking University, Beijing, China
| | - Jianyang Xie
- Department of Eye and Vision Sciences, University of Liverpool, Liverpool, United Kingdom
| | - Jinming Duan
- School of Computer Science, University of Birmingham, Birmingham, United Kingdom
| | - Martha Joddrell
- Liverpool Centre for Cardiovascular Science, University of Liverpool and Liverpool Heart & Chest Hospital, Liverpool, United Kingdom; Department of Cardiovascular and Metabolic Medicine, University of Liverpool, Liverpool, United Kingdom
| | - Savita Madhusudhan
- St Paul's Eye Unit, Liverpool University Hospitals NHS Foundation Trust, Liverpool, United Kingdom
| | - Tunde Peto
- School of Medicine, Dentistry and Biomedical Sciences, Queen's University Belfast, Belfast, United Kingdom
| | - Yitian Zhao
- Institute of Biomedical Engineering, Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Science, Ningbo, China; Ningbo Eye Hospital, Ningbo, China.
| | - Yalin Zheng
- Department of Eye and Vision Sciences, University of Liverpool, Liverpool, United Kingdom; Liverpool Centre for Cardiovascular Science, University of Liverpool and Liverpool Heart & Chest Hospital, Liverpool, United Kingdom.
| |
Collapse
|
2
|
He H, Qiu J, Lin L, Cai Z, Cheng P, Tang X. JOINEDTrans: Prior guided multi-task transformer for joint optic disc/cup segmentation and fovea detection. Comput Biol Med 2024; 177:108613. [PMID: 38781644 DOI: 10.1016/j.compbiomed.2024.108613] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2023] [Revised: 01/18/2024] [Accepted: 05/11/2024] [Indexed: 05/25/2024]
Abstract
Deep learning-based image segmentation and detection models have largely improved the efficiency of analyzing retinal landmarks such as optic disc (OD), optic cup (OC), and fovea. However, factors including ophthalmic disease-related lesions and low image quality issues may severely complicate automatic OD/OC segmentation and fovea detection. Most existing works treat the identification of each landmark as a single task, and take into account no prior information. To address these issues, we propose a prior guided multi-task transformer framework for joint OD/OC segmentation and fovea detection, named JOINEDTrans. JOINEDTrans effectively combines various spatial features of the fundus images, relieving the structural distortions induced by lesions and other imaging issues. It contains a segmentation branch and a detection branch. To be noted, we employ an encoder with prior-learning in a vessel segmentation task to effectively exploit the positional relationship among vessel, OD/OC, and fovea, successfully incorporating spatial prior into the proposed JOINEDTrans framework. There are a coarse stage and a fine stage in JOINEDTrans. In the coarse stage, OD/OC coarse segmentation and fovea heatmap localization are obtained through a joint segmentation and detection module. In the fine stage, we crop regions of interest for subsequent refinement and use predictions obtained in the coarse stage to provide additional information for better performance and faster convergence. Experimental results demonstrate that JOINEDTrans outperforms existing state-of-the-art methods on the publicly available GAMMA, REFUGE, and PALM fundus image datasets. We make our code available at https://github.com/HuaqingHe/JOINEDTrans.
Collapse
Affiliation(s)
- Huaqing He
- Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, Guangdong, China; Jiaxing Research Institute, Southern University of Science and Technology, Jiaxing, Zhejiang, China.
| | - Jiaming Qiu
- Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, Guangdong, China.
| | - Li Lin
- Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, Guangdong, China; Jiaxing Research Institute, Southern University of Science and Technology, Jiaxing, Zhejiang, China; Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong, China.
| | - Zhiyuan Cai
- Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, Guangdong, China; Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong, China.
| | - Pujin Cheng
- Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, Guangdong, China; Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong, China.
| | - Xiaoying Tang
- Department of Electronic and Electrical Engineering, Southern University of Science and Technology, Shenzhen, Guangdong, China; Jiaxing Research Institute, Southern University of Science and Technology, Jiaxing, Zhejiang, China.
| |
Collapse
|
3
|
Liu X, Tan H, Wang W, Chen Z. Deep learning based retinal vessel segmentation and hypertensive retinopathy quantification using heterogeneous features cross-attention neural network. Front Med (Lausanne) 2024; 11:1377479. [PMID: 38841586 PMCID: PMC11150614 DOI: 10.3389/fmed.2024.1377479] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2024] [Accepted: 05/09/2024] [Indexed: 06/07/2024] Open
Abstract
Retinal vessels play a pivotal role as biomarkers in the detection of retinal diseases, including hypertensive retinopathy. The manual identification of these retinal vessels is both resource-intensive and time-consuming. The fidelity of vessel segmentation in automated methods directly depends on the fundus images' quality. In instances of sub-optimal image quality, applying deep learning-based methodologies emerges as a more effective approach for precise segmentation. We propose a heterogeneous neural network combining the benefit of local semantic information extraction of convolutional neural network and long-range spatial features mining of transformer network structures. Such cross-attention network structure boosts the model's ability to tackle vessel structures in the retinal images. Experiments on four publicly available datasets demonstrate our model's superior performance on vessel segmentation and the big potential of hypertensive retinopathy quantification.
Collapse
Affiliation(s)
- Xinghui Liu
- School of Clinical Medicine, Guizhou Medical University, Guiyang, China
- Department of Cardiovascular Medicine, Guizhou Provincial People's Hospital, Guiyang, China
| | - Hongwen Tan
- Department of Cardiovascular Medicine, Guizhou Provincial People's Hospital, Guiyang, China
| | - Wu Wang
- Electrical Engineering College, Guizhou University, Guiyang, China
| | - Zhangrong Chen
- School of Clinical Medicine, Guizhou Medical University, Guiyang, China
- Department of Cardiovascular Medicine, The Affiliated Hospital of Guizhou Medical University, Guiyang, China
| |
Collapse
|
4
|
Jiang Q, Ye H, Yang B, Cao F. Label-Decoupled Medical Image Segmentation With Spatial-Channel Graph Convolution and Dual Attention Enhancement. IEEE J Biomed Health Inform 2024; 28:2830-2841. [PMID: 38376972 DOI: 10.1109/jbhi.2024.3367756] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/22/2024]
Abstract
Deep learning-based methods have been widely used in medical image segmentation recently. However, existing works are usually difficult to simultaneously capture global long-range information from images and topological correlations among feature maps. Further, medical images often suffer from blurred target edges. Accordingly, this paper proposes a novel medical image segmentation framework named a label-decoupled network with spatial-channel graph convolution and dual attention enhancement mechanism (LADENet for short). It constructs learnable adjacency matrices and utilizes graph convolutions to effectively capture global long-range information on spatial locations and topological dependencies between different channels in an image. Then a label-decoupled strategy based on distance transformation is introduced to decouple an original segmentation label into a body label and an edge label for supervising the body branch and edge branch. Again, a dual attention enhancement mechanism, designing a body attention block in the body branch and an edge attention block in the edge branch, is built to promote the learning ability of spatial region and boundary features. Besides, a feature interactor is devised to fully consider the information interaction between the body and edge branches to improve segmentation performance. Experiments on benchmark datasets reveal the superiority of LADENet compared to state-of-the-art approaches.
Collapse
|
5
|
Gibbon S, Muniz-Terrera G, Yii FSL, Hamid C, Cox S, Maccormick IJC, Tatham AJ, Ritchie C, Trucco E, Dhillon B, MacGillivray TJ. PallorMetrics: Software for Automatically Quantifying Optic Disc Pallor in Fundus Photographs, and Associations With Peripapillary RNFL Thickness. Transl Vis Sci Technol 2024; 13:20. [PMID: 38780955 PMCID: PMC11127490 DOI: 10.1167/tvst.13.5.20] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/08/2023] [Accepted: 04/10/2024] [Indexed: 05/25/2024] Open
Abstract
Purpose We sough to develop an automatic method of quantifying optic disc pallor in fundus photographs and determine associations with peripapillary retinal nerve fiber layer (pRNFL) thickness. Methods We used deep learning to segment the optic disc, fovea, and vessels in fundus photographs, and measured pallor. We assessed the relationship between pallor and pRNFL thickness derived from optical coherence tomography scans in 118 participants. Separately, we used images diagnosed by clinical inspection as pale (n = 45) and assessed how measurements compared with healthy controls (n = 46). We also developed automatic rejection thresholds and tested the software for robustness to camera type, image format, and resolution. Results We developed software that automatically quantified disc pallor across several zones in fundus photographs. Pallor was associated with pRNFL thickness globally (β = -9.81; standard error [SE] = 3.16; P < 0.05), in the temporal inferior zone (β = -29.78; SE = 8.32; P < 0.01), with the nasal/temporal ratio (β = 0.88; SE = 0.34; P < 0.05), and in the whole disc (β = -8.22; SE = 2.92; P < 0.05). Furthermore, pallor was significantly higher in the patient group. Last, we demonstrate the analysis to be robust to camera type, image format, and resolution. Conclusions We developed software that automatically locates and quantifies disc pallor in fundus photographs and found associations between pallor measurements and pRNFL thickness. Translational Relevance We think our method will be useful for the identification, monitoring, and progression of diseases characterized by disc pallor and optic atrophy, including glaucoma, compression, and potentially in neurodegenerative disorders.
Collapse
Affiliation(s)
- Samuel Gibbon
- Centre for Clinical Brain Sciences, Edinburgh, UK
- Robert O Curle Ophthalmology Suite, Institute for Regeneration and Repair, University of Edinburgh, UK, Edinburgh, UK
| | | | - Fabian S. L. Yii
- Centre for Clinical Brain Sciences, Edinburgh, UK
- Robert O Curle Ophthalmology Suite, Institute for Regeneration and Repair, University of Edinburgh, UK, Edinburgh, UK
| | | | - Simon Cox
- Lothian Birth Cohorts, Department of Psychology, University of Edinburgh, Edinburgh, UK
| | - Ian J. C. Maccormick
- Centre for Inflammation Research, University of Edinburgh, Edinburgh, UK
- Institute for Adaptive and Neural Computation, University of Edinburgh, Edinburgh, UK
| | - Andrew J. Tatham
- Centre for Clinical Brain Sciences, Edinburgh, UK
- Princess Alexandra Eye Pavilion, Chalmers Street, Edinburgh, UK
| | - Craig Ritchie
- Centre for Clinical Brain Sciences, Edinburgh, UK
- Centre for Dementia Prevention, University of Edinburgh, Edinburgh, UK
| | - Emanuele Trucco
- VAMPIRE Project, Computing (SSEN), University of Dundee, Dundee, UK
| | - Baljean Dhillon
- Centre for Clinical Brain Sciences, Edinburgh, UK
- Princess Alexandra Eye Pavilion, Chalmers Street, Edinburgh, UK
| | - Thomas J. MacGillivray
- Centre for Clinical Brain Sciences, Edinburgh, UK
- Robert O Curle Ophthalmology Suite, Institute for Regeneration and Repair, University of Edinburgh, UK, Edinburgh, UK
- VAMPIRE Project, Edinburgh Clinical Research facility, University of Edinburgh, Edinburgh, UK
| |
Collapse
|
6
|
Mozaffari J, Amirkhani A, Shokouhi SB. ColonGen: an efficient polyp segmentation system for generalization improvement using a new comprehensive dataset. Phys Eng Sci Med 2024; 47:309-325. [PMID: 38224384 DOI: 10.1007/s13246-023-01368-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2023] [Accepted: 12/06/2023] [Indexed: 01/16/2024]
Abstract
Colorectal cancer (CRC) is one of the most common causes of cancer-related deaths. While polyp detection is important for diagnosing CRC, high miss rates for polyps have been reported during colonoscopy. Most deep learning methods extract features from images using convolutional neural networks (CNNs). In recent years, vision transformer (ViT) models have been employed for image processing and have been successful in image segmentation. It is possible to improve image processing by using transformer models that can extract spatial location information, and CNNs that are capable of aggregating local information. Despite this, recent research shows limited effectiveness in increasing data diversity and generalization accuracy. This paper investigates the generalization proficiency of polyp image segmentation based on transformer architecture and proposes a novel approach using two different ViT architectures. This allows the model to learn representations from different perspectives, which can then be combined to create a richer feature representation. Additionally, a more universal and comprehensive dataset has been derived from the datasets presented in the related research, which can be used for improving generalizations. We first evaluated the generalization of our proposed model using three distinct training-testing scenarios. Our experimental results demonstrate that our ColonGen-V1 outperforms other state-of-the-art methods in all scenarios. As a next step, we used the comprehensive dataset for improving the performance of the model against in- and out-of-domain data. The results show that our ColonGen-V2 outperforms state-of-the-art studies by 5.1%, 1.3%, and 1.1% in ETIS-Larib, Kvasir-Seg, and CVC-ColonDB datasets, respectively. The inclusive dataset and the model introduced in this paper are available to the public through this link: https://github.com/javadmozaffari/Polyp_segmentation .
Collapse
Affiliation(s)
- Javad Mozaffari
- School of Electrical Engineering, Iran University of Science and Technology, Tehran, 16846-13114, Iran
| | - Abdollah Amirkhani
- School of Automotive Engineering, Iran University of Science and Technology, Tehran, 16846-13114, Iran.
| | - Shahriar B Shokouhi
- School of Electrical Engineering, Iran University of Science and Technology, Tehran, 16846-13114, Iran
| |
Collapse
|
7
|
Chen Y, Yang W, Lu J, Sun J, Rao L, Zhao H, Peng X, Ni D. A modified U-net with graph representation for dose prediction in esophageal cancer radiotherapy plans. Comput Med Imaging Graph 2024; 111:102318. [PMID: 38088017 DOI: 10.1016/j.compmedimag.2023.102318] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2023] [Revised: 11/29/2023] [Accepted: 11/30/2023] [Indexed: 01/08/2024]
Abstract
The manual design of esophageal cancer radiotherapy plan is time-consuming and labor-intensive. Automatic planning (AP) is prevalent nowadays to increase physicists' work efficiency. Because of the intuitiveness of dose distribution in AP evaluation, obtaining reasonable dose prediction provides effective guarantees to generate a satisfactory AP. Existing fully convolutional network-based methods for predicting dose distribution in esophageal cancer radiotherapy plans often capture features in a limited receptive field. Additionally, the correlations between voxel pairs are often ignored. This work modifies the U-net architecture and exploits graph convolution to capture long-range information for dose prediction in esophageal cancer plans. Meanwhile, attention mechanism gets correlations between planning target volume (PTV) and organs at risk, and adaptively learns their feature weights. Finally, a novel loss function that considers features between voxel pairs is used to highlight the predictions. 152 subjects with prescription doses of 50 Gy or 60 Gy are collected in this study. The mean absolute error and standard deviation of conformity index, homogeneity index, and max dose for PTV achieved by the proposed method are 0.036 ± 0.030, 0.036 ± 0.027, and 0.930 ± 1.162, respectively, which outperform other state-of-the-art models. The superior performance demonstrates that our proposed method has great potential for AP generation.
Collapse
Affiliation(s)
- Yanlin Chen
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China
| | - Wei Yang
- School of Biomedical Engineering, Southern Medical University, Guangzhou, China
| | - Jiayang Lu
- Department of Radiation Oncology, Cancer Hospital of Shantou University Medical College, Shantou, China
| | - Jinyan Sun
- School of Medicine, Foshan University, Foshan, China
| | - Linshang Rao
- Shenzhen Mindray Bio-Medical Electronics Co., Ltd., Shenzhen, China
| | - Huanmiao Zhao
- School of Biomedical Engineering, Southern Medical University, Guangzhou, China
| | - Xun Peng
- Department of Radiation Oncology, Cancer Hospital of Shantou University Medical College, Shantou, China.
| | - Dong Ni
- National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, School of Biomedical Engineering, Health Science Center, Shenzhen University, Shenzhen, China; Medical UltraSound Image Computing (MUSIC) Lab, Shenzhen University, Shenzhen, China.
| |
Collapse
|
8
|
Zhang H, Li X, Li Z, Huang D, Zhang L. Estimation of Particle Location in Granular Materials Based on Graph Neural Networks. MICROMACHINES 2023; 14:714. [PMID: 37420946 DOI: 10.3390/mi14040714] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/21/2023] [Revised: 03/20/2023] [Accepted: 03/21/2023] [Indexed: 07/09/2023]
Abstract
Particle locations determine the whole structure of a granular system, which is crucial to understanding various anomalous behaviors in glasses and amorphous solids. How to accurately determine the coordinates of each particle in such materials within a short time has always been a challenge. In this paper, we use an improved graph convolutional neural network to estimate the particle locations in two-dimensional photoelastic granular materials purely from the knowledge of the distances for each particle, which can be estimated in advance via a distance estimation algorithm. The robustness and effectiveness of our model are verified by testing other granular systems with different disorder degrees, as well as systems with different configurations. In this study, we attempt to provide a new route to the structural information of granular systems irrelevant to dimensionality, compositions, or other material properties.
Collapse
Affiliation(s)
- Hang Zhang
- School of Automation, Central South University, Changsha 410083, China
| | - Xingqiao Li
- School of Automation, Central South University, Changsha 410083, China
| | - Zirui Li
- School of Automation, Central South University, Changsha 410083, China
| | - Duan Huang
- School of Computer Science and Engineering, Central South University, Changsha 410083, China
| | - Ling Zhang
- School of Automation, Central South University, Changsha 410083, China
| |
Collapse
|
9
|
Meng Y, Bridge J, Addison C, Wang M, Merritt C, Franks S, Mackey M, Messenger S, Sun R, Fitzmaurice T, McCann C, Li Q, Zhao Y, Zheng Y. Bilateral adaptive graph convolutional network on CT based Covid-19 diagnosis with uncertainty-aware consensus-assisted multiple instance learning. Med Image Anal 2023; 84:102722. [PMID: 36574737 PMCID: PMC9753459 DOI: 10.1016/j.media.2022.102722] [Citation(s) in RCA: 10] [Impact Index Per Article: 10.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2022] [Revised: 10/17/2022] [Accepted: 12/02/2022] [Indexed: 12/23/2022]
Abstract
Coronavirus disease (COVID-19) has caused a worldwide pandemic, putting millions of people's health and lives in jeopardy. Detecting infected patients early on chest computed tomography (CT) is critical in combating COVID-19. Harnessing uncertainty-aware consensus-assisted multiple instance learning (UC-MIL), we propose to diagnose COVID-19 using a new bilateral adaptive graph-based (BA-GCN) model that can use both 2D and 3D discriminative information in 3D CT volumes with arbitrary number of slices. Given the importance of lung segmentation for this task, we have created the largest manual annotation dataset so far with 7,768 slices from COVID-19 patients, and have used it to train a 2D segmentation model to segment the lungs from individual slices and mask the lungs as the regions of interest for the subsequent analyses. We then used the UC-MIL model to estimate the uncertainty of each prediction and the consensus between multiple predictions on each CT slice to automatically select a fixed number of CT slices with reliable predictions for the subsequent model reasoning. Finally, we adaptively constructed a BA-GCN with vertices from different granularity levels (2D and 3D) to aggregate multi-level features for the final diagnosis with the benefits of the graph convolution network's superiority to tackle cross-granularity relationships. Experimental results on three largest COVID-19 CT datasets demonstrated that our model can produce reliable and accurate COVID-19 predictions using CT volumes with any number of slices, which outperforms existing approaches in terms of learning and generalisation ability. To promote reproducible research, we have made the datasets, including the manual annotations and cleaned CT dataset, as well as the implementation code, available at https://doi.org/10.5281/zenodo.6361963.
Collapse
Affiliation(s)
- Yanda Meng
- Department of Eye and Vision Science, University of Liverpool, Liverpool, United Kingdom
| | - Joshua Bridge
- Department of Eye and Vision Science, University of Liverpool, Liverpool, United Kingdom
| | - Cliff Addison
- Advanced Research Computing, University of Liverpool, Liverpool, United Kingdom
| | - Manhui Wang
- Advanced Research Computing, University of Liverpool, Liverpool, United Kingdom
| | | | - Stu Franks
- Alces Flight Limited, Bicester, United Kingdom
| | - Maria Mackey
- Amazon Web Services, 60 Holborn Viaduct, London, United Kingdom
| | - Steve Messenger
- Amazon Web Services, 60 Holborn Viaduct, London, United Kingdom
| | - Renrong Sun
- Department of Radiology, Hubei Provincial Hospital of Integrated Chinese and Western Medicine, Hubei University of Chinese Medicine, Wuhan, China
| | - Thomas Fitzmaurice
- Adult Cystic Fibrosis Unit, Liverpool Heart and Chest Hospital NHS Foundation Trust, Liverpool, United Kingdom
| | - Caroline McCann
- Radiology, Liverpool Heart and Chest Hospital NHS Foundation Trust, United Kingdom
| | - Qiang Li
- The Affiliated People’s Hospital of Ningbo University, Ningbo, China
| | - Yitian Zhao
- The Affiliated People's Hospital of Ningbo University, Ningbo, China; Cixi Institute of Biomedical Engineering, Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Science, Ningbo, China.
| | - Yalin Zheng
- Department of Eye and Vision Science, University of Liverpool, Liverpool, United Kingdom; Liverpool Centre for Cardiovascular Science, University of Liverpool and Liverpool Heart & Chest Hospital, Liverpool, United Kingdom.
| |
Collapse
|
10
|
Meng Y, Zhang H, Zhao Y, Gao D, Hamill B, Patri G, Peto T, Madhusudhan S, Zheng Y. Dual Consistency Enabled Weakly and Semi-Supervised Optic Disc and Cup Segmentation With Dual Adaptive Graph Convolutional Networks. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:416-429. [PMID: 36044486 DOI: 10.1109/tmi.2022.3203318] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Glaucoma is a progressive eye disease that results in permanent vision loss, and the vertical cup to disc ratio (vCDR) in colour fundus images is essential in glaucoma screening and assessment. Previous fully supervised convolution neural networks segment the optic disc (OD) and optic cup (OC) from color fundus images and then calculate the vCDR offline. However, they rely on a large set of labeled masks for training, which is expensive and time-consuming to acquire. To address this, we propose a weakly and semi-supervised graph-based network that investigates geometric associations and domain knowledge between segmentation probability maps (PM), modified signed distance function representations (mSDF), and boundary region of interest characteristics (B-ROI) in three aspects. Firstly, we propose a novel Dual Adaptive Graph Convolutional Network (DAGCN) to reason the long-range features of the PM and the mSDF w.r.t. the regional uniformity. Secondly, we propose a dual consistency regularization-based semi-supervised learning paradigm. The regional consistency between the PM and the mSDF, and the marginal consistency between the derived B-ROI from each of them boost the proposed model's performance due to the inherent geometric associations. Thirdly, we exploit the task-specific domain knowledge via the oval shapes of OD & OC, where a differentiable vCDR estimating layer is proposed. Furthermore, without additional annotations, the supervision on vCDR serves as weakly-supervisions for segmentation tasks. Experiments on six large-scale datasets demonstrate our model's superior performance on OD & OC segmentation and vCDR estimation. The implementation code has been made available.https://github.com/smallmax00/Dual_Adaptive_Graph_Reasoning.
Collapse
|
11
|
Zhan X, Liu J, Long H, Zhu J, Tang H, Gou F, Wu J. An Intelligent Auxiliary Framework for Bone Malignant Tumor Lesion Segmentation in Medical Image Analysis. Diagnostics (Basel) 2023; 13:diagnostics13020223. [PMID: 36673032 PMCID: PMC9858155 DOI: 10.3390/diagnostics13020223] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/25/2022] [Revised: 12/17/2022] [Accepted: 01/04/2023] [Indexed: 01/11/2023] Open
Abstract
Bone malignant tumors are metastatic and aggressive, with poor treatment outcomes and prognosis. Rapid and accurate diagnosis is crucial for limb salvage and increasing the survival rate. There is a lack of research on deep learning to segment bone malignant tumor lesions in medical images with complex backgrounds and blurred boundaries. Therefore, we propose a new intelligent auxiliary framework for the medical image segmentation of bone malignant tumor lesions, which consists of a supervised edge-attention guidance segmentation network (SEAGNET). We design a boundary key points selection module to supervise the learning of edge attention in the model to retain fine-grained edge feature information. We precisely locate malignant tumors by instance segmentation networks while extracting feature maps of tumor lesions in medical images. The rich contextual-dependent information in the feature map is captured by mixed attention to better understand the uncertainty and ambiguity of the boundary, and edge attention learning is used to guide the segmentation network to focus on the fuzzy boundary of the tumor region. We implement extensive experiments on real-world medical data to validate our model. It validates the superiority of our method over the latest segmentation methods, achieving the best performance in terms of the Dice similarity coefficient (0.967), precision (0.968), and accuracy (0.996). The results prove the important contribution of the framework in assisting doctors to improve the accuracy of diagnosis and clinical efficiency.
Collapse
Affiliation(s)
- Xiangbing Zhan
- State Key Laboratory of Public Big Data, College of Computer Science and Technology, Guizhou University, Guiyang 550025, China
| | - Jun Liu
- The Second People’s Hospital of Huaihua, Huaihua 418000, China
- Correspondence: (J.L.); (H.L.); (J.W.)
| | - Huiyun Long
- State Key Laboratory of Public Big Data, College of Computer Science and Technology, Guizhou University, Guiyang 550025, China
- Correspondence: (J.L.); (H.L.); (J.W.)
| | - Jun Zhu
- The First People’s Hospital of Huaihua, Huaihua 418000, China
- Collaborative Innovation Center for Medical Artificial Intelligence and Big Data Decision Making Assistance, Hunan University of Medicine, Huaihua 418000, China
| | - Haoyu Tang
- The First People’s Hospital of Huaihua, Huaihua 418000, China
| | - Fangfang Gou
- State Key Laboratory of Public Big Data, College of Computer Science and Technology, Guizhou University, Guiyang 550025, China
- The First People’s Hospital of Huaihua, Huaihua 418000, China
- Collaborative Innovation Center for Medical Artificial Intelligence and Big Data Decision Making Assistance, Hunan University of Medicine, Huaihua 418000, China
| | - Jia Wu
- State Key Laboratory of Public Big Data, College of Computer Science and Technology, Guizhou University, Guiyang 550025, China
- The First People’s Hospital of Huaihua, Huaihua 418000, China
- Collaborative Innovation Center for Medical Artificial Intelligence and Big Data Decision Making Assistance, Hunan University of Medicine, Huaihua 418000, China
- Research Center for Artificial Intelligence, Monash University, Melbourne, VIC 3800, Australia
- Correspondence: (J.L.); (H.L.); (J.W.)
| |
Collapse
|
12
|
Narasimha Raju AS, Jayavel K, Rajalakshmi T. ColoRectalCADx: Expeditious Recognition of Colorectal Cancer with Integrated Convolutional Neural Networks and Visual Explanations Using Mixed Dataset Evidence. COMPUTATIONAL AND MATHEMATICAL METHODS IN MEDICINE 2022; 2022:8723957. [PMID: 36404909 PMCID: PMC9671728 DOI: 10.1155/2022/8723957] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/27/2022] [Accepted: 10/27/2022] [Indexed: 12/07/2023]
Abstract
Colorectal cancer typically affects the gastrointestinal tract within the human body. Colonoscopy is one of the most accurate methods of detecting cancer. The current system facilitates the identification of cancer by computer-assisted diagnosis (CADx) systems with a limited number of deep learning methods. It does not imply the depiction of mixed datasets for the functioning of the system. The proposed system, called ColoRectalCADx, is supported by deep learning (DL) models suitable for cancer research. The CADx system comprises five stages: convolutional neural networks (CNN), support vector machine (SVM), long short-term memory (LSTM), visual explanation such as gradient-weighted class activation mapping (Grad-CAM), and semantic segmentation phases. Here, the key components of the CADx system are equipped with 9 individual and 12 integrated CNNs, implying that the system consists mainly of investigational experiments with a total of 21 CNNs. In the subsequent phase, the CADx has a combination of CNNs of concatenated transfer learning functions associated with the machine SVM classification. Additional classification is applied to ensure effective transfer of results from CNN to LSTM. The system is mainly made up of a combination of CVC Clinic DB, Kvasir2, and Hyper Kvasir input as a mixed dataset. After CNN and LSTM, in advanced stage, malignancies are detected by using a better polyp recognition technique with Grad-CAM and semantic segmentation using U-Net. CADx results have been stored on Google Cloud for record retention. In these experiments, among all the CNNs, the individual CNN DenseNet-201 (87.1% training and 84.7% testing accuracies) and the integrated CNN ADaDR-22 (84.61% training and 82.17% testing accuracies) were the most efficient for cancer detection with the CNN+LSTM model. ColoRectalCADx accurately identifies cancer through individual CNN DesnseNet-201 and integrated CNN ADaDR-22. In Grad-CAM's visual explanations, CNN DenseNet-201 displays precise visualization of polyps, and CNN U-Net provides precise malignant polyps.
Collapse
Affiliation(s)
- Akella S. Narasimha Raju
- Department of Networking and Communications, School of Computing, SRM Institute of Science and Technology, Kattankulathur, 603203 Chennai, India
| | - Kayalvizhi Jayavel
- Department of Networking and Communications, School of Computing, SRM Institute of Science and Technology, Kattankulathur, 603203 Chennai, India
| | - T. Rajalakshmi
- Department of Electronics and Communication Engineering, School of Electrical and Electronics Engineering, SRM Institute of Science and Technology, Kattankulathur, 603203 Chennai, India
| |
Collapse
|
13
|
DGRUnit: Dual graph reasoning unit for brain tumor segmentation. Comput Biol Med 2022; 149:106079. [PMID: 36108413 DOI: 10.1016/j.compbiomed.2022.106079] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/23/2022] [Revised: 08/27/2022] [Accepted: 09/03/2022] [Indexed: 11/20/2022]
Abstract
Many fully automatic segmentation models have been created to solve the difficulty of brain tumor segmentation, thanks to the rapid growth of deep learning. However, few approaches focus on the long-range relationships and contextual interdependence in multimodal Magnetic Resonance (MR) images. In this paper, we propose a novel approach for brain tumor segmentation called the dual graph reasoning unit (DGRUnit). Two parallel graph reasoning modules are included in our proposed method: a spatial reasoning module and a channel reasoning module. The spatial reasoning module models the long-range spatial dependencies between distinct regions in an image using a graph convolutional network (GCN). The channel reasoning module uses a graph attention network (GAT) to model the rich contextual interdependencies between different channels with similar semantic representations. Our experimental results clearly demonstrate the superior performance of the proposed DGRUnit. The ablation study shows the flexibility and generalizability of our model, which can be easily integrated into a wide range of neural networks and further improve them. When compared to several state-of-the-art methods, experimental results show that the proposed approach significantly improves both visual inspection and quantitative metrics for brain tumor segmentation tasks.
Collapse
|
14
|
Zhang G, Sun B, Zhang Z, Pan J, Yang W, Liu Y. Multi-Model Domain Adaptation for Diabetic Retinopathy Classification. Front Physiol 2022; 13:918929. [PMID: 35845987 PMCID: PMC9284280 DOI: 10.3389/fphys.2022.918929] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2022] [Accepted: 05/12/2022] [Indexed: 01/01/2023] Open
Abstract
Diabetic retinopathy (DR) is one of the most threatening complications in diabetic patients, leading to permanent blindness without timely treatment. However, DR screening is not only a time-consuming task that requires experienced ophthalmologists but also easy to produce misdiagnosis. In recent years, deep learning techniques based on convolutional neural networks have attracted increasing research attention in medical image analysis, especially for DR diagnosis. However, dataset labeling is expensive work and it is necessary for existing deep-learning-based DR detection models. For this study, a novel domain adaptation method (multi-model domain adaptation) is developed for unsupervised DR classification in unlabeled retinal images. At the same time, it only exploits discriminative information from multiple source models without access to any data. In detail, we integrate a weight mechanism into the multi-model-based domain adaptation by measuring the importance of each source domain in a novel way, and a weighted pseudo-labeling strategy is attached to the source feature extractors for training the target DR classification model. Extensive experiments are performed on four source datasets (DDR, IDRiD, Messidor, and Messidor-2) to a target domain APTOS 2019, showing that MMDA produces competitive performance for present state-of-the-art methods for DR classification. As a novel DR detection approach, this article presents a new domain adaptation solution for medical image analysis when the source data is unavailable.
Collapse
Affiliation(s)
- Guanghua Zhang
- Department of Intelligence and Automation, Taiyuan University, Taiyuan, China
- Graphics and Imaging Laboratory, University of Girona, Girona, Spain
| | - Bin Sun
- Shanxi Eye Hospital, Taiyuan, China
| | | | - Jing Pan
- Department of Materials and Chemical Engineering, Taiyuan University, Taiyuan, China
| | - Weihua Yang
- Affiliated Eye Hospital, Nanjing Medical University, Nanjing, China
- *Correspondence: Yunfang Liu, ; Weihua Yang,
| | - Yunfang Liu
- The First Affiliated Hospital of Huzhou University, Huzhou, China
- *Correspondence: Yunfang Liu, ; Weihua Yang,
| |
Collapse
|
15
|
Zhang G, Sun B, Chen Z, Gao Y, Zhang Z, Li K, Yang W. Diabetic Retinopathy Grading by Deep Graph Correlation Network on Retinal Images Without Manual Annotations. Front Med (Lausanne) 2022; 9:872214. [PMID: 35492360 PMCID: PMC9046841 DOI: 10.3389/fmed.2022.872214] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/10/2022] [Accepted: 03/18/2022] [Indexed: 11/20/2022] Open
Abstract
Background Diabetic retinopathy, as a severe public health problem associated with vision loss, should be diagnosed early using an accurate screening tool. While many previous deep learning models have been proposed for this disease, they need sufficient professional annotation data to train the model, requiring more expensive and time-consuming screening skills. Method This study aims to economize manual power and proposes a deep graph correlation network (DGCN) to develop automated diabetic retinopathy grading without any professional annotations. DGCN involves the novel deep learning algorithm of a graph convolutional network to exploit inherent correlations from independent retinal image features learned by a convolutional neural network. Three designed loss functions of graph-center, pseudo-contrastive, and transformation-invariant constrain the optimisation and application of the DGCN model in an automated diabetic retinopathy grading task. Results To evaluate the DGCN model, this study employed EyePACS-1 and Messidor-2 sets to perform grading results. It achieved an accuracy of 89.9% (91.8%), sensitivity of 88.2% (90.2%), and specificity of 91.3% (93.0%) on EyePACS-1 (Messidor-2) data set with a confidence index of 95% and commendable effectiveness on receiver operating characteristic (ROC) curve and t-SNE plots. Conclusion The grading capability of this study is close to that of retina specialists, but superior to that of trained graders, which demonstrates that the proposed DGCN provides an innovative route for automated diabetic retinopathy grading and other computer-aided diagnostic systems.
Collapse
Affiliation(s)
- Guanghua Zhang
- Department of Intelligence and Automation, Taiyuan University, Taiyuan, China
- Graphics and Imaging Laboratory, University of Girona, Girona, Spain
| | - Bin Sun
- Shanxi Eye Hospital, Taiyuan, China
| | - Zhixian Chen
- Department of Intelligence and Automation, Taiyuan University, Taiyuan, China
| | - Yuxi Gao
- Shanxi Finance and Taxation College, Taiyuan, China
| | | | - Keran Li
- The Laboratory of Artificial Intelligence and Bigdata in Ophthalmology, The Affiliated Eye Hospital of Nanjing Medical University, Nanjing, China
- Keran Li,
| | - Weihua Yang
- The Laboratory of Artificial Intelligence and Bigdata in Ophthalmology, The Affiliated Eye Hospital of Nanjing Medical University, Nanjing, China
- *Correspondence: Weihua Yang,
| |
Collapse
|