1
|
Zhu H, Shu S, Zhang J. A cascaded FAS-UNet+ framework with iterative optimization strategy for segmentation of organs at risk. Med Biol Eng Comput 2024:10.1007/s11517-024-03208-7. [PMID: 39365519 DOI: 10.1007/s11517-024-03208-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2024] [Accepted: 09/17/2024] [Indexed: 10/05/2024]
Abstract
Segmentation of organs at risks (OARs) in the thorax plays a critical role in radiation therapy for lung and esophageal cancer. Although automatic segmentation of OARs has been extensively studied, it remains challenging due to the varying sizes and shapes of organs, as well as the low contrast between the target and background. This paper proposes a cascaded FAS-UNet+ framework, which integrates convolutional neural networks and nonlinear multi-grid theory to solve a modified Mumford-shah model for segmenting OARs. This framework is equipped with an enhanced iteration block, a coarse-to-fine multiscale architecture, an iterative optimization strategy, and a model ensemble technique. The enhanced iteration block aims to extract multiscale features, while the cascade module is used to refine coarse segmentation predictions. The iterative optimization strategy improves the network parameters to avoid unfavorable local minima. An efficient data augmentation method is also developed to train the network, which significantly improves its performance. During the prediction stage, a weighted ensemble technique combines predictions from multiple models to refine the final segmentation. The proposed cascaded FAS-UNet+ framework was evaluated on the SegTHOR dataset, and the results demonstrate significant improvements in Dice score and Hausdorff Distance (HD). The Dice scores were 95.22%, 95.68%, and HD values were 0.1024, and 0.1194 for the segmentations of the aorta and heart in the official unlabeled dataset, respectively. Our code and trained models are available at https://github.com/zhuhui100/C-FASUNet-plus .
Collapse
Affiliation(s)
- Hui Zhu
- School of Mathematics and Computational Science, Xiangtan University, Xiangtan, 411105, China
- School of Computational Science and Electronics, Hunan Institute of Engineering, Xiangtan, 411104, China
- Key Laboratory of Intelligent Computing and Information Processing of Ministry of Education, Xiangtan, Hunan, 411105, China
| | - Shi Shu
- School of Mathematics and Computational Science, Xiangtan University, Xiangtan, 411105, China
- Hunan Key Laboratory for Computation and Simulation in Science and Engineering, Xiangtan, Hunan, 411105, China
| | - Jianping Zhang
- School of Mathematics and Computational Science, Xiangtan University, Xiangtan, 411105, China.
- National Center for Applied Mathematics in Hunan, Xiangtan, Hunan, 411105, China.
| |
Collapse
|
2
|
Jiang J, Fu T, Liu J, Wang Y, Fan J, Song H, Xiao D, Wang Y, Yang J. Real-time simulation for multi-component biomechanical analysis using localized tissue constraint progressive transfer learning. J Mech Behav Biomed Mater 2024; 158:106682. [PMID: 39142234 DOI: 10.1016/j.jmbbm.2024.106682] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2024] [Revised: 07/29/2024] [Accepted: 08/08/2024] [Indexed: 08/16/2024]
Abstract
In virtual surgical training, it is crucial to achieve real-time, high-fidelity simulation of the tissue deformation. The anisotropic and nonlinear characteristics of the organ with multi-component make accurate real-time deformation simulation difficult. A localized tissue constraint progressive transfer learning method is proposed in this paper, where the base-compensated dual-output transfer learning strategy and the localized tissue constraint progressive learning architecture are developed. The proposed strategy enriches the multi-component biomechanical dataset to fully represent complex force-displacement with minimal high-quality data. Meanwhile, the proposed architecture adopts focused and progressive model to accurately describe tissues with varied biomechanical properties rather than singular homogeneous model. We made comparison with 4 state-of-the-art (SOTA) methods in simulating multi-component biomechanical deformations of organs with 100 pairs of testing data. Results show that the accuracy of our method is 50% higher than other methods in different validation matrix. And our method can stably simulate the deformations in 0.005 s per frame, which largely improves the computing efficiency.
Collapse
Affiliation(s)
- Jiaxi Jiang
- School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China
| | - Tianyu Fu
- School of Medical Technology, Beijing Institute of Technology, Beijing, 100081, China.
| | - Jiaqi Liu
- School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China
| | - Yuanyuan Wang
- School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China
| | - Jingfan Fan
- School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China
| | - Hong Song
- School of Computer Science and Technology, Beijing Institute of Technology, Beijing, 100081, China
| | - Deqiang Xiao
- School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China
| | - Yongtian Wang
- School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China.
| | - Jian Yang
- School of Optics and Photonics, Beijing Institute of Technology, Beijing, 100081, China.
| |
Collapse
|
3
|
Kumar A, Jiang H, Imran M, Valdes C, Leon G, Kang D, Nataraj P, Zhou Y, Weiss MD, Shao W. A flexible 2.5D medical image segmentation approach with in-slice and cross-slice attention. Comput Biol Med 2024; 182:109173. [PMID: 39317055 DOI: 10.1016/j.compbiomed.2024.109173] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/30/2024] [Revised: 08/18/2024] [Accepted: 09/17/2024] [Indexed: 09/26/2024]
Abstract
Deep learning has become the de facto method for medical image segmentation, with 3D segmentation models excelling in capturing complex 3D structures and 2D models offering high computational efficiency. However, segmenting 2.5D images, characterized by high in-plane resolution but lower through-plane resolution, presents significant challenges. While applying 2D models to individual slices of a 2.5D image is feasible, it fails to capture the spatial relationships between slices. On the other hand, 3D models face challenges such as resolution inconsistencies in 2.5D images, along with computational complexity and susceptibility to overfitting when trained with limited data. In this context, 2.5D models, which capture inter-slice correlations using only 2D neural networks, emerge as a promising solution due to their reduced computational demand and simplicity in implementation. In this paper, we introduce CSA-Net, a flexible 2.5D segmentation model capable of processing 2.5D images with an arbitrary number of slices. CSA-Net features an innovative Cross-Slice Attention (CSA) module that effectively captures 3D spatial information by learning long-range dependencies between the center slice (for segmentation) and its neighboring slices. Moreover, CSA-Net utilizes the self-attention mechanism to learn correlations among pixels within the center slice. We evaluated CSA-Net on three 2.5D segmentation tasks: (1) multi-class brain MR image segmentation, (2) binary prostate MR image segmentation, and (3) multi-class prostate MR image segmentation. CSA-Net outperformed leading 2D, 2.5D, and 3D segmentation methods across all three tasks, achieving average Dice coefficients and HD95 values of 0.897 and 1.40 mm for the brain dataset, 0.921 and 1.06 mm for the prostate dataset, and 0.659 and 2.70 mm for the ProstateX dataset, demonstrating its efficacy and superiority. Our code is publicly available at: https://github.com/mirthAI/CSA-Net.
Collapse
Affiliation(s)
- Amarjeet Kumar
- Department of Computer and Information Science and Engineering, University of Florida, Gainesville, FL, 32610, United States
| | - Hongxu Jiang
- Department of Electrical and Computer Engineering, University of Florida, Gainesville, FL, 32610, United States
| | - Muhammad Imran
- Department of Medicine, University of Florida, Gainesville, FL, 32610, United States
| | - Cyndi Valdes
- Department of Pediatrics, University of Florida, Gainesville, FL, 32610, United States
| | - Gabriela Leon
- College of Medicine, University of Florida, Gainesville, FL, 32610, United States
| | - Dahyun Kang
- College of Medicine, University of Florida, Gainesville, FL, 32610, United States
| | - Parvathi Nataraj
- Department of Pediatrics, University of Florida, Gainesville, FL, 32610, United States
| | - Yuyin Zhou
- Department of Computer Science and Engineering, University of California, Santa Cruz, CA, 95064, United States
| | - Michael D Weiss
- Department of Pediatrics, University of Florida, Gainesville, FL, 32610, United States
| | - Wei Shao
- Department of Medicine, University of Florida, Gainesville, FL, 32610, United States; Intelligent Clinical Care Center, University of Florida, Gainesville, FL, 32610, United States.
| |
Collapse
|
4
|
Orouskhani M, Firoozeh N, Wang H, Wang Y, Shi H, Li W, Sun B, Zhang J, Li X, Zhao H, Mossa-Basha M, Hwang JN, Zhu C. Morphology and Texture-Guided Deep Neural Network for Intracranial Aneurysm Segmentation in 3D TOF-MRA. Neuroinformatics 2024:10.1007/s12021-024-09683-5. [PMID: 39259472 DOI: 10.1007/s12021-024-09683-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 08/08/2024] [Indexed: 09/13/2024]
Abstract
This study concentrates on the segmentation of intracranial aneurysms, a pivotal aspect of diagnosis and treatment planning. We aim to overcome the inherent instance imbalance and morphological variability by introducing a novel morphology and texture loss reweighting approach. Our innovative method involves the incorporation of tailored weights within the loss function of deep neural networks. Specifically designed to account for aneurysm size, shape, and texture, this approach strategically guides the model to focus on capturing discriminative information from imbalanced features. The study conducted extensive experimentation utilizing ADAM and RENJI TOF-MRA datasets to validate the proposed approach. The results of our experimentation demonstrate the remarkable effectiveness of the introduced methodology in improving aneurysm segmentation accuracy. By dynamically adapting to the variances present in aneurysm features, our model showcases promising outcomes for accurate diagnostic insights. The nuanced consideration of morphological and textural nuances within the loss function proves instrumental in overcoming the challenge posed by instance imbalance. In conclusion, our study presents a nuanced solution to the intricate challenge of intracranial aneurysm segmentation. The proposed morphology and texture loss reweighting approach, with its tailored weights and dynamic adaptability, proves to be instrumental in enhancing segmentation precision. The promising outcomes from our experimentation suggest the potential for accurate diagnostic insights and informed treatment strategies, marking a significant advancement in this critical domain of medical imaging.
Collapse
Affiliation(s)
| | - Negar Firoozeh
- Department of Radiology, University of Washington, Seattle, WA, USA
| | - Huayu Wang
- Department of Radiology, University of Washington, Seattle, WA, USA
- Department of Electrical and Computer Engineering, University of Washington, Seattle, WA, USA
| | - Yan Wang
- Department of Radiology and Biomedical Imaging, University of California San Francisco, San Francisco, CA, USA
| | - Hanrui Shi
- Department of Radiology, University of Washington, Seattle, WA, USA
- Department of Electrical and Computer Engineering, University of Washington, Seattle, WA, USA
| | - Weijing Li
- Department of Radiology, University of Washington, Seattle, WA, USA
- Department of Electrical and Computer Engineering, University of Washington, Seattle, WA, USA
| | - Beibei Sun
- Department of Radiology, University of Washington, Seattle, WA, USA
- Department of Radiology, Ren Ji Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Jianjian Zhang
- Department of Radiology, Ren Ji Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Xiao Li
- Department of Radiology, Ren Ji Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Huilin Zhao
- Department of Radiology, Ren Ji Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | | | - Jenq-Neng Hwang
- Department of Electrical and Computer Engineering, University of Washington, Seattle, WA, USA
| | - Chengcheng Zhu
- Department of Radiology, University of Washington, Seattle, WA, USA.
- Harborview Medical Center, Seattle, WA, USA.
| |
Collapse
|
5
|
Henninger S, Kellner M, Rombach B, Reiterer A. Reducing Training Data Using Pre-Trained Foundation Models: A Case Study on Traffic Sign Segmentation Using the Segment Anything Model. J Imaging 2024; 10:220. [PMID: 39330440 PMCID: PMC11433296 DOI: 10.3390/jimaging10090220] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/19/2024] [Revised: 09/02/2024] [Accepted: 09/05/2024] [Indexed: 09/28/2024] Open
Abstract
The utilization of robust, pre-trained foundation models enables simple adaptation to specific ongoing tasks. In particular, the recently developed Segment Anything Model (SAM) has demonstrated impressive results in the context of semantic segmentation. Recognizing that data collection is generally time-consuming and costly, this research aims to determine whether the use of these foundation models can reduce the need for training data. To assess the models' behavior under conditions of reduced training data, five test datasets for semantic segmentation will be utilized. This study will concentrate on traffic sign segmentation to analyze the results in comparison to Mask R-CNN: the field's leading model. The findings indicate that SAM does not surpass the leading model for this specific task, regardless of the quantity of training data. Nevertheless, a knowledge-distilled student architecture derived from SAM exhibits no reduction in accuracy when trained on data that have been reduced by 95%.
Collapse
Affiliation(s)
- Sofia Henninger
- Fraunhofer Institute for Physical Measurement Techniques IPM, 79110 Freiburg, Germany
| | - Maximilian Kellner
- Fraunhofer Institute for Physical Measurement Techniques IPM, 79110 Freiburg, Germany
- Department of Sustainable Systems Engineering INATECH, Albert Ludwigs University Freiburg, 79110 Freiburg, Germany
| | - Benedikt Rombach
- Fraunhofer Institute for Physical Measurement Techniques IPM, 79110 Freiburg, Germany
| | - Alexander Reiterer
- Fraunhofer Institute for Physical Measurement Techniques IPM, 79110 Freiburg, Germany
- Department of Sustainable Systems Engineering INATECH, Albert Ludwigs University Freiburg, 79110 Freiburg, Germany
| |
Collapse
|
6
|
Yousefirizi F, Shiri I, O JH, Bloise I, Martineau P, Wilson D, Bénard F, Sehn LH, Savage KJ, Zaidi H, Uribe CF, Rahmim A. Semi-supervised learning towards automated segmentation of PET images with limited annotations: application to lymphoma patients. Phys Eng Sci Med 2024; 47:833-849. [PMID: 38512435 DOI: 10.1007/s13246-024-01408-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2023] [Accepted: 02/18/2024] [Indexed: 03/23/2024]
Abstract
Manual segmentation poses a time-consuming challenge for disease quantification, therapy evaluation, treatment planning, and outcome prediction. Convolutional neural networks (CNNs) hold promise in accurately identifying tumor locations and boundaries in PET scans. However, a major hurdle is the extensive amount of supervised and annotated data necessary for training. To overcome this limitation, this study explores semi-supervised approaches utilizing unlabeled data, specifically focusing on PET images of diffuse large B-cell lymphoma (DLBCL) and primary mediastinal large B-cell lymphoma (PMBCL) obtained from two centers. We considered 2-[18F]FDG PET images of 292 patients PMBCL (n = 104) and DLBCL (n = 188) (n = 232 for training and validation, and n = 60 for external testing). We harnessed classical wisdom embedded in traditional segmentation methods, such as the fuzzy clustering loss function (FCM), to tailor the training strategy for a 3D U-Net model, incorporating both supervised and unsupervised learning approaches. Various supervision levels were explored, including fully supervised methods with labeled FCM and unified focal/Dice loss, unsupervised methods with robust FCM (RFCM) and Mumford-Shah (MS) loss, and semi-supervised methods combining FCM with supervised Dice loss (MS + Dice) or labeled FCM (RFCM + FCM). The unified loss function yielded higher Dice scores (0.73 ± 0.11; 95% CI 0.67-0.8) than Dice loss (p value < 0.01). Among the semi-supervised approaches, RFCM + αFCM (α = 0.3) showed the best performance, with Dice score of 0.68 ± 0.10 (95% CI 0.45-0.77), outperforming MS + αDice for any supervision level (any α) (p < 0.01). Another semi-supervised approach with MS + αDice (α = 0.2) achieved Dice score of 0.59 ± 0.09 (95% CI 0.44-0.76) surpassing other supervision levels (p < 0.01). Given the time-consuming nature of manual delineations and the inconsistencies they may introduce, semi-supervised approaches hold promise for automating medical imaging segmentation workflows.
Collapse
Affiliation(s)
- Fereshteh Yousefirizi
- Department of Integrative Oncology, BC Cancer Research Institute, Vancouver, BC, Canada.
| | - Isaac Shiri
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, Switzerland
| | - Joo Hyun O
- College of Medicine, Seoul St. Mary's Hospital, The Catholic University of Korea, Seoul, Republic of Korea
| | | | | | - Don Wilson
- BC Cancer, Vancouver, BC, Canada
- Department of Radiology, University of British Columbia, Vancouver, Canada
| | | | - Laurie H Sehn
- BC Cancer, Vancouver, BC, Canada
- Centre for Lymphoid Cancer, BC Cancer, Vancouver, Canada
| | - Kerry J Savage
- BC Cancer, Vancouver, BC, Canada
- Centre for Lymphoid Cancer, BC Cancer, Vancouver, Canada
| | - Habib Zaidi
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, Geneva, Switzerland
- University Medical Center Groningen, University of Groningens, Groningen, Netherlands
- Department of Nuclear Medicine, University of Southern Denmark, Vancouver, Odense, Denmark
- University Research and Innovation Center, Óbuda University, Budapest, Hungary
| | - Carlos F Uribe
- Department of Integrative Oncology, BC Cancer Research Institute, Vancouver, BC, Canada
- BC Cancer, Vancouver, BC, Canada
- Department of Radiology, University of British Columbia, Vancouver, Canada
| | - Arman Rahmim
- Department of Integrative Oncology, BC Cancer Research Institute, Vancouver, BC, Canada
- BC Cancer, Vancouver, BC, Canada
- Department of Radiology, University of British Columbia, Vancouver, Canada
- Departments of Physics and Biomedical Engineering, University of British Columbia, Vancouver, Canada
| |
Collapse
|
7
|
d'Albenzio G, Kamkova Y, Naseem R, Ullah M, Colonnese S, Cheikh FA, Kumar RP. A dual-encoder double concatenation Y-shape network for precise volumetric liver and lesion segmentation. Comput Biol Med 2024; 179:108870. [PMID: 39024904 DOI: 10.1016/j.compbiomed.2024.108870] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/26/2023] [Revised: 07/07/2024] [Accepted: 07/08/2024] [Indexed: 07/20/2024]
Abstract
Accurate segmentation of the liver and tumors from CT volumes is crucial for hepatocellular carcinoma diagnosis and pre-operative resection planning. Despite advances in deep learning-based methods for abdominal CT images, fully-automated segmentation remains challenging due to class imbalance and structural variations, often requiring cascaded approaches that incur significant computational costs. In this paper, we present the Dual-Encoder Double Concatenation Network (DEDC-Net) for simultaneous segmentation of the liver and its tumors. DEDC-Net leverages both residual and skip connections to enhance feature reuse and optimize performance in liver and tumor segmentation tasks. Extensive qualitative and quantitative experiments on the LiTS dataset demonstrate that DEDC-Net outperforms existing state-of-the-art liver segmentation methods. An ablation study was conducted to evaluate different encoder backbones - specifically VGG19 and ResNet - and the impact of incorporating an attention mechanism. Our results indicate that DEDC-Net, without any additional attention gates, achieves a superior mean Dice Score (DS) of 0.898 for liver segmentation. Moreover, integrating residual connections into one encoder yielded the highest DS for tumor segmentation tasks. The robustness of our proposed network was further validated on two additional, unseen CT datasets: IDCARDb-01 and COMET. Our model demonstrated superior lesion segmentation capabilities, particularly on IRCADb-01, achieving a DS of 0.629. The code implementation is publicly available at this website.
Collapse
Affiliation(s)
- Gabriella d'Albenzio
- The Intervention Center, Oslo University Hospital, 0slo, Norway; Department of Informatics, University of Oslo, Oslo, Norway.
| | - Yuliia Kamkova
- Department of Informatics, University of Oslo, Oslo, Norway; Department of Research and Development, Division of Emergencies and Critical Care, Oslo University Hospital, Oslo, Norway
| | - Rabia Naseem
- COMSATS, University Islamabad, Islamabad, Pakistan
| | - Mohib Ullah
- Department of Computer Science, Norwegian University of Science and Technology, Gjøvik, Norway
| | - Stefania Colonnese
- Department of Information Engineering, Electronics and Telecommunications (DIET), La Sapienza University of Rome, Rome, Italy
| | - Faouzi Alaya Cheikh
- Department of Computer Science, Norwegian University of Science and Technology, Gjøvik, Norway
| | | |
Collapse
|
8
|
Zhong S, Wang W, Feng Q, Zhang Y, Ning Z. Cross-view discrepancy-dependency network for volumetric medical image segmentation. Med Image Anal 2024; 99:103329. [PMID: 39236632 DOI: 10.1016/j.media.2024.103329] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/17/2024] [Revised: 07/28/2024] [Accepted: 08/26/2024] [Indexed: 09/07/2024]
Abstract
The limited data poses a crucial challenge for deep learning-based volumetric medical image segmentation, and many methods have tried to represent the volume by its subvolumes (i.e., multi-view slices) for alleviating this issue. However, such methods generally sacrifice inter-slice spatial continuity. Currently, a promising avenue involves incorporating multi-view information into the network to enhance volume representation learning, but most existing studies tend to overlook the discrepancy and dependency across different views, ultimately limiting the potential of multi-view representations. To this end, we propose a cross-view discrepancy-dependency network (CvDd-Net) to task with volumetric medical image segmentation, which exploits multi-view slice prior to assist volume representation learning and explore view discrepancy and view dependency for performance improvement. Specifically, we develop a discrepancy-aware morphology reinforcement (DaMR) module to effectively learn view-specific representation by mining morphological information (i.e., boundary and position of object). Besides, we design a dependency-aware information aggregation (DaIA) module to adequately harness the multi-view slice prior, enhancing individual view representations of the volume and integrating them based on cross-view dependency. Extensive experiments on four medical image datasets (i.e., Thyroid, Cervix, Pancreas, and Glioma) demonstrate the efficacy of the proposed method on both fully-supervised and semi-supervised tasks.
Collapse
Affiliation(s)
- Shengzhou Zhong
- School of Biomedical Engineering, Southern Medical University, Guangzhou, Guangdong, 510515, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Guangzhou, Guangdong, 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou 510515, China.
| | - Wenxu Wang
- School of Biomedical Engineering, Southern Medical University, Guangzhou, Guangdong, 510515, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Guangzhou, Guangdong, 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou 510515, China
| | - Qianjin Feng
- School of Biomedical Engineering, Southern Medical University, Guangzhou, Guangdong, 510515, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Guangzhou, Guangdong, 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou 510515, China
| | - Yu Zhang
- School of Biomedical Engineering, Southern Medical University, Guangzhou, Guangdong, 510515, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Guangzhou, Guangdong, 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou 510515, China.
| | - Zhenyuan Ning
- School of Biomedical Engineering, Southern Medical University, Guangzhou, Guangdong, 510515, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Guangzhou, Guangdong, 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou 510515, China.
| |
Collapse
|
9
|
Wang W, Mao Q, Tian Y, Zhang Y, Xiang Z, Ren L. FMD-UNet: fine-grained feature squeeze and multiscale cascade dilated semantic aggregation dual-decoder UNet for COVID-19 lung infection segmentation from CT images. Biomed Phys Eng Express 2024; 10:055031. [PMID: 39142295 DOI: 10.1088/2057-1976/ad6f12] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2024] [Accepted: 08/14/2024] [Indexed: 08/16/2024]
Abstract
With the advancement of computer-aided diagnosis, the automatic segmentation of COVID-19 infection areas holds great promise for assisting in the timely diagnosis and recovery of patients in clinical practice. Currently, methods relying on U-Net face challenges in effectively utilizing fine-grained semantic information from input images and bridging the semantic gap between the encoder and decoder. To address these issues, we propose an FMD-UNet dual-decoder U-Net network for COVID-19 infection segmentation, which integrates a Fine-grained Feature Squeezing (FGFS) decoder and a Multi-scale Dilated Semantic Aggregation (MDSA) decoder. The FGFS decoder produces fine feature maps through the compression of fine-grained features and a weighted attention mechanism, guiding the model to capture detailed semantic information. The MDSA decoder consists of three hierarchical MDSA modules designed for different stages of input information. These modules progressively fuse different scales of dilated convolutions to process the shallow and deep semantic information from the encoder, and use the extracted feature information to bridge the semantic gaps at various stages, this design captures extensive contextual information while decoding and predicting segmentation, thereby suppressing the increase in model parameters. To better validate the robustness and generalizability of the FMD-UNet, we conducted comprehensive performance evaluations and ablation experiments on three public datasets, and achieved leading Dice Similarity Coefficient (DSC) scores of 84.76, 78.56 and 61.99% in COVID-19 infection segmentation, respectively. Compared to previous methods, the FMD-UNet has fewer parameters and shorter inference time, which also demonstrates its competitiveness.
Collapse
Affiliation(s)
- Wenfeng Wang
- School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai, 201620, People's Republic of China
| | - Qi Mao
- School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai, 201620, People's Republic of China
| | - Yi Tian
- School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai, 201620, People's Republic of China
| | - Yan Zhang
- School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai, 201620, People's Republic of China
| | - Zhenwu Xiang
- School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai, 201620, People's Republic of China
| | - Lijia Ren
- School of Electronic and Electrical Engineering, Shanghai University of Engineering Science, Shanghai, 201620, People's Republic of China
| |
Collapse
|
10
|
Zhou Y, Xie Y, Cai N, Liang Y, Gong R, Wang P. mm3DSNet: multi-scale and multi-feedforward self-attention 3D segmentation network for CT scans of hepatobiliary ducts. Med Biol Eng Comput 2024:10.1007/s11517-024-03183-z. [PMID: 39177918 DOI: 10.1007/s11517-024-03183-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/19/2023] [Accepted: 08/06/2024] [Indexed: 08/24/2024]
Abstract
Image segmentation is a key step of the 3D reconstruction of the hepatobiliary duct tree, which is significant for preoperative planning. In this paper, a novel 3D U-Net variant is designed for CT image segmentation of hepatobiliary ducts from the abdominal CT scans, which is composed of a 3D encoder-decoder and a 3D multi-feedforward self-attention module (MFSAM). To well sufficient semantic and spatial features with high inference speed, the 3D ConvNeXt block is designed as the 3D extension of the 2D ConvNeXt. To improve the ability of semantic feature extraction, the MFSAM is designed to transfer the semantic and spatial features at different scales from the encoder to the decoder. Also, to balance the losses for the voxels and the edges of the hepatobiliary ducts, a boundary-aware overlap cross-entropy loss is proposed by combining the cross-entropy loss, the Dice loss, and the boundary loss. Experimental results indicate that the proposed method is superior to some existing deep networks as well as the radiologist without rich experience in terms of CT segmentation of hepatobiliary ducts, with a segmentation performance of 76.54% Dice and 6.56 HD.
Collapse
Affiliation(s)
- Yinghong Zhou
- School of Information Engineering, Guangdong University of Technology, Guangzhou, 510006, China
| | - Yiying Xie
- School of Information Engineering, Guangdong University of Technology, Guangzhou, 510006, China
| | - Nian Cai
- School of Information Engineering, Guangdong University of Technology, Guangzhou, 510006, China.
| | - Yuchen Liang
- School of Information Engineering, Guangdong University of Technology, Guangzhou, 510006, China
| | - Ruifeng Gong
- School of Information Engineering, Guangdong University of Technology, Guangzhou, 510006, China
| | - Ping Wang
- Department of Hepatobiliary Surgery in the First Affiliated Hospital of Guangzhou Medical University, Guangzhou, 510120, China.
| |
Collapse
|
11
|
Pak DH, Liu M, Kim T, Ozturk C, McKay R, Roche ET, Gleason R, Duncan JS. Robust automated calcification meshing for personalized cardiovascular biomechanics. NPJ Digit Med 2024; 7:213. [PMID: 39143242 PMCID: PMC11324740 DOI: 10.1038/s41746-024-01202-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2024] [Accepted: 07/26/2024] [Indexed: 08/16/2024] Open
Abstract
Calcification has significant influence over cardiovascular diseases and interventions. Detailed characterization of calcification is thus desired for predictive modeling, but calcium deposits on cardiovascular structures are still often manually reconstructed for physics-driven simulations. This poses a major bottleneck for large-scale adoption of computational simulations for research or clinical use. To address this, we propose an end-to-end automated image-to-mesh algorithm that enables robust incorporation of patient-specific calcification onto a given cardiovascular tissue mesh. The algorithm provides a substantial speed-up from several hours of manual meshing to ~1 min of automated computation, and it solves an important problem that cannot be addressed with recent template-based meshing techniques. We validated our final calcified tissue meshes with extensive simulations, demonstrating our ability to accurately model patient-specific aortic stenosis and Transcatheter Aortic Valve Replacement. Our method may serve as an important tool for accelerating the development and usage of personalized cardiovascular biomechanics.
Collapse
Affiliation(s)
- Daniel H Pak
- Yale University, 300 Cedar St, New Haven, CT, 06511, USA.
| | - Minliang Liu
- Texas Tech University, 805 Boston Avenue, Lubbock, TX, 79409, USA
| | - Theodore Kim
- Yale University, 300 Cedar St, New Haven, CT, 06511, USA
| | - Caglar Ozturk
- Massachusetts Institute of Technology, 45 Carleton St, Cambridge, MA, 02142, USA
- University of Southampton, University Road, Southampton, SO17 1BJ, UK
| | - Raymond McKay
- Hartford Hospital, 85 Seymour St, Hartford, CT, 06106, USA
| | - Ellen T Roche
- Massachusetts Institute of Technology, 45 Carleton St, Cambridge, MA, 02142, USA
| | - Rudolph Gleason
- Georgia Institute of Technology, 315 Ferst Dr NW, Atlanta, GA, 30332, USA
| | - James S Duncan
- Yale University, 300 Cedar St, New Haven, CT, 06511, USA
| |
Collapse
|
12
|
Huang Y, Gomaa A, Höfler D, Schubert P, Gaipl U, Frey B, Fietkau R, Bert C, Putz F. Principles of artificial intelligence in radiooncology. Strahlenther Onkol 2024:10.1007/s00066-024-02272-0. [PMID: 39105746 DOI: 10.1007/s00066-024-02272-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/26/2024] [Accepted: 06/17/2024] [Indexed: 08/07/2024]
Abstract
PURPOSE In the rapidly expanding field of artificial intelligence (AI) there is a wealth of literature detailing the myriad applications of AI, particularly in the realm of deep learning. However, a review that elucidates the technical principles of deep learning as relevant to radiation oncology in an easily understandable manner is still notably lacking. This paper aims to fill this gap by providing a comprehensive guide to the principles of deep learning that is specifically tailored toward radiation oncology. METHODS In light of the extensive variety of AI methodologies, this review selectively concentrates on the specific domain of deep learning. It emphasizes the principal categories of deep learning models and delineates the methodologies for training these models effectively. RESULTS This review initially delineates the distinctions between AI and deep learning as well as between supervised and unsupervised learning. Subsequently, it elucidates the fundamental principles of major deep learning models, encompassing multilayer perceptrons (MLPs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), transformers, generative adversarial networks (GANs), diffusion-based generative models, and reinforcement learning. For each category, it presents representative networks alongside their specific applications in radiation oncology. Moreover, the review outlines critical factors essential for training deep learning models, such as data preprocessing, loss functions, optimizers, and other pivotal training parameters including learning rate and batch size. CONCLUSION This review provides a comprehensive overview of deep learning principles tailored toward radiation oncology. It aims to enhance the understanding of AI-based research and software applications, thereby bridging the gap between complex technological concepts and clinical practice in radiation oncology.
Collapse
Affiliation(s)
- Yixing Huang
- Department of Radiation Oncology, Universitätsklinikum Erlangen, Friedrich-Alexander Universität Erlangen-Nürnberg, 91054, Erlangen, Germany.
- Comprehensive Cancer Center Erlangen-EMN (CCC ER-EMN), 91054, Erlangen, Germany.
| | - Ahmed Gomaa
- Department of Radiation Oncology, Universitätsklinikum Erlangen, Friedrich-Alexander Universität Erlangen-Nürnberg, 91054, Erlangen, Germany
- Comprehensive Cancer Center Erlangen-EMN (CCC ER-EMN), 91054, Erlangen, Germany
| | - Daniel Höfler
- Department of Radiation Oncology, Universitätsklinikum Erlangen, Friedrich-Alexander Universität Erlangen-Nürnberg, 91054, Erlangen, Germany
- Comprehensive Cancer Center Erlangen-EMN (CCC ER-EMN), 91054, Erlangen, Germany
| | - Philipp Schubert
- Department of Radiation Oncology, Universitätsklinikum Erlangen, Friedrich-Alexander Universität Erlangen-Nürnberg, 91054, Erlangen, Germany
- Comprehensive Cancer Center Erlangen-EMN (CCC ER-EMN), 91054, Erlangen, Germany
| | - Udo Gaipl
- Comprehensive Cancer Center Erlangen-EMN (CCC ER-EMN), 91054, Erlangen, Germany
- Translational Radiobiology, Department of Radiation Oncology, Universitätsklinikum Erlangen, Friedrich-Alexander Universität Erlangen-Nürnberg, 91054, Erlangen, Germany
| | - Benjamin Frey
- Comprehensive Cancer Center Erlangen-EMN (CCC ER-EMN), 91054, Erlangen, Germany
- Translational Radiobiology, Department of Radiation Oncology, Universitätsklinikum Erlangen, Friedrich-Alexander Universität Erlangen-Nürnberg, 91054, Erlangen, Germany
| | - Rainer Fietkau
- Department of Radiation Oncology, Universitätsklinikum Erlangen, Friedrich-Alexander Universität Erlangen-Nürnberg, 91054, Erlangen, Germany
- Comprehensive Cancer Center Erlangen-EMN (CCC ER-EMN), 91054, Erlangen, Germany
| | - Christoph Bert
- Department of Radiation Oncology, Universitätsklinikum Erlangen, Friedrich-Alexander Universität Erlangen-Nürnberg, 91054, Erlangen, Germany
- Comprehensive Cancer Center Erlangen-EMN (CCC ER-EMN), 91054, Erlangen, Germany
| | - Florian Putz
- Department of Radiation Oncology, Universitätsklinikum Erlangen, Friedrich-Alexander Universität Erlangen-Nürnberg, 91054, Erlangen, Germany
- Comprehensive Cancer Center Erlangen-EMN (CCC ER-EMN), 91054, Erlangen, Germany
| |
Collapse
|
13
|
Boneš E, Gergolet M, Bohak C, Lesar Ž, Marolt M. Automatic Segmentation and Alignment of Uterine Shapes from 3D Ultrasound Data. Comput Biol Med 2024; 178:108794. [PMID: 38941903 DOI: 10.1016/j.compbiomed.2024.108794] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/31/2023] [Revised: 06/18/2024] [Accepted: 06/19/2024] [Indexed: 06/30/2024]
Abstract
BACKGROUND The uterus is the most important organ in the female reproductive system. Its shape plays a critical role in fertility and pregnancy outcomes. Advances in medical imaging, such as 3D ultrasound, have significantly improved the exploration of the female genital tract, thereby enhancing gynecological healthcare. Despite well-documented data for organs like the liver and heart, large-scale studies on the uterus are lacking. Existing classifications, such as VCUAM and ESHRE/ESGE, provide different definitions for normal uterine shapes but are not based on real-world measurements. Moreover, the lack of comprehensive datasets significantly hinders research in this area. Our research, part of the larger NURSE study, aims to fill this gap by establishing the shape of a normal uterus using real-world 3D vaginal ultrasound scans. This will facilitate research into uterine shape abnormalities associated with infertility and recurrent miscarriages. METHODS We developed an automated system for the segmentation and alignment of uterine shapes from 3D ultrasound data, which consists of two steps: automatic segmentation of the uteri in 3D ultrasound scans using deep learning techniques, and alignment of the resulting shapes with standard geometrical approaches, enabling the extraction of the normal shape for future analysis. The system was trained and validated on a comprehensive dataset of 3D ultrasound images from multiple medical centers. Its performance was evaluated by comparing the automated results with manual annotations provided by expert clinicians. RESULTS The presented approach demonstrated high accuracy in segmenting and aligning uterine shapes from 3D ultrasound data. The segmentation achieved an average Dice similarity coefficient (DSC) of 0.90. Our method for aligning uterine shapes showed minimal translation and rotation errors compared to traditional methods, with the preliminary average shape exhibiting characteristics consistent with expert findings of a normal uterus. CONCLUSION We have presented an approach to automatically segment and align uterine shapes from 3D ultrasound data. We trained a deep learning nnU-Net model that achieved high accuracy and proposed an alignment method using a combination of standard geometrical techniques. Additionally, we have created a publicly available dataset of 3D transvaginal ultrasound volumes with manual annotations of uterine cavities to support further research and development in this field. The dataset and the trained models are available at https://github.com/UL-FRI-LGM/UterUS.
Collapse
Affiliation(s)
- Eva Boneš
- University of Ljubljana, Faculty of Computer and Information Science, Večna pot 113, Ljubljana, 1000, Slovenia.
| | - Marco Gergolet
- University of Ljubljana, Faculty of Medicine, Vrazov trg 2, Ljubljana, 1000, Slovenia.
| | - Ciril Bohak
- University of Ljubljana, Faculty of Computer and Information Science, Večna pot 113, Ljubljana, 1000, Slovenia; King Abdullah University of Science and Technology, Visual Computing Center, Thuwal, 23955-6900, Saudi Arabia.
| | - Žiga Lesar
- University of Ljubljana, Faculty of Computer and Information Science, Večna pot 113, Ljubljana, 1000, Slovenia.
| | - Matija Marolt
- University of Ljubljana, Faculty of Computer and Information Science, Večna pot 113, Ljubljana, 1000, Slovenia.
| |
Collapse
|
14
|
Song Y, Teoh JYC, Choi KS, Qin J. Dynamic Loss Weighting for Multiorgan Segmentation in Medical Images. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024; 35:10651-10662. [PMID: 37027749 DOI: 10.1109/tnnls.2023.3243241] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
Deep neural networks often suffer from performance inconsistency for multiorgan segmentation in medical images; some organs are segmented far worse than others. The main reason might be organs with different levels of learning difficulty for segmentation mapping, due to variations such as size, texture complexity, shape irregularity, and imaging quality. In this article, we propose a principled class-reweighting algorithm, termed dynamic loss weighting, which dynamically assigns a larger loss weight to organs if they are discriminated as more difficult to learn according to the data and network's status, for forcing the network to learn from them more to maximally promote the performance consistency. This new algorithm uses an extra autoencoder to measure the discrepancy between the segmentation network's output and the ground truth and dynamically estimates the loss weight of organs per the contribution of the organ to the new updated discrepancy. It can capture the variation in organs' learning difficult during training, and it is neither sensitive to data's property nor dependent on human priors. We evaluate this algorithm in two multiorgan segmentation tasks: abdominal organs and head-neck structures, on publicly available datasets, with positive results obtained from extensive experiments which confirm the validity and effectiveness. Source codes are available at: https://github.com/YouyiSong/Dynamic-Loss-Weighting.
Collapse
|
15
|
Hsiao CH, Lin FYS, Sun TL, Liao YY, Wu CH, Lai YC, Wu HP, Liu PR, Xiao BR, Chen CH, Huang Y. Precision and Robust Models on Healthcare Institution Federated Learning for Predicting HCC on Portal Venous CT Images. IEEE J Biomed Health Inform 2024; 28:4674-4687. [PMID: 38739503 DOI: 10.1109/jbhi.2024.3400599] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/16/2024]
Abstract
Hepatocellular carcinoma (HCC), the most common type of liver cancer, poses significant challenges in detection and diagnosis. Medical imaging, especially computed tomography (CT), is pivotal in non-invasively identifying this disease, requiring substantial expertise for interpretation. This research introduces an innovative strategy that integrates two-dimensional (2D) and three-dimensional (3D) deep learning models within a federated learning (FL) framework for precise segmentation of liver and tumor regions in medical images. The study utilized 131 CT scans from the Liver Tumor Segmentation (LiTS) challenge and demonstrated the superior efficiency and accuracy of the proposed Hybrid-ResUNet model with a Dice score of 0.9433 and an AUC of 0.9965 compared to ResNet and EfficientNet models. This FL approach is beneficial for conducting large-scale clinical trials while safeguarding patient privacy across healthcare settings. It facilitates active engagement in problem-solving, data collection, model development, and refinement. The study also addresses data imbalances in the FL context, showing resilience and highlighting local models' robust performance. Future research will concentrate on refining federated learning algorithms and their incorporation into the continuous implementation and deployment (CI/CD) processes in AI system operations, emphasizing the dynamic involvement of clients. We recommend a collaborative human-AI endeavor to enhance feature extraction and knowledge transfer. These improvements are intended to boost equitable and efficient data collaboration across various sectors in practical scenarios, offering a crucial guide for forthcoming research in medical AI.
Collapse
|
16
|
Chen Y, Gao S, Tu P, Chen X. Automatic 3D Teeth Reconstruction From Five Intra-Oral Photos Using Parametric Teeth Model. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 2024; 30:4780-4791. [PMID: 37204961 DOI: 10.1109/tvcg.2023.3277914] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/21/2023]
Abstract
Orthodontic treatment is a lengthy process that requires regular in-person dental monitoring, making remote dental monitoring a viable alternative when face-to-face consultation is not possible. In this study, we propose an improved 3D teeth reconstruction framework that automatically restores the shape, arrangement, and dental occlusion of upper and lower teeth from five intra-oral photographs to aid orthodontists in visualizing the condition of patients in virtual consultations. The framework comprises a parametric model that leverages statistical shape modeling to describe the shape and arrangement of teeth, a modified U-net that extracts teeth contours from intra-oral images, and an iterative process that alternates between finding point correspondences and optimizing a compound loss function to fit the parametric teeth model to predicted teeth contours. We perform a five-fold cross-validation on a dataset of 95 orthodontic cases and report an average Chamfer distance of 1.0121 mm2 and an average Dice similarity coefficient of 0.7672 on all the test samples in the cross-validation, demonstrating a significant improvement compared with the previous work. Our teeth reconstruction framework provides a feasible solution for visualizing 3D teeth models in remote orthodontic consultations.
Collapse
|
17
|
Parry R, Wright K, Bellinge JW, Ebert MA, Rowshanfarzad P, Francis RJ, Schultz CJ. Training and assessing convolutional neural network performance in automatic vascular segmentation using Ga-68 DOTATATE PET/CT. THE INTERNATIONAL JOURNAL OF CARDIOVASCULAR IMAGING 2024:10.1007/s10554-024-03171-2. [PMID: 38967895 DOI: 10.1007/s10554-024-03171-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/21/2023] [Accepted: 06/22/2024] [Indexed: 07/06/2024]
Abstract
To evaluate a convolutional neural network's performance (nnU-Net) in the assessment of vascular contours, calcification and PET tracer activity using Ga-68 DOTATATE PET/CT. Patients who underwent Ga-68 DOTATATE PET/CT imaging over a 12-month period for neuroendocrine investigation were included. Manual cardiac and aortic segmentations were performed by an experienced observer. Scans were randomly allocated in ratio 64:16:20 for training, validation and testing of the nnU-Net model. PET tracer uptake and calcium scoring were compared between segmentation methods and different observers. 116 patients (53.5% female) with a median age of 64.5 years (range 23-79) were included. There were strong, positive correlations between all segmentations (mostly r > 0.98). There were no significant differences between manual and AI segmentation of SUVmean for global cardiac (mean ± SD 0.71 ± 0.22 vs. 0.71 ± 0.22; mean diff 0.001 ± 0.008, p > 0.05), ascending aorta (mean ± SD 0.44 ± 0.14 vs. 0.44 ± 0.14; mean diff 0.002 ± 0.01, p > 0.05), aortic arch (mean ± SD 0.44 ± 0.10 vs. 0.43 ± 0.10; mean diff 0.008 ± 0.16, p > 0.05) and descending aorta (mean ± SD < 0.001; 0.58 ± 0.12 vs. 0.57 ± 0.12; mean diff 0.01 ± 0.03, p > 0.05) contours. There was excellent agreement between the majority of manual and AI segmentation measures (r ≥ 0.80) and in all vascular contour calcium scores. Compared with the manual segmentation approach, the CNN required a significantly lower workflow time. AI segmentation of vascular contours using nnU-Net resulted in very similar measures of PET tracer uptake and vascular calcification when compared to an experienced observer and significantly reduced workflow time.
Collapse
Affiliation(s)
- R Parry
- School of Medicine, The University of Western Australia, Perth, Australia.
- Department of Cardiology, Royal Perth Hospital, Perth, Australia.
| | - K Wright
- School of Physics, Mathematics and Computing, The University of Western Australia, Crawley, WA, Australia
| | - J W Bellinge
- School of Medicine, The University of Western Australia, Perth, Australia
- Department of Cardiology, Royal Perth Hospital, Perth, Australia
| | - M A Ebert
- School of Physics, Mathematics and Computing, The University of Western Australia, Crawley, WA, Australia
- Department of Radiation Oncology, Sir Charles Gairdner Hospital, Perth, Australia
- School of Medicine and Population Health, University of Wisconsin, Madison, WI, USA
| | - P Rowshanfarzad
- School of Physics, Mathematics and Computing, The University of Western Australia, Crawley, WA, Australia
| | - R J Francis
- School of Medicine, The University of Western Australia, Perth, Australia
- Department of Nuclear Medicine, Sir Charles Gairdner Hospital, Perth, Australia
| | - C J Schultz
- School of Medicine, The University of Western Australia, Perth, Australia
- Department of Cardiology, Royal Perth Hospital, Perth, Australia
| |
Collapse
|
18
|
Jain S, Dhir R, Sikka G. View adaptive unified self-supervised technique for abdominal organ segmentation. Comput Biol Med 2024; 177:108659. [PMID: 38823366 DOI: 10.1016/j.compbiomed.2024.108659] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/02/2023] [Revised: 03/05/2024] [Accepted: 05/24/2024] [Indexed: 06/03/2024]
Abstract
Automatic abdominal organ segmentation is an essential prerequisite for accurate volumetric analysis, disease diagnosis, and tracking by medical practitioners. However, the deformable shapes, variable locations, overlapping with nearby organs, and similar contrast make the segmentation challenging. Moreover, the requirement of a large manually labeled dataset makes it harder. Hence, a semi-supervised contrastive learning approach is utilized to perform the automatic abdominal organ segmentation. Existing 3D deep learning models based on contrastive learning are not able to capture the 3D context of medical volumetric data along three planes/views: axial, sagittal, and coronal views. In this work, a semi-supervised view-adaptive unified model (VAU-model) is proposed to make the 3D deep learning model as view-adaptive to learn 3D context along each view in a unified manner. This method utilizes the novel optimization function that assists the 3D model to learn the 3D context of volumetric medical data along each view in a single model. The effectiveness of the proposed approach is validated on the three types of datasets: BTCV, NIH, and MSD quantitatively and qualitatively. The results demonstrate that the VAU model achieves an average Dice score of 81.61% which is a 3.89% improvement compared to the previous best results for pancreas segmentation in multi-organ dataset BTCV. It also achieves an average Dice score of 77.76% and 76.76% for the pancreas under the single organ non-pathological NIH dataset, and pathological MSD dataset.
Collapse
Affiliation(s)
- Suchi Jain
- Computer Science and Engineering, Dr. B.R. Ambedkar National Institute of Technology, Jalandhar, Punjab, 144008, India.
| | - Renu Dhir
- Computer Science and Engineering, Dr. B.R. Ambedkar National Institute of Technology, Jalandhar, Punjab, 144008, India
| | - Geeta Sikka
- Computer Science and Engineering, National Institute of Technology, Delhi, 110036, India
| |
Collapse
|
19
|
Liu X, Qu L, Xie Z, Zhao J, Shi Y, Song Z. Towards more precise automatic analysis: a systematic review of deep learning-based multi-organ segmentation. Biomed Eng Online 2024; 23:52. [PMID: 38851691 PMCID: PMC11162022 DOI: 10.1186/s12938-024-01238-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2023] [Accepted: 04/11/2024] [Indexed: 06/10/2024] Open
Abstract
Accurate segmentation of multiple organs in the head, neck, chest, and abdomen from medical images is an essential step in computer-aided diagnosis, surgical navigation, and radiation therapy. In the past few years, with a data-driven feature extraction approach and end-to-end training, automatic deep learning-based multi-organ segmentation methods have far outperformed traditional methods and become a new research topic. This review systematically summarizes the latest research in this field. We searched Google Scholar for papers published from January 1, 2016 to December 31, 2023, using keywords "multi-organ segmentation" and "deep learning", resulting in 327 papers. We followed the PRISMA guidelines for paper selection, and 195 studies were deemed to be within the scope of this review. We summarized the two main aspects involved in multi-organ segmentation: datasets and methods. Regarding datasets, we provided an overview of existing public datasets and conducted an in-depth analysis. Concerning methods, we categorized existing approaches into three major classes: fully supervised, weakly supervised and semi-supervised, based on whether they require complete label information. We summarized the achievements of these methods in terms of segmentation accuracy. In the discussion and conclusion section, we outlined and summarized the current trends in multi-organ segmentation.
Collapse
Affiliation(s)
- Xiaoyu Liu
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, 138 Yixueyuan Road, Shanghai, 200032, People's Republic of China
- Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Shanghai, 200032, China
| | - Linhao Qu
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, 138 Yixueyuan Road, Shanghai, 200032, People's Republic of China
- Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Shanghai, 200032, China
| | - Ziyue Xie
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, 138 Yixueyuan Road, Shanghai, 200032, People's Republic of China
- Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Shanghai, 200032, China
| | - Jiayue Zhao
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, 138 Yixueyuan Road, Shanghai, 200032, People's Republic of China
- Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Shanghai, 200032, China
| | - Yonghong Shi
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, 138 Yixueyuan Road, Shanghai, 200032, People's Republic of China.
- Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Shanghai, 200032, China.
| | - Zhijian Song
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, 138 Yixueyuan Road, Shanghai, 200032, People's Republic of China.
- Shanghai Key Laboratory of Medical Image Computing and Computer Assisted Intervention, Shanghai, 200032, China.
| |
Collapse
|
20
|
Casali N, Scalco E, Taccogna MG, Lauretani F, Porcelli S, Ciuni A, Mastropietro A, Rizzo G. Positional contrastive learning for improved thigh muscle segmentation in MR images. NMR IN BIOMEDICINE 2024:e5197. [PMID: 38822595 DOI: 10.1002/nbm.5197] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/28/2023] [Revised: 05/02/2024] [Accepted: 05/13/2024] [Indexed: 06/03/2024]
Abstract
The accurate segmentation of individual muscles is essential for quantitative MRI analysis of thigh images. Deep learning methods have achieved state-of-the-art results in segmentation, but they require large numbers of labeled data to perform well. However, labeling individual thigh muscles slice by slice for numerous volumes is a laborious and time-consuming task, which limits the availability of annotated datasets. To address this challenge, self-supervised learning (SSL) emerges as a promising technique to enhance model performance by pretraining the model on unlabeled data. A recent approach, called positional contrastive learning, exploits the information given by the axial position of the slices to learn features transferable on the segmentation task. The aim of this work was to propose positional contrastive SSL for the segmentation of individual thigh muscles from MRI acquisitions in a population of elderly healthy subjects and to evaluate it on different levels of limited annotated data. An unlabeled dataset of 72 T1w MRI thigh acquisitions was available for SSL pretraining, while a labeled dataset of 52 volumes was employed for the final segmentation task, split into training and test sets. The effectiveness of SSL pretraining to fine-tune a U-Net architecture for thigh muscle segmentation was compared with that of a randomly initialized model (RND), considering an increasing number of annotated volumes (S = 1, 2, 5, 10, 20, 30, 40). Our results demonstrated that SSL yields substantial improvements in Dice similarity coefficient (DSC) when using a very limited number of labeled volumes (e.g., forS $$ S $$ = 1, DSC 0.631 versus 0.530 for SSL and RND, respectively). Moreover, enhancements are achievable even when utilizing the full number of labeled subjects, with DSC = 0.927 for SSL and 0.924 for RND. In conclusion, positional contrastive SSL was effective in obtaining more accurate thigh muscle segmentation, even with a very low number of labeled data, with a potential impact of speeding up the annotation process in clinics.
Collapse
Affiliation(s)
- Nicola Casali
- Institute of Intelligent Industrial Technologies and Systems for Advanced Manufacturing, National Research Council, Milan, Italy
- Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, Italy
| | - Elisa Scalco
- Institute of Biomedical Technologies, National Research Council, Segrate, Italy
| | | | - Fulvio Lauretani
- Department of Medicine and Surgery, University of Parma, Parma, Italy
- Geriatric Clinic Unit, Geriatric-Rehabilitation Department, Parma University Hospital, Parma, Italy
| | - Simone Porcelli
- Department of Molecular Medicine, University of Pavia, Pavia, Italy
| | - Andrea Ciuni
- Department of Radiologic Sciences, Parma University Hospital, Parma, Italy
| | - Alfonso Mastropietro
- Institute of Intelligent Industrial Technologies and Systems for Advanced Manufacturing, National Research Council, Milan, Italy
| | - Giovanna Rizzo
- Institute of Intelligent Industrial Technologies and Systems for Advanced Manufacturing, National Research Council, Milan, Italy
| |
Collapse
|
21
|
Wang Z, Cao N, Sun J, Zhang H, Zhang S, Ding J, Xie K, Gao L, Ni X. Uncertainty estimation- and attention-based semi-supervised models for automatically delineate clinical target volume in CBCT images of breast cancer. Radiat Oncol 2024; 19:66. [PMID: 38811994 DOI: 10.1186/s13014-024-02455-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2023] [Accepted: 05/14/2024] [Indexed: 05/31/2024] Open
Abstract
OBJECTIVES Accurate segmentation of the clinical target volume (CTV) of CBCT images can observe the changes of CTV during patients' radiotherapy, and lay a foundation for the subsequent implementation of adaptive radiotherapy (ART). However, segmentation is challenging due to the poor quality of CBCT images and difficulty in obtaining target volumes. An uncertainty estimation- and attention-based semi-supervised model called residual convolutional block attention-uncertainty aware mean teacher (RCBA-UAMT) was proposed to delineate the CTV in cone-beam computed tomography (CBCT) images of breast cancer automatically. METHODS A total of 60 patients who undergone radiotherapy after breast-conserving surgery were enrolled in this study, which involved 60 planning CTs and 380 CBCTs. RCBA-UAMT was proposed by integrating residual and attention modules in the backbone network 3D UNet. The attention module can adjust channel and spatial weights of the extracted image features. The proposed design can train the model and segment CBCT images with a small amount of labeled data (5%, 10%, and 20%) and a large amount of unlabeled data. Four types of evaluation metrics, namely, dice similarity coefficient (DSC), Jaccard, average surface distance (ASD), and 95% Hausdorff distance (95HD), are used to assess the model segmentation performance quantitatively. RESULTS The proposed method achieved average DSC, Jaccard, 95HD, and ASD of 82%, 70%, 8.93, and 1.49 mm for CTV delineation on CBCT images of breast cancer, respectively. Compared with the three classical methods of mean teacher, uncertainty-aware mean-teacher and uncertainty rectified pyramid consistency, DSC and Jaccard increased by 7.89-9.33% and 14.75-16.67%, respectively, while 95HD and ASD decreased by 33.16-67.81% and 36.05-75.57%, respectively. The comparative experiment results of the labeled data with different proportions (5%, 10% and 20%) showed significant differences in the DSC, Jaccard, and 95HD evaluation indexes in the labeled data with 5% versus 10% and 5% versus 20%. Moreover, no significant differences were observed in the labeled data with 10% versus 20% among all evaluation indexes. Therefore, we can use only 10% labeled data to achieve the experimental objective. CONCLUSIONS Using the proposed RCBA-UAMT, the CTV of breast cancer CBCT images can be delineated reliably with a small amount of labeled data. These delineated images can be used to observe the changes in CTV and lay the foundation for the follow-up implementation of ART.
Collapse
Affiliation(s)
- Ziyi Wang
- Department of Radiotherapy Oncology, Changzhou No. 2 People's Hospital, Nanjing Medical University, Gehu Road 68#, Wujin District, Changzhou, 213003, Jiangsu, China
- Jiangsu Province Engineering Research Center of Medical Physics, Changzhou, 213003, China
- Medical Physics Research Center, Nanjing Medical University, Changzhou, 213003, China
- Key Laboratory of Medical Physics in Changzhou, Changzhou, 213003, China
| | - Nannan Cao
- Department of Radiotherapy Oncology, Changzhou No. 2 People's Hospital, Nanjing Medical University, Gehu Road 68#, Wujin District, Changzhou, 213003, Jiangsu, China
- Jiangsu Province Engineering Research Center of Medical Physics, Changzhou, 213003, China
- Medical Physics Research Center, Nanjing Medical University, Changzhou, 213003, China
- Key Laboratory of Medical Physics in Changzhou, Changzhou, 213003, China
| | - Jiawei Sun
- Department of Radiotherapy Oncology, Changzhou No. 2 People's Hospital, Nanjing Medical University, Gehu Road 68#, Wujin District, Changzhou, 213003, Jiangsu, China
- Jiangsu Province Engineering Research Center of Medical Physics, Changzhou, 213003, China
- Medical Physics Research Center, Nanjing Medical University, Changzhou, 213003, China
- Key Laboratory of Medical Physics in Changzhou, Changzhou, 213003, China
| | - Heng Zhang
- Department of Radiotherapy Oncology, Changzhou No. 2 People's Hospital, Nanjing Medical University, Gehu Road 68#, Wujin District, Changzhou, 213003, Jiangsu, China
- Jiangsu Province Engineering Research Center of Medical Physics, Changzhou, 213003, China
- Medical Physics Research Center, Nanjing Medical University, Changzhou, 213003, China
- Key Laboratory of Medical Physics in Changzhou, Changzhou, 213003, China
| | - Sai Zhang
- Department of Radiotherapy Oncology, Changzhou No. 2 People's Hospital, Nanjing Medical University, Gehu Road 68#, Wujin District, Changzhou, 213003, Jiangsu, China
- Jiangsu Province Engineering Research Center of Medical Physics, Changzhou, 213003, China
- Medical Physics Research Center, Nanjing Medical University, Changzhou, 213003, China
- Key Laboratory of Medical Physics in Changzhou, Changzhou, 213003, China
| | - Jiangyi Ding
- Department of Radiotherapy Oncology, Changzhou No. 2 People's Hospital, Nanjing Medical University, Gehu Road 68#, Wujin District, Changzhou, 213003, Jiangsu, China
- Jiangsu Province Engineering Research Center of Medical Physics, Changzhou, 213003, China
- Medical Physics Research Center, Nanjing Medical University, Changzhou, 213003, China
- Key Laboratory of Medical Physics in Changzhou, Changzhou, 213003, China
| | - Kai Xie
- Department of Radiotherapy Oncology, Changzhou No. 2 People's Hospital, Nanjing Medical University, Gehu Road 68#, Wujin District, Changzhou, 213003, Jiangsu, China
- Jiangsu Province Engineering Research Center of Medical Physics, Changzhou, 213003, China
- Medical Physics Research Center, Nanjing Medical University, Changzhou, 213003, China
- Key Laboratory of Medical Physics in Changzhou, Changzhou, 213003, China
| | - Liugang Gao
- Department of Radiotherapy Oncology, Changzhou No. 2 People's Hospital, Nanjing Medical University, Gehu Road 68#, Wujin District, Changzhou, 213003, Jiangsu, China
- Jiangsu Province Engineering Research Center of Medical Physics, Changzhou, 213003, China
- Medical Physics Research Center, Nanjing Medical University, Changzhou, 213003, China
- Key Laboratory of Medical Physics in Changzhou, Changzhou, 213003, China
| | - Xinye Ni
- Department of Radiotherapy Oncology, Changzhou No. 2 People's Hospital, Nanjing Medical University, Gehu Road 68#, Wujin District, Changzhou, 213003, Jiangsu, China.
- Jiangsu Province Engineering Research Center of Medical Physics, Changzhou, 213003, China.
- Medical Physics Research Center, Nanjing Medical University, Changzhou, 213003, China.
- Key Laboratory of Medical Physics in Changzhou, Changzhou, 213003, China.
| |
Collapse
|
22
|
Li S, Wang H, Meng Y, Zhang C, Song Z. Multi-organ segmentation: a progressive exploration of learning paradigms under scarce annotation. Phys Med Biol 2024; 69:11TR01. [PMID: 38479023 DOI: 10.1088/1361-6560/ad33b5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2023] [Accepted: 03/13/2024] [Indexed: 05/21/2024]
Abstract
Precise delineation of multiple organs or abnormal regions in the human body from medical images plays an essential role in computer-aided diagnosis, surgical simulation, image-guided interventions, and especially in radiotherapy treatment planning. Thus, it is of great significance to explore automatic segmentation approaches, among which deep learning-based approaches have evolved rapidly and witnessed remarkable progress in multi-organ segmentation. However, obtaining an appropriately sized and fine-grained annotated dataset of multiple organs is extremely hard and expensive. Such scarce annotation limits the development of high-performance multi-organ segmentation models but promotes many annotation-efficient learning paradigms. Among these, studies on transfer learning leveraging external datasets, semi-supervised learning including unannotated datasets and partially-supervised learning integrating partially-labeled datasets have led the dominant way to break such dilemmas in multi-organ segmentation. We first review the fully supervised method, then present a comprehensive and systematic elaboration of the 3 abovementioned learning paradigms in the context of multi-organ segmentation from both technical and methodological perspectives, and finally summarize their challenges and future trends.
Collapse
Affiliation(s)
- Shiman Li
- Digital Medical Research Center, School of Basic Medical Science, Fudan University, Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, People's Republic of China
| | - Haoran Wang
- Digital Medical Research Center, School of Basic Medical Science, Fudan University, Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, People's Republic of China
| | - Yucong Meng
- Digital Medical Research Center, School of Basic Medical Science, Fudan University, Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, People's Republic of China
| | - Chenxi Zhang
- Digital Medical Research Center, School of Basic Medical Science, Fudan University, Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, People's Republic of China
| | - Zhijian Song
- Digital Medical Research Center, School of Basic Medical Science, Fudan University, Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention, Shanghai 200032, People's Republic of China
| |
Collapse
|
23
|
Liu Z, Qi Q, Wang S, Zhai G. A novel approach to the detection of facial wrinkles: Database, detection algorithm, and evaluation metrics. Comput Biol Med 2024; 174:108431. [PMID: 38626507 DOI: 10.1016/j.compbiomed.2024.108431] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/29/2024] [Revised: 03/19/2024] [Accepted: 04/07/2024] [Indexed: 04/18/2024]
Abstract
Skin wrinkles result from intrinsic aging processes and extrinsic influences, including prolonged exposure to ultraviolet radiation and tobacco smoking. Hence, the identification of wrinkles holds significant importance in skin aging and medical aesthetic investigation. Nevertheless, current methods lack the comprehensiveness to identify facial wrinkles, particularly those that may appear insignificant. Furthermore, the current assessment techniques neglect to consider the blurred boundary of wrinkles and cannot differentiate images with varying resolutions. This research introduces a novel wrinkle detection algorithm and a distance-based loss function to identify full-face wrinkles. Furthermore, we develop a wrinkle detection evaluation metric that assesses outcomes based on curve, location, and gradient similarity. We collected and annotated a dataset for wrinkle detection consisting of 1021 images of Chinese faces. The dataset will be made publicly available to further promote wrinkle detection research. The research demonstrates a substantial enhancement in detecting subtle wrinkles through implementing the proposed method. Furthermore, the suggested evaluation procedure effectively considers the indistinct boundaries of wrinkles and is applicable to images with various resolutions.
Collapse
Affiliation(s)
- Zijia Liu
- Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, 800 Dongchuan RD, Shanghai, 200240, China.
| | - Quan Qi
- Shanghai Institute of Nutrition and Health, Chinese Academy of Sciences, 320 Yueyang RD, Shanghai, 200031, China.
| | - Sijia Wang
- Shanghai Institute of Nutrition and Health, Chinese Academy of Sciences, 320 Yueyang RD, Shanghai, 200031, China.
| | - Guangtao Zhai
- Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, 800 Dongchuan RD, Shanghai, 200240, China.
| |
Collapse
|
24
|
Chen C, Chen Y, Li X, Ning H, Xiao R. Linear semantic transformation for semi-supervised medical image segmentation. Comput Biol Med 2024; 173:108331. [PMID: 38522252 DOI: 10.1016/j.compbiomed.2024.108331] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/29/2024] [Revised: 02/29/2024] [Accepted: 03/17/2024] [Indexed: 03/26/2024]
Abstract
Medical image segmentation is a focus research and foundation in developing intelligent medical systems. Recently, deep learning for medical image segmentation has become a standard process and succeeded significantly, promoting the development of reconstruction, and surgical planning of disease diagnosis. However, semantic learning is often inefficient owing to the lack of supervision of feature maps, resulting in that high-quality segmentation models always rely on numerous and accurate data annotations. Learning robust semantic representation in latent spaces remains a challenge. In this paper, we propose a novel semi-supervised learning framework to learn vital attributes in medical images, which constructs generalized representation from diverse semantics to realize medical image segmentation. We first build a self-supervised learning part that achieves context recovery by reconstructing space and intensity of medical images, which conduct semantic representation for feature maps. Subsequently, we combine semantic-rich feature maps and utilize simple linear semantic transformation to convert them into image segmentation. The proposed framework was tested using five medical segmentation datasets. Quantitative assessments indicate the highest scores of our method on IXI (73.78%), ScaF (47.50%), COVID-19-Seg (50.72%), PC-Seg (65.06%), and Brain-MR (72.63%) datasets. Finally, we compared our method with the latest semi-supervised learning methods and obtained 77.15% and 75.22% DSC values, respectively, ranking first on two representative datasets. The experimental results not only proved that the proposed linear semantic transformation was effectively applied to medical image segmentation, but also presented its simplicity and ease-of-use to pursue robust segmentation in semi-supervised learning. Our code is now open at: https://github.com/QingYunA/Linear-Semantic-Transformation-for-Semi-Supervised-Medical-Image-Segmentation.
Collapse
Affiliation(s)
- Cheng Chen
- School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, 100083, China
| | - Yunqing Chen
- School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, 100083, China
| | - Xiaoheng Li
- School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, 100083, China
| | - Huansheng Ning
- School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, 100083, China
| | - Ruoxiu Xiao
- School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, 100083, China; Shunde Innovation School, University of Science and Technology Beijing, Foshan, 100024, China.
| |
Collapse
|
25
|
Rachmadi MF, Byra M, Skibbe H. A new family of instance-level loss functions for improving instance-level segmentation and detection of white matter hyperintensities in routine clinical brain MRI. Comput Biol Med 2024; 174:108414. [PMID: 38599072 DOI: 10.1016/j.compbiomed.2024.108414] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/17/2023] [Revised: 03/16/2024] [Accepted: 04/03/2024] [Indexed: 04/12/2024]
Abstract
In this study, we introduce "instance loss functions", a new family of loss functions designed to enhance the training of neural networks in the instance-level segmentation and detection of objects in biomedical image data, particularly those of varied numbers and sizes. Intended to be utilized conjointly with traditional loss functions, these proposed functions, prioritize object instances over pixel-by-pixel comparisons. The specific functions, the instance segmentation loss (Linstance), the instance center loss (Lcenter), the false instance rate loss (Lfalse), and the instance proximity loss (Lproximity), serve distinct purposes. Specifically, Linstance improves instance-wise segmentation quality, Lcenter enhances segmentation quality of small instances, Lfalse minimizes the rate of false and missed detections across varied instance sizes, and Lproximity improves detection quality by pulling predicted instances towards the ground truth instances. Through the task of segmenting white matter hyperintensities (WMH) in brain MRI, we benchmarked our proposed instance loss functions, both individually and in combination via an ensemble inference models approach, against traditional pixel-level loss functions. Data were sourced from the Alzheimer's Disease Neuroimaging Initiative (ADNI) and the WMH Segmentation Challenge datasets, which exhibit significant variation in WMH instance sizes. Empirical evaluations demonstrate that combining two instance-level loss functions through ensemble inference models outperforms models using other loss function on both the ADNI and WMH Segmentation Challenge datasets for the segmentation and detection of WMH instances. Further, applying these functions to the segmentation of nuclei in histopathology images demonstrated their effectiveness and generalizability beyond WMH, improving performance even in contexts with less severe instance imbalance.
Collapse
Affiliation(s)
- Muhammad Febrian Rachmadi
- Brain Image Analysis Unit, RIKEN Center for Brain Science, Wako-shi, Japan; Faculty of Computer Science, Universitas Indonesia, Depok, Indonesia.
| | - Michal Byra
- Brain Image Analysis Unit, RIKEN Center for Brain Science, Wako-shi, Japan; Institute of Fundamental Technological Research, Polish Academy of Sciences, Warsaw, Poland
| | - Henrik Skibbe
- Brain Image Analysis Unit, RIKEN Center for Brain Science, Wako-shi, Japan
| |
Collapse
|
26
|
He K, Peng B, Yu W, Liu Y, Liu S, Cheng J, Dai Y. A Novel Mis-Seg-Focus Loss Function Based on a Two-Stage nnU-Net Framework for Accurate Brain Tissue Segmentation. Bioengineering (Basel) 2024; 11:427. [PMID: 38790294 PMCID: PMC11118222 DOI: 10.3390/bioengineering11050427] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2024] [Revised: 04/14/2024] [Accepted: 04/24/2024] [Indexed: 05/26/2024] Open
Abstract
Brain tissue segmentation plays a critical role in the diagnosis, treatment, and study of brain diseases. Accurately identifying these boundaries is essential for improving segmentation accuracy. However, distinguishing boundaries between different brain tissues can be challenging, as they often overlap. Existing deep learning methods primarily calculate the overall segmentation results without adequately addressing local regions, leading to error propagation and mis-segmentation along boundaries. In this study, we propose a novel mis-segmentation-focused loss function based on a two-stage nnU-Net framework. Our approach aims to enhance the model's ability to handle ambiguous boundaries and overlapping anatomical structures, thereby achieving more accurate brain tissue segmentation results. Specifically, the first stage targets the identification of mis-segmentation regions using a global loss function, while the second stage involves defining a mis-segmentation loss function to adaptively adjust the model, thus improving its capability to handle ambiguous boundaries and overlapping anatomical structures. Experimental evaluations on two datasets demonstrate that our proposed method outperforms existing approaches both quantitatively and qualitatively.
Collapse
Affiliation(s)
- Keyi He
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215163, China; (K.H.); (B.P.); (Y.L.); (S.L.)
- The School of Electrical and Electronic Engineering, Changchun University of Technology, Changchun 130012, China;
| | - Bo Peng
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215163, China; (K.H.); (B.P.); (Y.L.); (S.L.)
| | - Weibo Yu
- The School of Electrical and Electronic Engineering, Changchun University of Technology, Changchun 130012, China;
| | - Yan Liu
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215163, China; (K.H.); (B.P.); (Y.L.); (S.L.)
| | - Surui Liu
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215163, China; (K.H.); (B.P.); (Y.L.); (S.L.)
| | - Jian Cheng
- State Key Laboratory of Complex & Critical Software Environment, Beihang University, Beijing 100191, China
- International Innovation Institute, Beihang University, 166 Shuanghongqiao Street, Pingyao Town, Yuhang District, Hangzhou 311115, China
| | - Yakang Dai
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou 215163, China; (K.H.); (B.P.); (Y.L.); (S.L.)
| |
Collapse
|
27
|
He S, Li Q, Li X, Zhang M. SALW-Net: a lightweight convolutional neural network based on self-adjusting loss function for spine MR image segmentation. Med Biol Eng Comput 2024; 62:1247-1264. [PMID: 38172324 DOI: 10.1007/s11517-023-02963-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2023] [Accepted: 10/30/2023] [Indexed: 01/05/2024]
Abstract
Segmentation of intervertebral discs and vertebrae from spine magnetic resonance (MR) images is essential to aid diagnosis algorithms for lumbar disc herniation. Convolutional neural networks (CNN) are effective methods, but often require high computational costs. Designing a lightweight CNN is more suitable for medical sites lacking high-computing power devices, yet due to the unbalanced pixel distribution in spine MR images, the segmentation is often sub-optimal. To address this issue, a lightweight spine segmentation CNN based on a self-adjusting loss function, which is named SALW-Net, is proposed in this study. For SALW-Net, the self-adjusting loss function could dynamically adjust the loss weights of the two branches according to the differences in segmentation results and labels during the training; thus, the ability for learning unbalanced pixels is enhanced. Two separate datasets are used to evaluate the proposed SALW-Net. Specifically, the proposed SALW-Net has fewer parameter numbers than U-net (only 2%) but achieves higher evaluation scores than that of U-net (the average DSC score of SALW-Net is 0.8781, and that of U-net is 0.8482). In addition, the practicality validation for SALW-Net is also proceeding, including deploying the model on a lightweight device and producing an aid diagnosis algorithm based on segmentation results. This means our SALW-Net has clinical application potential for assisted diagnosis in low computational power scenarios.
Collapse
Affiliation(s)
- Siyuan He
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, Jilin, 130022, China
| | - Qi Li
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, Jilin, 130022, China.
- Zhongshan Institute of Changchun University of Science and Technology, Zhongshan, Guangdong, 528437, China.
| | - Xianda Li
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, Jilin, 130022, China
| | - Mengchao Zhang
- Department of Radiology, China-Japan Union Hospital of Jilin University, Changchun, Jilin, 130033, China.
| |
Collapse
|
28
|
Egebjerg JM, Szomek M, Thaysen K, Juhl AD, Kozakijevic S, Werner S, Pratsch C, Schneider G, Kapishnikov S, Ekman A, Röttger R, Wüstner D. Automated quantification of vacuole fusion and lipophagy in Saccharomyces cerevisiae from fluorescence and cryo-soft X-ray microscopy data using deep learning. Autophagy 2024; 20:902-922. [PMID: 37908116 PMCID: PMC11062380 DOI: 10.1080/15548627.2023.2270378] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/06/2023] [Accepted: 10/02/2023] [Indexed: 11/02/2023] Open
Abstract
During starvation in the yeast Saccharomyces cerevisiae vacuolar vesicles fuse and lipid droplets (LDs) can become internalized into the vacuole in an autophagic process named lipophagy. There is a lack of tools to quantitatively assess starvation-induced vacuole fusion and lipophagy in intact cells with high resolution and throughput. Here, we combine soft X-ray tomography (SXT) with fluorescence microscopy and use a deep-learning computational approach to visualize and quantify these processes in yeast. We focus on yeast homologs of mammalian NPC1 (NPC intracellular cholesterol transporter 1; Ncr1 in yeast) and NPC2 proteins, whose dysfunction leads to Niemann Pick type C (NPC) disease in humans. We developed a convolutional neural network (CNN) model which classifies fully fused versus partially fused vacuoles based on fluorescence images of stained cells. This CNN, named Deep Yeast Fusion Network (DYFNet), revealed that cells lacking Ncr1 (ncr1∆ cells) or Npc2 (npc2∆ cells) have a reduced capacity for vacuole fusion. Using a second CNN model, we implemented a pipeline named LipoSeg to perform automated instance segmentation of LDs and vacuoles from high-resolution reconstructions of X-ray tomograms. From that, we obtained 3D renderings of LDs inside and outside of the vacuole in a fully automated manner and additionally measured droplet volume, number, and distribution. We find that ncr1∆ and npc2∆ cells could ingest LDs into vacuoles normally but showed compromised degradation of LDs and accumulation of lipid vesicles inside vacuoles. Our new method is versatile and allows for analysis of vacuole fusion, droplet size and lipophagy in intact cells.Abbreviations: BODIPY493/503: 4,4-difluoro-1,3,5,7,8-pentamethyl-4-bora-3a,4a-diaza-s-Indacene; BPS: bathophenanthrolinedisulfonic acid disodium salt hydrate; CNN: convolutional neural network; DHE; dehydroergosterol; npc2∆, yeast deficient in Npc2; DSC, Dice similarity coefficient; EM, electron microscopy; EVs, extracellular vesicles; FIB-SEM, focused ion beam milling-scanning electron microscopy; FM 4-64, N-(3-triethylammoniumpropyl)-4-(6-[4-{diethylamino} phenyl] hexatrienyl)-pyridinium dibromide; LDs, lipid droplets; Ncr1, yeast homolog of human NPC1 protein; ncr1∆, yeast deficient in Ncr1; NPC, Niemann Pick type C; NPC2, Niemann Pick type C homolog; OD600, optical density at 600 nm; ReLU, rectifier linear unit; PPV, positive predictive value; NPV, negative predictive value; MCC, Matthews correlation coefficient; SXT, soft X-ray tomography; UV, ultraviolet; YPD, yeast extract peptone dextrose.
Collapse
Affiliation(s)
- Jacob Marcus Egebjerg
- Department of Biochemistry and Molecular Biology, University of Southern Denmark, Odense M, Denmark
- Department of Mathematics and Computer Science, University of Southern Denmark, Odense M, Denmark
| | - Maria Szomek
- Department of Biochemistry and Molecular Biology, University of Southern Denmark, Odense M, Denmark
| | - Katja Thaysen
- Department of Biochemistry and Molecular Biology, University of Southern Denmark, Odense M, Denmark
| | - Alice Dupont Juhl
- Department of Biochemistry and Molecular Biology, University of Southern Denmark, Odense M, Denmark
| | - Suzana Kozakijevic
- Department of Biochemistry and Molecular Biology, University of Southern Denmark, Odense M, Denmark
| | - Stephan Werner
- Department of X‑Ray Microscopy, Helmholtz-Zentrum Berlin, Germany and Humboldt-Universität zu Berlin, Institut für Physik, Berlin, Germany
| | - Christoph Pratsch
- Department of X‑Ray Microscopy, Helmholtz-Zentrum Berlin, Germany and Humboldt-Universität zu Berlin, Institut für Physik, Berlin, Germany
| | - Gerd Schneider
- Department of X‑Ray Microscopy, Helmholtz-Zentrum Berlin, Germany and Humboldt-Universität zu Berlin, Institut für Physik, Berlin, Germany
| | - Sergey Kapishnikov
- SiriusXT, 9A Holly Ave. Stillorgan Industrial Park, Blackrock, Co, Dublin, Ireland
| | - Axel Ekman
- Department of Biological and Environmental Science and Nanoscience Centre, University of Jyväskylä, Jyväskylä, Finland
| | - Richard Röttger
- Department of Mathematics and Computer Science, University of Southern Denmark, Odense M, Denmark
| | - Daniel Wüstner
- Department of Biochemistry and Molecular Biology, University of Southern Denmark, Odense M, Denmark
| |
Collapse
|
29
|
He S, Li Q, Li X, Zhang M. A Lightweight Convolutional Neural Network Based on Dynamic Level-Set Loss Function for Spine MR Image Segmentation. J Magn Reson Imaging 2024; 59:1438-1453. [PMID: 37382232 DOI: 10.1002/jmri.28877] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2023] [Revised: 06/09/2023] [Accepted: 06/09/2023] [Indexed: 06/30/2023] Open
Abstract
BACKGROUND Spine MR image segmentation is important foundation for computer-aided diagnostic (CAD) algorithms of spine disorders. Convolutional neural networks segment effectively, but require high computational costs. PURPOSE To design a lightweight model based on dynamic level-set loss function for high segmentation performance. STUDY TYPE Retrospective. POPULATION Four hundred forty-eight subjects (3163 images) from two separate datasets. Dataset-1: 276 subjects/994 images (53.26% female, mean age 49.02 ± 14.09), all for disc degeneration screening, 188 had disc degeneration, 67 had herniated disc. Dataset-2: public dataset with 172 subjects/2169 images, 142 patients with vertebral degeneration, 163 patients with disc degeneration. FIELD STRENGTH/SEQUENCE T2 weighted turbo spin echo sequences at 3T. ASSESSMENT Dynamic Level-set Net (DLS-Net) was compared with four mainstream (including U-net++) and four lightweight models, and manual label made by five radiologists (vertebrae, discs, spinal fluid) used as segmentation evaluation standard. Five-fold cross-validation are used for all experiments. Based on segmentation, a CAD algorithm of lumbar disc was designed for assessing DLS-Net's practicality, and the text annotation (normal, bulging, or herniated) from medical history data were used as evaluation standard. STATISTICAL TESTS All segmentation models were evaluated with DSC, accuracy, precision, and AUC. The pixel numbers of segmented results were compared with manual label using paired t-tests, with P < 0.05 indicating significance. The CAD algorithm was evaluated with accuracy of lumbar disc diagnosis. RESULTS With only 1.48% parameters of U-net++, DLS-Net achieved similar accuracy in both datasets (Dataset-1: DSC 0.88 vs. 0.89, AUC 0.94 vs. 0.94; Dataset-2: DSC 0.86 vs. 0.86, AUC 0.93 vs. 0.93). The segmentation results of DLS-Net showed no significant differences with manual labels in pixel numbers for discs (Dataset-1: 1603.30 vs. 1588.77, P = 0.22; Dataset-2: 863.61 vs. 886.4, P = 0.14) and vertebrae (Dataset-1: 3984.28 vs. 3961.94, P = 0.38; Dataset-2: 4806.91 vs. 4732.85, P = 0.21). Based on DLS-Net's segmentation results, the CAD algorithm achieved higher accuracy than using non-cropped MR images (87.47% vs. 61.82%). DATA CONCLUSION The proposed DLS-Net has fewer parameters but achieves similar accuracy to U-net++, helps CAD algorithm achieve higher accuracy, which facilitates wider application. EVIDENCE LEVEL 2 TECHNICAL EFFICACY: Stage 1.
Collapse
Affiliation(s)
- Siyuan He
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| | - Qi Li
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
- Zhongshan Institute of Changchun University of Science and Technology, Zhongshan, China
| | - Xianda Li
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| | - Mengchao Zhang
- Department of Radiology, China-Japan Union Hospital of Jilin University, Changchun, China
| |
Collapse
|
30
|
Zhang Y, Yu M, Tong C, Zhao Y, Han J. CA-UNet Segmentation Makes a Good Ischemic Stroke Risk Prediction. Interdiscip Sci 2024; 16:58-72. [PMID: 37626263 DOI: 10.1007/s12539-023-00583-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/10/2023] [Revised: 07/13/2023] [Accepted: 07/19/2023] [Indexed: 08/27/2023]
Abstract
Stroke is still the World's second major factor of death, as well as the third major factor of death and disability. Ischemic stroke is a type of stroke, in which early detection and treatment are the keys to preventing ischemic strokes. However, due to the limitation of privacy protection and labeling difficulties, there are only a few studies on the intelligent automatic diagnosis of stroke or ischemic stroke, and the results are unsatisfactory. Therefore, we collect some data and propose a 3D carotid Computed Tomography Angiography (CTA) image segmentation model called CA-UNet for fully automated extraction of carotid arteries. We explore the number of down-sampling times applicable to carotid segmentation and design a multi-scale loss function to resolve the loss of detailed features during the process of down-sampling. Moreover, based on CA-Unet, we propose an ischemic stroke risk prediction model to predict the risk in patients using their 3D CTA images, electronic medical records, and medical history. We have validated the efficacy of our segmentation model and prediction model through comparison tests. Our method can provide reliable diagnoses and results that benefit patients and medical professionals.
Collapse
Affiliation(s)
- Yuqi Zhang
- School of Computer Science and Engineering, Beihang University, Beijing, China
- State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China
| | - Mengbo Yu
- School of Computer Science and Engineering, Beihang University, Beijing, China
- State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China
| | - Chao Tong
- School of Computer Science and Engineering, Beihang University, Beijing, China.
- State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China.
| | - Yanqing Zhao
- Department of Interventional Radiology and Vascular Surgery, Peking University Third Hospital, Beijing, China
| | - Jintao Han
- Department of Interventional Radiology and Vascular Surgery, Peking University Third Hospital, Beijing, China
| |
Collapse
|
31
|
Quinton F, Presles B, Leclerc S, Nodari G, Lopez O, Chevallier O, Pellegrinelli J, Vrigneaud JM, Popoff R, Meriaudeau F, Alberini JL. Navigating the nuances: comparative analysis and hyperparameter optimisation of neural architectures on contrast-enhanced MRI for liver and liver tumour segmentation. Sci Rep 2024; 14:3522. [PMID: 38347017 PMCID: PMC10861452 DOI: 10.1038/s41598-024-53528-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2023] [Accepted: 02/01/2024] [Indexed: 02/15/2024] Open
Abstract
In medical imaging, accurate segmentation is crucial to improving diagnosis, treatment, or both. However, navigating the multitude of available architectures for automatic segmentation can be overwhelming, making it challenging to determine the appropriate type of architecture and tune the most crucial parameters during dataset optimisation. To address this problem, we examined and refined seven distinct architectures for segmenting the liver, as well as liver tumours, with a restricted training collection of 60 3D contrast-enhanced magnetic resonance images (CE-MRI) from the ATLAS dataset. Included in these architectures are convolutional neural networks (CNNs), transformers, and hybrid CNN/transformer architectures. Bayesian search techniques were used for hyperparameter tuning to hasten convergence to the optimal parameter mixes while also minimising the number of trained models. It was unexpected that hybrid models, which typically exhibit superior performance on larger datasets, would exhibit comparable performance to CNNs. The optimisation of parameters contributed to better segmentations, resulting in an average increase of 1.7% and 5.0% in liver and tumour segmentation Dice coefficients, respectively. In conclusion, the findings of this study indicate that hybrid CNN/transformer architectures may serve as a practical substitute for CNNs even in small datasets. This underscores the significance of hyperparameter optimisation.
Collapse
Affiliation(s)
- Felix Quinton
- Institut de Chimie Moléculaire de l'Université de Bourgogne, ICMUB UMR CNRS 6302, Université Bourgogne, 21000, Dijon, France.
| | - Benoit Presles
- Institut de Chimie Moléculaire de l'Université de Bourgogne, ICMUB UMR CNRS 6302, Université Bourgogne, 21000, Dijon, France
| | - Sarah Leclerc
- Institut de Chimie Moléculaire de l'Université de Bourgogne, ICMUB UMR CNRS 6302, Université Bourgogne, 21000, Dijon, France
| | - Guillaume Nodari
- Service de Médecine Nucléaire, Centre Georges-François Leclerc, 21000, Dijon, France
| | - Olivier Lopez
- Service de Radiologie et Imagerie Medicale Diagnostique et Therapeutique, Centre Hospitalier Universitaire, 21000, Dijon, France
| | - Olivier Chevallier
- Service de Radiologie et Imagerie Medicale Diagnostique et Therapeutique, Centre Hospitalier Universitaire, 21000, Dijon, France
| | - Julie Pellegrinelli
- Service de Médecine Nucléaire, Centre Georges-François Leclerc, 21000, Dijon, France
| | - Jean-Marc Vrigneaud
- Institut de Chimie Moléculaire de l'Université de Bourgogne, ICMUB UMR CNRS 6302, Université Bourgogne, 21000, Dijon, France
- Service de Médecine Nucléaire, Centre Georges-François Leclerc, 21000, Dijon, France
| | - Romain Popoff
- Institut de Chimie Moléculaire de l'Université de Bourgogne, ICMUB UMR CNRS 6302, Université Bourgogne, 21000, Dijon, France
- Service de Médecine Nucléaire, Centre Georges-François Leclerc, 21000, Dijon, France
| | - Fabrice Meriaudeau
- Institut de Chimie Moléculaire de l'Université de Bourgogne, ICMUB UMR CNRS 6302, Université Bourgogne, 21000, Dijon, France
| | - Jean-Louis Alberini
- Institut de Chimie Moléculaire de l'Université de Bourgogne, ICMUB UMR CNRS 6302, Université Bourgogne, 21000, Dijon, France
- Service de Médecine Nucléaire, Centre Georges-François Leclerc, 21000, Dijon, France
| |
Collapse
|
32
|
Mei C, Yang X, Zhou M, Zhang S, Chen H, Yang X, Wang L. Semi-supervised image segmentation using a residual-driven mean teacher and an exponential Dice loss. Artif Intell Med 2024; 148:102757. [PMID: 38325920 DOI: 10.1016/j.artmed.2023.102757] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2023] [Revised: 11/13/2023] [Accepted: 12/29/2023] [Indexed: 02/09/2024]
Abstract
Semi-supervised segmentation plays an important role in computer vision and medical image analysis and can alleviate the burden of acquiring abundant expert-annotated images. In this paper, we developed a residual-driven semi-supervised segmentation method (termed RDMT) based on the classical mean teacher (MT) framework by introducing a novel model-level residual perturbation and an exponential Dice (eDice) loss. The introduced perturbation was integrated into the exponential moving average (EMA) scheme to enhance the performance of the MT, while the eDice loss was used to improve the detection sensitivity of a given network to object boundaries. We validated the developed method by applying it to segment 3D Left Atrium (LA) and 2D optic cup (OC) from the public LASC and REFUGE datasets based on the V-Net and U-Net, respectively. Extensive experiments demonstrated that the developed method achieved the average Dice score of 0.8776 and 0.7751, when trained on 10% and 20% labeled images, respectively for the LA and OC regions depicted on the LASC and REFUGE datasets. It significantly outperformed the MT and can compete with several existing semi-supervised segmentation methods (i.e., HCMT, UAMT, DTC and SASS).
Collapse
Affiliation(s)
- Chenyang Mei
- School of Ophthalmology & Optometry, Eye Hospital, Wenzhou Medical University, Wenzhou 325027, China
| | - Xiaoguo Yang
- Department of Neurology, Wenzhou People's Hospital, The Third Affiliated Hospital of Shanghai University, Wenzhou 325041, China
| | - Mi Zhou
- Department of Neurology, Wenzhou People's Hospital, The Third Affiliated Hospital of Shanghai University, Wenzhou 325041, China; School of Medicine, Shanghai University, Shanghai 200444, China
| | - Shaodan Zhang
- School of Ophthalmology & Optometry, Eye Hospital, Wenzhou Medical University, Wenzhou 325027, China
| | - Hao Chen
- School of Ophthalmology & Optometry, Eye Hospital, Wenzhou Medical University, Wenzhou 325027, China
| | - Xiaokai Yang
- Department of Neurology, Wenzhou People's Hospital, The Third Affiliated Hospital of Shanghai University, Wenzhou 325041, China; School of Medicine, Shanghai University, Shanghai 200444, China.
| | - Lei Wang
- School of Ophthalmology & Optometry, Eye Hospital, Wenzhou Medical University, Wenzhou 325027, China.
| |
Collapse
|
33
|
Niu Z, O'Farrell A, Li J, Reffsin S, Jain N, Dardani I, Goyal Y, Raj A. Piscis: a novel loss estimator of the F1 score enables accurate spot detection in fluorescence microscopy images via deep learning. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.01.31.578123. [PMID: 38352551 PMCID: PMC10862914 DOI: 10.1101/2024.01.31.578123] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 02/24/2024]
Abstract
Single-molecule RNA fluorescence in situ hybridization (RNA FISH)-based spatial transcriptomics methods have enabled the accurate quantification of gene expression at single-cell resolution by visualizing transcripts as diffraction-limited spots. While these methods generally scale to large samples, image analysis remains challenging, often requiring manual parameter tuning. We present Piscis, a fully automatic deep learning algorithm for spot detection trained using a novel loss function, the SmoothF1 loss, that approximates the F1 score to directly penalize false positives and false negatives but remains differentiable and hence usable for training by deep learning approaches. Piscis was trained and tested on a diverse dataset composed of 358 manually annotated experimental RNA FISH images representing multiple cell types and 240 additional synthetic images. Piscis outperforms other state-of-the-art spot detection methods, enabling accurate, high-throughput analysis of RNA FISH-derived imaging data without the need for manual parameter tuning.
Collapse
Affiliation(s)
- Zijian Niu
- Department of Chemistry, School of Arts and Sciences, University of Pennsylvania, Philadelphia, PA, USA
- Department of Physics and Astronomy, School of Arts and Sciences, University of Pennsylvania, Philadelphia, PA, USA
| | - Aoife O'Farrell
- Department of Bioengineering, School of Engineering and Applied Sciences, University of Pennsylvania, Philadelphia, PA, USA
| | - Jingxin Li
- Genetics and Epigenetics, Cell and Molecular Biology Graduate Group, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Sam Reffsin
- Department of Bioengineering, School of Engineering and Applied Sciences, University of Pennsylvania, Philadelphia, PA, USA
| | - Naveen Jain
- Genetics and Epigenetics, Cell and Molecular Biology Graduate Group, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Ian Dardani
- Department of Bioengineering, School of Engineering and Applied Sciences, University of Pennsylvania, Philadelphia, PA, USA
| | - Yogesh Goyal
- Department of Bioengineering, School of Engineering and Applied Sciences, University of Pennsylvania, Philadelphia, PA, USA
- Department of Cell and Developmental Biology, Feinberg School of Medicine, Northwestern University, Chicago, IL, USA
- Center for Synthetic Biology, Northwestern University, Chicago, IL, USA
- Robert H. Lurie Comprehensive Cancer Center, Feinberg School of Medicine, Northwestern University, Chicago, IL, USA
| | - Arjun Raj
- Department of Bioengineering, School of Engineering and Applied Sciences, University of Pennsylvania, Philadelphia, PA, USA
- Department of Genetics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| |
Collapse
|
34
|
Ma J, He Y, Li F, Han L, You C, Wang B. Segment anything in medical images. Nat Commun 2024; 15:654. [PMID: 38253604 PMCID: PMC10803759 DOI: 10.1038/s41467-024-44824-z] [Citation(s) in RCA: 47] [Impact Index Per Article: 47.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/24/2023] [Accepted: 01/05/2024] [Indexed: 01/24/2024] Open
Abstract
Medical image segmentation is a critical component in clinical practice, facilitating accurate diagnosis, treatment planning, and disease monitoring. However, existing methods, often tailored to specific modalities or disease types, lack generalizability across the diverse spectrum of medical image segmentation tasks. Here we present MedSAM, a foundation model designed for bridging this gap by enabling universal medical image segmentation. The model is developed on a large-scale medical image dataset with 1,570,263 image-mask pairs, covering 10 imaging modalities and over 30 cancer types. We conduct a comprehensive evaluation on 86 internal validation tasks and 60 external validation tasks, demonstrating better accuracy and robustness than modality-wise specialist models. By delivering accurate and efficient segmentation across a wide spectrum of tasks, MedSAM holds significant potential to expedite the evolution of diagnostic tools and the personalization of treatment plans.
Collapse
Affiliation(s)
- Jun Ma
- Peter Munk Cardiac Centre, University Health Network, Toronto, ON, Canada
- Department of Laboratory Medicine and Pathobiology, University of Toronto, Toronto, ON, Canada
- Vector Institute, Toronto, ON, Canada
| | - Yuting He
- Department of Computer Science, Western University, London, ON, Canada
| | - Feifei Li
- Peter Munk Cardiac Centre, University Health Network, Toronto, ON, Canada
| | - Lin Han
- Tandon School of Engineering, New York University, New York, NY, USA
| | - Chenyu You
- Department of Electrical Engineering, Yale University, New Haven, CT, USA
| | - Bo Wang
- Peter Munk Cardiac Centre, University Health Network, Toronto, ON, Canada.
- Department of Laboratory Medicine and Pathobiology, University of Toronto, Toronto, ON, Canada.
- Vector Institute, Toronto, ON, Canada.
- Department of Computer Science, University of Toronto, Toronto, ON, Canada.
- UHN AI Hub, Toronto, ON, Canada.
| |
Collapse
|
35
|
Xing G, Wang S, Gao J, Li X. Real-time reliable semantic segmentation of thyroid nodules in ultrasound images. Phys Med Biol 2024; 69:025016. [PMID: 38048630 DOI: 10.1088/1361-6560/ad1210] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/12/2023] [Accepted: 12/04/2023] [Indexed: 12/06/2023]
Abstract
Objective.Low efficiency in medical image segmentation is a common issue that limits computer-aided diagnosis development. Due to the varying positions and sizes of nodules, it is not easy to accurately segment ultrasound images. This study aims to propose a segmentation model that maintains high efficiency while improving accuracy.Approach. We propose a novel layer that integrates the advantages of dense connectivity, dilated convolution, and factorized filters to maintain excellent efficiency while improving accuracy. Dense connectivity optimizes feature reuse, dilated convolution redesigns layers, and factorized convolution improves efficiency. Moreover, we propose a loss function optimization method from a pixel perspective to increase the network's accuracy further.Main results.Experiments on the Thyroid dataset show that our method achieves 81.70% intersection-over-union (IoU), 90.50% true positive rate (TPR), and 0.25% false positive rate (FPR). In terms of accuracy, our method outperforms the state-of-the-art methods, with twice faster inference and nearly 400 times fewer parameters. Meanwhile, in a test on an External Thyroid dataset, our method achieves 77.03% IoU, 82.10% TPR, and 0.16% FPR, demonstrating our proposed model's robustness.Significance.We propose a real-time semantic segmentation architecture for thyroid nodule segmentation in ultrasound images called fully convolution dense dilated network (FCDDN). Our method runs fast with a few parameters and is suitable for medical devices requiring real-time segmentation.
Collapse
Affiliation(s)
- Guangxin Xing
- State Key Laboratory of Precision Measuring Technology and Instruments, Tianjin University, Tianjin, People's Republic of China
| | - Shuaijie Wang
- College of Intelligence and Computing, Tianjin University, Tianjin, People's Republic of China
- Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin, People's Republic of China
- Tianjin Key Laboratory of Advanced Networking, Tianjin, People's Republic of China
| | - Jie Gao
- College of Intelligence and Computing, Tianjin University, Tianjin, People's Republic of China
- Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin, People's Republic of China
- Tianjin Key Laboratory of Advanced Networking, Tianjin, People's Republic of China
| | - Xuewei Li
- College of Intelligence and Computing, Tianjin University, Tianjin, People's Republic of China
- Tianjin Key Laboratory of Cognitive Computing and Application, Tianjin, People's Republic of China
- Tianjin Key Laboratory of Advanced Networking, Tianjin, People's Republic of China
| |
Collapse
|
36
|
Yang L, Lei Y, Huang Z, Geng M, Liu Z, Wang B, Luo D, Huang W, Liang D, Pang Z, Hu Z. An interactive nuclei segmentation framework with Voronoi diagrams and weighted convex difference for cervical cancer pathology images. Phys Med Biol 2024; 69:025021. [PMID: 37972412 DOI: 10.1088/1361-6560/ad0d44] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/01/2023] [Accepted: 11/16/2023] [Indexed: 11/19/2023]
Abstract
Objective.Nuclei segmentation is crucial for pathologists to accurately classify and grade cancer. However, this process faces significant challenges, such as the complex background structures in pathological images, the high-density distribution of nuclei, and cell adhesion.Approach.In this paper, we present an interactive nuclei segmentation framework that increases the precision of nuclei segmentation. Our framework incorporates expert monitoring to gather as much prior information as possible and accurately segment complex nucleus images through limited pathologist interaction, where only a small portion of the nucleus locations in each image are labeled. The initial contour is determined by the Voronoi diagram generated from the labeled points, which is then input into an optimized weighted convex difference model to regularize partition boundaries in an image. Specifically, we provide theoretical proof of the mathematical model, stating that the objective function monotonically decreases. Furthermore, we explore a postprocessing stage that incorporates histograms, which are simple and easy to handle and prevent arbitrariness and subjectivity in individual choices.Main results.To evaluate our approach, we conduct experiments on both a cervical cancer dataset and a nasopharyngeal cancer dataset. The experimental results demonstrate that our approach achieves competitive performance compared to other methods.Significance.The Voronoi diagram in the paper serves as prior information for the active contour, providing positional information for individual cells. Moreover, the active contour model achieves precise segmentation results while offering mathematical interpretability.
Collapse
Affiliation(s)
- Lin Yang
- Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, People's Republic of China
- College of Mathematics and Statistics, Henan University, Kaifeng 475004, People's Republic of China
| | - Yuanyuan Lei
- Department of Pathology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital & Shenzhen Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Shenzhen, 518116, People's Republic of China
| | - Zhenxing Huang
- Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, People's Republic of China
| | - Mengxiao Geng
- Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, People's Republic of China
- College of Mathematics and Statistics, Henan University, Kaifeng 475004, People's Republic of China
| | - Zhou Liu
- Department of Radiology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital & Shenzhen Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Shenzhen, 518116, People's Republic of China
| | - Baijie Wang
- Department of Radiology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital & Shenzhen Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Shenzhen, 518116, People's Republic of China
| | - Dehong Luo
- Department of Radiology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital & Shenzhen Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Shenzhen, 518116, People's Republic of China
| | - Wenting Huang
- Department of Pathology, National Cancer Center/National Clinical Research Center for Cancer/Cancer Hospital & Shenzhen Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College, Shenzhen, 518116, People's Republic of China
| | - Dong Liang
- Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, People's Republic of China
- Key Laboratory of Biomedical Imaging Science and System, Chinese Academy of Sciences, Shenzhen 518055, People's Republic of China
| | - Zhifeng Pang
- College of Mathematics and Statistics, Henan University, Kaifeng 475004, People's Republic of China
- Hubei Key Laboratory of Computational Science, Wuhan University, Wuhan 430072, People's Republic of China
| | - Zhanli Hu
- Lauterbur Research Center for Biomedical Imaging, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518055, People's Republic of China
- Key Laboratory of Biomedical Imaging Science and System, Chinese Academy of Sciences, Shenzhen 518055, People's Republic of China
| |
Collapse
|
37
|
Veeramani N, Jayaraman P, Krishankumar R, Ravichandran KS, Gandomi AH. DDCNN-F: double decker convolutional neural network 'F' feature fusion as a medical image classification framework. Sci Rep 2024; 14:676. [PMID: 38182607 PMCID: PMC10770172 DOI: 10.1038/s41598-023-49721-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/23/2023] [Accepted: 12/11/2023] [Indexed: 01/07/2024] Open
Abstract
Melanoma is a severe skin cancer that involves abnormal cell development. This study aims to provide a new feature fusion framework for melanoma classification that includes a novel 'F' Flag feature for early detection. This novel 'F' indicator efficiently distinguishes benign skin lesions from malignant ones known as melanoma. The article proposes an architecture that is built in a Double Decker Convolutional Neural Network called DDCNN future fusion. The network's deck one, known as a Convolutional Neural Network (CNN), finds difficult-to-classify hairy images using a confidence factor termed the intra-class variance score. These hirsute image samples are combined to form a Baseline Separated Channel (BSC). By eliminating hair and using data augmentation techniques, the BSC is ready for analysis. The network's second deck trains the pre-processed BSC and generates bottleneck features. The bottleneck features are merged with features generated from the ABCDE clinical bio indicators to promote classification accuracy. Different types of classifiers are fed to the resulting hybrid fused features with the novel 'F' Flag feature. The proposed system was trained using the ISIC 2019 and ISIC 2020 datasets to assess its performance. The empirical findings expose that the DDCNN feature fusion strategy for exposing malignant melanoma achieved a specificity of 98.4%, accuracy of 93.75%, precision of 98.56%, and Area Under Curve (AUC) value of 0.98. This study proposes a novel approach that can accurately identify and diagnose fatal skin cancer and outperform other state-of-the-art techniques, which is attributed to the DDCNN 'F' Feature fusion framework. Also, this research ascertained improvements in several classifiers when utilising the 'F' indicator, resulting in the highest specificity of + 7.34%.
Collapse
Affiliation(s)
- Nirmala Veeramani
- School of Computing, SASTRA Deemed to Be University, Thanjavur, India
| | | | - Raghunathan Krishankumar
- Information Technology Systems and Analytics Area, Indian Institute of Management Bodh Gaya, Bodh Gaya, Bihar, 824234, India
| | | | - Amir H Gandomi
- Faculty of Engineering and Information Technology, University of Technology Sydney, Ultimo, NSW, Australia.
- University Research and Innovation Center (EKIK), Obuda University, Buddapest, Hungary.
| |
Collapse
|
38
|
Kato S, Hotta K. Adaptive t-vMF dice loss: An effective expansion of dice loss for medical image segmentation. Comput Biol Med 2024; 168:107695. [PMID: 38061152 DOI: 10.1016/j.compbiomed.2023.107695] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2022] [Revised: 10/30/2023] [Accepted: 11/06/2023] [Indexed: 01/10/2024]
Abstract
Dice loss is widely used for medical image segmentation, and many improved loss functions have been proposed. However, further Dice loss improvements are still possible. In this study, we reconsidered the use of Dice loss and discovered that Dice loss can be rewritten in the loss function using the cosine similarity through a simple equation transformation. Using this knowledge, we present a novel t-vMF Dice loss based on the t-vMF similarity instead of the cosine similarity. Based on the t-vMF similarity, our proposed Dice loss is formulated in a more compact similarity loss function than the original Dice loss. Furthermore, we present an effective algorithm that automatically determines the parameter κ for the t-vMF similarity using a validation accuracy, called Adaptive t-vMF Dice loss. Using this algorithm, it is possible to apply more compact similarities for easy classes and wider similarities for difficult classes, and we are able to achieve adaptive training based on the accuracy of each class. We evaluated binary segmentation datasets of CVC-ClinicDB and Kvasir-SEG, and multi-class segmentation datasets of Automated Cardiac Diagnosis Challenge and Synapse multi-organ segmentation. Through experiments conducted on four datasets using a five-fold cross-validation, we confirmed that the Dice score coefficient (DSC) was further improved in comparison with the original Dice loss and other loss functions.
Collapse
Affiliation(s)
- Sota Kato
- Department of Electrical, Information, Materials and Materials Engineering, Meijo University, Tempaku-ku, Nagoya, 468-8502, Aichi, Japan.
| | - Kazuhiro Hotta
- Department of Electrical and Electronic Engineering, Meijo University, Nagoya, Japan.
| |
Collapse
|
39
|
Qian L, Wen C, Li Y, Hu Z, Zhou X, Xia X, Kim SH. Multi-scale context UNet-like network with redesigned skip connections for medical image segmentation. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 243:107885. [PMID: 37897988 DOI: 10.1016/j.cmpb.2023.107885] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/19/2022] [Revised: 10/22/2023] [Accepted: 10/24/2023] [Indexed: 10/30/2023]
Abstract
BACKGROUND AND OBJECTIVE Medical image segmentation has garnered significant research attention in the neural network community as a fundamental requirement for developing intelligent medical assistant systems. A series of UNet-like networks with an encoder-decoder architecture have achieved remarkable success in medical image segmentation. Among these networks, UNet2+ (UNet++) and UNet3+ (UNet+++) have introduced redesigned skip connections, dense skip connections, and full-scale skip connections, respectively, surpassing the performance of the original UNet. However, UNet2+ lacks comprehensive information obtained from the entire scale, which hampers its ability to learn organ placement and boundaries. Similarly, due to the limited number of neurons in its structure, UNet3+ fails to effectively segment small objects when trained with a small number of samples. METHOD In this study, we propose UNet_sharp (UNet#), a novel network topology named after the "#" symbol, which combines dense skip connections and full-scale skip connections. In the decoder sub-network, UNet# can effectively integrate feature maps of different scales and capture fine-grained features and coarse-grained semantics from the entire scale. This approach enhances the understanding of organ and lesion positions and enables accurate boundary segmentation. We employ deep supervision for model pruning to accelerate testing and enable mobile device deployment. Additionally, we construct two classification-guided modules to reduce false positives and improve segmentation accuracy. RESULTS Compared to current UNet-like networks, our proposed method achieves the highest Intersection over Union (IoU) values ((92.67±0.96)%, (92.38±1.29)%, (95.36±1.22)%, (74.01±2.03)%) and F1 scores ((91.64±1.86)%, (95.70±2.16)%, (97.34±2.76)%, (84.77±2.65)%) on the semantic segmentation tasks of nuclei, brain tumors, liver, and lung nodules, respectively. CONCLUSIONS The experimental results demonstrate that the reconstructed skip connections in UNet successfully incorporate multi-scale contextual semantic information. Compared to most state-of-the-art medical image segmentation models, our proposed method more accurately locates organs and lesions and precisely segments boundaries.
Collapse
Affiliation(s)
- Ledan Qian
- College of Mathematics and Physics, Wenzhou University, Wenzhou, 325035, Zhejiang, China
| | - Caiyun Wen
- The First Affiliated Hospital of Wenzhou Medical University, Wenzhou, 325035, Zhejiang, China
| | - Yi Li
- College of Computer Science and Artificial Intelligence, Wenzhou University, Wenzhou, 325035, Zhejiang, China
| | - Zhongyi Hu
- Key Laboratory of Intelligent Image Processing and Analysis, Wenzhou, 325035, Zhejiang, China
| | - Xiao Zhou
- Information Technology Center, Wenzhou University, Wenzhou, 325035, Zhejiang, China.
| | - Xiaonyu Xia
- College of Mathematics and Physics, Wenzhou University, Wenzhou, 325035, Zhejiang, China
| | - Soo-Hyung Kim
- College of AI Convergence, Chonnam National University, Gwangju, 61186, Korea
| |
Collapse
|
40
|
Xu B, Yang J, Hong P, Fan X, Sun Y, Zhang L, Yang B, Xu L, Avolio A. Coronary artery segmentation in CCTA images based on multi-scale feature learning. JOURNAL OF X-RAY SCIENCE AND TECHNOLOGY 2024; 32:973-991. [PMID: 38943423 DOI: 10.3233/xst-240093] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 07/01/2024]
Abstract
BACKGROUND Coronary artery segmentation is a prerequisite in computer-aided diagnosis of Coronary Artery Disease (CAD). However, segmentation of coronary arteries in Coronary Computed Tomography Angiography (CCTA) images faces several challenges. The current segmentation approaches are unable to effectively address these challenges and existing problems such as the need for manual interaction or low segmentation accuracy. OBJECTIVE A Multi-scale Feature Learning and Rectification (MFLR) network is proposed to tackle the challenges and achieve automatic and accurate segmentation of coronary arteries. METHODS The MFLR network introduces a multi-scale feature extraction module in the encoder to effectively capture contextual information under different receptive fields. In the decoder, a feature correction and fusion module is proposed, which employs high-level features containing multi-scale information to correct and guide low-level features, achieving fusion between the two-level features to further improve segmentation performance. RESULTS The MFLR network achieved the best performance on the dice similarity coefficient, Jaccard index, Recall, F1-score, and 95% Hausdorff distance, for both in-house and public datasets. CONCLUSION Experimental results demonstrate the superiority and good generalization ability of the MFLR approach. This study contributes to the accurate diagnosis and treatment of CAD, and it also informs other segmentation applications in medicine.
Collapse
Affiliation(s)
- Bu Xu
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Jinzhong Yang
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Peng Hong
- Software College, Northeastern University, Shenyang, China
| | - Xiaoxue Fan
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Yu Sun
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
- Department of Radiology, General Hospital of North Theater Command, Shenyang, China
| | - Libo Zhang
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
- Department of Radiology, General Hospital of North Theater Command, Shenyang, China
| | - Benqiang Yang
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
- Department of Radiology, General Hospital of North Theater Command, Shenyang, China
| | - Lisheng Xu
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
- Key Laboratory of Medical Image Computing, Ministry of Education, Shenyang, China
- Engineering Research Center of Medical Imaging and Intelligent Analysis, Ministry of Education, Shenyang, China
| | - Alberto Avolio
- Macquarie Medical School, Faculty of Medicine, Health and Human Sciences, Macquarie University, Sydney, Australia
| |
Collapse
|
41
|
Liu B, Dolz J, Galdran A, Kobbi R, Ben Ayed I. Do we really need dice? The hidden region-size biases of segmentation losses. Med Image Anal 2024; 91:103015. [PMID: 37918314 DOI: 10.1016/j.media.2023.103015] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/22/2022] [Revised: 09/24/2023] [Accepted: 10/24/2023] [Indexed: 11/04/2023]
Abstract
Most segmentation losses are arguably variants of the Cross-Entropy (CE) or Dice losses. On the surface, these two categories of losses (i.e., distribution based vs. geometry based) seem unrelated, and there is no clear consensus as to which category is a better choice, with varying performances for each across different benchmarks and applications. Furthermore, it is widely argued within the medical-imaging community that Dice and CE are complementary, which has motivated the use of compound CE-Dice losses. In this work, we provide a theoretical analysis, which shows that CE and Dice share a much deeper connection than previously thought. First, we show that, from a constrained-optimization perspective, they both decompose into two components, i.e., a similar ground-truth matching term, which pushes the predicted foreground regions towards the ground-truth, and a region-size penalty term imposing different biases on the size (or proportion) of the predicted regions. Then, we provide bound relationships and an information-theoretic analysis, which uncover hidden region-size biases: Dice has an intrinsic bias towards specific extremely imbalanced solutions, whereas CE implicitly encourages the ground-truth region proportions. Our theoretical results explain the wide experimental evidence in the medical-imaging literature, whereby Dice losses bring improvements for imbalanced segmentation. It also explains why CE dominates natural-image problems with diverse class proportions, in which case Dice might have difficulty adapting to different region-size distributions. Based on our theoretical analysis, we propose a principled and simple solution, which enables to control explicitly the region-size bias. The proposed method integrates CE with explicit terms based on L1 or the KL divergence, which encourage segmenting region proportions to match target class proportions, thereby mitigating class imbalance but without losing generality. Comprehensive experiments and ablation studies over different losses and applications validate our theoretical analysis, as well as the effectiveness of explicit and simple region-size terms. The code is available at https://github.com/by-liu/SegLossBias .
Collapse
Affiliation(s)
- Bingyuan Liu
- LIVIA, ÉTS Montréal, Canada; International Laboratory on Learning Systems (ILLS), McGill - ETS - MILA - CNRS - Université Paris-Saclay - CentraleSupélec, Canada.
| | - Jose Dolz
- LIVIA, ÉTS Montréal, Canada; International Laboratory on Learning Systems (ILLS), McGill - ETS - MILA - CNRS - Université Paris-Saclay - CentraleSupélec, Canada; Centre de Recherche du Centre Hospitalier de l'Université de Montréal (CRCHUM), Canada
| | | | | | - Ismail Ben Ayed
- LIVIA, ÉTS Montréal, Canada; International Laboratory on Learning Systems (ILLS), McGill - ETS - MILA - CNRS - Université Paris-Saclay - CentraleSupélec, Canada; Centre de Recherche du Centre Hospitalier de l'Université de Montréal (CRCHUM), Canada
| |
Collapse
|
42
|
Marchant T, Price G, McWilliam A, Henderson E, McSweeney D, van Herk M, Banfill K, Schmitt M, King J, Barker C, Faivre-Finn C. Assessment of heart-substructures auto-contouring accuracy for application in heart-sparing radiotherapy for lung cancer. BJR Open 2024; 6:tzae006. [PMID: 38737623 PMCID: PMC11087931 DOI: 10.1093/bjro/tzae006] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2023] [Revised: 12/14/2023] [Accepted: 02/14/2024] [Indexed: 05/14/2024] Open
Abstract
Objectives We validated an auto-contouring algorithm for heart substructures in lung cancer patients, aiming to establish its accuracy and reliability for radiotherapy (RT) planning. We focus on contouring an amalgamated set of subregions in the base of the heart considered to be a new organ at risk, the cardiac avoidance area (CAA), to enable maximum dose limit implementation in lung RT planning. Methods The study validates a deep-learning model specifically adapted for auto-contouring the CAA (which includes the right atrium, aortic valve root, and proximal segments of the left and right coronary arteries). Geometric, dosimetric, quantitative, and qualitative validation measures are reported. Comparison with manual contours, including assessment of interobserver variability, and robustness testing over 198 cases are also conducted. Results Geometric validation shows that auto-contouring performance lies within the expected range of manual observer variability despite being slightly poorer than the average of manual observers (mean surface distance for CAA of 1.6 vs 1.2 mm, dice similarity coefficient of 0.86 vs 0.88). Dosimetric validation demonstrates consistency between plans optimized using auto-contours and manual contours. Robustness testing confirms acceptable contours in all cases, with 80% rated as "Good" and the remaining 20% as "Useful." Conclusions The auto-contouring algorithm for heart substructures in lung cancer patients demonstrates acceptable and comparable performance to human observers. Advances in knowledge Accurate and reliable auto-contouring results for the CAA facilitate the implementation of a maximum dose limit to this region in lung RT planning, which has now been introduced in the routine setting at our institution.
Collapse
Affiliation(s)
- Tom Marchant
- Christie Medical Physics & Engineering, The Christie NHS Foundation Trust, Manchester, M20 4BX, United Kingdom
- Division of Cancer Sciences, The University of Manchester, Manchester, M13 9PL, United Kingdom
| | - Gareth Price
- Division of Cancer Sciences, The University of Manchester, Manchester, M13 9PL, United Kingdom
- Radiotherapy Related Research, The Christie NHS Foundation Trust, Manchester, M20 4BX, United Kingdom
| | - Alan McWilliam
- Division of Cancer Sciences, The University of Manchester, Manchester, M13 9PL, United Kingdom
- Radiotherapy Related Research, The Christie NHS Foundation Trust, Manchester, M20 4BX, United Kingdom
| | - Edward Henderson
- Division of Cancer Sciences, The University of Manchester, Manchester, M13 9PL, United Kingdom
- Radiotherapy Related Research, The Christie NHS Foundation Trust, Manchester, M20 4BX, United Kingdom
| | - Dónal McSweeney
- Division of Cancer Sciences, The University of Manchester, Manchester, M13 9PL, United Kingdom
- Radiotherapy Related Research, The Christie NHS Foundation Trust, Manchester, M20 4BX, United Kingdom
| | - Marcel van Herk
- Division of Cancer Sciences, The University of Manchester, Manchester, M13 9PL, United Kingdom
- Radiotherapy Related Research, The Christie NHS Foundation Trust, Manchester, M20 4BX, United Kingdom
| | - Kathryn Banfill
- Division of Cancer Sciences, The University of Manchester, Manchester, M13 9PL, United Kingdom
- Department of Clinical Oncology, The Christie NHS Foundation Trust, Manchester, M20 4BX, United Kingdom
| | - Matthias Schmitt
- Division of Cardiovascular Sciences, The University of Manchester, Manchester, M13 9PL, United Kingdom
- Department of Cardiology, Manchester University NHS Foundation Trust, Manchester, M13 9WL, United Kingdom
| | - Jennifer King
- Department of Clinical Oncology, The Christie NHS Foundation Trust, Manchester, M20 4BX, United Kingdom
| | - Claire Barker
- Department of Clinical Oncology, The Christie NHS Foundation Trust, Manchester, M20 4BX, United Kingdom
| | - Corinne Faivre-Finn
- Division of Cancer Sciences, The University of Manchester, Manchester, M13 9PL, United Kingdom
- Department of Clinical Oncology, The Christie NHS Foundation Trust, Manchester, M20 4BX, United Kingdom
| |
Collapse
|
43
|
Kato S, Hotta K. Expanded tube attention for tubular structure segmentation. Int J Comput Assist Radiol Surg 2023:10.1007/s11548-023-03038-2. [PMID: 38112883 DOI: 10.1007/s11548-023-03038-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/17/2023] [Accepted: 11/13/2023] [Indexed: 12/21/2023]
Abstract
PURPOSE Semantic segmentation of tubular structures, such as blood vessels and cell membranes, is a very difficult task, and it tends to break many predicted regions in the middle. This problem is due to the fact that tubular ground truth is very thin, and the number of pixels is extremely unbalanced compared to the background. METHODS We present a novel training method using pseudo-labels generated by morphological transformation. Furthermore, we present an attention module using thickened pseudo-labels, called the expanded tube attention (ETA) module. By using the ETA module, the network learns thickened regions based on pseudo-labels at first and then gradually learns thinned original regions while transferring information in the thickened regions as an attention map. RESULTS Through experiments conducted on retina vessel image datasets using various evaluation measures, we confirmed that the proposed method using ETA modules improved the clDice metric accuracy in comparison with the conventional methods. CONCLUSIONS We demonstrated that the proposed novel expanded tube attention module using thickened pseudo-labels can achieve easy-to-hard learning.
Collapse
Affiliation(s)
- Sota Kato
- Department of Electrical, Information, Materials and Materials Engineering, Meijo University, Tempaku-ku, Nagoya, Aichi, 468-8502, Japan.
| | - Kazuhiro Hotta
- Department of Electrical and Electronic Engineering, Meijo University, Tempaku-ku, Nagoya, Aichi, Japan
| |
Collapse
|
44
|
Chen Y, Yu L, Wang JY, Panjwani N, Obeid JP, Liu W, Liu L, Kovalchuk N, Gensheimer MF, Vitzthum LK, Beadle BM, Chang DT, Le QT, Han B, Xing L. Adaptive Region-Specific Loss for Improved Medical Image Segmentation. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2023; 45:13408-13421. [PMID: 37363838 PMCID: PMC11346301 DOI: 10.1109/tpami.2023.3289667] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/28/2023]
Abstract
Defining the loss function is an important part of neural network design and critically determines the success of deep learning modeling. A significant shortcoming of the conventional loss functions is that they weight all regions in the input image volume equally, despite the fact that the system is known to be heterogeneous (i.e., some regions can achieve high prediction performance more easily than others). Here, we introduce a region-specific loss to lift the implicit assumption of homogeneous weighting for better learning. We divide the entire volume into multiple sub-regions, each with an individualized loss constructed for optimal local performance. Effectively, this scheme imposes higher weightings on the sub-regions that are more difficult to segment, and vice versa. Furthermore, the regional false positive and false negative errors are computed for each input image during a training step and the regional penalty is adjusted accordingly to enhance the overall accuracy of the prediction. Using different public and in-house medical image datasets, we demonstrate that the proposed regionally adaptive loss paradigm outperforms conventional methods in the multi-organ segmentations, without any modification to the neural network architecture or additional data preparation.
Collapse
|
45
|
Yang S, Song ES, Lee ES, Kang SR, Yi WJ, Lee SP. Ceph-Net: automatic detection of cephalometric landmarks on scanned lateral cephalograms from children and adolescents using an attention-based stacked regression network. BMC Oral Health 2023; 23:803. [PMID: 37884918 PMCID: PMC10604948 DOI: 10.1186/s12903-023-03452-7] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/08/2023] [Accepted: 09/25/2023] [Indexed: 10/28/2023] Open
Abstract
BACKGROUND The success of cephalometric analysis depends on the accurate detection of cephalometric landmarks on scanned lateral cephalograms. However, manual cephalometric analysis is time-consuming and can cause inter- and intra-observer variability. The purpose of this study was to automatically detect cephalometric landmarks on scanned lateral cephalograms with low contrast and resolution using an attention-based stacked regression network (Ceph-Net). METHODS The main body of Ceph-Net compromised stacked fully convolutional networks (FCN) which progressively refined the detection of cephalometric landmarks on each FCN. By embedding dual attention and multi-path convolution modules in Ceph-Net, the network learned local and global context and semantic relationships between cephalometric landmarks. Additionally, the intermediate deep supervision in each FCN further boosted the training stability and the detection performance of cephalometric landmarks. RESULTS Ceph-Net showed a superior detection performance in mean radial error and successful detection rate, including accuracy improvements in cephalometric landmark detection located in low-contrast soft tissues compared with other detection networks. Moreover, Ceph-Net presented superior detection performance on the test dataset split by age from 8 to 16 years old. CONCLUSIONS Ceph-Net demonstrated an automatic and superior detection of cephalometric landmarks by successfully learning local and global context and semantic relationships between cephalometric landmarks in scanned lateral cephalograms with low contrast and resolutions.
Collapse
Affiliation(s)
- Su Yang
- Department of Applied Bioengineering, Graduate School of Convergence Science and Technology, Seoul National University, Seoul, South Korea
| | - Eun Sun Song
- Department of Oral Anatomy, Dental Research Institute, School of Dentistry, Seoul National University, Seoul, South Korea
| | - Eun Seung Lee
- Department of Oral Anatomy, Dental Research Institute, School of Dentistry, Seoul National University, Seoul, South Korea
| | - Se-Ryong Kang
- Department of Biomedical Radiation Sciences, Graduate School of Convergence Science and Technology, Seoul National University, Seoul, South Korea
| | - Won-Jin Yi
- Department of Applied Bioengineering, Graduate School of Convergence Science and Technology, Seoul National University, Seoul, South Korea.
- Department of Oral and Maxillofacial Radiology and Dental Research Institute, School of Dentistry, Seoul National University, Seoul, South Korea.
| | - Seung-Pyo Lee
- Department of Oral Anatomy, Dental Research Institute, School of Dentistry, Seoul National University, Seoul, South Korea.
| |
Collapse
|
46
|
AL Qurri A, Almekkawy M. Improved UNet with Attention for Medical Image Segmentation. SENSORS (BASEL, SWITZERLAND) 2023; 23:8589. [PMID: 37896682 PMCID: PMC10611347 DOI: 10.3390/s23208589] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/06/2023] [Revised: 10/01/2023] [Accepted: 10/13/2023] [Indexed: 10/29/2023]
Abstract
Medical image segmentation is crucial for medical image processing and the development of computer-aided diagnostics. In recent years, deep Convolutional Neural Networks (CNNs) have been widely adopted for medical image segmentation and have achieved significant success. UNet, which is based on CNNs, is the mainstream method used for medical image segmentation. However, its performance suffers owing to its inability to capture long-range dependencies. Transformers were initially designed for Natural Language Processing (NLP), and sequence-to-sequence applications have demonstrated the ability to capture long-range dependencies. However, their abilities to acquire local information are limited. Hybrid architectures of CNNs and Transformer, such as TransUNet, have been proposed to benefit from Transformer's long-range dependencies and CNNs' low-level details. Nevertheless, automatic medical image segmentation remains a challenging task due to factors such as blurred boundaries, the low-contrast tissue environment, and in the context of ultrasound, issues like speckle noise and attenuation. In this paper, we propose a new model that combines the strengths of both CNNs and Transformer, with network architectural improvements designed to enrich the feature representation captured by the skip connections and the decoder. To this end, we devised a new attention module called Three-Level Attention (TLA). This module is composed of an Attention Gate (AG), channel attention, and spatial normalization mechanism. The AG preserves structural information, whereas channel attention helps to model the interdependencies between channels. Spatial normalization employs the spatial coefficient of the Transformer to improve spatial attention akin to TransNorm. To further improve the skip connection and reduce the semantic gap, skip connections between the encoder and decoder were redesigned in a manner similar to that of the UNet++ dense connection. Moreover, deep supervision using a side-output channel was introduced, analogous to BASNet, which was originally used for saliency predictions. Two datasets from different modalities, a CT scan dataset and an ultrasound dataset, were used to evaluate the proposed UNet architecture. The experimental results showed that our model consistently improved the prediction performance of the UNet across different datasets.
Collapse
|
47
|
Li H, Zeng P, Bai C, Wang W, Yu Y, Yu P. PMJAF-Net: Pyramidal multi-scale joint attention and adaptive fusion network for explainable skin lesion segmentation. Comput Biol Med 2023; 165:107454. [PMID: 37716246 DOI: 10.1016/j.compbiomed.2023.107454] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/14/2023] [Revised: 08/18/2023] [Accepted: 09/04/2023] [Indexed: 09/18/2023]
Abstract
Traditional convolutional neural networks have achieved remarkable success in skin lesion segmentation. However, the successive pooling operations and convolutional spans reduce the feature resolution and hinder the dense prediction for spatial information, resulting in blurred boundaries, low accuracy and poor interpretability for irregular lesion segmentation under low contrast. To solve the above issues, a pyramidal multi-scale joint attention and adaptive fusion network for explainable (PMJAF-Net) skin lesion segmentation is proposed. Firstly, an adaptive spatial attention module is designed to establish the long-term correlation between pixels, enrich the global and local contextual information, and refine the detailed features. Subsequently, an efficient pyramidal multi-scale channel attention module is proposed to capture the multi-scale information and edge features by using the pyramidal module. Meanwhile, a channel attention module is devised to establish the long-term correlation between channels and highlight the most related feature channels to capture the multi-scale key information on each channel. Thereafter, a multi-scale adaptive fusion attention module is put forward to efficiently fuse the scale features at different decoding stages. Finally, a novel hybrid loss function based on region salient features and boundary quality is presented to guide the network to learn from map-level, patch-level and pixel-level and to accurately predict the lesion regions with clear boundaries. In addition, visualizing attention weight maps are utilized to visually enhance the interpretability of our proposed model. Comprehensive experiments are conducted on four public skin lesion datasets, and the results demonstrate that the proposed network outperforms the state-of-the-art methods, with the segmentation assessment evaluation metrics Dice, JI, and ACC improved to 92.65%, 87.86% and 96.26%, respectively.
Collapse
Affiliation(s)
- Haiyan Li
- School of Information, Yunnan University, Kunming, 650504, China
| | - Peng Zeng
- School of Information, Yunnan University, Kunming, 650504, China
| | - Chongbin Bai
- Otolaryngology Department, Honghe Prefecture Second People's Hospital, Jianshui, 654300, China
| | - Wei Wang
- School of Software, Yunnan University, Kunming, 650504, China.
| | - Ying Yu
- School of Information, Yunnan University, Kunming, 650504, China
| | - Pengfei Yu
- School of Information, Yunnan University, Kunming, 650504, China
| |
Collapse
|
48
|
Hu P, Zhou H, Yan T, Miu H, Xiao F, Zhu X, Shu L, Yang S, Jin R, Dou W, Ren B, Zhu L, Liu W, Zhang Y, Zeng K, Ye M, Lv S, Wu M, Deng G, Hu R, Zhan R, Chen Q, Zhang D, Zhu X. Deep learning-assisted identification and quantification of aneurysmal subarachnoid hemorrhage in non-contrast CT scans: Development and external validation of Hybrid 2D/3D UNet. Neuroimage 2023; 279:120321. [PMID: 37574119 DOI: 10.1016/j.neuroimage.2023.120321] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/27/2023] [Revised: 08/01/2023] [Accepted: 08/10/2023] [Indexed: 08/15/2023] Open
Abstract
Accurate stroke assessment and consequent favorable clinical outcomes rely on the early identification and quantification of aneurysmal subarachnoid hemorrhage (aSAH) in non-contrast computed tomography (NCCT) images. However, hemorrhagic lesions can be complex and difficult to distinguish manually. To solve these problems, here we propose a novel Hybrid 2D/3D UNet deep-learning framework for automatic aSAH identification and quantification in NCCT images. We evaluated 1824 consecutive patients admitted with aSAH to four hospitals in China between June 2018 and May 2022. Accuracy and precision, Dice scores and intersection over union (IoU), and interclass correlation coefficients (ICC) were calculated to assess model performance, segmentation performance, and correlations between automatic and manual segmentation, respectively. A total of 1355 patients with aSAH were enrolled: 931, 101, 179, and 144 in four datasets, of whom 326 were scanned with Siemens, 640 with Philips, and 389 with GE Medical Systems scanners. Our proposed deep-learning method accurately identified (accuracies 0.993-0.999) and segmented (Dice scores 0.550-0.897) hemorrhage in both the internal and external datasets, even combinations of hemorrhage subtypes. We further developed a convenient AI-assisted platform based on our algorithm to assist clinical workflows, whose performance was comparable to manual measurements by experienced neurosurgeons (ICCs 0.815-0.957) but with greater efficiency and reduced cost. While this tool has not yet been prospectively tested in clinical practice, our innovative hybrid network algorithm and platform can accurately identify and quantify aSAH, paving the way for fast and cheap NCCT interpretation and a reliable AI-based approach to expedite clinical decision-making for aSAH patients.
Collapse
Affiliation(s)
- Ping Hu
- Department of Neurosurgery, The Second Affiliated Hospital of Nanchang University, Nanchang, Jiangxi 330006, China; Jiangxi Key Laboratory of Neurological Tumors and Cerebrovascular Diseases, Nanchang, Jiangxi 330006, China; Jiangxi Health Commission Key Laboratory of Neurological Medicine, Nanchang, Jiangxi 330006, China; Institute of Neuroscience, Nanchang University, Nanchang, Jiangxi 330006, China
| | - Haizhu Zhou
- School of Physics and Technology, Wuhan University, Wuhan, Hubei 430060, China
| | - Tengfeng Yan
- Department of Neurosurgery, The Second Affiliated Hospital of Nanchang University, Nanchang, Jiangxi 330006, China; Jiangxi Key Laboratory of Neurological Tumors and Cerebrovascular Diseases, Nanchang, Jiangxi 330006, China; Jiangxi Health Commission Key Laboratory of Neurological Medicine, Nanchang, Jiangxi 330006, China; Institute of Neuroscience, Nanchang University, Nanchang, Jiangxi 330006, China
| | - Hongping Miu
- Department of Neurosurgery, Southwest Hospital, Third Military Medical University (Army Medical University), Chongqing 400038, China
| | - Feng Xiao
- Department of Neurosurgery, First Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, Zhejiang 310003, China
| | - Xinyi Zhu
- Department of Neurosurgery, Renmin Hospital of Wuhan University, Wuhan, Hubei 430060, China
| | - Lei Shu
- Department of Neurosurgery, The Second Affiliated Hospital of Nanchang University, Nanchang, Jiangxi 330006, China; Jiangxi Key Laboratory of Neurological Tumors and Cerebrovascular Diseases, Nanchang, Jiangxi 330006, China; Jiangxi Health Commission Key Laboratory of Neurological Medicine, Nanchang, Jiangxi 330006, China; Institute of Neuroscience, Nanchang University, Nanchang, Jiangxi 330006, China
| | - Shuang Yang
- School of Physics and Technology, Wuhan University, Wuhan, Hubei 430060, China
| | - Ruiyun Jin
- Department of Neurosurgery, The Second Affiliated Hospital of Nanchang University, Nanchang, Jiangxi 330006, China
| | - Wenlei Dou
- Department of Neurosurgery, The Second Affiliated Hospital of Nanchang University, Nanchang, Jiangxi 330006, China
| | - Baoyu Ren
- Department of Neurosurgery, The Second Affiliated Hospital of Nanchang University, Nanchang, Jiangxi 330006, China
| | - Lizhen Zhu
- Department of Neurosurgery, The Second Affiliated Hospital of Nanchang University, Nanchang, Jiangxi 330006, China
| | - Wanrong Liu
- Department of Neurosurgery, The Second Affiliated Hospital of Nanchang University, Nanchang, Jiangxi 330006, China
| | - Yihan Zhang
- Department of Neurosurgery, The Second Affiliated Hospital of Nanchang University, Nanchang, Jiangxi 330006, China
| | - Kaisheng Zeng
- Department of Neurosurgery, The Second Affiliated Hospital of Nanchang University, Nanchang, Jiangxi 330006, China
| | - Minhua Ye
- Department of Neurosurgery, The Second Affiliated Hospital of Nanchang University, Nanchang, Jiangxi 330006, China
| | - Shigang Lv
- Department of Neurosurgery, The Second Affiliated Hospital of Nanchang University, Nanchang, Jiangxi 330006, China
| | - Miaojing Wu
- Department of Neurosurgery, The Second Affiliated Hospital of Nanchang University, Nanchang, Jiangxi 330006, China
| | - Gang Deng
- Department of Neurosurgery, Renmin Hospital of Wuhan University, Wuhan, Hubei 430060, China
| | - Rong Hu
- Department of Neurosurgery, Southwest Hospital, Third Military Medical University (Army Medical University), Chongqing 400038, China
| | - Renya Zhan
- Department of Neurosurgery, First Affiliated Hospital, School of Medicine, Zhejiang University, Hangzhou, Zhejiang 310003, China
| | - Qianxue Chen
- Department of Neurosurgery, Renmin Hospital of Wuhan University, Wuhan, Hubei 430060, China
| | - Dong Zhang
- School of Physics and Technology, Wuhan University, Wuhan, Hubei 430060, China
| | - Xingen Zhu
- Department of Neurosurgery, The Second Affiliated Hospital of Nanchang University, Nanchang, Jiangxi 330006, China; Jiangxi Key Laboratory of Neurological Tumors and Cerebrovascular Diseases, Nanchang, Jiangxi 330006, China; Jiangxi Health Commission Key Laboratory of Neurological Medicine, Nanchang, Jiangxi 330006, China; Institute of Neuroscience, Nanchang University, Nanchang, Jiangxi 330006, China.
| |
Collapse
|
49
|
Müller-Franzes G, Müller-Franzes F, Huck L, Raaff V, Kemmer E, Khader F, Arasteh ST, Lemainque T, Kather JN, Nebelung S, Kuhl C, Truhn D. Fibroglandular tissue segmentation in breast MRI using vision transformers: a multi-institutional evaluation. Sci Rep 2023; 13:14207. [PMID: 37648728 PMCID: PMC10468506 DOI: 10.1038/s41598-023-41331-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/19/2023] [Accepted: 08/24/2023] [Indexed: 09/01/2023] Open
Abstract
Accurate and automatic segmentation of fibroglandular tissue in breast MRI screening is essential for the quantification of breast density and background parenchymal enhancement. In this retrospective study, we developed and evaluated a transformer-based neural network for breast segmentation (TraBS) in multi-institutional MRI data, and compared its performance to the well established convolutional neural network nnUNet. TraBS and nnUNet were trained and tested on 200 internal and 40 external breast MRI examinations using manual segmentations generated by experienced human readers. Segmentation performance was assessed in terms of the Dice score and the average symmetric surface distance. The Dice score for nnUNet was lower than for TraBS on the internal testset (0.909 ± 0.069 versus 0.916 ± 0.067, P < 0.001) and on the external testset (0.824 ± 0.144 versus 0.864 ± 0.081, P = 0.004). Moreover, the average symmetric surface distance was higher (= worse) for nnUNet than for TraBS on the internal (0.657 ± 2.856 versus 0.548 ± 2.195, P = 0.001) and on the external testset (0.727 ± 0.620 versus 0.584 ± 0.413, P = 0.03). Our study demonstrates that transformer-based networks improve the quality of fibroglandular tissue segmentation in breast MRI compared to convolutional-based models like nnUNet. These findings might help to enhance the accuracy of breast density and parenchymal enhancement quantification in breast MRI screening.
Collapse
Affiliation(s)
- Gustav Müller-Franzes
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany
| | - Fritz Müller-Franzes
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany
| | - Luisa Huck
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany
| | - Vanessa Raaff
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany
| | - Eva Kemmer
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany
| | - Firas Khader
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany
| | - Soroosh Tayebi Arasteh
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany
| | - Teresa Lemainque
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany
| | - Jakob Nikolas Kather
- Else Kroener Fresenius Center for Digital Health, Technical University, Dresden, Germany
- Department of Medicine III, University Hospital RWTH, Aachen, Germany
| | - Sven Nebelung
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany
| | - Christiane Kuhl
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany
| | - Daniel Truhn
- Department of Diagnostic and Interventional Radiology, University Hospital RWTH, Aachen, Germany.
| |
Collapse
|
50
|
Yousef R, Khan S, Gupta G, Albahlal BM, Alajlan SA, Ali A. Bridged-U-Net-ASPP-EVO and Deep Learning Optimization for Brain Tumor Segmentation. Diagnostics (Basel) 2023; 13:2633. [PMID: 37627893 PMCID: PMC10453237 DOI: 10.3390/diagnostics13162633] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2023] [Revised: 07/25/2023] [Accepted: 07/28/2023] [Indexed: 08/27/2023] Open
Abstract
Brain tumor segmentation from Magnetic Resonance Images (MRI) is considered a big challenge due to the complexity of brain tumor tissues, and segmenting these tissues from the healthy tissues is an even more tedious challenge when manual segmentation is undertaken by radiologists. In this paper, we have presented an experimental approach to emphasize the impact and effectiveness of deep learning elements like optimizers and loss functions towards a deep learning optimal solution for brain tumor segmentation. We evaluated our performance results on the most popular brain tumor datasets (MICCAI BraTS 2020 and RSNA-ASNR-MICCAI BraTS 2021). Furthermore, a new Bridged U-Net-ASPP-EVO was introduced that exploits Atrous Spatial Pyramid Pooling to enhance capturing multi-scale information to help in segmenting different tumor sizes, Evolving Normalization layers, squeeze and excitation residual blocks, and the max-average pooling for down sampling. Two variants of this architecture were constructed (Bridged U-Net_ASPP_EVO v1 and Bridged U-Net_ASPP_EVO v2). The best results were achieved using these two models when compared with other state-of-the-art models; we have achieved average segmentation dice scores of 0.84, 0.85, and 0.91 from variant1, and 0.83, 0.86, and 0.92 from v2 for the Enhanced Tumor (ET), Tumor Core (TC), and Whole Tumor (WT) tumor sub-regions, respectively, in the BraTS 2021validation dataset.
Collapse
Affiliation(s)
- Rammah Yousef
- Yogananda School of AI, Computers and Data Sciences, Shoolini University, Solan 173229, India
| | - Shakir Khan
- College of Computer and Information Sciences, Imam Mohammad Ibn Saud Islamic University (IMSIU), Riyadh 11432, Saudi Arabia (S.A.A.)
- Department of Computer Science and Engineering, University Centre for Research and Development, Chandigarh University, Mohali 140413, India
| | - Gaurav Gupta
- Yogananda School of AI, Computers and Data Sciences, Shoolini University, Solan 173229, India
| | - Bader M. Albahlal
- College of Computer and Information Sciences, Imam Mohammad Ibn Saud Islamic University (IMSIU), Riyadh 11432, Saudi Arabia (S.A.A.)
| | - Saad Abdullah Alajlan
- College of Computer and Information Sciences, Imam Mohammad Ibn Saud Islamic University (IMSIU), Riyadh 11432, Saudi Arabia (S.A.A.)
| | - Aleem Ali
- Department of Computer Science and Engineering, University Centre for Research and Development, Chandigarh University, Mohali 140413, India
| |
Collapse
|