1
|
Li L, Jiang C, Yu L, Zeng X, Zheng S. Efficient model-informed co-segmentation of tumors on PET/CT driven by clustering and classification information. Comput Biol Med 2024; 180:108980. [PMID: 39137668 DOI: 10.1016/j.compbiomed.2024.108980] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2024] [Revised: 07/18/2024] [Accepted: 08/01/2024] [Indexed: 08/15/2024]
Abstract
Automatic tumor segmentation via positron emission tomography (PET) and computed tomography (CT) images plays a critical role in the prevention, diagnosis, and treatment of this disease via radiation oncology. However, segmenting these tumors is challenging due to the heterogeneity of grayscale levels and fuzzy boundaries. To address these issues, in this paper, an efficient model-informed PET/CT tumor co-segmentation method that combines fuzzy C-means clustering and Bayesian classification information is proposed. To alleviate the grayscale heterogeneity of multi-modal images, in this method, a novel grayscale similar region term is designed based on the background region information of PET and the foreground region information of CT. An edge stop function is innovatively presented to enhance the localization of fuzzy edges by incorporating the fuzzy C-means clustering strategy. To improve the segmentation accuracy further, a unique data fidelity term is introduced based on PET images by combining the distribution characteristics of pixel points in PET images. Finally, experimental validation on datasets of head and neck tumor (HECKTOR) and non-small cell lung cancer (NSCLC) demonstrated impressive values for three key evaluation metrics, including DSC, RVD and HD5, achieved impressive values of 0.85, 5.32, and 0.17, respectively. These compelling results indicate that image segmentation methods based on mathematical models exhibit outstanding performance in handling grayscale heterogeneity and fuzzy boundaries in multi-modal images.
Collapse
Affiliation(s)
- Laquan Li
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, 400065, China
| | - Chuangbo Jiang
- School of Science, Chongqing University of Posts and Telecommunications, Chongqing, 400065, China
| | - Lei Yu
- Emergency Department, The Second Affiliated Hospital of Chongqing Medical University, Chongqing, 400010, China
| | - Xianhua Zeng
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, 400065, China
| | - Shenhai Zheng
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, 400065, China.
| |
Collapse
|
2
|
Huynh BN, Groendahl AR, Tomic O, Liland KH, Knudtsen IS, Hoebers F, van Elmpt W, Dale E, Malinen E, Futsaether CM. Deep learning with uncertainty estimation for automatic tumor segmentation in PET/CT of head and neck cancers: impact of model complexity, image processing and augmentation. Biomed Phys Eng Express 2024; 10:055038. [PMID: 39127060 DOI: 10.1088/2057-1976/ad6dcd] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2024] [Accepted: 08/09/2024] [Indexed: 08/12/2024]
Abstract
Objective.Target volumes for radiotherapy are usually contoured manually, which can be time-consuming and prone to inter- and intra-observer variability. Automatic contouring by convolutional neural networks (CNN) can be fast and consistent but may produce unrealistic contours or miss relevant structures. We evaluate approaches for increasing the quality and assessing the uncertainty of CNN-generated contours of head and neck cancers with PET/CT as input.Approach.Two patient cohorts with head and neck squamous cell carcinoma and baseline18F-fluorodeoxyglucose positron emission tomography and computed tomography images (FDG-PET/CT) were collected retrospectively from two centers. The union of manual contours of the gross primary tumor and involved nodes was used to train CNN models for generating automatic contours. The impact of image preprocessing, image augmentation, transfer learning and CNN complexity, architecture, and dimension (2D or 3D) on model performance and generalizability across centers was evaluated. A Monte Carlo dropout technique was used to quantify and visualize the uncertainty of the automatic contours.Main results. CNN models provided contours with good overlap with the manually contoured ground truth (median Dice Similarity Coefficient: 0.75-0.77), consistent with reported inter-observer variations and previous auto-contouring studies. Image augmentation and model dimension, rather than model complexity, architecture, or advanced image preprocessing, had the largest impact on model performance and cross-center generalizability. Transfer learning on a limited number of patients from a separate center increased model generalizability without decreasing model performance on the original training cohort. High model uncertainty was associated with false positive and false negative voxels as well as low Dice coefficients.Significance.High quality automatic contours can be obtained using deep learning architectures that are not overly complex. Uncertainty estimation of the predicted contours shows potential for highlighting regions of the contour requiring manual revision or flagging segmentations requiring manual inspection and intervention.
Collapse
Affiliation(s)
- Bao Ngoc Huynh
- Faculty of Science and Technology, Norwegian University of Life Sciences, Ås, Norway
| | - Aurora Rosvoll Groendahl
- Faculty of Science and Technology, Norwegian University of Life Sciences, Ås, Norway
- Section of Oncology, Vestre Viken Hospital Trust, Drammen, Norway
| | - Oliver Tomic
- Faculty of Science and Technology, Norwegian University of Life Sciences, Ås, Norway
| | - Kristian Hovde Liland
- Faculty of Science and Technology, Norwegian University of Life Sciences, Ås, Norway
| | - Ingerid Skjei Knudtsen
- Department of Circulation and Medical Imaging, Norwegian University of Science and Technology, Trondheim, Norway
| | - Frank Hoebers
- Department of Radiation Oncology (MAASTRO), GROW School for Oncology and Reproduction, Maastricht, Netherlands
| | - Wouter van Elmpt
- Department of Radiation Oncology (MAASTRO), GROW School for Oncology and Reproduction, Maastricht, Netherlands
| | - Einar Dale
- Department of Oncology, Oslo University Hospital, Oslo, Norway
| | - Eirik Malinen
- Department of Medical Physics, Oslo University Hospital, Oslo, Norway
- Department of Physics, University of Oslo, Oslo, Norway
| | | |
Collapse
|
3
|
Cho MJ, Hwang D, Yie SY, Lee JS. Multi-modal co-learning with attention mechanism for head and neck tumor segmentation on 18FDG PET-CT. EJNMMI Phys 2024; 11:67. [PMID: 39052194 PMCID: PMC11272764 DOI: 10.1186/s40658-024-00670-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/16/2023] [Accepted: 07/12/2024] [Indexed: 07/27/2024] Open
Abstract
PURPOSE Effective radiation therapy requires accurate segmentation of head and neck cancer, one of the most common types of cancer. With the advancement of deep learning, people have come up with various methods that use positron emission tomography-computed tomography to get complementary information. However, these approaches are computationally expensive because of the separation of feature extraction and fusion functions and do not make use of the high sensitivity of PET. We propose a new deep learning-based approach to alleviate these challenges. METHODS We proposed a tumor region attention module that fully exploits the high sensitivity of PET and designed a network that learns the correlation between the PET and CT features using squeeze-and-excitation normalization (SE Norm) without separating the feature extraction and fusion functions. In addition, we introduce multi-scale context fusion, which exploits contextual information from different scales. RESULTS The HECKTOR challenge 2021 dataset was used for training and testing. The proposed model outperformed the state-of-the-art models for medical image segmentation; in particular, the dice similarity coefficient increased by 8.78% compared to U-net. CONCLUSION The proposed network segmented the complex shape of the tumor better than the state-of-the-art medical image segmentation methods, accurately distinguishing between tumor and non-tumor regions.
Collapse
Affiliation(s)
- Min Jeong Cho
- Interdisciplinary Program in Bioengineering, Seoul National University College of Engineering, Seoul, 03080, South Korea
- Department of Nuclear Medicine, Seoul National University College of Medicine, 103 Daehak-ro, Jongno-gu, Seoul, 03080, South Korea
- Integrated Major in Innovative Medical Science, Seoul National Graduate School, Seoul, South Korea
| | - Donghwi Hwang
- Department of Nuclear Medicine, Seoul National University College of Medicine, 103 Daehak-ro, Jongno-gu, Seoul, 03080, South Korea
- Department of Biomedical Sciences, Seoul National University College of Medicine, Seoul, 03080, South Korea
| | - Si Young Yie
- Interdisciplinary Program in Bioengineering, Seoul National University College of Engineering, Seoul, 03080, South Korea
- Department of Nuclear Medicine, Seoul National University College of Medicine, 103 Daehak-ro, Jongno-gu, Seoul, 03080, South Korea
- Integrated Major in Innovative Medical Science, Seoul National Graduate School, Seoul, South Korea
| | - Jae Sung Lee
- Interdisciplinary Program in Bioengineering, Seoul National University College of Engineering, Seoul, 03080, South Korea.
- Department of Nuclear Medicine, Seoul National University College of Medicine, 103 Daehak-ro, Jongno-gu, Seoul, 03080, South Korea.
- Integrated Major in Innovative Medical Science, Seoul National Graduate School, Seoul, South Korea.
- Department of Biomedical Sciences, Seoul National University College of Medicine, Seoul, 03080, South Korea.
- Brightonix Imaging Inc, Seoul, 04782, South Korea.
| |
Collapse
|
4
|
Wang H, Wang T, Hao Y, Ding S, Feng J. Breast tumor segmentation via deep correlation analysis of multi-sequence MRI. Med Biol Eng Comput 2024:10.1007/s11517-024-03166-0. [PMID: 39031329 DOI: 10.1007/s11517-024-03166-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/19/2023] [Accepted: 07/03/2024] [Indexed: 07/22/2024]
Abstract
Precise segmentation of breast tumors from MRI is crucial for breast cancer diagnosis, as it allows for detailed calculation of tumor characteristics such as shape, size, and edges. Current segmentation methodologies face significant challenges in accurately modeling the complex interrelationships inherent in multi-sequence MRI data. This paper presents a hybrid deep network framework with three interconnected modules, aimed at efficiently integrating and exploiting the spatial-temporal features among multiple MRI sequences for breast tumor segmentation. The first module involves an advanced multi-sequence encoder with a densely connected architecture, separating the encoding pathway into multiple streams for individual MRI sequences. To harness the intricate correlations between different sequence features, we propose a sequence-awareness and temporal-awareness method that adeptly fuses spatial-temporal features of MRI in the second multi-scale feature embedding module. Finally, the decoder module engages in the upsampling of feature maps, meticulously refining the resolution to achieve highly precise segmentation of breast tumors. In contrast to other popular methods, the proposed method learns the interrelationships inherent in multi-sequence MRI. We justify the proposed method through extensive experiments. It achieves notable improvements in segmentation performance, with Dice Similarity Coefficient (DSC), Intersection over Union (IoU), and Positive Predictive Value (PPV) scores of 80.57%, 74.08%, and 84.74% respectively.
Collapse
Affiliation(s)
- Hongyu Wang
- School of Computer Science and Technology, Xi'an University of Posts and Telecommunications, Xi'an, Shaanxi, 710121, China.
- Shaanxi Key Laboratory of Network Data Analysis and Intelligent Processing, Xi'an University of Posts and Telecommunications, Xi'an, Shaanxi, 710121, China.
- Xi'an Key Laboratory of Big Data and Intelligent Computing, Xi'an University of Posts and Telecommunications, Xi'an, Shaanxi, 710121, China.
| | - Tonghui Wang
- Department of Information Science and Technology, Northwest University, Xi'an, Shaanxi, 7101127, China
| | - Yanfang Hao
- School of Computer Science and Technology, Xi'an University of Posts and Telecommunications, Xi'an, Shaanxi, 710121, China
- Shaanxi Key Laboratory of Network Data Analysis and Intelligent Processing, Xi'an University of Posts and Telecommunications, Xi'an, Shaanxi, 710121, China
- Xi'an Key Laboratory of Big Data and Intelligent Computing, Xi'an University of Posts and Telecommunications, Xi'an, Shaanxi, 710121, China
| | - Songtao Ding
- School of Computer Science and Technology, Xi'an University of Posts and Telecommunications, Xi'an, Shaanxi, 710121, China
- Shaanxi Key Laboratory of Network Data Analysis and Intelligent Processing, Xi'an University of Posts and Telecommunications, Xi'an, Shaanxi, 710121, China
- Xi'an Key Laboratory of Big Data and Intelligent Computing, Xi'an University of Posts and Telecommunications, Xi'an, Shaanxi, 710121, China
| | - Jun Feng
- Department of Information Science and Technology, Northwest University, Xi'an, Shaanxi, 7101127, China.
| |
Collapse
|
5
|
Yue Y, Li N, Zhang G, Xing W, Zhu Z, Liu X, Song S, Ta D. A transformer-guided cross-modality adaptive feature fusion framework for esophageal gross tumor volume segmentation. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 251:108216. [PMID: 38761412 DOI: 10.1016/j.cmpb.2024.108216] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/01/2023] [Revised: 04/17/2024] [Accepted: 05/06/2024] [Indexed: 05/20/2024]
Abstract
BACKGROUND AND OBJECTIVE Accurate segmentation of esophageal gross tumor volume (GTV) indirectly enhances the efficacy of radiotherapy for patients with esophagus cancer. In this domain, learning-based methods have been employed to fuse cross-modality positron emission tomography (PET) and computed tomography (CT) images, aiming to improve segmentation accuracy. This fusion is essential as it combines functional metabolic information from PET with anatomical information from CT, providing complementary information. While the existing three-dimensional (3D) segmentation method has achieved state-of-the-art (SOTA) performance, it typically relies on pure-convolution architectures, limiting its ability to capture long-range spatial dependencies due to convolution's confinement to a local receptive field. To address this limitation and further enhance esophageal GTV segmentation performance, this work proposes a transformer-guided cross-modality adaptive feature fusion network, referred to as TransAttPSNN, which is based on cross-modality PET/CT scans. METHODS Specifically, we establish an attention progressive semantically-nested network (AttPSNN) by incorporating the convolutional attention mechanism into the progressive semantically-nested network (PSNN). Subsequently, we devise a plug-and-play transformer-guided cross-modality adaptive feature fusion model, which is inserted between the multi-scale feature counterparts of a two-stream AttPSNN backbone (one for the PET modality flow and another for the CT modality flow), resulting in the proposed TransAttPSNN architecture. RESULTS Through extensive four-fold cross-validation experiments on the clinical PET/CT cohort. The proposed approach acquires a Dice similarity coefficient (DSC) of 0.76 ± 0.13, a Hausdorff distance (HD) of 9.38 ± 8.76 mm, and a Mean surface distance (MSD) of 1.13 ± 0.94 mm, outperforming the SOTA competing methods. The qualitative results show a satisfying consistency with the lesion areas. CONCLUSIONS The devised transformer-guided cross-modality adaptive feature fusion module integrates the strengths of PET and CT, effectively enhancing the segmentation performance of esophageal GTV. The proposed TransAttPSNN has further advanced the research of esophageal GTV segmentation.
Collapse
Affiliation(s)
- Yaoting Yue
- Department of Biomedical Engineering, School of Information Science and Technology, Fudan University, Shanghai 200438, PR China
| | - Nan Li
- Department of Nuclear Medicine, Fudan University Shanghai Cancer Center, Shanghai 201321, PR China
| | - Gaobo Zhang
- Department of Biomedical Engineering, School of Information Science and Technology, Fudan University, Shanghai 200438, PR China
| | - Wenyu Xing
- Academy for Engineering and Technology, Fudan University, Shanghai 200433, PR China
| | - Zhibin Zhu
- Department of Biomedical Engineering, School of Information Science and Technology, Fudan University, Shanghai 200438, PR China; School of Physics and Electromechanical Engineering, Hexi University, Zhangye 734000, Gansu, PR China
| | - Xin Liu
- Academy for Engineering and Technology, Fudan University, Shanghai 200433, PR China.
| | - Shaoli Song
- Department of Nuclear Medicine, Fudan University Shanghai Cancer Center, Shanghai 201321, PR China.
| | - Dean Ta
- Department of Biomedical Engineering, School of Information Science and Technology, Fudan University, Shanghai 200438, PR China; Academy for Engineering and Technology, Fudan University, Shanghai 200433, PR China.
| |
Collapse
|
6
|
Zou Z, Zou B, Kui X, Chen Z, Li Y. DGCBG-Net: A dual-branch network with global cross-modal interaction and boundary guidance for tumor segmentation in PET/CT images. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 250:108125. [PMID: 38631130 DOI: 10.1016/j.cmpb.2024.108125] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/14/2023] [Revised: 02/24/2024] [Accepted: 03/07/2024] [Indexed: 04/19/2024]
Abstract
BACKGROUND AND OBJECTIVES Automatic tumor segmentation plays a crucial role in cancer diagnosis and treatment planning. Computed tomography (CT) and positron emission tomography (PET) are extensively employed for their complementary medical information. However, existing methods ignore bilateral cross-modal interaction of global features during feature extraction, and they underutilize multi-stage tumor boundary features. METHODS To address these limitations, we propose a dual-branch tumor segmentation network based on global cross-modal interaction and boundary guidance in PET/CT images (DGCBG-Net). DGCBG-Net consists of 1) a global cross-modal interaction module that extracts global contextual information from PET/CT images and promotes bilateral cross-modal interaction of global feature; 2) a shared multi-path downsampling module that learns complementary features from PET/CT modalities to mitigate the impact of misleading features and decrease the loss of discriminative features during downsampling; 3) a boundary prior-guided branch that extracts potential boundary features from CT images at multiple stages, assisting the semantic segmentation branch in improving the accuracy of tumor boundary segmentation. RESULTS Extensive experiments are conducted on STS and Hecktor 2022 datasets to evaluate the proposed method. The average Dice scores of our DGCB-Net on the two datasets are 80.33% and 79.29%, with average IOU scores of 67.64% and 70.18%. DGCB-Net outperformed the current state-of-the-art methods with a 1.77% higher Dice score and a 2.12% higher IOU score. CONCLUSIONS Extensive experimental results demonstrate that DGCBG-Net outperforms existing segmentation methods, and is competitive to state-of-arts.
Collapse
Affiliation(s)
- Ziwei Zou
- School of Computer Science and Engineering, Central South University, No. 932, Lushan South Road, ChangSha, 410083, China
| | - Beiji Zou
- School of Computer Science and Engineering, Central South University, No. 932, Lushan South Road, ChangSha, 410083, China
| | - Xiaoyan Kui
- School of Computer Science and Engineering, Central South University, No. 932, Lushan South Road, ChangSha, 410083, China.
| | - Zhi Chen
- School of Computer Science and Engineering, Central South University, No. 932, Lushan South Road, ChangSha, 410083, China
| | - Yang Li
- School of Informatics, Hunan University of Chinese Medicine, No. 300, Xueshi Road, ChangSha, 410208, China
| |
Collapse
|
7
|
Rong Y, Chen Q, Fu Y, Yang X, Al-Hallaq HA, Wu QJ, Yuan L, Xiao Y, Cai B, Latifi K, Benedict SH, Buchsbaum JC, Qi XS. NRG Oncology Assessment of Artificial Intelligence Deep Learning-Based Auto-segmentation for Radiation Therapy: Current Developments, Clinical Considerations, and Future Directions. Int J Radiat Oncol Biol Phys 2024; 119:261-280. [PMID: 37972715 PMCID: PMC11023777 DOI: 10.1016/j.ijrobp.2023.10.033] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/08/2023] [Revised: 09/16/2023] [Accepted: 10/14/2023] [Indexed: 11/19/2023]
Abstract
Deep learning neural networks (DLNN) in Artificial intelligence (AI) have been extensively explored for automatic segmentation in radiotherapy (RT). In contrast to traditional model-based methods, data-driven AI-based models for auto-segmentation have shown high accuracy in early studies in research settings and controlled environment (single institution). Vendor-provided commercial AI models are made available as part of the integrated treatment planning system (TPS) or as a stand-alone tool that provides streamlined workflow interacting with the main TPS. These commercial tools have drawn clinics' attention thanks to their significant benefit in reducing the workload from manual contouring and shortening the duration of treatment planning. However, challenges occur when applying these commercial AI-based segmentation models to diverse clinical scenarios, particularly in uncontrolled environments. Contouring nomenclature and guideline standardization has been the main task undertaken by the NRG Oncology. AI auto-segmentation holds the potential clinical trial participants to reduce interobserver variations, nomenclature non-compliance, and contouring guideline deviations. Meanwhile, trial reviewers could use AI tools to verify contour accuracy and compliance of those submitted datasets. In recognizing the growing clinical utilization and potential of these commercial AI auto-segmentation tools, NRG Oncology has formed a working group to evaluate the clinical utilization and potential of commercial AI auto-segmentation tools. The group will assess in-house and commercially available AI models, evaluation metrics, clinical challenges, and limitations, as well as future developments in addressing these challenges. General recommendations are made in terms of the implementation of these commercial AI models, as well as precautions in recognizing the challenges and limitations.
Collapse
Affiliation(s)
- Yi Rong
- Mayo Clinic Arizona, Phoenix, AZ
| | - Quan Chen
- City of Hope Comprehensive Cancer Center Duarte, CA
| | - Yabo Fu
- Memorial Sloan Kettering Cancer Center, Commack, NY
| | | | | | | | - Lulin Yuan
- Virginia Commonwealth University, Richmond, VA
| | - Ying Xiao
- University of Pennsylvania/Abramson Cancer Center, Philadelphia, PA
| | - Bin Cai
- The University of Texas Southwestern Medical Center, Dallas, TX
| | | | - Stanley H Benedict
- University of California Davis Comprehensive Cancer Center, Sacramento, CA
| | | | - X Sharon Qi
- University of California Los Angeles, Los Angeles, CA
| |
Collapse
|
8
|
Carles M, Kuhn D, Fechter T, Baltas D, Mix M, Nestle U, Grosu AL, Martí-Bonmatí L, Radicioni G, Gkika E. Development and evaluation of two open-source nnU-Net models for automatic segmentation of lung tumors on PET and CT images with and without respiratory motion compensation. Eur Radiol 2024:10.1007/s00330-024-10751-2. [PMID: 38662100 DOI: 10.1007/s00330-024-10751-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/07/2024] [Revised: 02/22/2024] [Accepted: 03/28/2024] [Indexed: 04/26/2024]
Abstract
OBJECTIVES In lung cancer, one of the main limitations for the optimal integration of the biological and anatomical information derived from Positron Emission Tomography (PET) and Computed Tomography (CT) is the time and expertise required for the evaluation of the different respiratory phases. In this study, we present two open-source models able to automatically segment lung tumors on PET and CT, with and without motion compensation. MATERIALS AND METHODS This study involved time-bin gated (4D) and non-gated (3D) PET/CT images from two prospective lung cancer cohorts (Trials 108237 and 108472) and one retrospective. For model construction, the ground truth (GT) was defined by consensus of two experts, and the nnU-Net with 5-fold cross-validation was applied to 560 4D-images for PET and 100 3D-images for CT. The test sets included 270 4D- images and 19 3D-images for PET and 80 4D-images and 27 3D-images for CT, recruited at 10 different centres. RESULTS In the performance evaluation with the multicentre test sets, the Dice Similarity Coefficients (DSC) obtained for our PET model were DSC(4D-PET) = 0.74 ± 0.06, improving 19% relative to the DSC between experts and DSC(3D-PET) = 0.82 ± 0.11. The performance for CT was DSC(4D-CT) = 0.61 ± 0.28 and DSC(3D-CT) = 0.63 ± 0.34, improving 4% and 15% relative to DSC between experts. CONCLUSIONS Performance evaluation demonstrated that the automatic segmentation models have the potential to achieve accuracy comparable to manual segmentation and thus hold promise for clinical application. The resulting models can be freely downloaded and employed to support the integration of 3D- or 4D- PET/CT and to facilitate the evaluation of its impact on lung cancer clinical practice. CLINICAL RELEVANCE STATEMENT We provide two open-source nnU-Net models for the automatic segmentation of lung tumors on PET/CT to facilitate the optimal integration of biological and anatomical information in clinical practice. The models have superior performance compared to the variability observed in manual segmentations by the different experts for images with and without motion compensation, allowing to take advantage in the clinical practice of the more accurate and robust 4D-quantification. KEY POINTS Lung tumor segmentation on PET/CT imaging is limited by respiratory motion and manual delineation is time consuming and suffer from inter- and intra-variability. Our segmentation models had superior performance compared to the manual segmentations by different experts. Automating PET image segmentation allows for easier clinical implementation of biological information.
Collapse
Affiliation(s)
- Montserrat Carles
- La Fe Health Research Institute, Biomedical Imaging Research Group (GIBI230-PREBI) and Imaging La Fe node at Distributed Network for Biomedical Imaging (ReDIB) Unique Scientific and Technical Infra-structures (ICTS), Valencia, Spain.
| | - Dejan Kuhn
- Department of Radiation Oncology, Division of Medical Physics, University Medical Center Freiburg, Faculty of Medicine, Freiburg, Germany
- German Cancer Consortium (DKTK), German Cancer Research Center (DKFZ), Partner Site Freiburg, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Tobias Fechter
- Department of Radiation Oncology, Division of Medical Physics, University Medical Center Freiburg, Faculty of Medicine, Freiburg, Germany
- German Cancer Consortium (DKTK), German Cancer Research Center (DKFZ), Partner Site Freiburg, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Dimos Baltas
- Department of Radiation Oncology, Division of Medical Physics, University Medical Center Freiburg, Faculty of Medicine, Freiburg, Germany
- German Cancer Consortium (DKTK), German Cancer Research Center (DKFZ), Partner Site Freiburg, German Cancer Research Center (DKFZ), Heidelberg, Germany
| | - Michael Mix
- Department of Nuclear Medicine, Faculty of Medicine, University Medical Center Freiburg, Freiburg, Germany
| | - Ursula Nestle
- German Cancer Consortium (DKTK), German Cancer Research Center (DKFZ), Partner Site Freiburg, German Cancer Research Center (DKFZ), Heidelberg, Germany
- Department of Radiation Oncology, Faculty of Medicine, University Medical Center Freiburg, Freiburg, Germany
- Department of Radiation Oncology, Kliniken Maria Hilf GmbH Moenchengladbach, Moechengladbach, Germany
| | - Anca L Grosu
- German Cancer Consortium (DKTK), German Cancer Research Center (DKFZ), Partner Site Freiburg, German Cancer Research Center (DKFZ), Heidelberg, Germany
- Department of Radiation Oncology, Faculty of Medicine, University Medical Center Freiburg, Freiburg, Germany
| | - Luis Martí-Bonmatí
- La Fe Health Research Institute, Biomedical Imaging Research Group (GIBI230-PREBI) and Imaging La Fe node at Distributed Network for Biomedical Imaging (ReDIB) Unique Scientific and Technical Infra-structures (ICTS), Valencia, Spain
| | - Gianluca Radicioni
- German Cancer Consortium (DKTK), German Cancer Research Center (DKFZ), Partner Site Freiburg, German Cancer Research Center (DKFZ), Heidelberg, Germany
- Department of Radiation Oncology, Faculty of Medicine, University Medical Center Freiburg, Freiburg, Germany
| | - Eleni Gkika
- German Cancer Consortium (DKTK), German Cancer Research Center (DKFZ), Partner Site Freiburg, German Cancer Research Center (DKFZ), Heidelberg, Germany
- Department of Radiation Oncology, Faculty of Medicine, University Medical Center Freiburg, Freiburg, Germany
| |
Collapse
|
9
|
Yuan L, An L, Zhu Y, Duan C, Kong W, Jiang P, Yu QQ. Machine Learning in Diagnosis and Prognosis of Lung Cancer by PET-CT. Cancer Manag Res 2024; 16:361-375. [PMID: 38699652 PMCID: PMC11063459 DOI: 10.2147/cmar.s451871] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2023] [Accepted: 04/16/2024] [Indexed: 05/05/2024] Open
Abstract
As a disease with high morbidity and high mortality, lung cancer has seriously harmed people's health. Therefore, early diagnosis and treatment are more important. PET/CT is usually used to obtain the early diagnosis, staging, and curative effect evaluation of tumors, especially lung cancer, due to the heterogeneity of tumors and the differences in artificial image interpretation and other reasons, it also fails to entirely reflect the real situation of tumors. Artificial intelligence (AI) has been applied to all aspects of life. Machine learning (ML) is one of the important ways to realize AI. With the help of the ML method used by PET/CT imaging technology, there are many studies in the diagnosis and treatment of lung cancer. This article summarizes the application progress of ML based on PET/CT in lung cancer, in order to better serve the clinical. In this study, we searched PubMed using machine learning, lung cancer, and PET/CT as keywords to find relevant articles in the past 5 years or more. We found that PET/CT-based ML approaches have achieved significant results in the detection, delineation, classification of pathology, molecular subtyping, staging, and response assessment with survival and prognosis of lung cancer, which can provide clinicians a powerful tool to support and assist in critical daily clinical decisions. However, ML has some shortcomings such as slightly poor repeatability and reliability.
Collapse
Affiliation(s)
- Lili Yuan
- Jining NO.1 People’s Hospital, Shandong First Medical University, Jining, People’s Republic of China
| | - Lin An
- Jining NO.1 People’s Hospital, Shandong First Medical University, Jining, People’s Republic of China
| | - Yandong Zhu
- Jining NO.1 People’s Hospital, Shandong First Medical University, Jining, People’s Republic of China
| | - Chongling Duan
- Jining NO.1 People’s Hospital, Shandong First Medical University, Jining, People’s Republic of China
| | - Weixiang Kong
- Jining NO.1 People’s Hospital, Shandong First Medical University, Jining, People’s Republic of China
| | - Pei Jiang
- Translational Pharmaceutical Laboratory, Jining NO.1 People’s Hospital, Shandong First Medical University, Jining, People’s Republic of China
| | - Qing-Qing Yu
- Jining NO.1 People’s Hospital, Shandong First Medical University, Jining, People’s Republic of China
| |
Collapse
|
10
|
Liu Z, Mhlanga JC, Xia H, Siegel BA, Jha AK. Need for Objective Task-Based Evaluation of Image Segmentation Algorithms for Quantitative PET: A Study with ACRIN 6668/RTOG 0235 Multicenter Clinical Trial Data. J Nucl Med 2024; 65:jnumed.123.266018. [PMID: 38360049 PMCID: PMC10924158 DOI: 10.2967/jnumed.123.266018] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/12/2023] [Revised: 12/19/2023] [Accepted: 12/19/2023] [Indexed: 02/17/2024] Open
Abstract
Reliable performance of PET segmentation algorithms on clinically relevant tasks is required for their clinical translation. However, these algorithms are typically evaluated using figures of merit (FoMs) that are not explicitly designed to correlate with clinical task performance. Such FoMs include the Dice similarity coefficient (DSC), the Jaccard similarity coefficient (JSC), and the Hausdorff distance (HD). The objective of this study was to investigate whether evaluating PET segmentation algorithms using these task-agnostic FoMs yields interpretations consistent with evaluation on clinically relevant quantitative tasks. Methods: We conducted a retrospective study to assess the concordance in the evaluation of segmentation algorithms using the DSC, JSC, and HD and on the tasks of estimating the metabolic tumor volume (MTV) and total lesion glycolysis (TLG) of primary tumors from PET images of patients with non-small cell lung cancer. The PET images were collected from the American College of Radiology Imaging Network 6668/Radiation Therapy Oncology Group 0235 multicenter clinical trial data. The study was conducted in 2 contexts: (1) evaluating conventional segmentation algorithms, namely those based on thresholding (SUVmax40% and SUVmax50%), boundary detection (Snakes), and stochastic modeling (Markov random field-Gaussian mixture model); (2) evaluating the impact of network depth and loss function on the performance of a state-of-the-art U-net-based segmentation algorithm. Results: Evaluation of conventional segmentation algorithms based on the DSC, JSC, and HD showed that SUVmax40% significantly outperformed SUVmax50%. However, SUVmax40% yielded lower accuracy on the tasks of estimating MTV and TLG, with a 51% and 54% increase, respectively, in the ensemble normalized bias. Similarly, the Markov random field-Gaussian mixture model significantly outperformed Snakes on the basis of the task-agnostic FoMs but yielded a 24% increased bias in estimated MTV. For the U-net-based algorithm, our evaluation showed that although the network depth did not significantly alter the DSC, JSC, and HD values, a deeper network yielded substantially higher accuracy in the estimated MTV and TLG, with a decreased bias of 91% and 87%, respectively. Additionally, whereas there was no significant difference in the DSC, JSC, and HD values for different loss functions, up to a 73% and 58% difference in the bias of the estimated MTV and TLG, respectively, existed. Conclusion: Evaluation of PET segmentation algorithms using task-agnostic FoMs could yield findings discordant with evaluation on clinically relevant quantitative tasks. This study emphasizes the need for objective task-based evaluation of image segmentation algorithms for quantitative PET.
Collapse
Affiliation(s)
- Ziping Liu
- Department of Biomedical Engineering, Washington University, St. Louis, Missouri
| | - Joyce C Mhlanga
- Mallinckrodt Institute of Radiology, Washington University School of Medicine, St. Louis, Missouri; and
| | - Huitian Xia
- Department of Biomedical Engineering, Washington University, St. Louis, Missouri
| | - Barry A Siegel
- Mallinckrodt Institute of Radiology, Washington University School of Medicine, St. Louis, Missouri; and
- Alvin J. Siteman Cancer Center, Washington University School of Medicine, St. Louis, Missouri
| | - Abhinav K Jha
- Department of Biomedical Engineering, Washington University, St. Louis, Missouri;
- Mallinckrodt Institute of Radiology, Washington University School of Medicine, St. Louis, Missouri; and
- Alvin J. Siteman Cancer Center, Washington University School of Medicine, St. Louis, Missouri
| |
Collapse
|
11
|
Tarai S, Lundström E, Sjöholm T, Jönsson H, Korenyushkin A, Ahmad N, Pedersen MA, Molin D, Enblad G, Strand R, Ahlström H, Kullberg J. Improved automated tumor segmentation in whole-body 3D scans using multi-directional 2D projection-based priors. Heliyon 2024; 10:e26414. [PMID: 38390107 PMCID: PMC10882139 DOI: 10.1016/j.heliyon.2024.e26414] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2023] [Revised: 02/09/2024] [Accepted: 02/13/2024] [Indexed: 02/24/2024] Open
Abstract
Early cancer detection, guided by whole-body imaging, is important for the overall survival and well-being of the patients. While various computer-assisted systems have been developed to expedite and enhance cancer diagnostics and longitudinal monitoring, the detection and segmentation of tumors, especially from whole-body scans, remain challenging. To address this, we propose a novel end-to-end automated framework that first generates a tumor probability distribution map (TPDM), incorporating prior information about the tumor characteristics (e.g. size, shape, location). Subsequently, the TPDM is integrated with a state-of-the-art 3D segmentation network along with the original PET/CT or PET/MR images. This aims to produce more meaningful tumor segmentation masks compared to using the baseline 3D segmentation network alone. The proposed method was evaluated on three independent cohorts (autoPET, CAR-T, cHL) of images containing different cancer forms, obtained with different imaging modalities, and acquisition parameters and lesions annotated by different experts. The evaluation demonstrated the superiority of our proposed method over the baseline model by significant margins in terms of Dice coefficient, and lesion-wise sensitivity and precision. Many of the extremely small tumor lesions (i.e. the most difficult to segment) were missed by the baseline model but detected by the proposed model without additional false positives, resulting in clinically more relevant assessments. On average, an improvement of 0.0251 (autoPET), 0.144 (CAR-T), and 0.0528 (cHL) in overall Dice was observed. In conclusion, the proposed TPDM-based approach can be integrated with any state-of-the-art 3D UNET with potentially more accurate and robust segmentation results.
Collapse
Affiliation(s)
- Sambit Tarai
- Department of Surgical Sciences, Uppsala University, SE-75185, Uppsala, Sweden
| | - Elin Lundström
- Department of Surgical Sciences, Uppsala University, SE-75185, Uppsala, Sweden
| | - Therese Sjöholm
- Department of Surgical Sciences, Uppsala University, SE-75185, Uppsala, Sweden
| | - Hanna Jönsson
- Department of Surgical Sciences, Uppsala University, SE-75185, Uppsala, Sweden
| | | | - Nouman Ahmad
- Department of Surgical Sciences, Uppsala University, SE-75185, Uppsala, Sweden
| | - Mette A Pedersen
- Department of Nuclear Medicine & PET-Centre, Aarhus University Hospital, 8200 Aarhus N, Denmark
- Department of Biomedicine, Aarhus University, 8000 Aarhus C, Denmark
- Steno Diabetes Center Aarhus, Aarhus University Hospital, 8200 Aarhus N, Denmark
| | - Daniel Molin
- Department of Immunology, Genetics and Pathology, Uppsala University, SE-75185 Uppsala, Sweden
| | - Gunilla Enblad
- Department of Immunology, Genetics and Pathology, Uppsala University, SE-75185 Uppsala, Sweden
| | - Robin Strand
- Department of Information Technology, Uppsala University, SE-75237, Uppsala, Sweden
| | - Håkan Ahlström
- Department of Surgical Sciences, Uppsala University, SE-75185, Uppsala, Sweden
- Antaros Medical AB, SE-43153, Mölndal, Sweden
| | - Joel Kullberg
- Department of Surgical Sciences, Uppsala University, SE-75185, Uppsala, Sweden
- Antaros Medical AB, SE-43153, Mölndal, Sweden
| |
Collapse
|
12
|
Xue H, Yao Y, Teng Y. Multi-modal tumor segmentation methods based on deep learning: a narrative review. Quant Imaging Med Surg 2024; 14:1122-1140. [PMID: 38223046 PMCID: PMC10784092 DOI: 10.21037/qims-23-818] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2023] [Accepted: 10/19/2023] [Indexed: 01/16/2024]
Abstract
Background and Objective Automatic tumor segmentation is a critical component in clinical diagnosis and treatment. Although single-modal imaging provides useful information, multi-modal imaging provides a more comprehensive understanding of the tumor. Multi-modal tumor segmentation has been an essential topic in medical image processing. With the remarkable performance of deep learning (DL) methods in medical image analysis, multi-modal tumor segmentation based on DL has attracted significant attention. This study aimed to provide an overview of recent DL-based multi-modal tumor segmentation methods. Methods In in the PubMed and Google Scholar databases, the keywords "multi-modal", "deep learning", and "tumor segmentation" were used to systematically search English articles in the past 5 years. The date range was from 1 January 2018 to 1 June 2023. A total of 78 English articles were reviewed. Key Content and Findings We introduce public datasets, evaluation methods, and multi-modal data processing. We also summarize common DL network structures, techniques, and multi-modal image fusion methods used in different tumor segmentation tasks. Finally, we conclude this study by presenting perspectives for future research. Conclusions In multi-modal tumor segmentation tasks, DL technique is a powerful method. With the fusion methods of different modal data, the DL framework can effectively use the characteristics of different modal data to improve the accuracy of tumor segmentation.
Collapse
Affiliation(s)
- Hengzhi Xue
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
| | - Yudong Yao
- Department of Electrical and Computer Engineering, Stevens Institute of Technology, Hoboken, NJ, USA
- Research Institute for Medical and Biological Engineering, Ningbo University, Ningbo, China
| | - Yueyang Teng
- College of Medicine and Biological Information Engineering, Northeastern University, Shenyang, China
- Key Laboratory of Intelligent Computing in Medical Image, Ministry of Education, Shenyang, China
| |
Collapse
|
13
|
Fallahpoor M, Chakraborty S, Pradhan B, Faust O, Barua PD, Chegeni H, Acharya R. Deep learning techniques in PET/CT imaging: A comprehensive review from sinogram to image space. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 243:107880. [PMID: 37924769 DOI: 10.1016/j.cmpb.2023.107880] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/31/2023] [Revised: 10/16/2023] [Accepted: 10/21/2023] [Indexed: 11/06/2023]
Abstract
Positron emission tomography/computed tomography (PET/CT) is increasingly used in oncology, neurology, cardiology, and emerging medical fields. The success stems from the cohesive information that hybrid PET/CT imaging offers, surpassing the capabilities of individual modalities when used in isolation for different malignancies. However, manual image interpretation requires extensive disease-specific knowledge, and it is a time-consuming aspect of physicians' daily routines. Deep learning algorithms, akin to a practitioner during training, extract knowledge from images to facilitate the diagnosis process by detecting symptoms and enhancing images. This acquired knowledge aids in supporting the diagnosis process through symptom detection and image enhancement. The available review papers on PET/CT imaging have a drawback as they either included additional modalities or examined various types of AI applications. However, there has been a lack of comprehensive investigation specifically focused on the highly specific use of AI, and deep learning, on PET/CT images. This review aims to fill that gap by investigating the characteristics of approaches used in papers that employed deep learning for PET/CT imaging. Within the review, we identified 99 studies published between 2017 and 2022 that applied deep learning to PET/CT images. We also identified the best pre-processing algorithms and the most effective deep learning models reported for PET/CT while highlighting the current limitations. Our review underscores the potential of deep learning (DL) in PET/CT imaging, with successful applications in lesion detection, tumor segmentation, and disease classification in both sinogram and image spaces. Common and specific pre-processing techniques are also discussed. DL algorithms excel at extracting meaningful features, and enhancing accuracy and efficiency in diagnosis. However, limitations arise from the scarcity of annotated datasets and challenges in explainability and uncertainty. Recent DL models, such as attention-based models, generative models, multi-modal models, graph convolutional networks, and transformers, are promising for improving PET/CT studies. Additionally, radiomics has garnered attention for tumor classification and predicting patient outcomes. Ongoing research is crucial to explore new applications and improve the accuracy of DL models in this rapidly evolving field.
Collapse
Affiliation(s)
- Maryam Fallahpoor
- Centre for Advanced Modelling and Geospatial Information Systems (CAMGIS), School of Civil and Environmental Engineering, University of Technology Sydney, Ultimo, NSW 2007, Australia
| | - Subrata Chakraborty
- Centre for Advanced Modelling and Geospatial Information Systems (CAMGIS), School of Civil and Environmental Engineering, University of Technology Sydney, Ultimo, NSW 2007, Australia; School of Science and Technology, Faculty of Science, Agriculture, Business and Law, University of New England, Armidale, NSW 2351, Australia
| | - Biswajeet Pradhan
- Centre for Advanced Modelling and Geospatial Information Systems (CAMGIS), School of Civil and Environmental Engineering, University of Technology Sydney, Ultimo, NSW 2007, Australia; Earth Observation Centre, Institute of Climate Change, Universiti Kebangsaan Malaysia, Bangi 43600, Malaysia.
| | - Oliver Faust
- School of Computing and Information Science, Anglia Ruskin University Cambridge Campus, United Kingdom
| | - Prabal Datta Barua
- School of Science and Technology, Faculty of Science, Agriculture, Business and Law, University of New England, Armidale, NSW 2351, Australia; Faculty of Engineering and Information Technology, University of Technology Sydney, Australia; School of Business (Information Systems), Faculty of Business, Education, Law & Arts, University of Southern Queensland, Australia
| | | | - Rajendra Acharya
- School of Mathematics, Physics and Computing, University of Southern Queensland, Toowoomba, QLD, Australia
| |
Collapse
|
14
|
Hussain D, Al-Masni MA, Aslam M, Sadeghi-Niaraki A, Hussain J, Gu YH, Naqvi RA. Revolutionizing tumor detection and classification in multimodality imaging based on deep learning approaches: Methods, applications and limitations. JOURNAL OF X-RAY SCIENCE AND TECHNOLOGY 2024; 32:857-911. [PMID: 38701131 DOI: 10.3233/xst-230429] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/05/2024]
Abstract
BACKGROUND The emergence of deep learning (DL) techniques has revolutionized tumor detection and classification in medical imaging, with multimodal medical imaging (MMI) gaining recognition for its precision in diagnosis, treatment, and progression tracking. OBJECTIVE This review comprehensively examines DL methods in transforming tumor detection and classification across MMI modalities, aiming to provide insights into advancements, limitations, and key challenges for further progress. METHODS Systematic literature analysis identifies DL studies for tumor detection and classification, outlining methodologies including convolutional neural networks (CNNs), recurrent neural networks (RNNs), and their variants. Integration of multimodality imaging enhances accuracy and robustness. RESULTS Recent advancements in DL-based MMI evaluation methods are surveyed, focusing on tumor detection and classification tasks. Various DL approaches, including CNNs, YOLO, Siamese Networks, Fusion-Based Models, Attention-Based Models, and Generative Adversarial Networks, are discussed with emphasis on PET-MRI, PET-CT, and SPECT-CT. FUTURE DIRECTIONS The review outlines emerging trends and future directions in DL-based tumor analysis, aiming to guide researchers and clinicians toward more effective diagnosis and prognosis. Continued innovation and collaboration are stressed in this rapidly evolving domain. CONCLUSION Conclusions drawn from literature analysis underscore the efficacy of DL approaches in tumor detection and classification, highlighting their potential to address challenges in MMI analysis and their implications for clinical practice.
Collapse
Affiliation(s)
- Dildar Hussain
- Department of Artificial Intelligence and Data Science, Sejong University, Seoul, Korea
| | - Mohammed A Al-Masni
- Department of Artificial Intelligence and Data Science, Sejong University, Seoul, Korea
| | - Muhammad Aslam
- Department of Artificial Intelligence and Data Science, Sejong University, Seoul, Korea
| | - Abolghasem Sadeghi-Niaraki
- Department of Computer Science & Engineering and Convergence Engineering for Intelligent Drone, XR Research Center, Sejong University, Seoul, Korea
| | - Jamil Hussain
- Department of Artificial Intelligence and Data Science, Sejong University, Seoul, Korea
| | - Yeong Hyeon Gu
- Department of Artificial Intelligence and Data Science, Sejong University, Seoul, Korea
| | - Rizwan Ali Naqvi
- Department of Intelligent Mechatronics Engineering, Sejong University, Seoul, Korea
| |
Collapse
|
15
|
Yue Y, Li N, Xing W, Zhang G, Liu X, Zhu Z, Song S, Ta D. Condition control training-based ConVMLP-ResU-Net for semantic segmentation of esophageal cancer in 18F-FDG PET/CT images. Phys Eng Sci Med 2023; 46:1643-1658. [PMID: 37910383 DOI: 10.1007/s13246-023-01327-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2023] [Accepted: 08/28/2023] [Indexed: 11/03/2023]
Abstract
The precise delineation of esophageal gross tumor volume (GTV) on medical images can promote the radiotherapy effect of esophagus cancer. This work is intended to explore effective learning-based methods to tackle the challenging auto-segmentation problem of esophageal GTV. By employing the progressive hierarchical reasoning mechanism (PHRM), we devised a simple yet effective two-stage deep framework, ConVMLP-ResU-Net. Thereinto, the front-end ConVMLP integrates convolution (ConV) and multi-layer perceptrons (MLP) to capture localized and long-range spatial information, thus making ConVMLP excel in the location and coarse shape prediction of esophageal GTV. According to the PHRM, the front-end ConVMLP should have a strong generalization ability to ensure that the back-end ResU-Net has correct and valid reasoning. Therefore, a condition control training algorithm was proposed to control the training process of ConVMLP for a robust front end. Afterward, the back-end ResU-Net benefits from the yielded mask by ConVMLP to conduct a finer expansive segmentation to output the final result. Extensive experiments were carried out on a clinical cohort, which included 1138 pairs of 18F-FDG positron emission tomography/computed tomography (PET/CT) images. We report the Dice similarity coefficient, Hausdorff distance, and Mean surface distance as 0.82 ± 0.13, 4.31 ± 7.91 mm, and 1.42 ± 3.69 mm, respectively. The predicted contours visually have good agreements with the ground truths. The devised ConVMLP is apt at locating the esophageal GTV with correct initial shape prediction and hence facilitates the finer segmentation of the back-end ResU-Net. Both the qualitative and quantitative results validate the effectiveness of the proposed method.
Collapse
Affiliation(s)
- Yaoting Yue
- Center for Biomedical Engineering, School of Information Science and Technology, Fudan University, Shanghai, China
| | - Nan Li
- Department of Nuclear Medicine, Fudan University Shanghai Cancer Center, Shanghai, China
| | - Wenyu Xing
- Center for Biomedical Engineering, School of Information Science and Technology, Fudan University, Shanghai, China
| | - Gaobo Zhang
- Center for Biomedical Engineering, School of Information Science and Technology, Fudan University, Shanghai, China
| | - Xin Liu
- Academy for Engineering and Technology, Fudan University, Shanghai, China
| | - Zhibin Zhu
- School of Physics and Electromechanical Engineering, Hexi University, Zhangye, Gansu, China.
| | - Shaoli Song
- Department of Nuclear Medicine, Fudan University Shanghai Cancer Center, Shanghai, China.
| | - Dean Ta
- Center for Biomedical Engineering, School of Information Science and Technology, Fudan University, Shanghai, China
- Academy for Engineering and Technology, Fudan University, Shanghai, China
| |
Collapse
|
16
|
Alshmrani GM, Ni Q, Jiang R, Muhammed N. Hyper-Dense_Lung_Seg: Multimodal-Fusion-Based Modified U-Net for Lung Tumour Segmentation Using Multimodality of CT-PET Scans. Diagnostics (Basel) 2023; 13:3481. [PMID: 37998617 PMCID: PMC10670323 DOI: 10.3390/diagnostics13223481] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2023] [Revised: 11/09/2023] [Accepted: 11/15/2023] [Indexed: 11/25/2023] Open
Abstract
The majority of cancer-related deaths globally are due to lung cancer, which also has the second-highest mortality rate. The segmentation of lung tumours, treatment evaluation, and tumour stage classification have become significantly more accessible with the advent of PET/CT scans. With the advent of PET/CT scans, it is possible to obtain both functioning and anatomic data during a single examination. However, integrating images from different modalities can indeed be time-consuming for medical professionals and remains a challenging task. This challenge arises from several factors, including differences in image acquisition techniques, image resolutions, and the inherent variations in the spectral and temporal data captured by different imaging modalities. Artificial Intelligence (AI) methodologies have shown potential in the automation of image integration and segmentation. To address these challenges, multimodal fusion approach-based U-Net architecture (early fusion, late fusion, dense fusion, hyper-dense fusion, and hyper-dense VGG16 U-Net) are proposed for lung tumour segmentation. Dice scores of 73% show that hyper-dense VGG16 U-Net is superior to the other four proposed models. The proposed method can potentially aid medical professionals in detecting lung cancer at an early stage.
Collapse
Affiliation(s)
- Goram Mufarah Alshmrani
- School of Computing and Commutations, Lancaster University, Lancaster LA1 4YW, UK; (Q.N.); (R.J.)
- College of Computing and Information Technology, University of Bisha, Bisha 67714, Saudi Arabia
| | - Qiang Ni
- School of Computing and Commutations, Lancaster University, Lancaster LA1 4YW, UK; (Q.N.); (R.J.)
| | - Richard Jiang
- School of Computing and Commutations, Lancaster University, Lancaster LA1 4YW, UK; (Q.N.); (R.J.)
| | - Nada Muhammed
- Computers and Control Engineering Department, Faculty of Engineering, Tanta University, Tanta 31733, Egypt;
| |
Collapse
|
17
|
Li L, Tan J, Yu L, Li C, Nan H, Zheng S. LSAM: L2-norm self-attention and latent space feature interaction for automatic 3D multi-modal head and neck tumor segmentation. Phys Med Biol 2023; 68:225004. [PMID: 37852283 DOI: 10.1088/1361-6560/ad04a8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/11/2023] [Accepted: 10/18/2023] [Indexed: 10/20/2023]
Abstract
Objective.Head and neck (H&N) cancers are prevalent globally, and early and accurate detection is absolutely crucial for timely and effective treatment. However, the segmentation of H&N tumors is challenging due to the similar density of the tumors and surrounding tissues in CT images. While positron emission computed tomography (PET) images provide information about the metabolic activity of the tissue and can distinguish between lesion regions and normal tissue. But they are limited by their low spatial resolution. To fully leverage the complementary information from PET and CT images, we propose a novel and innovative multi-modal tumor segmentation method specifically designed for H&N tumor segmentation.Approach.The proposed novel and innovative multi-modal tumor segmentation network (LSAM) consists of two key learning modules, namely L2-Norm self-attention and latent space feature interaction, which exploit the high sensitivity of PET images and the anatomical information of CT images. These two advanced modules contribute to a powerful 3D segmentation network based on a U-shaped structure. The well-designed segmentation method can integrate complementary features from different modalities at multiple scales, thereby improving the feature interaction between modalities.Main results.We evaluated the proposed method on the public HECKTOR PET-CT dataset, and the experimental results demonstrate that the proposed method convincingly outperforms existing H&N tumor segmentation methods in terms of key evaluation metrics, including DSC (0.8457), Jaccard (0.7756), RVD (0.0938), and HD95 (11.75).Significance.The innovative Self-Attention mechanism based on L2-Norm offers scalability and is effective in reducing the impact of outliers on the performance of the model. And the novel method for multi-scale feature interaction based on Latent Space utilizes the learning process in the encoder phase to achieve the best complementary effects among different modalities.
Collapse
Affiliation(s)
- Laquan Li
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China
- School of Science, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China
| | - Jiaxin Tan
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China
| | - Lei Yu
- Emergency Department, The Second Affiliated Hospital of Chongqing Medical University, Chongqing, People's Republic of China
| | - Chunwen Li
- Emergency Department, The Second Affiliated Hospital of Chongqing Medical University, Chongqing, People's Republic of China
| | - Hai Nan
- College of Computer Science and Engineering, Chongqing University of Technology, Chongqing, People's Republic of China
| | - Shenhai Zheng
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China
| |
Collapse
|
18
|
Feuerecker B, Heimer MM, Geyer T, Fabritius MP, Gu S, Schachtner B, Beyer L, Ricke J, Gatidis S, Ingrisch M, Cyran CC. Artificial Intelligence in Oncological Hybrid Imaging. Nuklearmedizin 2023; 62:296-305. [PMID: 37802057 DOI: 10.1055/a-2157-6810] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/08/2023]
Abstract
BACKGROUND Artificial intelligence (AI) applications have become increasingly relevant across a broad spectrum of settings in medical imaging. Due to the large amount of imaging data that is generated in oncological hybrid imaging, AI applications are desirable for lesion detection and characterization in primary staging, therapy monitoring, and recurrence detection. Given the rapid developments in machine learning (ML) and deep learning (DL) methods, the role of AI will have significant impact on the imaging workflow and will eventually improve clinical decision making and outcomes. METHODS AND RESULTS The first part of this narrative review discusses current research with an introduction to artificial intelligence in oncological hybrid imaging and key concepts in data science. The second part reviews relevant examples with a focus on applications in oncology as well as discussion of challenges and current limitations. CONCLUSION AI applications have the potential to leverage the diagnostic data stream with high efficiency and depth to facilitate automated lesion detection, characterization, and therapy monitoring to ultimately improve quality and efficiency throughout the medical imaging workflow. The goal is to generate reproducible, structured, quantitative diagnostic data for evidence-based therapy guidance in oncology. However, significant challenges remain regarding application development, benchmarking, and clinical implementation. KEY POINTS · Hybrid imaging generates a large amount of multimodality medical imaging data with high complexity and depth.. · Advanced tools are required to enable fast and cost-efficient processing along the whole radiology value chain.. · AI applications promise to facilitate the assessment of oncological disease in hybrid imaging with high quality and efficiency for lesion detection, characterization, and response assessment. The goal is to generate reproducible, structured, quantitative diagnostic data for evidence-based oncological therapy guidance.. · Selected applications in three oncological entities (lung, prostate, and neuroendocrine tumors) demonstrate how AI algorithms may impact imaging-based tasks in hybrid imaging and potentially guide clinical decision making..
Collapse
Affiliation(s)
- Benedikt Feuerecker
- Department of Radiology, University Hospital, LMU Munich, Munich, Germany
- German Cancer Research Center (DKFZ), Partner site Munich, DKTK German Cancer Consortium, Munich, Germany
| | - Maurice M Heimer
- Department of Radiology, University Hospital, LMU Munich, Munich, Germany
| | - Thomas Geyer
- Department of Radiology, University Hospital, LMU Munich, Munich, Germany
| | | | - Sijing Gu
- Department of Radiology, University Hospital, LMU Munich, Munich, Germany
| | | | - Leonie Beyer
- Department of Nuclear Medicine, University Hospital, LMU Munich, Munich, Germany
| | - Jens Ricke
- Department of Radiology, University Hospital, LMU Munich, Munich, Germany
| | - Sergios Gatidis
- Department of Radiology, University Hospital Tübingen, Tübingen, Germany
- MPI, Max Planck Institute for Intelligent Systems, Tübingen, Germany
| | - Michael Ingrisch
- Department of Radiology, University Hospital, LMU Munich, Munich, Germany
| | - Clemens C Cyran
- Department of Radiology, University Hospital, LMU Munich, Munich, Germany
| |
Collapse
|
19
|
Zhang W, Ray S. From coarse to fine: a deep 3D probability volume contours framework for tumour segmentation and dose painting in PET images. FRONTIERS IN RADIOLOGY 2023; 3:1225215. [PMID: 37745205 PMCID: PMC10512384 DOI: 10.3389/fradi.2023.1225215] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 05/18/2023] [Accepted: 08/21/2023] [Indexed: 09/26/2023]
Abstract
With the increasing integration of functional imaging techniques like Positron Emission Tomography (PET) into radiotherapy (RT) practices, a paradigm shift in cancer treatment methodologies is underway. A fundamental step in RT planning is the accurate segmentation of tumours based on clinical diagnosis. Furthermore, novel tumour control methods, such as intensity modulated radiation therapy (IMRT) dose painting, demand the precise delineation of multiple intensity value contours to ensure optimal tumour dose distribution. Recently, convolutional neural networks (CNNs) have made significant strides in 3D image segmentation tasks, most of which present the output map at a voxel-wise level. However, because of information loss in subsequent downsampling layers, they frequently fail to precisely identify precise object boundaries. Moreover, in the context of dose painting strategies, there is an imperative need for reliable and precise image segmentation techniques to delineate high recurrence-risk contours. To address these challenges, we introduce a 3D coarse-to-fine framework, integrating a CNN with a kernel smoothing-based probability volume contour approach (KsPC). This integrated approach generates contour-based segmentation volumes, mimicking expert-level precision and providing accurate probability contours crucial for optimizing dose painting/IMRT strategies. Our final model, named KsPC-Net, leverages a CNN backbone to automatically learn parameters in the kernel smoothing process, thereby obviating the need for user-supplied tuning parameters. The 3D KsPC-Net exploits the strength of KsPC to simultaneously identify object boundaries and generate corresponding probability volume contours, which can be trained within an end-to-end framework. The proposed model has demonstrated promising performance, surpassing state-of-the-art models when tested against the MICCAI 2021 challenge dataset (HECKTOR).
Collapse
Affiliation(s)
- Wenhui Zhang
- School of Mathematics and Statistics, University of Glasgow, Glasgow, United Kingdom
| | | |
Collapse
|
20
|
Li L, Jiang C, Wang PSP, Zheng S. 3D PET/CT Tumor Co-Segmentation Based on Background Subtraction Hybrid Active Contour Model. INT J PATTERN RECOGN 2023; 37. [DOI: 10.1142/s0218001423570069] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/15/2024]
Abstract
Accurate tumor segmentation in medical images plays an important role in clinical diagnosis and disease analysis. However, medical images usually have great complexity, such as low contrast of computed tomography (CT) or low spatial resolution of positron emission tomography (PET). In the actual radiotherapy plan, multimodal imaging technology, such as PET/CT, is often used. PET images provide basic metabolic information and CT images provide anatomical details. In this paper, we propose a 3D PET/CT tumor co-segmentation framework based on active contour model. First, a new edge stop function (ESF) based on PET image and CT image is defined, which combines the grayscale standard deviation information of the image and is more effective for blurry medical image edges. Second, we propose a background subtraction model to solve the problem of uneven grayscale level in medical images. Apart from that, the calculation format adopts the level set algorithm based on the additive operator splitting (AOS) format. The solution is unconditionally stable and eliminates the dependence on time step size. Experimental results on a dataset of 50 pairs of PET/CT images of non-small cell lung cancer patients show that the proposed method has a good performance for tumor segmentation.
Collapse
Affiliation(s)
- Laquan Li
- School of Science, Chongqing University of Posts and Telecommunications, Chongqing 400065, P. R. China
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, P. R. China
| | - Chuangbo Jiang
- School of Science, Chongqing University of Posts and Telecommunications, Chongqing 400065, P. R. China
| | - Patrick Shen-Pei Wang
- College of Computer and Information Science, Northeastern University, Boston 02115, USA
| | - Shenhai Zheng
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, P. R. China
- College of Computer Science, Chongqing University, Chongqing 400044, P. R. China
| |
Collapse
|
21
|
Currie GM, Rohren EM. Radiation Dosimetry, Artificial Intelligence and Digital Twins: Old Dog, New Tricks. Semin Nucl Med 2023; 53:457-466. [PMID: 36379728 DOI: 10.1053/j.semnuclmed.2022.10.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2022] [Accepted: 10/28/2022] [Indexed: 11/14/2022]
Abstract
Developments in artificial intelligence, particularly convolutional neural networks and deep learning, have the potential for problem solving that has previously confounded human intelligence. Accurate prediction of radiation dosimetry pre-treatment with scope to adjust dosing for optimal target and non-target tissue doses is consistent with striving for improved the outcomes of precision medicine. The combination of artificial intelligence and production of digital twins could provide an avenue for an individualised therapy doses and enhanced outcomes in theranostics. While there are barriers to overcome, the maturity of individual technologies (i.e. radiation dosimetry, artificial intelligence, theranostics and digital twins) places these approaches within reach.
Collapse
Affiliation(s)
- Geoffrey M Currie
- Charles Sturt University, NSW, Australia; Baylor College of Medicine, TX.
| | - Eric M Rohren
- Charles Sturt University, NSW, Australia; Baylor College of Medicine, TX
| |
Collapse
|
22
|
Wu Y, Jiang H, Pang W. MSRA-Net: Tumor segmentation network based on Multi-scale Residual Attention. Comput Biol Med 2023; 158:106818. [PMID: 36966557 DOI: 10.1016/j.compbiomed.2023.106818] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/16/2022] [Revised: 03/08/2023] [Accepted: 03/20/2023] [Indexed: 03/31/2023]
Abstract
Automatic Medical segmentation of medical images is an important part in the field of computer medical diagnosis, among which tumor segmentation is an important branch of medical image segmentation. Accurate automatic segmentation method is very important in medical diagnosis and treatment. Positron emission computed tomography (PET) and X-ray computed tomography (CT) images are widely used in medical image segmentation to help doctors accurately locate information such as tumor location and shape, providing metabolic and anatomical information, respectively. At present, PET/CT images have not been effectively combined in the research of medical image segmentation, and the complementary semantic information between the superficial and deep layers of neural network has not been ensured. To solve the above problems, this paper proposed a Multi-scale Residual Attention network (MSRA-Net) for tumor segmentation of PET/CT. We first use an attention-fusion based approach to automatically learn the tumor-related areas of PET images and weaken the irrelevant area. Then, the segmentation results of PET branch are processed to optimize the segmentation results of CT branch by using attention mechanism. The proposed neural network (MSRA-Net) can effectively fuse PET image and CT image, which can improve the precision of tumor segmentation by using complementary information of the multi-modal image, and reduce the uncertainty of single modal image segmentation. Proposed model uses multi-scale attention mechanism and residual module, which fuse multi-scale features to form complementary features of different scales. We compare with state-of-the-art medical image segmentation methods. The experiment showed that the Dice coefficient of the proposed network in soft tissue sarcoma and lymphoma datasets increased by 8.5% and 6.1% respectively compared with UNet, showing a significant improvement.
Collapse
|
23
|
Ramachandran P, Eswarlal T, Lehman M, Colbert Z. Assessment of Optimizers and their Performance in Autosegmenting Lung Tumors. J Med Phys 2023; 48:129-135. [PMID: 37576091 PMCID: PMC10419743 DOI: 10.4103/jmp.jmp_54_23] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/25/2023] [Revised: 05/06/2023] [Accepted: 05/14/2023] [Indexed: 08/15/2023] Open
Abstract
Purpose Optimizers are widely utilized across various domains to enhance desired outcomes by either maximizing or minimizing objective functions. In the context of deep learning, they help to minimize the loss function and improve model's performance. This study aims to evaluate the accuracy of different optimizers employed for autosegmentation of non-small cell lung cancer (NSCLC) target volumes on thoracic computed tomography images utilized in oncology. Materials and Methods The study utilized 112 patients, comprising 92 patients from "The Cancer Imaging Archive" (TCIA) and 20 of our local clinical patients, to evaluate the efficacy of various optimizers. The gross tumor volume was selected as the foreground mask for training and testing the models. Of the 92 TCIA patients, 57 were used for training and validation, and the remaining 35 for testing using nnU-Net. The performance of the final model was further evaluated on the 20 local clinical patient datasets. Six different optimizers, namely AdaDelta, AdaGrad, Adam, NAdam, RMSprop, and stochastic gradient descent (SGD), were investigated. To assess the agreement between the predicted volume and the ground truth, several metrics including Dice similarity coefficient (DSC), Jaccard index, sensitivity, precision, Hausdorff distance (HD), 95th percentile Hausdorff distance (HD95), and average symmetric surface distance (ASSD) were utilized. Results The DSC values for AdaDelta, AdaGrad, Adam, NAdam, RMSprop, and SGD were 0.75, 0.84, 0.85, 0.84, 0.83, and 0.81, respectively, for the TCIA test data. However, when the model trained on TCIA datasets was applied to the clinical datasets, the DSC, HD, HD95, and ASSD metrics showed a statistically significant decrease in performance compared to the TCIA test datasets, indicating the presence of image and/or mask heterogeneity between the data sources. Conclusion The choice of optimizer in deep learning is a critical factor that can significantly impact the performance of autosegmentation models. However, it is worth noting that the behavior of optimizers may vary when applied to new clinical datasets, which can lead to changes in models' performance. Therefore, selecting the appropriate optimizer for a specific task is essential to ensure optimal performance and generalizability of the model to different datasets.
Collapse
Affiliation(s)
- Prabhakar Ramachandran
- Department of Radiation Oncology, Cancer Services, Princess Alexandra Hospital, Woolloongabba, Queensland, Australia
| | - Tamma Eswarlal
- Department of Engineering Mathematics, College of Engineering, Koneru Lakshmaiah Education Foundation, Vaddeswaram, Andhra Pradesh, India
| | - Margot Lehman
- Department of Radiation Oncology, Cancer Services, Princess Alexandra Hospital, Woolloongabba, Queensland, Australia
| | - Zachery Colbert
- Department of Radiation Oncology, Cancer Services, Princess Alexandra Hospital, Woolloongabba, Queensland, Australia
| |
Collapse
|
24
|
Systematic Review of Tumor Segmentation Strategies for Bone Metastases. Cancers (Basel) 2023; 15:cancers15061750. [PMID: 36980636 PMCID: PMC10046265 DOI: 10.3390/cancers15061750] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2023] [Revised: 03/09/2023] [Accepted: 03/10/2023] [Indexed: 03/18/2023] Open
Abstract
Purpose: To investigate the segmentation approaches for bone metastases in differentiating benign from malignant bone lesions and characterizing malignant bone lesions. Method: The literature search was conducted in Scopus, PubMed, IEEE and MedLine, and Web of Science electronic databases following the guidelines of Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA). A total of 77 original articles, 24 review articles, and 1 comparison paper published between January 2010 and March 2022 were included in the review. Results: The results showed that most studies used neural network-based approaches (58.44%) and CT-based imaging (50.65%) out of 77 original articles. However, the review highlights the lack of a gold standard for tumor boundaries and the need for manual correction of the segmentation output, which largely explains the absence of clinical translation studies. Moreover, only 19 studies (24.67%) specifically mentioned the feasibility of their proposed methods for use in clinical practice. Conclusion: Development of tumor segmentation techniques that combine anatomical information and metabolic activities is encouraging despite not having an optimal tumor segmentation method for all applications or can compensate for all the difficulties built into data limitations.
Collapse
|
25
|
Wang F, Cheng C, Cao W, Wu Z, Wang H, Wei W, Yan Z, Liu Z. MFCNet: A multi-modal fusion and calibration networks for 3D pancreas tumor segmentation on PET-CT images. Comput Biol Med 2023; 155:106657. [PMID: 36791551 DOI: 10.1016/j.compbiomed.2023.106657] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2022] [Revised: 01/29/2023] [Accepted: 02/09/2023] [Indexed: 02/12/2023]
Abstract
In clinical diagnosis, positron emission tomography and computed tomography (PET-CT) images containing complementary information are fused. Tumor segmentation based on multi-modal PET-CT images is an important part of clinical diagnosis and treatment. However, the existing current PET-CT tumor segmentation methods mainly focus on positron emission tomography (PET) and computed tomography (CT) feature fusion, which weakens the specificity of the modality. In addition, the information interaction between different modal images is usually completed by simple addition or concatenation operations, but this has the disadvantage of introducing irrelevant information during the multi-modal semantic feature fusion, so effective features cannot be highlighted. To overcome this problem, this paper propose a novel Multi-modal Fusion and Calibration Networks (MFCNet) for tumor segmentation based on three-dimensional PET-CT images. First, a Multi-modal Fusion Down-sampling Block (MFDB) with a residual structure is developed. The proposed MFDB can fuse complementary features of multi-modal images while retaining the unique features of different modal images. Second, a Multi-modal Mutual Calibration Block (MMCB) based on the inception structure is designed. The MMCB can guide the network to focus on a tumor region by combining different branch decoding features using the attention mechanism and extracting multi-scale pathological features using a convolution kernel of different sizes. The proposed MFCNet is verified on both the public dataset (Head and Neck cancer) and the in-house dataset (pancreas cancer). The experimental results indicate that on the public and in-house datasets, the average Dice values of the proposed multi-modal segmentation network are 74.14% and 76.20%, while the average Hausdorff distances are 6.41 and 6.84, respectively. In addition, the experimental results show that the proposed MFCNet outperforms the state-of-the-art methods on the two datasets.
Collapse
Affiliation(s)
- Fei Wang
- Institute of Biomedical Engineering, School of Communication and Information Engineering, Shanghai University, Shanghai, 200444, China; Department of Medical Imaging, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, 215163, China
| | - Chao Cheng
- Department of Nuclear Medicine, The First Affiliated Hospital of Naval Medical University(Changhai Hospital), Shanghai, 200433, China
| | - Weiwei Cao
- Department of Medical Imaging, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, 215163, China
| | - Zhongyi Wu
- Department of Medical Imaging, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, 215163, China
| | - Heng Wang
- School of Electronic and Information Engineering, Changchun University of Science and Technology, Changchun, 130022, China
| | - Wenting Wei
- School of Electronic and Information Engineering, Changchun University of Science and Technology, Changchun, 130022, China
| | - Zhuangzhi Yan
- Institute of Biomedical Engineering, School of Communication and Information Engineering, Shanghai University, Shanghai, 200444, China.
| | - Zhaobang Liu
- Department of Medical Imaging, Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, 215163, China.
| |
Collapse
|
26
|
Yue Y, Li N, Zhang G, Zhu Z, Liu X, Song S, Ta D. Automatic segmentation of esophageal gross tumor volume in 18F-FDG PET/CT images via GloD-LoATUNet. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 229:107266. [PMID: 36470035 DOI: 10.1016/j.cmpb.2022.107266] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/30/2022] [Revised: 10/08/2022] [Accepted: 11/22/2022] [Indexed: 06/17/2023]
Abstract
BACKGROUND AND OBJECTIVE For esophageal squamous cell carcinoma, radiotherapy is one of the primary treatments. During the planning before radiotherapy, the intractable task is to precisely delineate the esophageal gross tumor volume (GTV) on medical images. In current clinical practice, the manual delineation suffers from high intra- and inter-rater variability, while also exhausting the oncologists on a treadmill. There is an urgent demand for effective computer-aided automatic segmentation methods. To this end, we designed a novel deep network, dubbed as GloD-LoATUNet. METHODS GloD-LoATUNet follows the effective U-shape structure. On the contractile path, the global deformable dense attention transformer (GloDAT), local attention transformer (LoAT), and convolution blocks are integrated to model long-range dependencies and localized information. On the center bridge and the expanding path, convolution blocks are adopted to upsample the extracted representations for pixel-wise semantic prediction. Between the peer-to-peer counterparts, enhanced skip connections are built to compensate for the lost spatial information and dependencies. By exploiting complementary strengths of the GloDAT, LoAT, and convolution, GloD-LoATUNet has remarkable representation learning capabilities, performing well in the prediction of the small and variable esophageal GTV. RESULTS The proposed approach was validated in the clinical positron emission tomography/computed tomography (PET/CT) cohort. For 4 different data partitions, we report the Dice similarity coefficient (DSC), Hausdorff distance (HD), and Mean surface distance (MSD) as: 0.83±0.13, 4.88±9.16 mm, and 1.40±4.11 mm; 0.84±0.12, 6.89±12.04 mm, and 1.18±3.02 mm; 0.84±0.13, 3.89±7.64 mm, and 1.28±3.68 mm; 0.86±0.09, 3.71±4.79 mm, and 0.90±0.37 mm; respectively. The predicted contours present a desirable consistency with the ground truth. CONCLUSIONS The inspiring results confirm the accuracy and generalizability of the proposed model, demonstrating the potential for automatic segmentation of esophageal GTV in clinical practice.
Collapse
Affiliation(s)
- Yaoting Yue
- Center for Biomedical Engineering, School of Information Science and Technology, Fudan University, Shanghai 200438, China; Human Phenome Institute, Fudan University, Shanghai 201203, China
| | - Nan Li
- Department of Nuclear Medicine, Fudan University Shanghai Cancer Center, Shanghai 201321, China
| | - Gaobo Zhang
- Center for Biomedical Engineering, School of Information Science and Technology, Fudan University, Shanghai 200438, China
| | - Zhibin Zhu
- School of Physics and Electromechanical Engineering, Hexi University, Zhangye 734000, Gansu, China.
| | - Xin Liu
- Academy for Engineering and Technology, Fudan University, Shanghai 200433, China
| | - Shaoli Song
- Department of Nuclear Medicine, Fudan University Shanghai Cancer Center, Shanghai 201321, China.
| | - Dean Ta
- Center for Biomedical Engineering, School of Information Science and Technology, Fudan University, Shanghai 200438, China; Academy for Engineering and Technology, Fudan University, Shanghai 200433, China.
| |
Collapse
|
27
|
Wang S, Mahon R, Weiss E, Jan N, Taylor RJ, McDonagh PR, Quinn B, Yuan L. Automated Lung Cancer Segmentation Using a PET and CT Dual-Modality Deep Learning Neural Network. Int J Radiat Oncol Biol Phys 2023; 115:529-539. [PMID: 35934160 DOI: 10.1016/j.ijrobp.2022.07.2312] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/05/2021] [Revised: 06/16/2022] [Accepted: 07/28/2022] [Indexed: 01/11/2023]
Abstract
PURPOSE To develop an automated lung tumor segmentation method for radiation therapy planning based on deep learning and dual-modality positron emission tomography (PET) and computed tomography (CT) images. METHODS AND MATERIALS A 3-dimensional (3D) convolutional neural network using inputs from diagnostic PETs and simulation CTs was constructed with 2 parallel convolution paths for independent feature extraction at multiple resolution levels and a single deconvolution path. At each resolution level, the extracted features from the convolution arms were concatenated and fed through the skip connections into the deconvolution path that produced the tumor segmentation. Our network was trained/validated/tested by a 3:1:1 split on 290 pairs of PET and CT images from patients with lung cancer treated at our clinic, with manual physician contours as the ground truth. A stratified training strategy based on the magnitude of the gross tumor volume (GTV) was investigated to improve performance, especially for small tumors. Multiple radiation oncologists assessed the clinical acceptability of the network-produced segmentations. RESULTS The mean Dice similarity coefficient, Hausdorff distance, and bidirectional local distance comparing manual versus automated contours were 0.79 ± 0.10, 5.8 ± 3.2 mm, and 2.8 ± 1.5 mm for the unstratified 3D dual-modality model. Stratification delivered the best results when the model for the large GTVs (>25 mL) was trained with all-size GTVs and the model for the small GTVs (<25 mL) was trained with small GTVs only. The best combined Dice similarity coefficient, Hausdorff distance, and bidirectional local distance from the 2 stratified models on their corresponding test data sets were 0.83 ± 0.07, 5.9 ± 2.5 mm, and 2.8 ± 1.4 mm, respectively. In the multiobserver review, 91.25% manual versus 88.75% automatic contours were accepted or accepted with modifications. CONCLUSIONS By using an expansive clinical PET and CT image database and a dual-modality architecture, the proposed 3D network with a novel GTVbased stratification strategy generated clinically useful lung cancer contours that were highly acceptable on physician review.
Collapse
Affiliation(s)
- Siqiu Wang
- Department of Radiation Oncology, Virginia Commonwealth University, Richmond, Virginia
| | - Rebecca Mahon
- Washington University School of Medicine in St Louis, St Louis, Missouri
| | - Elisabeth Weiss
- Department of Radiation Oncology, Virginia Commonwealth University, Richmond, Virginia
| | - Nuzhat Jan
- Department of Radiation Oncology, Virginia Commonwealth University, Richmond, Virginia
| | - Ross James Taylor
- Department of Radiation Oncology, Virginia Commonwealth University, Richmond, Virginia
| | - Philip Reed McDonagh
- Department of Radiation Oncology, Virginia Commonwealth University, Richmond, Virginia
| | - Bridget Quinn
- Department of Radiation Oncology, Virginia Commonwealth University, Richmond, Virginia
| | - Lulin Yuan
- Department of Radiation Oncology, Virginia Commonwealth University, Richmond, Virginia.
| |
Collapse
|
28
|
Feuerecker B, Heimer MM, Geyer T, Fabritius MP, Gu S, Schachtner B, Beyer L, Ricke J, Gatidis S, Ingrisch M, Cyran CC. Artificial Intelligence in Oncological Hybrid Imaging. ROFO-FORTSCHR RONTG 2023; 195:105-114. [PMID: 36170852 DOI: 10.1055/a-1909-7013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/04/2023]
Abstract
BACKGROUND Artificial intelligence (AI) applications have become increasingly relevant across a broad spectrum of settings in medical imaging. Due to the large amount of imaging data that is generated in oncological hybrid imaging, AI applications are desirable for lesion detection and characterization in primary staging, therapy monitoring, and recurrence detection. Given the rapid developments in machine learning (ML) and deep learning (DL) methods, the role of AI will have significant impact on the imaging workflow and will eventually improve clinical decision making and outcomes. METHODS AND RESULTS The first part of this narrative review discusses current research with an introduction to artificial intelligence in oncological hybrid imaging and key concepts in data science. The second part reviews relevant examples with a focus on applications in oncology as well as discussion of challenges and current limitations. CONCLUSION AI applications have the potential to leverage the diagnostic data stream with high efficiency and depth to facilitate automated lesion detection, characterization, and therapy monitoring to ultimately improve quality and efficiency throughout the medical imaging workflow. The goal is to generate reproducible, structured, quantitative diagnostic data for evidence-based therapy guidance in oncology. However, significant challenges remain regarding application development, benchmarking, and clinical implementation. KEY POINTS · Hybrid imaging generates a large amount of multimodality medical imaging data with high complexity and depth.. · Advanced tools are required to enable fast and cost-efficient processing along the whole radiology value chain.. · AI applications promise to facilitate the assessment of oncological disease in hybrid imaging with high quality and efficiency for lesion detection, characterization, and response assessment. The goal is to generate reproducible, structured, quantitative diagnostic data for evidence-based oncological therapy guidance.. · Selected applications in three oncological entities (lung, prostate, and neuroendocrine tumors) demonstrate how AI algorithms may impact imaging-based tasks in hybrid imaging and potentially guide clinical decision making.. CITATION FORMAT · Feuerecker B, Heimer M, Geyer T et al. Artificial Intelligence in Oncological Hybrid Imaging. Fortschr Röntgenstr 2023; 195: 105 - 114.
Collapse
Affiliation(s)
- Benedikt Feuerecker
- Department of Radiology, University Hospital, LMU Munich, Munich, Germany.,German Cancer Research Center (DKFZ), Partner site Munich, DKTK German Cancer Consortium, Munich, Germany
| | - Maurice M Heimer
- Department of Radiology, University Hospital, LMU Munich, Munich, Germany
| | - Thomas Geyer
- Department of Radiology, University Hospital, LMU Munich, Munich, Germany
| | | | - Sijing Gu
- Department of Radiology, University Hospital, LMU Munich, Munich, Germany
| | | | - Leonie Beyer
- Department of Nuclear Medicine, University Hospital, LMU Munich, Munich, Germany
| | - Jens Ricke
- Department of Radiology, University Hospital, LMU Munich, Munich, Germany
| | - Sergios Gatidis
- Department of Radiology, University Hospital Tübingen, Tübingen, Germany.,MPI, Max Planck Institute for Intelligent Systems, Tübingen, Germany
| | - Michael Ingrisch
- Department of Radiology, University Hospital, LMU Munich, Munich, Germany
| | - Clemens C Cyran
- Department of Radiology, University Hospital, LMU Munich, Munich, Germany
| |
Collapse
|
29
|
Liu Z, Mhlanga JC, Siegel BA, Jha AK. Need for objective task-based evaluation of AI-based segmentation methods for quantitative PET. PROCEEDINGS OF SPIE--THE INTERNATIONAL SOCIETY FOR OPTICAL ENGINEERING 2023; 12467:124670R. [PMID: 37990707 PMCID: PMC10659582 DOI: 10.1117/12.2647894] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/23/2023]
Abstract
Artificial intelligence (AI)-based methods are showing substantial promise in segmenting oncologic positron emission tomography (PET) images. For clinical translation of these methods, assessing their performance on clinically relevant tasks is important. However, these methods are typically evaluated using metrics that may not correlate with the task performance. One such widely used metric is the Dice score, a figure of merit that measures the spatial overlap between the estimated segmentation and a reference standard (e.g., manual segmentation). In this work, we investigated whether evaluating AI-based segmentation methods using Dice scores yields a similar interpretation as evaluation on the clinical tasks of quantifying metabolic tumor volume (MTV) and total lesion glycolysis (TLG) of primary tumor from PET images of patients with non-small cell lung cancer. The investigation was conducted via a retrospective analysis with the ECOG-ACRIN 6668/RTOG 0235 multi-center clinical trial data. Specifically, we evaluated different structures of a commonly used AI-based segmentation method using both Dice scores and the accuracy in quantifying MTV/TLG. Our results show that evaluation using Dice scores can lead to findings that are inconsistent with evaluation using the task-based figure of merit. Thus, our study motivates the need for objective task-based evaluation of AI-based segmentation methods for quantitative PET.
Collapse
Affiliation(s)
- Ziping Liu
- Department of Biomedical Engineering, Washington University in St. Louis, St. Louis, MO, USA
| | - Joyce C. Mhlanga
- Mallinckrodt Institute of Radiology, Washington University School of Medicine, St. Louis, MO, USA
| | - Barry A. Siegel
- Mallinckrodt Institute of Radiology, Washington University School of Medicine, St. Louis, MO, USA
- Alvin J. Siteman Cancer Center, Washington University School of Medicine, St. Louis, MO, USA
| | - Abhinav K. Jha
- Department of Biomedical Engineering, Washington University in St. Louis, St. Louis, MO, USA
- Mallinckrodt Institute of Radiology, Washington University School of Medicine, St. Louis, MO, USA
- Alvin J. Siteman Cancer Center, Washington University School of Medicine, St. Louis, MO, USA
| |
Collapse
|
30
|
Zheng S, Tan J, Jiang C, Li L. Automated multi-modal Transformer network (AMTNet) for 3D medical images segmentation. Phys Med Biol 2023; 68. [PMID: 36595252 DOI: 10.1088/1361-6560/aca74c] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/21/2022] [Accepted: 11/29/2022] [Indexed: 12/05/2022]
Abstract
Objective.Over the past years, convolutional neural networks based methods have dominated the field of medical image segmentation. But the main drawback of these methods is that they have difficulty representing long-range dependencies. Recently, the Transformer has demonstrated super performance in computer vision and has also been successfully applied to medical image segmentation because of the self-attention mechanism and long-range dependencies encoding on images. To the best of our knowledge, only a few works focus on cross-modalities of image segmentation using the Transformer. Hence, the main objective of this study was to design, propose and validate a deep learning method to extend the application of Transformer to multi-modality medical image segmentation.Approach.This paper proposes a novel automated multi-modal Transformer network termed AMTNet for 3D medical image segmentation. Especially, the network is a well-modeled U-shaped network architecture where many effective and significant changes have been made in the feature encoding, fusion, and decoding parts. The encoding part comprises 3D embedding, 3D multi-modal Transformer, and 3D Co-learn down-sampling blocks. Symmetrically, the 3D Transformer block, upsampling block, and 3D-expanding blocks are included in the decoding part. In addition, a Transformer-based adaptive channel interleaved Transformer feature fusion module is designed to fully fuse features of different modalities.Main results.We provide a comprehensive experimental analysis of the Prostate and BraTS2021 datasets. The results show that our method achieves an average DSC of 0.907 and 0.851 (0.734 for ET, 0.895 for TC, and 0.924 for WT) on these two datasets, respectively. These values show that AMTNet yielded significant improvements over the state-of-the-art segmentation networks.Significance.The proposed 3D segmentation network exploits complementary features of different modalities during the feature extraction process at multiple scales to increase the 3D feature representations and improve the segmentation efficiency. This powerful network enriches the research of the Transformer to multi-modal medical image segmentation.
Collapse
Affiliation(s)
- Shenhai Zheng
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, People's Republic of China.,Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing 400065, People's Republic of China
| | - Jiaxin Tan
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, People's Republic of China
| | - Chuangbo Jiang
- School of Science, Chongqing University of Posts and Telecommunications, Chongqing 400065, People's Republic of China
| | - Laquan Li
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing 400065, People's Republic of China.,School of Science, Chongqing University of Posts and Telecommunications, Chongqing 400065, People's Republic of China
| |
Collapse
|
31
|
Lei Y, Wang T, Jeong JJ, Janopaul-Naylor J, Kesarwala AH, Roper J, Tian S, Bradley JD, Liu T, Higgins K, Yang X. Automated lung tumor delineation on positron emission tomography/computed tomography via a hybrid regional network. Med Phys 2023; 50:274-283. [PMID: 36203393 PMCID: PMC9868056 DOI: 10.1002/mp.16001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/16/2021] [Revised: 09/20/2022] [Accepted: 09/20/2022] [Indexed: 01/26/2023] Open
Abstract
BACKGROUND Multimodality positron emission tomography/computed tomography (PET/CT) imaging combines the anatomical information of CT with the functional information of PET. In the diagnosis and treatment of many cancers, such as non-small cell lung cancer (NSCLC), PET/CT imaging allows more accurate delineation of tumor or involved lymph nodes for radiation planning. PURPOSE In this paper, we propose a hybrid regional network method of automatically segmenting lung tumors from PET/CT images. METHODS The hybrid regional network architecture synthesizes the functional and anatomical information from the two image modalities, whereas the mask regional convolutional neural network (R-CNN) and scoring fine-tune the regional location and quality of the output segmentation. This model consists of five major subnetworks, that is, a dual feature representation network (DFRN), a regional proposal network (RPN), a specific tumor-wise R-CNN, a mask-Net, and a score head. Given a PET/CT image as inputs, the DFRN extracts feature maps from the PET and CT images. Then, the RPN and R-CNN work together to localize lung tumors and reduce the image size and feature map size by removing irrelevant regions. The mask-Net is used to segment tumor within a volume-of-interest (VOI) with a score head evaluating the segmentation performed by the mask-Net. Finally, the segmented tumor within the VOI was mapped back to the volumetric coordinate system based on the location information derived via the RPN and R-CNN. We trained, validated, and tested the proposed neural network using 100 PET/CT images of patients with NSCLC. A fivefold cross-validation study was performed. The segmentation was evaluated with two indicators: (1) multiple metrics, including the Dice similarity coefficient, Jacard, 95th percentile Hausdorff distance, mean surface distance (MSD), residual mean square distance, and center-of-mass distance; (2) Bland-Altman analysis and volumetric Pearson correlation analysis. RESULTS In fivefold cross-validation, this method achieved Dice and MSD of 0.84 ± 0.15 and 1.38 ± 2.2 mm, respectively. A new PET/CT can be segmented in 1 s by this model. External validation on The Cancer Imaging Archive dataset (63 PET/CT images) indicates that the proposed model has superior performance compared to other methods. CONCLUSION The proposed method shows great promise to automatically delineate NSCLC tumors on PET/CT images, thereby allowing for a more streamlined clinical workflow that is faster and reduces physician effort.
Collapse
Affiliation(s)
- Yang Lei
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, School of Medicine, Atlanta, Georgia, USA
| | - Tonghe Wang
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, School of Medicine, Atlanta, Georgia, USA
| | - Jiwoong J Jeong
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, School of Medicine, Atlanta, Georgia, USA
| | - James Janopaul-Naylor
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, School of Medicine, Atlanta, Georgia, USA
| | - Aparna H Kesarwala
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, School of Medicine, Atlanta, Georgia, USA
| | - Justin Roper
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, School of Medicine, Atlanta, Georgia, USA
| | - Sibo Tian
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, School of Medicine, Atlanta, Georgia, USA
| | - Jeffrey D Bradley
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, School of Medicine, Atlanta, Georgia, USA
| | - Tian Liu
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, School of Medicine, Atlanta, Georgia, USA
| | - Kristin Higgins
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, School of Medicine, Atlanta, Georgia, USA
| | - Xiaofeng Yang
- Department of Radiation Oncology and Winship Cancer Institute, Emory University, School of Medicine, Atlanta, Georgia, USA
| |
Collapse
|
32
|
Jarabek L, Jamsek J, Cuderman A, Rep S, Hocevar M, Kocjan T, Jensterle M, Spiclin Z, Macek Lezaic Z, Cvetko F, Lezaic L. Detection and localization of hyperfunctioning parathyroid glands on [ 18F]fluorocholine PET/ CT using deep learning - model performance and comparison to human experts. Radiol Oncol 2022; 56:440-452. [PMID: 36503715 PMCID: PMC9784363 DOI: 10.2478/raon-2022-0037] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/21/2022] [Accepted: 08/22/2022] [Indexed: 12/15/2022] Open
Abstract
BACKGROUND In the setting of primary hyperparathyroidism (PHPT), [18F]fluorocholine PET/CT (FCH-PET) has excellent diagnostic performance, with experienced practitioners achieving 97.7% accuracy in localising hyperfunctioning parathyroid tissue (HPTT). Due to the relative triviality of the task for human readers, we explored the performance of deep learning (DL) methods for HPTT detection and localisation on FCH-PET images in the setting of PHPT. PATIENTS AND METHODS We used a dataset of 93 subjects with PHPT imaged using FCH-PET, of which 74 subjects had visible HPTT while 19 controls had no visible HPTT on FCH-PET. A conventional Resnet10 as well as a novel mPETResnet10 DL model were trained and tested to detect (present, not present) and localise (upper left, lower left, upper right or lower right) HPTT. Our mPETResnet10 architecture also contained a region-of-interest masking algorithm that we evaluated qualitatively in order to try to explain the model's decision process. RESULTS The models detected the presence of HPTT with an accuracy of 83% and determined the quadrant of HPTT with an accuracy of 74%. The DL methods performed statistically worse (p < 0.001) in both tasks compared to human readers, who localise HPTT with the accuracy of 97.7%. The produced region-of-interest mask, while not showing a consistent added value in the qualitative evaluation of model's decision process, had correctly identified the foreground PET signal. CONCLUSIONS Our experiment is the first reported use of DL analysis of FCH-PET in PHPT. We have shown that it is possible to utilize DL methods with FCH-PET to detect and localize HPTT. Given our small dataset of 93 subjects, results are nevertheless promising for further research.
Collapse
Affiliation(s)
- Leon Jarabek
- Department of Radiology, General Hospital Novo Mesto, Novo MestoSlovenia
| | - Jan Jamsek
- Department for Nuclear Medicine, University Medical CentreLjubljana, Slovenia
| | - Anka Cuderman
- Department for Nuclear Medicine, University Medical CentreLjubljana, Slovenia
| | - Sebastijan Rep
- Department for Nuclear Medicine, University Medical CentreLjubljana, Slovenia
- Faculty of Health Sciences, University of Ljubljana, Ljubljana, Slovenia
| | - Marko Hocevar
- Department of Surgical Oncology, Institute of Oncology, LjubljanaSlovenia
- Faculty of Medicine, University of Ljubljana, Ljubljana, Slovenia
| | - Tomaz Kocjan
- Faculty of Medicine, University of Ljubljana, Ljubljana, Slovenia
- Department for Endocrinology, Diabetes and Metabolic Diseases, University Medical CentreLjubljana, Slovenia
| | - Mojca Jensterle
- Faculty of Medicine, University of Ljubljana, Ljubljana, Slovenia
- Department for Endocrinology, Diabetes and Metabolic Diseases, University Medical CentreLjubljana, Slovenia
| | - Ziga Spiclin
- Faculty of Electrical Engineering, University of Ljubljana, LjubljanaSlovenia
| | | | - Filip Cvetko
- Faculty of Medicine, University of Ljubljana, Ljubljana, Slovenia
| | - Luka Lezaic
- Department for Nuclear Medicine, University Medical CentreLjubljana, Slovenia
- Faculty of Medicine, University of Ljubljana, Ljubljana, Slovenia
| |
Collapse
|
33
|
Zhang X, Zhang B, Deng S, Meng Q, Chen X, Xiang D. Cross modality fusion for modality-specific lung tumor segmentation in PET-CT images. Phys Med Biol 2022; 67. [DOI: 10.1088/1361-6560/ac994e] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2022] [Accepted: 10/11/2022] [Indexed: 11/09/2022]
Abstract
Abstract
Although positron emission tomography-computed tomography (PET-CT) images have been widely used, it is still challenging to accurately segment the lung tumor. The respiration, movement and imaging modality lead to large modality discrepancy of the lung tumors between PET images and CT images. To overcome these difficulties, a novel network is designed to simultaneously obtain the corresponding lung tumors of PET images and CT images. The proposed network can fuse the complementary information and preserve modality-specific features of PET images and CT images. Due to the complementarity between PET images and CT images, the two modality images should be fused for automatic lung tumor segmentation. Therefore, cross modality decoding blocks are designed to extract modality-specific features of PET images and CT images with the constraints of the other modality. The edge consistency loss is also designed to solve the problem of blurred boundaries of PET images and CT images. The proposed method is tested on 126 PET-CT images with non-small cell lung cancer, and Dice similarity coefficient scores of lung tumor segmentation reach 75.66 ± 19.42 in CT images and 79.85 ± 16.76 in PET images, respectively. Extensive comparisons with state-of-the-art lung tumor segmentation methods have also been performed to demonstrate the superiority of the proposed network.
Collapse
|
34
|
Savjani RR, Lauria M, Bose S, Deng J, Yuan Y, Andrearczyk V. Automated Tumor Segmentation in Radiotherapy. Semin Radiat Oncol 2022; 32:319-329. [DOI: 10.1016/j.semradonc.2022.06.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022]
|
35
|
Automatic lung tumor segmentation from CT images using improved 3D densely connected UNet. Med Biol Eng Comput 2022; 60:3311-3323. [DOI: 10.1007/s11517-022-02667-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/28/2020] [Accepted: 09/12/2022] [Indexed: 11/25/2022]
|
36
|
Yuan C, Shi Q, Huang X, Wang L, He Y, Li B, Zhao W, Qian D. Multimodal deep learning model on interim [ 18F]FDG PET/CT for predicting primary treatment failure in diffuse large B-cell lymphoma. Eur Radiol 2022; 33:77-88. [PMID: 36029345 DOI: 10.1007/s00330-022-09031-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/14/2022] [Revised: 05/30/2022] [Accepted: 07/13/2022] [Indexed: 11/28/2022]
Abstract
OBJECTIVES The prediction of primary treatment failure (PTF) is necessary for patients with diffuse large B-cell lymphoma (DLBCL) since it serves as a prominent means for improving front-line outcomes. Using interim 18F-fluoro-2-deoxyglucose ([18F]FDG) positron emission tomography/computed tomography (PET/CT) imaging data, we aimed to construct multimodal deep learning (MDL) models to predict possible PTF in low-risk DLBCL. METHODS Initially, 205 DLBCL patients undergoing interim [18F]FDG PET/CT scans and the front-line standard of care were included in the primary dataset for model development. Then, 44 other patients were included in the external dataset for generalization evaluation. Based on the powerful backbone of the Conv-LSTM network, we incorporated five different multimodal fusion strategies (pixel intermixing, separate channel, separate branch, quantitative weighting, and hybrid learning) to make full use of PET/CT features and built five corresponding MDL models. Moreover, we found the best model, that is, the hybrid learning model, and optimized it by integrating the contrastive training objective to further improve its prediction performance. RESULTS The final model with contrastive objective optimization, named the contrastive hybrid learning model, performed best, with an accuracy of 91.22% and an area under the receiver operating characteristic curve (AUC) of 0.926, in the primary dataset. In the external dataset, its accuracy and AUC remained at 88.64% and 0.925, respectively, indicating its good generalization ability. CONCLUSIONS The proposed model achieved good performance, validated the predictive value of interim PET/CT, and holds promise for directing individualized clinical treatment. KEY POINTS • The proposed multimodal models achieved accurate prediction of primary treatment failure in DLBCL patients. • Using an appropriate feature-level fusion strategy can make the same class close to each other regardless of the modal heterogeneity of the data source domain and positively impact the prediction performance. • Deep learning validated the predictive value of interim PET/CT in a way that exceeded human capabilities.
Collapse
Affiliation(s)
- Cheng Yuan
- School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, 200040, China
| | - Qing Shi
- Shanghai Institute of Hematology, State Key Laboratory of Medical Genomics, National Research Center for Translational Medicine at Shanghai, Ruijin Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, 200025, China
| | - Xinyun Huang
- Department of Nuclear Medicine, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200025, China
| | - Li Wang
- Shanghai Institute of Hematology, State Key Laboratory of Medical Genomics, National Research Center for Translational Medicine at Shanghai, Ruijin Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, 200025, China
| | - Yang He
- Shanghai Institute of Hematology, State Key Laboratory of Medical Genomics, National Research Center for Translational Medicine at Shanghai, Ruijin Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, 200025, China
| | - Biao Li
- Department of Nuclear Medicine, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai, 200025, China.
| | - Weili Zhao
- Shanghai Institute of Hematology, State Key Laboratory of Medical Genomics, National Research Center for Translational Medicine at Shanghai, Ruijin Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, 200025, China.
| | - Dahong Qian
- School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, 200040, China.
| |
Collapse
|
37
|
Manimegalai P, Suresh Kumar R, Valsalan P, Dhanagopal R, Vasanth Raj PT, Christhudass J. 3D Convolutional Neural Network Framework with Deep Learning for Nuclear Medicine. SCANNING 2022; 2022:9640177. [PMID: 35924105 PMCID: PMC9308558 DOI: 10.1155/2022/9640177] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/14/2022] [Accepted: 06/27/2022] [Indexed: 05/15/2023]
Abstract
Though artificial intelligence (AI) has been used in nuclear medicine for more than 50 years, more progress has been made in deep learning (DL) and machine learning (ML), which have driven the development of new AI abilities in the field. ANNs are used in both deep learning and machine learning in nuclear medicine. Alternatively, if 3D convolutional neural network (CNN) is used, the inputs may be the actual images that are being analyzed, rather than a set of inputs. In nuclear medicine, artificial intelligence reimagines and reengineers the field's therapeutic and scientific capabilities. Understanding the concepts of 3D CNN and U-Net in the context of nuclear medicine provides for a deeper engagement with clinical and research applications, as well as the ability to troubleshoot problems when they emerge. Business analytics, risk assessment, quality assurance, and basic classifications are all examples of simple ML applications. General nuclear medicine, SPECT, PET, MRI, and CT may benefit from more advanced DL applications for classification, detection, localization, segmentation, quantification, and radiomic feature extraction utilizing 3D CNNs. An ANN may be used to analyze a small dataset at the same time as traditional statistical methods, as well as bigger datasets. Nuclear medicine's clinical and research practices have been largely unaffected by the introduction of artificial intelligence (AI). Clinical and research landscapes have been fundamentally altered by the advent of 3D CNN and U-Net applications. Nuclear medicine professionals must now have at least an elementary understanding of AI principles such as neural networks (ANNs) and convolutional neural networks (CNNs).
Collapse
Affiliation(s)
- P. Manimegalai
- Department of Biomedical Engineering, Karunya Institute of Technology and Sciences, Coimbatore, India
| | - R. Suresh Kumar
- Center for System Design, Chennai Institute of Technology, Chennai, India
| | - Prajoona Valsalan
- Department of Electrical and Computer Engineering, Dhofar University, Salalah, Oman
| | - R. Dhanagopal
- Center for System Design, Chennai Institute of Technology, Chennai, India
| | - P. T. Vasanth Raj
- Center for System Design, Chennai Institute of Technology, Chennai, India
| | - Jerome Christhudass
- Department of Biomedical Engineering, Karunya Institute of Technology and Sciences, Coimbatore, India
| |
Collapse
|
38
|
Nakao T, Hanaoka S, Nomura Y, Hayashi N, Abe O. Anomaly detection in chest 18F-FDG PET/CT by Bayesian deep learning. Jpn J Radiol 2022; 40:730-739. [PMID: 35094221 PMCID: PMC9252947 DOI: 10.1007/s11604-022-01249-2] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/26/2021] [Accepted: 01/11/2022] [Indexed: 12/25/2022]
Abstract
PURPOSE To develop an anomaly detection system in PET/CT with the tracer 18F-fluorodeoxyglucose (FDG) that requires only normal PET/CT images for training and can detect abnormal FDG uptake at any location in the chest region. MATERIALS AND METHODS We trained our model based on a Bayesian deep learning framework using 1878 PET/CT scans with no abnormal findings. Our model learns the distribution of standard uptake values in these normal training images and detects out-of-normal uptake regions. We evaluated this model using 34 scans showing focal abnormal FDG uptake in the chest region. This evaluation dataset includes 28 pulmonary and 17 extrapulmonary abnormal FDG uptake foci. We performed per-voxel and per-slice receiver operating characteristic (ROC) analyses and per-lesion free-response receiver operating characteristic analysis. RESULTS Our model showed an area under the ROC curve of 0.992 on discriminating abnormal voxels and 0.852 on abnormal slices. Our model detected 41 of 45 (91.1%) of the abnormal FDG uptake foci with 12.8 false positives per scan (FPs/scan), which include 26 of 28 pulmonary and 15 of 17 extrapulmonary abnormalities. The sensitivity at 3.0 FPs/scan was 82.2% (37/45). CONCLUSION Our model trained only with normal PET/CT images successfully detected both pulmonary and extrapulmonary abnormal FDG uptake in the chest region.
Collapse
Affiliation(s)
- Takahiro Nakao
- Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8655, Japan.
| | - Shouhei Hanaoka
- Department of Radiology, The University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo-ku, Tokyo, Japan
| | - Yukihiro Nomura
- Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8655, Japan
- Center for Frontier Medical Engineering, Chiba University, 1-33 Yayoicho, Inage-ku, Chiba, Japan
| | - Naoto Hayashi
- Department of Computational Diagnostic Radiology and Preventive Medicine, The University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo-ku, Tokyo, 113-8655, Japan
| | - Osamu Abe
- Department of Radiology, The University of Tokyo Hospital, 7-3-1 Hongo, Bunkyo-ku, Tokyo, Japan
- Division of Radiology and Biomedical Engineering, Graduate School of Medicine, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, Japan
| |
Collapse
|
39
|
Wang R, Guo J, Zhou Z, Wang K, Gou S, Xu R, Sher D, Wang J. Locoregional recurrence prediction in head and neck cancer based on multi-modality and multi-view feature expansion. Phys Med Biol 2022; 67. [DOI: 10.1088/1361-6560/ac72f0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2021] [Accepted: 05/24/2022] [Indexed: 12/09/2022]
Abstract
Abstract
Objective. Locoregional recurrence (LRR) is one of the leading causes of treatment failure in head and neck (H&N) cancer. Accurately predicting LRR after radiotherapy is essential to achieving better treatment outcomes for patients with H&N cancer through developing personalized treatment strategies. We aim to develop an end-to-end multi-modality and multi-view feature extension method (MMFE) to predict LRR in H&N cancer. Approach. Deep learning (DL) has been widely used for building prediction models and has achieved great success. Nevertheless, 2D-based DL models inherently fail to utilize the contextual information from adjacent slices, while complicated 3D models have a substantially larger number of parameters, which require more training samples, memory and computing resources. In the proposed MMFE scheme, through the multi-view feature expansion and projection dimension reduction operations, we are able to reduce the model complexity while preserving volumetric information. Additionally, we designed a multi-modality convolutional neural network that can be trained in an end-to-end manner and can jointly optimize the use of deep features of CT, PET and clinical data to improve the model’s prediction ability. Main results. The dataset included 206 eligible patients, of which, 49 had LRR while 157 did not. The proposed MMFE method obtained a higher AUC value than the other four methods. The best prediction result was achieved when using all three modalities, which yielded an AUC value of 0.81. Significance. Comparison experiments demonstrated the superior performance of the MMFE as compared to other 2D/3D-DL-based methods. By combining CT, PET and clinical features, the MMFE could potentially identify H&N cancer patients at high risk for LRR such that personalized treatment strategy can be developed accordingly.
Collapse
|
40
|
Dirks I, Keyaerts M, Neyns B, Vandemeulebroucke J. Computer-aided detection and segmentation of malignant melanoma lesions on whole-body 18F-FDG PET/CT using an interpretable deep learning approach. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2022; 221:106902. [PMID: 35636357 DOI: 10.1016/j.cmpb.2022.106902] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/10/2021] [Revised: 04/27/2022] [Accepted: 05/21/2022] [Indexed: 06/15/2023]
Abstract
BACKGROUND AND OBJECTIVE In oncology, 18-fluorodeoxyglucose (18F-FDG) positron emission tomography (PET) / computed tomography (CT) is widely used to identify and analyse metabolically-active tumours. The combination of the high sensitivity and specificity from 18F-FDG PET and the high resolution from CT makes accurate assessment of disease status and treatment response possible. Since cancer is a systemic disease, whole-body imaging is of high interest. Moreover, whole-body metabolic tumour burden is emerging as a promising new biomarker predicting outcome for innovative immunotherapy in different tumour types. However, this comes with certain challenges such as the large amount of data for manual reading, different appearance of lesions across the body and cumbersome reporting, hampering its use in clinical routine. Automation of the reading can facilitate the process, maximise the information retrieved from the images and support clinicians in making treatment decisions. METHODS This work proposes a fully automated system for lesion detection and segmentation on whole-body 18F-FDG PET/CT. The novelty of the method stems from the fact that the same two-step approach used when manually reading the images was adopted, consisting of an intensity-based thresholding on PET followed by a classification that specifies which regions represent normal physiological uptake and which are malignant tissue. The dataset contained 69 patients treated for malignant melanoma. Baseline and follow-up scans together offered 267 images for training and testing. RESULTS On an unseen dataset of 53 PET/CT images, a median F1-score of 0.7500 was achieved with, on average, 1.566 false positive lesions per scan. Metabolically-active tumours were segmented with a median dice score of 0.8493 and absolute volume difference of 0.2986 ml. CONCLUSIONS The proposed fully automated method for the segmentation and detection of metabolically-active lesions on whole-body 18F-FDG PET/CT achieved competitive results. Moreover, it was compared to a direct segmentation approach which it outperformed for all metrics.
Collapse
Affiliation(s)
- Ine Dirks
- Vrije Universiteit Brussel (VUB), Department of Electronics and Informatics (ETRO), Brussels, Belgium; imec, Leuven, Belgium.
| | - Marleen Keyaerts
- Vrije Universiteit Brussel (VUB), Universitair Ziekenhuis Brussel (UZ Brussel), Department of Nuclear Medicine, Brussels, Belgium
| | - Bart Neyns
- Vrije Universiteit Brussel (VUB), Universitair Ziekenhuis Brussel (UZ Brussel), Department of Medical Oncology, Brussels, Belgium
| | - Jef Vandemeulebroucke
- Vrije Universiteit Brussel (VUB), Department of Electronics and Informatics (ETRO), Brussels, Belgium; imec, Leuven, Belgium; Vrije Universiteit Brussel (VUB), Universitair Ziekenhuis Brussel (UZ Brussel), Department of Radiology, Brussels, Belgium
| |
Collapse
|
41
|
Jiang J, Rimner A, Deasy JO, Veeraraghavan H. Unpaired Cross-Modality Educed Distillation (CMEDL) for Medical Image Segmentation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2022; 41:1057-1068. [PMID: 34855590 PMCID: PMC9128665 DOI: 10.1109/tmi.2021.3132291] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/15/2023]
Abstract
Accurate and robust segmentation of lung cancers from CT, even those located close to mediastinum, is needed to more accurately plan and deliver radiotherapy and to measure treatment response. Therefore, we developed a new cross-modality educed distillation (CMEDL) approach, using unpaired CT and MRI scans, whereby an informative teacher MRI network guides a student CT network to extract features that signal the difference between foreground and background. Our contribution eliminates two requirements of distillation methods: (i) paired image sets by using an image to image (I2I) translation and (ii) pre-training of the teacher network with a large training set by using concurrent training of all networks. Our framework uses an end-to-end trained unpaired I2I translation, teacher, and student segmentation networks. Architectural flexibility of our framework is demonstrated using 3 segmentation and 2 I2I networks. Networks were trained with 377 CT and 82 T2w MRI from different sets of patients, with independent validation (N = 209 tumors) and testing (N = 609 tumors) datasets. Network design, methods to combine MRI with CT information, distillation learning under informative (MRI to CT), weak (CT to MRI) and equal teacher (MRI to MRI), and ablation tests were performed. Accuracy was measured using Dice similarity (DSC), surface Dice (sDSC), and Hausdorff distance at the 95th percentile (HD95). The CMEDL approach was significantly (p < 0.001) more accurate (DSC of 0.77 vs. 0.73) than non-CMEDL methods with an informative teacher for CT lung tumor, with a weak teacher (DSC of 0.84 vs. 0.81) for MRI lung tumor, and with equal teacher (DSC of 0.90 vs. 0.88) for MRI multi-organ segmentation. CMEDL also reduced inter-rater lung tumor segmentation variabilities.
Collapse
|
42
|
Qiao X, Jiang C, Li P, Yuan Y, Zeng Q, Bi L, Song S, Kim J, Feng DD, Huang Q. Improving Breast Tumor Segmentation in PET via Attentive Transformation Based Normalization. IEEE J Biomed Health Inform 2022; 26:3261-3271. [PMID: 35377850 DOI: 10.1109/jbhi.2022.3164570] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
Positron Emission Tomography (PET) has become a preferred imaging modality for cancer diagnosis, radiotherapy planning, and treatment responses monitoring. Accurate and automatic tumor segmentation is the fundamental requirement for these clinical applications. Deep convolutional neural networks have become the state-of-the-art in PET tumor segmentation. The normalization process is one of the key components for accelerating network training and improving the performance of the network. However, existing normalization methods either introduce batch noise into the instance PET image by calculating statistics on batch level or introduce background noise into every single pixel by sharing the same learnable parameters spatially. In this paper, we proposed an attentive transformation (AT)-based normalization method for PET tumor segmentation. We exploit the distinguishability of breast tumor in PET images and dynamically generate dedicated and pixel-dependent learnable parameters in normalization via the transformation on a combination of channel-wise and spatial-wise attentive responses. The attentive learnable parameters allow to re-calibrate features pixel-by-pixel to focus on the high-uptake area while attenuating the background noise of PET images. Our experimental results on two real clinical datasets show that the AT-based normalization method improves breast tumor segmentation performance when compared with the existing normalization methods.
Collapse
|
43
|
Astley JR, Wild JM, Tahir BA. Deep learning in structural and functional lung image analysis. Br J Radiol 2022; 95:20201107. [PMID: 33877878 PMCID: PMC9153705 DOI: 10.1259/bjr.20201107] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/16/2022] Open
Abstract
The recent resurgence of deep learning (DL) has dramatically influenced the medical imaging field. Medical image analysis applications have been at the forefront of DL research efforts applied to multiple diseases and organs, including those of the lungs. The aims of this review are twofold: (i) to briefly overview DL theory as it relates to lung image analysis; (ii) to systematically review the DL research literature relating to the lung image analysis applications of segmentation, reconstruction, registration and synthesis. The review was conducted following the Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines. 479 studies were initially identified from the literature search with 82 studies meeting the eligibility criteria. Segmentation was the most common lung image analysis DL application (65.9% of papers reviewed). DL has shown impressive results when applied to segmentation of the whole lung and other pulmonary structures. DL has also shown great potential for applications in image registration, reconstruction and synthesis. However, the majority of published studies have been limited to structural lung imaging with only 12.9% of reviewed studies employing functional lung imaging modalities, thus highlighting significant opportunities for further research in this field. Although the field of DL in lung image analysis is rapidly expanding, concerns over inconsistent validation and evaluation strategies, intersite generalisability, transparency of methodological detail and interpretability need to be addressed before widespread adoption in clinical lung imaging workflow.
Collapse
Affiliation(s)
| | - Jim M Wild
- Department of Oncology and Metabolism, The University of Sheffield, Sheffield, United Kingdom
| | | |
Collapse
|
44
|
Segmentation and Quantitative Analysis of Photoacoustic Imaging: A Review. PHOTONICS 2022. [DOI: 10.3390/photonics9030176] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/04/2022]
Abstract
Photoacoustic imaging is an emerging biomedical imaging technique that combines optical contrast and ultrasound resolution to create unprecedented light absorption contrast in deep tissue. Thanks to its fusional imaging advantages, photoacoustic imaging can provide multiple structural and functional insights into biological tissues such as blood vasculatures and tumors and monitor the kinetic movements of hemoglobin and lipids. To better visualize and analyze the regions of interest, segmentation and quantitative analyses were used to extract several biological factors, such as the intensity level changes, diameter, and tortuosity of the tissues. Over the past 10 years, classical segmentation methods and advances in deep learning approaches have been utilized in research investigations. In this review, we provide a comprehensive review of segmentation and quantitative methods that have been developed to process photoacoustic imaging in preclinical and clinical experiments. We focus on the parametric reliability of quantitative analysis for semantic and instance-level segmentation. We also introduce the similarities and alternatives of deep learning models in qualitative measurements using classical segmentation methods for photoacoustic imaging.
Collapse
|
45
|
Punn NS, Agarwal S. Modality specific U-Net variants for biomedical image segmentation: a survey. Artif Intell Rev 2022; 55:5845-5889. [PMID: 35250146 PMCID: PMC8886195 DOI: 10.1007/s10462-022-10152-1] [Citation(s) in RCA: 28] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/09/2022] [Indexed: 02/06/2023]
Abstract
With the advent of advancements in deep learning approaches, such as deep convolution neural network, residual neural network, adversarial network; U-Net architectures are most widely utilized in biomedical image segmentation to address the automation in identification and detection of the target regions or sub-regions. In recent studies, U-Net based approaches have illustrated state-of-the-art performance in different applications for the development of computer-aided diagnosis systems for early diagnosis and treatment of diseases such as brain tumor, lung cancer, alzheimer, breast cancer, etc., using various modalities. This article contributes in presenting the success of these approaches by describing the U-Net framework, followed by the comprehensive analysis of the U-Net variants by performing (1) inter-modality, and (2) intra-modality categorization to establish better insights into the associated challenges and solutions. Besides, this article also highlights the contribution of U-Net based frameworks in the ongoing pandemic, severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) also known as COVID-19. Finally, the strengths and similarities of these U-Net variants are analysed along with the challenges involved in biomedical image segmentation to uncover promising future research directions in this area.
Collapse
|
46
|
Kao YS, Yang J. Deep learning-based auto-segmentation of lung tumor PET/CT scans: a systematic review. Clin Transl Imaging 2022. [DOI: 10.1007/s40336-022-00482-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/24/2022]
|
47
|
Protonotarios NE, Katsamenis I, Sykiotis S, Dikaios N, Kastis GA, Chatziioannou SN, Metaxas M, Doulamis N, Doulamis A. A few-shot U-Net deep learning model for lung cancer lesion segmentation via PET/CT imaging. Biomed Phys Eng Express 2022; 8. [PMID: 35144242 DOI: 10.1088/2057-1976/ac53bd] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/01/2021] [Accepted: 02/10/2022] [Indexed: 11/12/2022]
Abstract
Over the past few years, positron emission tomography/computed tomography (PET/CT) imaging for computer-aided diagnosis has received increasing attention. Supervised deep learning architectures are usually employed for the detection of abnormalities, with anatomical localization, especially in the case of CT scans. However, the main limitations of the supervised learning paradigm include (i) large amounts of data required for model training, and (ii) the assumption of fixed network weights upon training completion, implying that the performance of the model cannot be further improved after training. In order to overcome these limitations, we apply a few-shot learning (FSL) scheme. Contrary to traditional deep learning practices, in FSL the model is provided with less data during training. The model then utilizes end-user feedback after training to constantly improve its performance. We integrate FSL in a U-Net architecture for lung cancer lesion segmentation on PET/CT scans, allowing for dynamic model weight fine-tuning and resulting in an online supervised learning scheme. Constant online readjustments of the model weights according to the user's feedback, increase the detection and classification accuracy, especially in cases where low detection performance is encountered. Our proposed method is validated on the Lung-PET-CT-DX TCIA database. PET/CT scans from 87 patients were included in the dataset and were acquired 60 minutes after intravenous18F-FDG injection. Experimental results indicate the superiority of our approach compared to other state of the art methods.
Collapse
Affiliation(s)
- Nicholas E Protonotarios
- Department of Applied Mathematics and Theoretical Physics (DAMTP), University of Cambridge, University of Cambridge, Cambridge, CB3 0WA, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| | - Iason Katsamenis
- School of Rural and Surveying Engineering, National Technical University of Athens, 9, Heroon Polytechniou, Zografou, Attica, 157 73, GREECE
| | - Stavros Sykiotis
- School of Rural and Surveying Engineering, National Technical University of Athens, 9, Heroon Polytechniou, Zografou, Attica, 157 73, GREECE
| | - Nikolaos Dikaios
- Mathematics Research Center, Academy of Athens, 4, Soranou Efesiou, Athens, 115 27, GREECE
| | - George Anthony Kastis
- Mathematics Research Center, Academy of Athens, 4, Soranou Efesiou, Athens, Attica, 115 27, GREECE
| | - Sofia N Chatziioannou
- PET/CT, Biomedical Research Foundation of the Academy of Athens, 4, Soranou Efesiou, Athens, Attica, 115 27, GREECE
| | - Marinos Metaxas
- PET/CT, Biomedical Research Foundation of the Academy of Athens, 4, Soranou Efesiou, Athens, Attica, 115 27, GREECE
| | - Nikolaos Doulamis
- School of Rural and Surveying Engineering, National Technical University of Athens, 9, Heroon Polytechniou, Zografou, Attica, 157 73, GREECE
| | - Anastasios Doulamis
- School of Rural and Surveying Engineering, National Technical University of Athens, 9, Heroon Polytechniou, Zografou, Attica, 157 73, GREECE
| |
Collapse
|
48
|
Yu X, Jin F, Luo H, Lei Q, Wu Y. Gross Tumor Volume Segmentation for Stage III NSCLC Radiotherapy Using 3D ResSE-Unet. Technol Cancer Res Treat 2022; 21:15330338221090847. [PMID: 35443832 PMCID: PMC9047806 DOI: 10.1177/15330338221090847] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/22/2023] Open
Abstract
INTRODUCTION Radiotherapy is one of the most effective ways to treat lung cancer. Accurately delineating the gross target volume is a key step in the radiotherapy process. In current clinical practice, the target area is still delineated manually by radiologists, which is time-consuming and laborious. However, these problems can be better solved by deep learning-assisted automatic segmentation methods. METHODS In this paper, a 3D CNN model named 3D ResSE-Unet is proposed for gross tumor volume segmentation for stage III NSCLC radiotherapy. This model is based on 3D Unet and combines residual connection and channel attention mechanisms. Three-dimensional convolution operation and encoding-decoding structure are used to mine three-dimensional spatial information of tumors from computed tomography data. Inspired by ResNet and SE-Net, residual connection and channel attention mechanisms are used to improve segmentation performance. A total of 214 patients with stage III NSCLC were collected selectively and 148 cases were randomly selected as the training set, 30 cases as the validation set, and 36 cases as the testing set. The segmentation performance of models was evaluated by the testing set. In addition, the segmentation results of different depths of 3D Unet were analyzed. And the performance of 3D ResSE-Unet was compared with 3D Unet, 3D Res-Unet, and 3D SE-Unet. RESULTS Compared with other depths, 3D Unet with four downsampling depths is more suitable for our work. Compared with 3D Unet, 3D Res-Unet, and 3D SE-Unet, 3D ResSE-Unet can obtain superior results. Its dice similarity coefficient, 95th-percentile of Hausdorff distance, and average surface distance can reach 0.7367, 21.39mm, 4.962mm, respectively. And the average time cost of 3D ResSE-Unet to segment a patient is only about 10s. CONCLUSION The method proposed in this study provides a new tool for GTV auto-segmentation and may be useful for lung cancer radiotherapy.
Collapse
Affiliation(s)
- Xinhao Yu
- College of Bioengineering, 47913Chongqing University, Chongqing, China.,Department of radiation oncology, 605425Chongqing University Cancer Hospital, Chongqing, China
| | - Fu Jin
- Department of radiation oncology, 605425Chongqing University Cancer Hospital, Chongqing, China
| | - HuanLi Luo
- Department of radiation oncology, 605425Chongqing University Cancer Hospital, Chongqing, China
| | - Qianqian Lei
- Department of radiation oncology, 605425Chongqing University Cancer Hospital, Chongqing, China
| | - Yongzhong Wu
- Department of radiation oncology, 605425Chongqing University Cancer Hospital, Chongqing, China
| |
Collapse
|
49
|
Moreau N, Rousseau C, Fourcade C, Santini G, Brennan A, Ferrer L, Lacombe M, Guillerminet C, Colombié M, Jézéquel P, Campone M, Normand N, Rubeaux M. Automatic Segmentation of Metastatic Breast Cancer Lesions on 18F-FDG PET/CT Longitudinal Acquisitions for Treatment Response Assessment. Cancers (Basel) 2021; 14:101. [PMID: 35008265 PMCID: PMC8750371 DOI: 10.3390/cancers14010101] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/10/2021] [Revised: 12/16/2021] [Accepted: 12/21/2021] [Indexed: 11/21/2022] Open
Abstract
Metastatic breast cancer patients receive lifelong medication and are regularly monitored for disease progression. The aim of this work was to (1) propose networks to segment breast cancer metastatic lesions on longitudinal whole-body PET/CT and (2) extract imaging biomarkers from the segmentations and evaluate their potential to determine treatment response. Baseline and follow-up PET/CT images of 60 patients from the EPICUREseinmeta study were used to train two deep-learning models to segment breast cancer metastatic lesions: One for baseline images and one for follow-up images. From the automatic segmentations, four imaging biomarkers were computed and evaluated: SULpeak, Total Lesion Glycolysis (TLG), PET Bone Index (PBI) and PET Liver Index (PLI). The first network obtained a mean Dice score of 0.66 on baseline acquisitions. The second network obtained a mean Dice score of 0.58 on follow-up acquisitions. SULpeak, with a 32% decrease between baseline and follow-up, was the biomarker best able to assess patients' response (sensitivity 87%, specificity 87%), followed by TLG (43% decrease, sensitivity 73%, specificity 81%) and PBI (8% decrease, sensitivity 69%, specificity 69%). Our networks constitute promising tools for the automatic segmentation of lesions in patients with metastatic breast cancer allowing treatment response assessment with several biomarkers.
Collapse
Affiliation(s)
- Noémie Moreau
- LS2N, University of Nantes, CNRS, 44000 Nantes, France; (C.F.); (N.N.)
- Keosys Medical Imaging, 13 Imp. Serge Reggiani, 44815 Saint-Herblain, France; (G.S.); (A.B.); (M.R.)
| | - Caroline Rousseau
- CRCINA, University of Nantes, INSERM UMR1232, CNRS-ERL6001, 44000 Nantes, France; (C.R.); (P.J.)
- ICO Cancer Center, 49000 Angers, France; (L.F.); (M.L.); (C.G.); (M.C.); (M.C.)
| | - Constance Fourcade
- LS2N, University of Nantes, CNRS, 44000 Nantes, France; (C.F.); (N.N.)
- Keosys Medical Imaging, 13 Imp. Serge Reggiani, 44815 Saint-Herblain, France; (G.S.); (A.B.); (M.R.)
| | - Gianmarco Santini
- Keosys Medical Imaging, 13 Imp. Serge Reggiani, 44815 Saint-Herblain, France; (G.S.); (A.B.); (M.R.)
| | - Aislinn Brennan
- Keosys Medical Imaging, 13 Imp. Serge Reggiani, 44815 Saint-Herblain, France; (G.S.); (A.B.); (M.R.)
| | - Ludovic Ferrer
- ICO Cancer Center, 49000 Angers, France; (L.F.); (M.L.); (C.G.); (M.C.); (M.C.)
- CRCINA, University of Angers, INSERM UMR1232, CNRS-ERL6001, 49000 Angers, France
| | - Marie Lacombe
- ICO Cancer Center, 49000 Angers, France; (L.F.); (M.L.); (C.G.); (M.C.); (M.C.)
| | | | - Mathilde Colombié
- ICO Cancer Center, 49000 Angers, France; (L.F.); (M.L.); (C.G.); (M.C.); (M.C.)
| | - Pascal Jézéquel
- CRCINA, University of Nantes, INSERM UMR1232, CNRS-ERL6001, 44000 Nantes, France; (C.R.); (P.J.)
- ICO Cancer Center, 49000 Angers, France; (L.F.); (M.L.); (C.G.); (M.C.); (M.C.)
| | - Mario Campone
- ICO Cancer Center, 49000 Angers, France; (L.F.); (M.L.); (C.G.); (M.C.); (M.C.)
- CRCINA, University of Angers, INSERM UMR1232, CNRS-ERL6001, 49000 Angers, France
| | - Nicolas Normand
- LS2N, University of Nantes, CNRS, 44000 Nantes, France; (C.F.); (N.N.)
| | - Mathieu Rubeaux
- Keosys Medical Imaging, 13 Imp. Serge Reggiani, 44815 Saint-Herblain, France; (G.S.); (A.B.); (M.R.)
| |
Collapse
|
50
|
Oreiller V, Andrearczyk V, Jreige M, Boughdad S, Elhalawani H, Castelli J, Vallières M, Zhu S, Xie J, Peng Y, Iantsen A, Hatt M, Yuan Y, Ma J, Yang X, Rao C, Pai S, Ghimire K, Feng X, Naser MA, Fuller CD, Yousefirizi F, Rahmim A, Chen H, Wang L, Prior JO, Depeursinge A. Head and neck tumor segmentation in PET/CT: The HECKTOR challenge. Med Image Anal 2021; 77:102336. [PMID: 35016077 DOI: 10.1016/j.media.2021.102336] [Citation(s) in RCA: 58] [Impact Index Per Article: 19.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2021] [Revised: 10/13/2021] [Accepted: 12/14/2021] [Indexed: 12/23/2022]
Abstract
This paper relates the post-analysis of the first edition of the HEad and neCK TumOR (HECKTOR) challenge. This challenge was held as a satellite event of the 23rd International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI) 2020, and was the first of its kind focusing on lesion segmentation in combined FDG-PET and CT image modalities. The challenge's task is the automatic segmentation of the Gross Tumor Volume (GTV) of Head and Neck (H&N) oropharyngeal primary tumors in FDG-PET/CT images. To this end, the participants were given a training set of 201 cases from four different centers and their methods were tested on a held-out set of 53 cases from a fifth center. The methods were ranked according to the Dice Score Coefficient (DSC) averaged across all test cases. An additional inter-observer agreement study was organized to assess the difficulty of the task from a human perspective. 64 teams registered to the challenge, among which 10 provided a paper detailing their approach. The best method obtained an average DSC of 0.7591, showing a large improvement over our proposed baseline method and the inter-observer agreement, associated with DSCs of 0.6610 and 0.61, respectively. The automatic methods proved to successfully leverage the wealth of metabolic and structural properties of combined PET and CT modalities, significantly outperforming human inter-observer agreement level, semi-automatic thresholding based on PET images as well as other single modality-based methods. This promising performance is one step forward towards large-scale radiomics studies in H&N cancer, obviating the need for error-prone and time-consuming manual delineation of GTVs.
Collapse
Affiliation(s)
- Valentin Oreiller
- Institute of Information Systems, University of Applied Sciences Western Switzerland (HES-SO), Sierre, Switzerland; Department of Nuclear Medicine and Molecular Imaging, Centre Hospitalier Universitaire Vaudois (CHUV), Lausanne, Switzerland.
| | - Vincent Andrearczyk
- Institute of Information Systems, University of Applied Sciences Western Switzerland (HES-SO), Sierre, Switzerland
| | - Mario Jreige
- Department of Nuclear Medicine and Molecular Imaging, Centre Hospitalier Universitaire Vaudois (CHUV), Lausanne, Switzerland
| | - Sarah Boughdad
- Department of Nuclear Medicine and Molecular Imaging, Centre Hospitalier Universitaire Vaudois (CHUV), Lausanne, Switzerland
| | - Hesham Elhalawani
- Department of Radiation Oncology, Brigham and Women's Hospital and Dana-Farber Cancer Institute, Harvard Medical School, Boston, MA, USA
| | - Joel Castelli
- Radiotherapy Department, Cancer Institute Eugène Marquis, Rennes, France
| | - Martin Vallières
- Department of Computer Science, Université de Sherbrooke, Sherbrooke, Québec, Canada
| | - Simeng Zhu
- Department of Radiation Oncology, Henry Ford Cancer Institute, Detroit, MI, USA
| | - Juanying Xie
- School of Computer Science, Shaanxi Normal University, Xi'an 710119, PR China
| | - Ying Peng
- School of Computer Science, Shaanxi Normal University, Xi'an 710119, PR China
| | - Andrei Iantsen
- LaTIM, INSERM, UMR 1101, University Brest, Brest, France
| | - Mathieu Hatt
- LaTIM, INSERM, UMR 1101, University Brest, Brest, France
| | - Yading Yuan
- Department of Radiation Oncology, Icahn School of Medicine at Mount Sinai, New York, NY, USA
| | - Jun Ma
- Department of Mathematics, Nanjing University of Science and Technology, Jiangsu, China
| | - Xiaoping Yang
- Department of Mathematics, Nanjing University, Jiangsu, China
| | - Chinmay Rao
- Department of Radiation Oncology (Maastro), GROW School for Oncology, Maastricht University Medical Centre+, Maastricht, The Netherlands
| | - Suraj Pai
- Department of Radiation Oncology (Maastro), GROW School for Oncology, Maastricht University Medical Centre+, Maastricht, The Netherlands
| | | | - Xue Feng
- Carina Medical, Lexington, KY, 40513, USA; Department of Biomedical Engineering, University of Virginia, Charlottesville VA 22903, USA
| | - Mohamed A Naser
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, Texas 77030, USA
| | - Clifton D Fuller
- Department of Radiation Oncology, The University of Texas MD Anderson Cancer Center, Houston, Texas 77030, USA
| | - Fereshteh Yousefirizi
- Department of Integrative Oncology, BC Cancer Research Institute, Vancouver BC, Canada
| | - Arman Rahmim
- Department of Integrative Oncology, BC Cancer Research Institute, Vancouver BC, Canada
| | - Huai Chen
- Department of Automation, Institute of Image Processing and Pattern Recognition, Shangai Jiao Tong University, Shanghai 200240, People's Republic of China
| | - Lisheng Wang
- Department of Automation, Institute of Image Processing and Pattern Recognition, Shangai Jiao Tong University, Shanghai 200240, People's Republic of China
| | - John O Prior
- Department of Nuclear Medicine and Molecular Imaging, Centre Hospitalier Universitaire Vaudois (CHUV), Lausanne, Switzerland
| | - Adrien Depeursinge
- Institute of Information Systems, University of Applied Sciences Western Switzerland (HES-SO), Sierre, Switzerland; Department of Nuclear Medicine and Molecular Imaging, Centre Hospitalier Universitaire Vaudois (CHUV), Lausanne, Switzerland
| |
Collapse
|