1
|
Baldini G, Hosch R, Schmidt CS, Borys K, Kroll L, Koitka S, Haubold P, Pelka O, Nensa F, Haubold J. Addressing the Contrast Media Recognition Challenge: A Fully Automated Machine Learning Approach for Predicting Contrast Phases in CT Imaging. Invest Radiol 2024; 59:635-645. [PMID: 38436405 DOI: 10.1097/rli.0000000000001071] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/05/2024]
Abstract
OBJECTIVES Accurately acquiring and assigning different contrast-enhanced phases in computed tomography (CT) is relevant for clinicians and for artificial intelligence orchestration to select the most appropriate series for analysis. However, this information is commonly extracted from the CT metadata, which is often wrong. This study aimed at developing an automatic pipeline for classifying intravenous (IV) contrast phases and additionally for identifying contrast media in the gastrointestinal tract (GIT). MATERIALS AND METHODS This retrospective study used 1200 CT scans collected at the investigating institution between January 4, 2016 and September 12, 2022, and 240 CT scans from multiple centers from The Cancer Imaging Archive for external validation. The open-source segmentation algorithm TotalSegmentator was used to identify regions of interest (pulmonary artery, aorta, stomach, portal/splenic vein, liver, portal vein/hepatic veins, inferior vena cava, duodenum, small bowel, colon, left/right kidney, urinary bladder), and machine learning classifiers were trained with 5-fold cross-validation to classify IV contrast phases (noncontrast, pulmonary arterial, arterial, venous, and urographic) and GIT contrast enhancement. The performance of the ensembles was evaluated using the receiver operating characteristic area under the curve (AUC) and 95% confidence intervals (CIs). RESULTS For the IV phase classification task, the following AUC scores were obtained for the internal test set: 99.59% [95% CI, 99.58-99.63] for the noncontrast phase, 99.50% [95% CI, 99.49-99.52] for the pulmonary-arterial phase, 99.13% [95% CI, 99.10-99.15] for the arterial phase, 99.8% [95% CI, 99.79-99.81] for the venous phase, and 99.7% [95% CI, 99.68-99.7] for the urographic phase. For the external dataset, a mean AUC of 97.33% [95% CI, 97.27-97.35] and 97.38% [95% CI, 97.34-97.41] was achieved for all contrast phases for the first and second annotators, respectively. Contrast media in the GIT could be identified with an AUC of 99.90% [95% CI, 99.89-99.9] in the internal dataset, whereas in the external dataset, an AUC of 99.73% [95% CI, 99.71-99.73] and 99.31% [95% CI, 99.27-99.33] was achieved with the first and second annotator, respectively. CONCLUSIONS The integration of open-source segmentation networks and classifiers effectively classified contrast phases and identified GIT contrast enhancement using anatomical landmarks.
Collapse
Affiliation(s)
- Giulia Baldini
- From the Institute of Interventional and Diagnostic Radiology and Neuroradiology, University Hospital Essen, Essen, Germany (G.B., R.H., K.B., L.K., S.K., F.N., J.H.); Institute for Artificial Intelligence in Medicine, University Hospital Essen, Essen, Germany (G.B., R.H., C.S.S., K.B., L.K., S.K., O.P., F.N., J.H.); Institute for Transfusion Medicine, University Hospital Essen, Essen, Germany (C.S.S.); Department of Diagnostic and Interventional Radiology, Kliniken Essen-Mitte, Essen, Germany (P.H.); and Data Integration Center, Central IT Department, University Hospital Essen, Essen, Germany (O.P., F.N.)
| | | | | | | | | | | | | | | | | | | |
Collapse
|
2
|
Boneš E, Gergolet M, Bohak C, Lesar Ž, Marolt M. Automatic Segmentation and Alignment of Uterine Shapes from 3D Ultrasound Data. Comput Biol Med 2024; 178:108794. [PMID: 38941903 DOI: 10.1016/j.compbiomed.2024.108794] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/31/2023] [Revised: 06/18/2024] [Accepted: 06/19/2024] [Indexed: 06/30/2024]
Abstract
BACKGROUND The uterus is the most important organ in the female reproductive system. Its shape plays a critical role in fertility and pregnancy outcomes. Advances in medical imaging, such as 3D ultrasound, have significantly improved the exploration of the female genital tract, thereby enhancing gynecological healthcare. Despite well-documented data for organs like the liver and heart, large-scale studies on the uterus are lacking. Existing classifications, such as VCUAM and ESHRE/ESGE, provide different definitions for normal uterine shapes but are not based on real-world measurements. Moreover, the lack of comprehensive datasets significantly hinders research in this area. Our research, part of the larger NURSE study, aims to fill this gap by establishing the shape of a normal uterus using real-world 3D vaginal ultrasound scans. This will facilitate research into uterine shape abnormalities associated with infertility and recurrent miscarriages. METHODS We developed an automated system for the segmentation and alignment of uterine shapes from 3D ultrasound data, which consists of two steps: automatic segmentation of the uteri in 3D ultrasound scans using deep learning techniques, and alignment of the resulting shapes with standard geometrical approaches, enabling the extraction of the normal shape for future analysis. The system was trained and validated on a comprehensive dataset of 3D ultrasound images from multiple medical centers. Its performance was evaluated by comparing the automated results with manual annotations provided by expert clinicians. RESULTS The presented approach demonstrated high accuracy in segmenting and aligning uterine shapes from 3D ultrasound data. The segmentation achieved an average Dice similarity coefficient (DSC) of 0.90. Our method for aligning uterine shapes showed minimal translation and rotation errors compared to traditional methods, with the preliminary average shape exhibiting characteristics consistent with expert findings of a normal uterus. CONCLUSION We have presented an approach to automatically segment and align uterine shapes from 3D ultrasound data. We trained a deep learning nnU-Net model that achieved high accuracy and proposed an alignment method using a combination of standard geometrical techniques. Additionally, we have created a publicly available dataset of 3D transvaginal ultrasound volumes with manual annotations of uterine cavities to support further research and development in this field. The dataset and the trained models are available at https://github.com/UL-FRI-LGM/UterUS.
Collapse
Affiliation(s)
- Eva Boneš
- University of Ljubljana, Faculty of Computer and Information Science, Večna pot 113, Ljubljana, 1000, Slovenia.
| | - Marco Gergolet
- University of Ljubljana, Faculty of Medicine, Vrazov trg 2, Ljubljana, 1000, Slovenia.
| | - Ciril Bohak
- University of Ljubljana, Faculty of Computer and Information Science, Večna pot 113, Ljubljana, 1000, Slovenia; King Abdullah University of Science and Technology, Visual Computing Center, Thuwal, 23955-6900, Saudi Arabia.
| | - Žiga Lesar
- University of Ljubljana, Faculty of Computer and Information Science, Večna pot 113, Ljubljana, 1000, Slovenia.
| | - Matija Marolt
- University of Ljubljana, Faculty of Computer and Information Science, Večna pot 113, Ljubljana, 1000, Slovenia.
| |
Collapse
|
3
|
Goyanes E, de Moura J, Fernández-Vigo JI, García-Feijóo J, Novo J, Ortega M. 3D Features Fusion for Automated Segmentation of Fluid Regions in CSCR Patients: An OCT-based Photodynamic Therapy Response Analysis. JOURNAL OF IMAGING INFORMATICS IN MEDICINE 2024:10.1007/s10278-024-01190-y. [PMID: 39075249 DOI: 10.1007/s10278-024-01190-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/28/2023] [Revised: 05/02/2024] [Accepted: 05/03/2024] [Indexed: 07/31/2024]
Abstract
Central Serous Chorioretinopathy (CSCR) is a significant cause of vision impairment worldwide, with Photodynamic Therapy (PDT) emerging as a promising treatment strategy. The capability to precisely segment fluid regions in Optical Coherence Tomography (OCT) scans and predict the response to PDT treatment can substantially augment patient outcomes. This paper introduces a novel deep learning (DL) methodology for automated 3D segmentation of fluid regions in OCT scans, followed by a subsequent PDT response analysis for CSCR patients. Our approach utilizes the rich 3D contextual information from OCT scans to train a model that accurately delineates fluid regions. This model not only substantially reduces the time and effort required for segmentation but also offers a standardized technique, fostering further large-scale research studies. Additionally, by incorporating pre- and post-treatment OCT scans, our model is capable of predicting PDT response, hence enabling the formulation of personalized treatment strategies and optimized patient management. To validate our approach, we employed a robust dataset comprising 2,769 OCT scans (124 3D volumes), and the results obtained were significantly satisfactory, outperforming the current state-of-the-art methods. This research signifies an important milestone in the integration of DL advancements with practical clinical applications, propelling us a step closer towards improved management of CSCR. Furthermore, the methodologies and systems developed can be adapted and extrapolated to tackle similar challenges in the diagnosis and treatment of other retinal pathologies, favoring more comprehensive and personalized patient care.
Collapse
Affiliation(s)
- Elena Goyanes
- VARPA Research Group, Instituto de Investigación Biomédica de A Coruña (INIBIC), Universidade da Coruña, A Coruña, Spain
| | - Joaquim de Moura
- VARPA Research Group, Instituto de Investigación Biomédica de A Coruña (INIBIC), Universidade da Coruña, A Coruña, Spain.
| | - José I Fernández-Vigo
- Retina Unit, Ophthalmology Department, Hospital Clínico San Carlos, IdISSC, Madrid, Spain
| | - Julián García-Feijóo
- Retina Unit, Ophthalmology Department, Hospital Clínico San Carlos, IdISSC, Madrid, Spain
| | - Jorge Novo
- VARPA Research Group, Instituto de Investigación Biomédica de A Coruña (INIBIC), Universidade da Coruña, A Coruña, Spain
| | - Marcos Ortega
- VARPA Research Group, Instituto de Investigación Biomédica de A Coruña (INIBIC), Universidade da Coruña, A Coruña, Spain
| |
Collapse
|
4
|
Hu W, Yang S, Guo W, Xiao N, Yang X, Ren X. STC-UNet: renal tumor segmentation based on enhanced feature extraction at different network levels. BMC Med Imaging 2024; 24:179. [PMID: 39030510 PMCID: PMC11264758 DOI: 10.1186/s12880-024-01359-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2024] [Accepted: 07/08/2024] [Indexed: 07/21/2024] Open
Abstract
Renal tumors are one of the common diseases of urology, and precise segmentation of these tumors plays a crucial role in aiding physicians to improve diagnostic accuracy and treatment effectiveness. Nevertheless, inherent challenges associated with renal tumors, such as indistinct boundaries, morphological variations, and uncertainties in size and location, segmenting renal tumors accurately remains a significant challenge in the field of medical image segmentation. With the development of deep learning, substantial achievements have been made in the domain of medical image segmentation. However, existing models lack specificity in extracting features of renal tumors across different network hierarchies, which results in insufficient extraction of renal tumor features and subsequently affects the accuracy of renal tumor segmentation. To address this issue, we propose the Selective Kernel, Vision Transformer, and Coordinate Attention Enhanced U-Net (STC-UNet). This model aims to enhance feature extraction, adapting to the distinctive characteristics of renal tumors across various network levels. Specifically, the Selective Kernel modules are introduced in the shallow layers of the U-Net, where detailed features are more abundant. By selectively employing convolutional kernels of different scales, the model enhances its capability to extract detailed features of renal tumors across multiple scales. Subsequently, in the deeper layers of the network, where feature maps are smaller yet contain rich semantic information, the Vision Transformer modules are integrated in a non-patch manner. These assist the model in capturing long-range contextual information globally. Their non-patch implementation facilitates the capture of fine-grained features, thereby achieving collaborative enhancement of global-local information and ultimately strengthening the model's extraction of semantic features of renal tumors. Finally, in the decoder segment, the Coordinate Attention modules embedding positional information are proposed aiming to enhance the model's feature recovery and tumor region localization capabilities. Our model is validated on the KiTS19 dataset, and experimental results indicate that compared to the baseline model, STC-UNet shows improvements of 1.60%, 2.02%, 2.27%, 1.18%, 1.52%, and 1.35% in IoU, Dice, Accuracy, Precision, Recall, and F1-score, respectively. Furthermore, the experimental results demonstrate that the proposed STC-UNet method surpasses other advanced algorithms in both visual effectiveness and objective evaluation metrics.
Collapse
Affiliation(s)
- Wei Hu
- School of Electrical and Information Engineering of Zhengzhou University, Zhengzhou, China
| | - Shouyi Yang
- School of Electrical and Information Engineering of Zhengzhou University, Zhengzhou, China
| | - Weifeng Guo
- School of Electrical and Information Engineering of Zhengzhou University, Zhengzhou, China.
| | - Na Xiao
- Faculty of Engineering, Huanghe Science and Technology University, Zhengzhou, China
| | - Xiaopeng Yang
- Medical 3D Printing Center of the First Affiliated Hospital of Zhengzhou University, Zhengzhou, China.
| | - Xiangyang Ren
- Medical 3D Printing Center of the First Affiliated Hospital of Zhengzhou University, Zhengzhou, China.
| |
Collapse
|
5
|
Kumar K, Yeo AU, McIntosh L, Kron T, Wheeler G, Franich RD. Deep Learning Auto-Segmentation Network for Pediatric Computed Tomography Data Sets: Can We Extrapolate From Adults? Int J Radiat Oncol Biol Phys 2024; 119:1297-1306. [PMID: 38246249 DOI: 10.1016/j.ijrobp.2024.01.201] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/27/2023] [Revised: 12/10/2023] [Accepted: 01/07/2024] [Indexed: 01/23/2024]
Abstract
PURPOSE Artificial intelligence (AI)-based auto-segmentation models hold promise for enhanced efficiency and consistency in organ contouring for adaptive radiation therapy and radiation therapy planning. However, their performance on pediatric computed tomography (CT) data and cross-scanner compatibility remain unclear. This study aimed to evaluate the performance of AI-based auto-segmentation models trained on adult CT data when applied to pediatric data sets and explore the improvement in performance gained by including pediatric training data. It also examined their ability to accurately segment CT data acquired from different scanners. METHODS AND MATERIALS Using the nnU-Net framework, segmentation models were trained on data sets of adult, pediatric, and combined CT scans for 7 pelvic/thoracic organs. Each model was trained on 290 to 300 cases per category and organ. Training data sets included a combination of clinical data and several open repositories. The study incorporated a database of 459 pediatric (0-16 years) CT scans and 950 adults (>18 years), ensuring all scans had human expert ground-truth contours of the selected organs. Performance was evaluated based on Dice similarity coefficients (DSC) of the model-generated contours. RESULTS AI models trained exclusively on adult data underperformed on pediatric data, especially for the 0 to 2 age group: mean DSC was below 0.5 for the bladder and spleen. The addition of pediatric training data demonstrated significant improvement for all age groups, achieving a mean DSC of above 0.85 for all organs in every age group. Larger organs like the liver and kidneys maintained consistent performance for all models across age groups. No significant difference emerged in the cross-scanner performance evaluation, suggesting robust cross-scanner generalization. CONCLUSIONS For optimal segmentation across age groups, it is important to include pediatric data in the training of segmentation models. The successful cross-scanner generalization also supports the real-world clinical applicability of these AI models. This study emphasizes the significance of data set diversity in training robust AI systems for medical image interpretation tasks.
Collapse
Affiliation(s)
- Kartik Kumar
- Physical Sciences Department, Peter MacCallum Cancer Centre, Victoria, Australia; School of Science, RMIT University, Melbourne, Victoria, Australia
| | - Adam U Yeo
- Physical Sciences Department, Peter MacCallum Cancer Centre, Victoria, Australia; School of Science, RMIT University, Melbourne, Victoria, Australia; Sir Peter MacCallum Department of Oncology, University of Melbourne, Melbourne, Victoria, Australia
| | - Lachlan McIntosh
- Physical Sciences Department, Peter MacCallum Cancer Centre, Victoria, Australia; School of Science, RMIT University, Melbourne, Victoria, Australia
| | - Tomas Kron
- Physical Sciences Department, Peter MacCallum Cancer Centre, Victoria, Australia; School of Science, RMIT University, Melbourne, Victoria, Australia; Sir Peter MacCallum Department of Oncology, University of Melbourne, Melbourne, Victoria, Australia; Centre for Medical Radiation Physics, University of Wollongong, Wollongong, New South Wales, Australia
| | - Greg Wheeler
- Physical Sciences Department, Peter MacCallum Cancer Centre, Victoria, Australia; Sir Peter MacCallum Department of Oncology, University of Melbourne, Melbourne, Victoria, Australia
| | - Rick D Franich
- Physical Sciences Department, Peter MacCallum Cancer Centre, Victoria, Australia; School of Science, RMIT University, Melbourne, Victoria, Australia.
| |
Collapse
|
6
|
Zeng B, Wang H, Tao X, Shi H, Joskowicz L, Chen X. A bidirectional framework for fracture simulation and deformation-based restoration prediction in pelvic fracture surgical planning. Med Image Anal 2024; 97:103267. [PMID: 39053167 DOI: 10.1016/j.media.2024.103267] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2024] [Revised: 07/05/2024] [Accepted: 07/05/2024] [Indexed: 07/27/2024]
Abstract
Pelvic fracture is a severe trauma with life-threatening implications. Surgical reduction is essential for restoring the anatomical structure and functional integrity of the pelvis, requiring accurate preoperative planning. However, the complexity of pelvic fractures and limited data availability necessitate labor-intensive manual corrections in a clinical setting. We describe in this paper a novel bidirectional framework for automatic pelvic fracture surgical planning based on fracture simulation and structure restoration. Our fracture simulation method accounts for patient-specific pelvic structures, bone density information, and the randomness of fractures, enabling the generation of various types of fracture cases from healthy pelvises. Based on these features and on adversarial learning, we develop a novel structure restoration network to predict the deformation mapping in CT images before and after a fracture for the precise structural reconstruction of any fracture. Furthermore, a self-supervised strategy based on pelvic anatomical symmetry priors is developed to optimize the details of the restored pelvic structure. Finally, the restored pelvis is used as a template to generate a surgical reduction plan in which the fragments are repositioned in an efficient jigsaw puzzle registration manner. Extensive experiments on simulated and clinical datasets, including scans with metal artifacts, show that our method achieves good accuracy and robustness: a mean SSIM of 90.7% for restorations, with translational errors of 2.88 mm and rotational errors of 3.18°for reductions in real datasets. Our method takes 52.9 s to complete the surgical planning in the phantom study, representing a significant acceleration compared to standard clinical workflows. Our method may facilitate effective surgical planning for pelvic fractures tailored to individual patients in clinical settings.
Collapse
Affiliation(s)
- Bolun Zeng
- Institute of Biomedical Manufacturing and Life Quality Engineering, School of Mechanical Engineering, Shanghai Jiao Tong University, China
| | - Huixiang Wang
- Department of Orthopedics, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai, China
| | - Xingguang Tao
- Department of Orthopedics, Qingpu Branch of Zhongshan Hospital Affiliated to Fudan University, Shanghai, China
| | - Haochen Shi
- Institute of Biomedical Manufacturing and Life Quality Engineering, School of Mechanical Engineering, Shanghai Jiao Tong University, China
| | - Leo Joskowicz
- School of Computer Science and Engineering and the Edmond and Lily Safra Center for Brain Sciences, Hebrew University of Jerusalem, Jerusalem, Israel
| | - Xiaojun Chen
- Institute of Biomedical Manufacturing and Life Quality Engineering, School of Mechanical Engineering, Shanghai Jiao Tong University, China; Institute of Medical Robotics, Shanghai Jiao Tong University, Shanghai 200240, China.
| |
Collapse
|
7
|
Hild O, Berriet P, Nallet J, Salvi L, Lenoir M, Henriet J, Thiran JP, Auber F, Chaussy Y. Automation of Wilms' tumor segmentation by artificial intelligence. Cancer Imaging 2024; 24:83. [PMID: 38956718 PMCID: PMC11218149 DOI: 10.1186/s40644-024-00729-0] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/15/2024] [Accepted: 06/20/2024] [Indexed: 07/04/2024] Open
Abstract
BACKGROUND 3D reconstruction of Wilms' tumor provides several advantages but are not systematically performed because manual segmentation is extremely time-consuming. The objective of our study was to develop an artificial intelligence tool to automate the segmentation of tumors and kidneys in children. METHODS A manual segmentation was carried out by two experts on 14 CT scans. Then, the segmentation of Wilms' tumor and neoplastic kidney was automatically performed using the CNN U-Net and the same CNN U-Net trained according to the OV2ASSION method. The time saving for the expert was estimated depending on the number of sections automatically segmented. RESULTS When segmentations were performed manually by two experts, the inter-individual variability resulted in a Dice index of 0.95 for tumor and 0.87 for kidney. Fully automatic segmentation with the CNN U-Net yielded a poor Dice index of 0.69 for Wilms' tumor and 0.27 for kidney. With the OV2ASSION method, the Dice index varied depending on the number of manually segmented sections. For the segmentation of the Wilms' tumor and neoplastic kidney, it varied respectively from 0.97 to 0.94 for a gap of 1 (2 out of 3 sections performed manually) to 0.94 and 0.86 for a gap of 10 (1 section out of 6 performed manually). CONCLUSION Fully automated segmentation remains a challenge in the field of medical image processing. Although it is possible to use already developed neural networks, such as U-Net, we found that the results obtained were not satisfactory for segmentation of neoplastic kidneys or Wilms' tumors in children. We developed an innovative CNN U-Net training method that makes it possible to segment the kidney and its tumor with the same precision as an expert while reducing their intervention time by 80%.
Collapse
Affiliation(s)
- Olivier Hild
- Department of Pediatric Surgery, CHU Besançon, 3 boulevard Fleming, Besançon, F-25000, France
| | - Pierre Berriet
- Université de Franche-Comté, FEMTO-ST Institute, DISC, Besançon, F-25000, France
| | - Jérémie Nallet
- Department of Pediatric Surgery, CHU Besançon, 3 boulevard Fleming, Besançon, F-25000, France
| | - Lorédane Salvi
- Department of Pediatric Surgery, CHU Besançon, 3 boulevard Fleming, Besançon, F-25000, France
| | - Marion Lenoir
- Department of Radiology, CHU Besançon, Besançon, F-25000, France
| | - Julien Henriet
- Université de Franche-Comté, FEMTO-ST Institute, DISC, Besançon, F-25000, France
| | - Jean-Philippe Thiran
- Signal Processing Laboratory 5 (LTS5), Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, 1015, Switzerland
- University Hospital Center (CHUV) and University of Lausanne (UNIL), Lausanne, 1011, Switzerland
| | - Frédéric Auber
- Department of Pediatric Surgery, CHU Besançon, 3 boulevard Fleming, Besançon, F-25000, France
- Université de Franche-Comté, SINERGIES, Besançon, F-25000, France
| | - Yann Chaussy
- Department of Pediatric Surgery, CHU Besançon, 3 boulevard Fleming, Besançon, F-25000, France.
- Université de Franche-Comté, SINERGIES, Besançon, F-25000, France.
| |
Collapse
|
8
|
Tsanda A, Nickisch H, Wissel T, Klinder T, Knopp T, Grass M. Dose robustness of deep learning models for anatomic segmentation of computed tomography images. J Med Imaging (Bellingham) 2024; 11:044005. [PMID: 39099642 PMCID: PMC11293838 DOI: 10.1117/1.jmi.11.4.044005] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2023] [Revised: 04/23/2024] [Accepted: 07/10/2024] [Indexed: 08/06/2024] Open
Abstract
Purpose The trend towards lower radiation doses and advances in computed tomography (CT) reconstruction may impair the operation of pretrained segmentation models, giving rise to the problem of estimating the dose robustness of existing segmentation models. Previous studies addressing the issue suffer either from a lack of registered low- and full-dose CT images or from simplified simulations. Approach We employed raw data from full-dose acquisitions to simulate low-dose CT scans, avoiding the need to rescan a patient. The accuracy of the simulation is validated using a real CT scan of a phantom. We consider down to 20% reduction of radiation dose, for which we measure deviations of several pretrained segmentation models from the full-dose prediction. In addition, compatibility with existing denoising methods is considered. Results The results reveal the surprising robustness of the TotalSegmentator approach, showing minimal differences at the pixel level even without denoising. Less robust models show good compatibility with the denoising methods, which help to improve robustness in almost all cases. With denoising based on a convolutional neural network (CNN), the median Dice between low- and full-dose data does not fall below 0.9 (12 for the Hausdorff distance) for all but one model. We observe volatile results for labels with effective radii less than 19 mm and improved results for contrasted CT acquisitions. Conclusion The proposed approach facilitates clinically relevant analysis of dose robustness for human organ segmentation models. The results outline the robustness properties of a diverse set of models. Further studies are needed to identify the robustness of approaches for lesion segmentation and to rank the factors contributing to dose robustness.
Collapse
Affiliation(s)
- Artyom Tsanda
- Hamburg University of Technology, Institute for Biomedical Imaging, Hamburg, Germany
- Philips Innovative Technologies, Hamburg, Germany
| | | | | | | | - Tobias Knopp
- Hamburg University of Technology, Institute for Biomedical Imaging, Hamburg, Germany
- University Medical Center Hamburg-Eppendorf, Section for Biomedical Imaging, Hamburg, Germany
| | | |
Collapse
|
9
|
Li S, Zhou Z, Gao M, Liao Z, He K, Qu W, Li J, Kamel IR, Chu Q, Zhang Q, Li Z. Incremental value of automatically segmented perirenal adipose tissue for pathological grading of clear cell renal cell carcinoma: a multicenter cohort study. Int J Surg 2024; 110:4221-4230. [PMID: 38573065 PMCID: PMC11254242 DOI: 10.1097/js9.0000000000001358] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/17/2023] [Accepted: 03/04/2024] [Indexed: 04/05/2024]
Abstract
OBJECTIVES Accurate preoperative prediction of the pathological grade of clear cell renal cell carcinoma (ccRCC) is crucial for optimal treatment planning and patient outcomes. This study aims to develop and validate a deep-learning (DL) algorithm to automatically segment renal tumours, kidneys, and perirenal adipose tissue (PRAT) from computed tomography (CT) images and extract radiomics features to predict the pathological grade of ccRCC. METHODS In this cross-ethnic retrospective study, a total of 614 patients were divided into a training set (383 patients from the local hospital), an internal validation set (88 patients from the local hospital), and an external validation set (143 patients from the public dataset). A two-dimensional TransUNet-based DL model combined with the train-while-annotation method was trained for automatic volumetric segmentation of renal tumours, kidneys, and visceral adipose tissue (VAT) on images from two groups of datasets. PRAT was extracted using a dilation algorithm by calculating voxels of VAT surrounding the kidneys. Radiomics features were subsequently extracted from three regions of interest of CT images, adopting multiple filtering strategies. The least absolute shrinkage and selection operator (LASSO) regression was used for feature selection, and the support vector machine (SVM) for developing the pathological grading model. Ensemble learning was used for imbalanced data classification. Performance evaluation included the Dice coefficient for segmentation and metrics such as accuracy and area under curve (AUC) for classification. The WHO/International Society of Urological Pathology (ISUP) grading models were finally interpreted and visualized using the SHapley Additive exPlanations (SHAP) method. RESULTS For automatic segmentation, the mean Dice coefficient achieved 0.836 for renal tumours and 0.967 for VAT on the internal validation dataset. For WHO/ISUP grading, a model built with features of PRAT achieved a moderate AUC of 0.711 (95% CI, 0.604-0.802) in the internal validation set, coupled with a sensitivity of 0.400 and a specificity of 0.781. While model built with combination features of the renal tumour, kidney, and PRAT showed an AUC of 0.814 (95% CI, 0.717-0.889) in the internal validation set, with a sensitivity of 0.800 and a specificity of 0.753, significantly higher than the model built with features solely from tumour lesion (0.760; 95% CI, 0.657-0.845), with a sensitivity of 0.533 and a specificity of 0.767. CONCLUSION Automated segmentation of kidneys and visceral adipose tissue (VAT) through TransUNet combined with a conventional image morphology processing algorithm offers a standardized approach to extract PRAT with high reproducibility. The radiomics features of PRAT and tumour lesions, along with machine learning, accurately predict the pathological grade of ccRCC and reveal the incremental significance of PRAT in this prediction.
Collapse
Affiliation(s)
- Shichao Li
- Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology
| | - Ziling Zhou
- Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology
| | - Mengmeng Gao
- Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology
| | - Zhouyan Liao
- Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology
| | - Kangwen He
- Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology
| | - Weinuo Qu
- Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology
| | - Jiali Li
- Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology
| | - Ihab R Kamel
- Department of Radiology, University of Colorado Denver Anschutz Medical Campus, Aurora, CO, USA
| | - Qian Chu
- Department of Oncology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology, Wuhan, Hubei
| | - Qingpeng Zhang
- Department of Pharmacology and Pharmacy, LKS Faculty of Medicine, and the Musketeers Foundation Institute of Data Science, University of Hong Kong, Hong Kong, China
| | - Zhen Li
- Department of Radiology, Tongji Hospital, Tongji Medical College, Huazhong University of Science and Technology
| |
Collapse
|
10
|
Arabi H, Zaidi H. Contrastive Learning vs. Self-Learning vs. Deformable Data Augmentation in Semantic Segmentation of Medical Images. JOURNAL OF IMAGING INFORMATICS IN MEDICINE 2024:10.1007/s10278-024-01159-x. [PMID: 38858260 DOI: 10.1007/s10278-024-01159-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/24/2024] [Revised: 05/23/2024] [Accepted: 05/24/2024] [Indexed: 06/12/2024]
Abstract
To develop a robust segmentation model, encoding the underlying features/structures of the input data is essential to discriminate the target structure from the background. To enrich the extracted feature maps, contrastive learning and self-learning techniques are employed, particularly when the size of the training dataset is limited. In this work, we set out to investigate the impact of contrastive learning and self-learning on the performance of the deep learning-based semantic segmentation. To this end, three different datasets were employed used for brain tumor and hippocampus delineation from MR images (BraTS and Decathlon datasets, respectively) and kidney segmentation from CT images (Decathlon dataset). Since data augmentation techniques are also aimed at enhancing the performance of deep learning methods, a deformable data augmentation technique was proposed and compared with contrastive learning and self-learning frameworks. The segmentation accuracy for the three datasets was assessed with and without applying data augmentation, contrastive learning, and self-learning to individually investigate the impact of these techniques. The self-learning and deformable data augmentation techniques exhibited comparable performance with Dice indices of 0.913 ± 0.030 and 0.920 ± 0.022 for kidney segmentation, 0.890 ± 0.035 and 0.898 ± 0.027 for hippocampus segmentation, and 0.891 ± 0.045 and 0.897 ± 0.040 for lesion segmentation, respectively. These two approaches significantly outperformed the contrastive learning and the original model with Dice indices of 0.871 ± 0.039 and 0.868 ± 0.042 for kidney segmentation, 0.872 ± 0.045 and 0.865 ± 0.048 for hippocampus segmentation, and 0.870 ± 0.049 and 0.860 ± 0.058 for lesion segmentation, respectively. The combination of self-learning with deformable data augmentation led to a robust segmentation model with no outliers in the outcomes. This work demonstrated the beneficial impact of self-learning and deformable data augmentation on organ and lesion segmentation, where no additional training datasets are needed.
Collapse
Affiliation(s)
- Hossein Arabi
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, CH-1211, Geneva 4, Switzerland
| | - Habib Zaidi
- Division of Nuclear Medicine and Molecular Imaging, Geneva University Hospital, CH-1211, Geneva 4, Switzerland.
- Department of Nuclear Medicine and Molecular Imaging, University of Groningen, University Medical Center Groningen, 9700 RB, Groningen, Netherlands.
- Department of Nuclear Medicine, University of Southern Denmark, DK-500, Odense, Denmark.
- University Research and Innovation Center, Óbuda University, Budapest, Hungary.
| |
Collapse
|
11
|
Ji Z, Mu J, Liu J, Zhang H, Dai C, Zhang X, Ganchev I. ASD-Net: a novel U-Net based asymmetric spatial-channel convolution network for precise kidney and kidney tumor image segmentation. Med Biol Eng Comput 2024; 62:1673-1687. [PMID: 38326677 PMCID: PMC11076390 DOI: 10.1007/s11517-024-03025-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2023] [Accepted: 01/15/2024] [Indexed: 02/09/2024]
Abstract
Early intervention in tumors can greatly improve human survival rates. With the development of deep learning technology, automatic image segmentation has taken a prominent role in the field of medical image analysis. Manually segmenting kidneys on CT images is a tedious task, and due to the diversity of these images and varying technical skills of professionals, segmentation results can be inconsistent. To address this problem, a novel ASD-Net network is proposed in this paper for kidney and kidney tumor segmentation tasks. First, the proposed network employs newly designed Adaptive Spatial-channel Convolution Optimization (ASCO) blocks to capture anisotropic information in the images. Then, other newly designed blocks, i.e., Dense Dilated Enhancement Convolution (DDEC) blocks, are utilized to enhance feature propagation and reuse it across the network, thereby improving its segmentation accuracy. To allow the network to segment complex and small kidney tumors more effectively, the Atrous Spatial Pyramid Pooling (ASPP) module is incorporated in its middle layer. With its generalized pyramid feature, this module enables the network to better capture and understand context information at various scales within the images. In addition to this, the concurrent spatial and channel squeeze & excitation (scSE) attention mechanism is adopted to better comprehend and manage context information in the images. Additional encoding layers are also added to the base (U-Net) and connected to the original encoding layer through skip connections. The resultant enhanced U-Net structure allows for better extraction and merging of high-level and low-level features, further boosting the network's ability to restore segmentation details. In addition, the combined Binary Cross Entropy (BCE)-Dice loss is utilized as the network's loss function. Experiments, conducted on the KiTS19 dataset, demonstrate that the proposed ASD-Net network outperforms the existing segmentation networks according to all evaluation metrics used, except for recall in the case of kidney tumor segmentation, where it takes the second place after Attention-UNet.
Collapse
Affiliation(s)
- Zhanlin Ji
- Department of Artificial Intelligence, North China University of Science and Technology, Tangshan, 063009, People's Republic of China
| | - Juncheng Mu
- Department of Artificial Intelligence, North China University of Science and Technology, Tangshan, 063009, People's Republic of China
| | - Jianuo Liu
- Department of Artificial Intelligence, North China University of Science and Technology, Tangshan, 063009, People's Republic of China
| | - Haiyang Zhang
- Department of Computing, Xi'an Jiaotong-Liverpool University, Suzhou, People's Republic of China
| | - Chenxu Dai
- Department of Artificial Intelligence, North China University of Science and Technology, Tangshan, 063009, People's Republic of China
| | - Xueji Zhang
- School of Biomedical Engineering, Shenzhen University Health Science Center, Shenzhen, Guangdong, 518060, People's Republic of China.
| | - Ivan Ganchev
- Telecommunications Research Centre (TRC), University of Limerick, Limerick, V94 T9PX, Ireland.
- Department of Computer Systems, University of Plovdiv "Paisii Hilendarski", Plovdiv, 4000, Bulgaria.
- Institute of Mathematics and Informatics, Bulgarian Academy of Sciences, Sofia, 1040, Bulgaria.
| |
Collapse
|
12
|
Wood AM, Abdallah N, Heller N, Benidir T, Isensee F, Tejpaul R, Suk-Ouichai C, Curry C, You A, Remer E, Haywood S, Campbell S, Papanikolopoulos N, Weight C. Fully Automated Versions of Clinically Validated Nephrometry Scores Demonstrate Superior Predictive Utility versus Human Scores. BJU Int 2024; 133:690-698. [PMID: 38343198 PMCID: PMC11185291 DOI: 10.1111/bju.16276] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/12/2024]
Abstract
OBJECTIVE To automate the generation of three validated nephrometry scoring systems on preoperative computerised tomography (CT) scans by developing artificial intelligence (AI)-based image processing methods. Subsequently, we aimed to evaluate the ability of these scores to predict meaningful pathological and perioperative outcomes. PATIENTS AND METHODS A total of 300 patients with preoperative CT with early arterial contrast phase were identified from a cohort of 544 consecutive patients undergoing surgical extirpation for suspected renal cancer. A deep neural network approach was used to automatically segment kidneys and tumours, and then geometric algorithms were used to measure the components of the concordance index (C-Index), Preoperative Aspects and Dimensions Used for an Anatomical classification of renal tumours (PADUA), and tumour contact surface area (CSA) nephrometry scores. Human scores were independently calculated by medical personnel blinded to the AI scores. AI and human score agreement was assessed using linear regression and predictive abilities for meaningful outcomes were assessed using logistic regression and receiver operating characteristic curve analyses. RESULTS The median (interquartile range) age was 60 (51-68) years, and 40% were female. The median tumour size was 4.2 cm and 91.3% had malignant tumours. In all, 27% of the tumours were high stage, 37% high grade, and 63% of the patients underwent partial nephrectomy. There was significant agreement between human and AI scores on linear regression analyses (R ranged from 0.574 to 0.828, all P < 0.001). The AI-generated scores were equivalent or superior to human-generated scores for all examined outcomes including high-grade histology, high-stage tumour, indolent tumour, pathological tumour necrosis, and radical nephrectomy (vs partial nephrectomy) surgical approach. CONCLUSIONS Fully automated AI-generated C-Index, PADUA, and tumour CSA nephrometry scores are similar to human-generated scores and predict a wide variety of meaningful outcomes. Once validated, our results suggest that AI-generated nephrometry scores could be delivered automatically from a preoperative CT scan to a clinician and patient at the point of care to aid in decision making.
Collapse
Affiliation(s)
- Andrew M Wood
- Glickman Urological and Kidney Institute, Cleveland, OH, USA
| | - Nour Abdallah
- Glickman Urological and Kidney Institute, Cleveland, OH, USA
| | - Nicholas Heller
- Department of Computer Science and Engineering, University of Minnesota, Minneapolis, MN, USA
| | - Tarik Benidir
- Glickman Urological and Kidney Institute, Cleveland, OH, USA
| | - Fabian Isensee
- German Cancer Research Center (DKFZ) Heidelberg, University of Heidelberg, Heidelberg, Germany
| | - Resha Tejpaul
- Department of Computer Science and Engineering, University of Minnesota, Minneapolis, MN, USA
| | | | - Caleb Curry
- Glickman Urological and Kidney Institute, Cleveland, OH, USA
| | - Alex You
- Case Western Reserve University, Cleveland, OH, USA
| | - Erick Remer
- Department of Diagnostic Radiology, Imaging Institute Cleveland Clinic, Cleveland, OH, USA
| | - Samuel Haywood
- Glickman Urological and Kidney Institute, Cleveland, OH, USA
| | - Steven Campbell
- Glickman Urological and Kidney Institute, Cleveland, OH, USA
| | | | | |
Collapse
|
13
|
Chen J, Chen R, Chen L, Zhang L, Wang W, Zeng X. Kidney medicine meets computer vision: a bibliometric analysis. Int Urol Nephrol 2024:10.1007/s11255-024-04082-w. [PMID: 38814370 DOI: 10.1007/s11255-024-04082-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2024] [Accepted: 05/16/2024] [Indexed: 05/31/2024]
Abstract
BACKGROUND AND OBJECTIVE Rapid advances in computer vision (CV) have the potential to facilitate the examination, diagnosis, and treatment of diseases of the kidney. The bibliometric study aims to explore the research landscape and evolving research focus of the application of CV in kidney medicine research. METHODS The Web of Science Core Collection was utilized to identify publications related to the research or applications of CV technology in the field of kidney medicine from January 1, 1900, to December 31, 2022. We analyzed emerging research trends, highly influential publications and journals, prolific researchers, countries/regions, research institutions, co-authorship networks, and co-occurrence networks. Bibliographic information was analyzed and visualized using Python, Matplotlib, Seaborn, HistCite, and Vosviewer. RESULTS There was an increasing trend in the number of publications on CV-based kidney medicine research. These publications mainly focused on medical image processing, surgical procedures, medical image analysis/diagnosis, as well as the application and innovation of CV technology in medical imaging. The United States is currently the leading country in terms of the quantities of published articles and international collaborations, followed by China. Deep learning-based segmentation and machine learning-based texture analysis are the most commonly used techniques in this field. Regarding research hotspot trends, CV algorithms are shifting toward artificial intelligence, and research objects are expanding to encompass a wider range of kidney-related objects, with data dimensions used in research transitioning from 2D to 3D while simultaneously incorporating more diverse data modalities. CONCLUSION The present study provides a scientometric overview of the current progress in the research and application of CV technology in kidney medicine research. Through the use of bibliometric analysis and network visualization, we elucidate emerging trends, key sources, leading institutions, and popular topics. Our findings and analysis are expected to provide valuable insights for future research on the use of CV in kidney medicine research.
Collapse
Affiliation(s)
- Junren Chen
- Department of Nephrology and West China Biomedical Big Data Center, West China Hospital, Sichuan University, Chengdu, 610041, Sichuan, China
- School of Computer Science, Sichuan University, Chengdu, 610065, Sichuan, China
- Med-X Center for Informatics, Sichuan University, Chengdu, 610041, Sichuan, China
| | - Rui Chen
- The Department of Electronic Engineering, Tsinghua University, Beijing, 100084, China
| | - Liangyin Chen
- School of Computer Science, Sichuan University, Chengdu, 610065, Sichuan, China
| | - Lei Zhang
- School of Computer Science, Sichuan University, Chengdu, 610065, Sichuan, China
| | - Wei Wang
- School of Automation, Chengdu University of Information Technology, Chengdu, 610225, Sichuan, China
- School of Information and Communication Engineering, University of Electronic Science and Technology of China, Chengdu, 611731, Sichuan, China
| | - Xiaoxi Zeng
- Department of Nephrology and West China Biomedical Big Data Center, West China Hospital, Sichuan University, Chengdu, 610041, Sichuan, China.
- Med-X Center for Informatics, Sichuan University, Chengdu, 610041, Sichuan, China.
| |
Collapse
|
14
|
Scalco E, Pozzi S, Rizzo G, Lanzarone E. Uncertainty quantification in multi-class segmentation: Comparison between Bayesian and non-Bayesian approaches in a clinical perspective. Med Phys 2024. [PMID: 38808956 DOI: 10.1002/mp.17189] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/12/2024] [Revised: 04/17/2024] [Accepted: 05/12/2024] [Indexed: 05/30/2024] Open
Abstract
BACKGROUND Automatic segmentation techniques based on Convolutional Neural Networks (CNNs) are widely adopted to automatically identify any structure of interest from a medical image, as they are not time consuming and not subject to high intra- and inter-operator variability. However, the adoption of these approaches in clinical practice is slowed down by some factors, such as the difficulty in providing an accurate quantification of their uncertainty. PURPOSE This work aims to evaluate the uncertainty quantification provided by two Bayesian and two non-Bayesian approaches for a multi-class segmentation problem, and to compare the risk propensity among these approaches, considering CT images of patients affected by renal cancer (RC). METHODS Four uncertainty quantification approaches were implemented in this work, based on a benchmark CNN currently employed in medical image segmentation: two Bayesian CNNs with different regularizations (Dropout and DropConnect), named BDR and BDC, an ensemble method (Ens) and a test-time augmentation (TTA) method. They were compared in terms of segmentation accuracy, using the Dice score, uncertainty quantification, using the ratio of correct-certain pixels (RCC) and incorrect-uncertain pixels (RIU), and with respect to inter-observer variability in manual segmentation. They were trained with the Kidney and Kidney Tumor Segmentation Challenge launched in 2021 (Kits21), for which multi-class segmentations of kidney, RC, and cyst on 300 CT volumes are available. Moreover, they were tested considering this and other two public renal CT datasets. RESULTS Accuracy results achieved large differences across the structures of interest for all approaches, with an average Dice score of 0.92, 0.58, and 0.21 for kidney, tumor, and cyst, respectively. In terms of uncertainties, TTA provided the highest uncertainty, followed by Ens and BDC, whereas BDR provided the lowest, and minimized the number of incorrect certain pixels worse than the other approaches. Again, large differences were seen across the three structures in terms of RCC and RIU. These metrics were associated with different risk propensity, as BDR was the most risk-taking approach, able to provide higher accuracy in its prediction, but failing to assign uncertainty on incorrect segmentation in every case. The other three approaches were more conservative, providing large uncertainty regions, with the drawback of giving alert also on correct areas. Finally, the analysis of the inter-observer segmentation variability showed a significant variation among the four approaches on the external dataset, with BDR reporting the lowest agreement (Dice = 0.82), and TTA obtaining the highest score (Dice = 0.94). CONCLUSIONS Our outcomes highlight the importance of quantifying the segmentation uncertainty and that decision-makers can choose the approach most in line with the risk propensity degree required by the application and their policy.
Collapse
Affiliation(s)
- Elisa Scalco
- Institute of Biomedical Technologies (ITB), National Research Council (CNR), Segrate, Milan, Italy
| | - Silvia Pozzi
- Department of Management, Information and Production Engineering, University of Bergamo, Bergamo, Italy
| | - Giovanna Rizzo
- Institute Of Intelligent Industrial Technologies and Systems (STIIMA), National Research Council (CNR), Milan, Italy
| | - Ettore Lanzarone
- Department of Management, Information and Production Engineering, University of Bergamo, Bergamo, Italy
| |
Collapse
|
15
|
Yuan N, Zhang Y, Lv K, Liu Y, Yang A, Hu P, Yu H, Han X, Guo X, Li J, Wang T, Lei B, Ma G. HCA-DAN: hierarchical class-aware domain adaptive network for gastric tumor segmentation in 3D CT images. Cancer Imaging 2024; 24:63. [PMID: 38773670 PMCID: PMC11107051 DOI: 10.1186/s40644-024-00711-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/08/2023] [Accepted: 05/11/2024] [Indexed: 05/24/2024] Open
Abstract
BACKGROUND Accurate segmentation of gastric tumors from CT scans provides useful image information for guiding the diagnosis and treatment of gastric cancer. However, automated gastric tumor segmentation from 3D CT images faces several challenges. The large variation of anisotropic spatial resolution limits the ability of 3D convolutional neural networks (CNNs) to learn features from different views. The background texture of gastric tumor is complex, and its size, shape and intensity distribution are highly variable, which makes it more difficult for deep learning methods to capture the boundary. In particular, while multi-center datasets increase sample size and representation ability, they suffer from inter-center heterogeneity. METHODS In this study, we propose a new cross-center 3D tumor segmentation method named Hierarchical Class-Aware Domain Adaptive Network (HCA-DAN), which includes a new 3D neural network that efficiently bridges an Anisotropic neural network and a Transformer (AsTr) for extracting multi-scale context features from the CT images with anisotropic resolution, and a hierarchical class-aware domain alignment (HCADA) module for adaptively aligning multi-scale context features across two domains by integrating a class attention map with class-specific information. We evaluate the proposed method on an in-house CT image dataset collected from four medical centers and validate its segmentation performance in both in-center and cross-center test scenarios. RESULTS Our baseline segmentation network (i.e., AsTr) achieves best results compared to other 3D segmentation models, with a mean dice similarity coefficient (DSC) of 59.26%, 55.97%, 48.83% and 67.28% in four in-center test tasks, and with a DSC of 56.42%, 55.94%, 46.54% and 60.62% in four cross-center test tasks. In addition, the proposed cross-center segmentation network (i.e., HCA-DAN) obtains excellent results compared to other unsupervised domain adaptation methods, with a DSC of 58.36%, 56.72%, 49.25%, and 62.20% in four cross-center test tasks. CONCLUSIONS Comprehensive experimental results demonstrate that the proposed method outperforms compared methods on this multi-center database and is promising for routine clinical workflows.
Collapse
Affiliation(s)
- Ning Yuan
- Department of Medical Imaging, Heping Hospital Affiliated to Changzhi Medical College, Changzhi, China
| | - Yongtao Zhang
- School of Biomedical Engineering, Health Science Centers, National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Kuan Lv
- Peking University China-Japan Friendship School of Clinical Medicine, Beijing, China
| | - Yiyao Liu
- School of Biomedical Engineering, Health Science Centers, National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Aocai Yang
- Department of Radiology, China-Japan Friendship Hospital, No. 2 East Yinghua Road, Chaoyang District, Beijing, 100029, China
| | - Pianpian Hu
- Department of Radiology, China-Japan Friendship Hospital, No. 2 East Yinghua Road, Chaoyang District, Beijing, 100029, China
| | - Hongwei Yu
- Department of Radiology, China-Japan Friendship Hospital, No. 2 East Yinghua Road, Chaoyang District, Beijing, 100029, China
| | - Xiaowei Han
- Department of Radiology, The Affiliated Drum Tower Hospital of Nanjing University Medical School, Nanjing, China
| | - Xing Guo
- Department of Medical Imaging, Heping Hospital Affiliated to Changzhi Medical College, Changzhi, China
| | - Junfeng Li
- Department of Medical Imaging, Heping Hospital Affiliated to Changzhi Medical College, Changzhi, China
| | - Tianfu Wang
- School of Biomedical Engineering, Health Science Centers, National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
| | - Baiying Lei
- School of Biomedical Engineering, Health Science Centers, National-Regional Key Technology Engineering Laboratory for Medical Ultrasound, Marshall Laboratory of Biomedical Engineering, Shenzhen University, Shenzhen, China
- AI Research Center for Medical Image Analysis and Diagnosis, Shenzhen University, Guangdong, China
| | - Guolin Ma
- Department of Radiology, China-Japan Friendship Hospital, No. 2 East Yinghua Road, Chaoyang District, Beijing, 100029, China.
| |
Collapse
|
16
|
Koitka S, Baldini G, Kroll L, van Landeghem N, Pollok OB, Haubold J, Pelka O, Kim M, Kleesiek J, Nensa F, Hosch R. SAROS: A dataset for whole-body region and organ segmentation in CT imaging. Sci Data 2024; 11:483. [PMID: 38729970 PMCID: PMC11087485 DOI: 10.1038/s41597-024-03337-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2023] [Accepted: 05/01/2024] [Indexed: 05/12/2024] Open
Abstract
The Sparsely Annotated Region and Organ Segmentation (SAROS) dataset was created using data from The Cancer Imaging Archive (TCIA) to provide a large open-access CT dataset with high-quality annotations of body landmarks. In-house segmentation models were employed to generate annotation proposals on randomly selected cases from TCIA. The dataset includes 13 semantic body region labels (abdominal/thoracic cavity, bones, brain, breast implant, mediastinum, muscle, parotid/submandibular/thyroid glands, pericardium, spinal cord, subcutaneous tissue) and six body part labels (left/right arm/leg, head, torso). Case selection was based on the DICOM series description, gender, and imaging protocol, resulting in 882 patients (438 female) for a total of 900 CTs. Manual review and correction of proposals were conducted in a continuous quality control cycle. Only every fifth axial slice was annotated, yielding 20150 annotated slices from 28 data collections. For the reproducibility on downstream tasks, five cross-validation folds and a test set were pre-defined. The SAROS dataset serves as an open-access resource for training and evaluating novel segmentation models, covering various scanner vendors and diseases.
Collapse
Affiliation(s)
- Sven Koitka
- Institute of Interventional and Diagnostic Radiology and Neuroradiology, University Hospital Essen, Essen, Germany
- Institute for Artificial Intelligence in Medicine, University Hospital Essen, Essen, Germany
| | - Giulia Baldini
- Institute of Interventional and Diagnostic Radiology and Neuroradiology, University Hospital Essen, Essen, Germany
- Institute for Artificial Intelligence in Medicine, University Hospital Essen, Essen, Germany
| | - Lennard Kroll
- Institute of Interventional and Diagnostic Radiology and Neuroradiology, University Hospital Essen, Essen, Germany
| | - Natalie van Landeghem
- Institute of Interventional and Diagnostic Radiology and Neuroradiology, University Hospital Essen, Essen, Germany
| | - Olivia B Pollok
- Institute for Artificial Intelligence in Medicine, University Hospital Essen, Essen, Germany
| | - Johannes Haubold
- Institute of Interventional and Diagnostic Radiology and Neuroradiology, University Hospital Essen, Essen, Germany
| | - Obioma Pelka
- Institute for Artificial Intelligence in Medicine, University Hospital Essen, Essen, Germany
- Data Integration Center, Central IT Department, University Hospital Essen, Essen, Germany
| | - Moon Kim
- Institute for Artificial Intelligence in Medicine, University Hospital Essen, Essen, Germany
| | - Jens Kleesiek
- Institute for Artificial Intelligence in Medicine, University Hospital Essen, Essen, Germany
| | - Felix Nensa
- Institute of Interventional and Diagnostic Radiology and Neuroradiology, University Hospital Essen, Essen, Germany
- Institute for Artificial Intelligence in Medicine, University Hospital Essen, Essen, Germany
| | - René Hosch
- Institute of Interventional and Diagnostic Radiology and Neuroradiology, University Hospital Essen, Essen, Germany.
- Institute for Artificial Intelligence in Medicine, University Hospital Essen, Essen, Germany.
| |
Collapse
|
17
|
Liu H, Xu Z, Gao R, Li H, Wang J, Chabin G, Oguz I, Grbic S. COSST: Multi-Organ Segmentation With Partially Labeled Datasets Using Comprehensive Supervisions and Self-Training. IEEE TRANSACTIONS ON MEDICAL IMAGING 2024; 43:1995-2009. [PMID: 38224508 DOI: 10.1109/tmi.2024.3354673] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/17/2024]
Abstract
Deep learning models have demonstrated remarkable success in multi-organ segmentation but typically require large-scale datasets with all organs of interest annotated. However, medical image datasets are often low in sample size and only partially labeled, i.e., only a subset of organs are annotated. Therefore, it is crucial to investigate how to learn a unified model on the available partially labeled datasets to leverage their synergistic potential. In this paper, we systematically investigate the partial-label segmentation problem with theoretical and empirical analyses on the prior techniques. We revisit the problem from a perspective of partial label supervision signals and identify two signals derived from ground truth and one from pseudo labels. We propose a novel two-stage framework termed COSST, which effectively and efficiently integrates comprehensive supervision signals with self-training. Concretely, we first train an initial unified model using two ground truth-based signals and then iteratively incorporate the pseudo label signal to the initial model using self-training. To mitigate performance degradation caused by unreliable pseudo labels, we assess the reliability of pseudo labels via outlier detection in latent space and exclude the most unreliable pseudo labels from each self-training iteration. Extensive experiments are conducted on one public and three private partial-label segmentation tasks over 12 CT datasets. Experimental results show that our proposed COSST achieves significant improvement over the baseline method, i.e., individual networks trained on each partially labeled dataset. Compared to the state-of-the-art partial-label segmentation methods, COSST demonstrates consistent superior performance on various segmentation tasks and with different training data sizes.
Collapse
|
18
|
Dai C, Xiong Y, Zhu P, Yao L, Lin J, Yao J, Zhang X, Huang R, Wang R, Hou J, Wang K, Shi Z, Chen F, Guo J, Zeng M, Zhou J, Wang S. Deep Learning Assessment of Small Renal Masses at Contrast-enhanced Multiphase CT. Radiology 2024; 311:e232178. [PMID: 38742970 DOI: 10.1148/radiol.232178] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/16/2024]
Abstract
Background Accurate characterization of suspicious small renal masses is crucial for optimized management. Deep learning (DL) algorithms may assist with this effort. Purpose To develop and validate a DL algorithm for identifying benign small renal masses at contrast-enhanced multiphase CT. Materials and Methods Surgically resected renal masses measuring 3 cm or less in diameter at contrast-enhanced CT were included. The DL algorithm was developed by using retrospective data from one hospital between 2009 and 2021, with patients randomly allocated in a training and internal test set ratio of 8:2. Between 2013 and 2021, external testing was performed on data from five independent hospitals. A prospective test set was obtained between 2021 and 2022 from one hospital. Algorithm performance was evaluated by using the area under the receiver operating characteristic curve (AUC) and compared with the results of seven clinicians using the DeLong test. Results A total of 1703 patients (mean age, 56 years ± 12 [SD]; 619 female) with a single renal mass per patient were evaluated. The retrospective data set included 1063 lesions (874 in training set, 189 internal test set); the multicenter external test set included 537 lesions (12.3%, 66 benign) with 89 subcentimeter (≤1 cm) lesions (16.6%); and the prospective test set included 103 lesions (13.6%, 14 benign) with 20 (19.4%) subcentimeter lesions. The DL algorithm performance was comparable with that of urological radiologists: for the external test set, AUC was 0.80 (95% CI: 0.75, 0.85) versus 0.84 (95% CI: 0.78, 0.88) (P = .61); for the prospective test set, AUC was 0.87 (95% CI: 0.79, 0.93) versus 0.92 (95% CI: 0.86, 0.96) (P = .70). For subcentimeter lesions in the external test set, the algorithm and urological radiologists had similar AUC of 0.74 (95% CI: 0.63, 0.83) and 0.81 (95% CI: 0.68, 0.92) (P = .78), respectively. Conclusion The multiphase CT-based DL algorithm showed comparable performance with that of radiologists for identifying benign small renal masses, including lesions of 1 cm or less. Published under a CC BY 4.0 license. Supplemental material is available for this article.
Collapse
Affiliation(s)
- Chenchen Dai
- From the Departments of Radiology (C.D., P.Z., Z.S., M.Z., J.Z.), Urology (Y.X., J.G.), and Pathology (J.H.), Zhongshan Hospital, Fudan University, Shanghai, China; Shanghai Institute of Medical Imaging, Shanghai, China (C.D., P.Z., Z.S., M.Z.); Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China (L.Y., F.C.); Departments of Urology (J.L.) and Radiology (J.Z.), Xiamen Branch, Zhongshan Hospital, Fudan University, 668 Jinhu Road, Huli District, Xiamen 361015, China; Department of Urology, Zhangye People's Hospital affiliated to Hexi University, Zhangye, China (J.Y.); Department of Radiology, the First People's Hospital of Lianyungang, Lianyungang, China (X.Z.); Department of Radiology, Quanzhou First Hospital, Fujian Medical University, Quanzhou, China (R.H.); Department of Pathology, Sir Run Run Shaw Hospital, Hangzhou, China (R.W.); Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China (K.W., S.W.); Shanghai Key Laboratory of MICCAI, Shanghai, China (K.W., S.W.); Xiamen Municipal Clinical Research Center for Medical Imaging, Xiamen, China (J.Z.); and Xiamen Key Clinical Specialty, Xiamen, China (J.Z.)
| | - Ying Xiong
- From the Departments of Radiology (C.D., P.Z., Z.S., M.Z., J.Z.), Urology (Y.X., J.G.), and Pathology (J.H.), Zhongshan Hospital, Fudan University, Shanghai, China; Shanghai Institute of Medical Imaging, Shanghai, China (C.D., P.Z., Z.S., M.Z.); Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China (L.Y., F.C.); Departments of Urology (J.L.) and Radiology (J.Z.), Xiamen Branch, Zhongshan Hospital, Fudan University, 668 Jinhu Road, Huli District, Xiamen 361015, China; Department of Urology, Zhangye People's Hospital affiliated to Hexi University, Zhangye, China (J.Y.); Department of Radiology, the First People's Hospital of Lianyungang, Lianyungang, China (X.Z.); Department of Radiology, Quanzhou First Hospital, Fujian Medical University, Quanzhou, China (R.H.); Department of Pathology, Sir Run Run Shaw Hospital, Hangzhou, China (R.W.); Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China (K.W., S.W.); Shanghai Key Laboratory of MICCAI, Shanghai, China (K.W., S.W.); Xiamen Municipal Clinical Research Center for Medical Imaging, Xiamen, China (J.Z.); and Xiamen Key Clinical Specialty, Xiamen, China (J.Z.)
| | - Pingyi Zhu
- From the Departments of Radiology (C.D., P.Z., Z.S., M.Z., J.Z.), Urology (Y.X., J.G.), and Pathology (J.H.), Zhongshan Hospital, Fudan University, Shanghai, China; Shanghai Institute of Medical Imaging, Shanghai, China (C.D., P.Z., Z.S., M.Z.); Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China (L.Y., F.C.); Departments of Urology (J.L.) and Radiology (J.Z.), Xiamen Branch, Zhongshan Hospital, Fudan University, 668 Jinhu Road, Huli District, Xiamen 361015, China; Department of Urology, Zhangye People's Hospital affiliated to Hexi University, Zhangye, China (J.Y.); Department of Radiology, the First People's Hospital of Lianyungang, Lianyungang, China (X.Z.); Department of Radiology, Quanzhou First Hospital, Fujian Medical University, Quanzhou, China (R.H.); Department of Pathology, Sir Run Run Shaw Hospital, Hangzhou, China (R.W.); Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China (K.W., S.W.); Shanghai Key Laboratory of MICCAI, Shanghai, China (K.W., S.W.); Xiamen Municipal Clinical Research Center for Medical Imaging, Xiamen, China (J.Z.); and Xiamen Key Clinical Specialty, Xiamen, China (J.Z.)
| | - Linpeng Yao
- From the Departments of Radiology (C.D., P.Z., Z.S., M.Z., J.Z.), Urology (Y.X., J.G.), and Pathology (J.H.), Zhongshan Hospital, Fudan University, Shanghai, China; Shanghai Institute of Medical Imaging, Shanghai, China (C.D., P.Z., Z.S., M.Z.); Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China (L.Y., F.C.); Departments of Urology (J.L.) and Radiology (J.Z.), Xiamen Branch, Zhongshan Hospital, Fudan University, 668 Jinhu Road, Huli District, Xiamen 361015, China; Department of Urology, Zhangye People's Hospital affiliated to Hexi University, Zhangye, China (J.Y.); Department of Radiology, the First People's Hospital of Lianyungang, Lianyungang, China (X.Z.); Department of Radiology, Quanzhou First Hospital, Fujian Medical University, Quanzhou, China (R.H.); Department of Pathology, Sir Run Run Shaw Hospital, Hangzhou, China (R.W.); Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China (K.W., S.W.); Shanghai Key Laboratory of MICCAI, Shanghai, China (K.W., S.W.); Xiamen Municipal Clinical Research Center for Medical Imaging, Xiamen, China (J.Z.); and Xiamen Key Clinical Specialty, Xiamen, China (J.Z.)
| | - Jinglai Lin
- From the Departments of Radiology (C.D., P.Z., Z.S., M.Z., J.Z.), Urology (Y.X., J.G.), and Pathology (J.H.), Zhongshan Hospital, Fudan University, Shanghai, China; Shanghai Institute of Medical Imaging, Shanghai, China (C.D., P.Z., Z.S., M.Z.); Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China (L.Y., F.C.); Departments of Urology (J.L.) and Radiology (J.Z.), Xiamen Branch, Zhongshan Hospital, Fudan University, 668 Jinhu Road, Huli District, Xiamen 361015, China; Department of Urology, Zhangye People's Hospital affiliated to Hexi University, Zhangye, China (J.Y.); Department of Radiology, the First People's Hospital of Lianyungang, Lianyungang, China (X.Z.); Department of Radiology, Quanzhou First Hospital, Fujian Medical University, Quanzhou, China (R.H.); Department of Pathology, Sir Run Run Shaw Hospital, Hangzhou, China (R.W.); Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China (K.W., S.W.); Shanghai Key Laboratory of MICCAI, Shanghai, China (K.W., S.W.); Xiamen Municipal Clinical Research Center for Medical Imaging, Xiamen, China (J.Z.); and Xiamen Key Clinical Specialty, Xiamen, China (J.Z.)
| | - Jiaxi Yao
- From the Departments of Radiology (C.D., P.Z., Z.S., M.Z., J.Z.), Urology (Y.X., J.G.), and Pathology (J.H.), Zhongshan Hospital, Fudan University, Shanghai, China; Shanghai Institute of Medical Imaging, Shanghai, China (C.D., P.Z., Z.S., M.Z.); Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China (L.Y., F.C.); Departments of Urology (J.L.) and Radiology (J.Z.), Xiamen Branch, Zhongshan Hospital, Fudan University, 668 Jinhu Road, Huli District, Xiamen 361015, China; Department of Urology, Zhangye People's Hospital affiliated to Hexi University, Zhangye, China (J.Y.); Department of Radiology, the First People's Hospital of Lianyungang, Lianyungang, China (X.Z.); Department of Radiology, Quanzhou First Hospital, Fujian Medical University, Quanzhou, China (R.H.); Department of Pathology, Sir Run Run Shaw Hospital, Hangzhou, China (R.W.); Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China (K.W., S.W.); Shanghai Key Laboratory of MICCAI, Shanghai, China (K.W., S.W.); Xiamen Municipal Clinical Research Center for Medical Imaging, Xiamen, China (J.Z.); and Xiamen Key Clinical Specialty, Xiamen, China (J.Z.)
| | - Xue Zhang
- From the Departments of Radiology (C.D., P.Z., Z.S., M.Z., J.Z.), Urology (Y.X., J.G.), and Pathology (J.H.), Zhongshan Hospital, Fudan University, Shanghai, China; Shanghai Institute of Medical Imaging, Shanghai, China (C.D., P.Z., Z.S., M.Z.); Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China (L.Y., F.C.); Departments of Urology (J.L.) and Radiology (J.Z.), Xiamen Branch, Zhongshan Hospital, Fudan University, 668 Jinhu Road, Huli District, Xiamen 361015, China; Department of Urology, Zhangye People's Hospital affiliated to Hexi University, Zhangye, China (J.Y.); Department of Radiology, the First People's Hospital of Lianyungang, Lianyungang, China (X.Z.); Department of Radiology, Quanzhou First Hospital, Fujian Medical University, Quanzhou, China (R.H.); Department of Pathology, Sir Run Run Shaw Hospital, Hangzhou, China (R.W.); Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China (K.W., S.W.); Shanghai Key Laboratory of MICCAI, Shanghai, China (K.W., S.W.); Xiamen Municipal Clinical Research Center for Medical Imaging, Xiamen, China (J.Z.); and Xiamen Key Clinical Specialty, Xiamen, China (J.Z.)
| | - Risheng Huang
- From the Departments of Radiology (C.D., P.Z., Z.S., M.Z., J.Z.), Urology (Y.X., J.G.), and Pathology (J.H.), Zhongshan Hospital, Fudan University, Shanghai, China; Shanghai Institute of Medical Imaging, Shanghai, China (C.D., P.Z., Z.S., M.Z.); Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China (L.Y., F.C.); Departments of Urology (J.L.) and Radiology (J.Z.), Xiamen Branch, Zhongshan Hospital, Fudan University, 668 Jinhu Road, Huli District, Xiamen 361015, China; Department of Urology, Zhangye People's Hospital affiliated to Hexi University, Zhangye, China (J.Y.); Department of Radiology, the First People's Hospital of Lianyungang, Lianyungang, China (X.Z.); Department of Radiology, Quanzhou First Hospital, Fujian Medical University, Quanzhou, China (R.H.); Department of Pathology, Sir Run Run Shaw Hospital, Hangzhou, China (R.W.); Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China (K.W., S.W.); Shanghai Key Laboratory of MICCAI, Shanghai, China (K.W., S.W.); Xiamen Municipal Clinical Research Center for Medical Imaging, Xiamen, China (J.Z.); and Xiamen Key Clinical Specialty, Xiamen, China (J.Z.)
| | - Run Wang
- From the Departments of Radiology (C.D., P.Z., Z.S., M.Z., J.Z.), Urology (Y.X., J.G.), and Pathology (J.H.), Zhongshan Hospital, Fudan University, Shanghai, China; Shanghai Institute of Medical Imaging, Shanghai, China (C.D., P.Z., Z.S., M.Z.); Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China (L.Y., F.C.); Departments of Urology (J.L.) and Radiology (J.Z.), Xiamen Branch, Zhongshan Hospital, Fudan University, 668 Jinhu Road, Huli District, Xiamen 361015, China; Department of Urology, Zhangye People's Hospital affiliated to Hexi University, Zhangye, China (J.Y.); Department of Radiology, the First People's Hospital of Lianyungang, Lianyungang, China (X.Z.); Department of Radiology, Quanzhou First Hospital, Fujian Medical University, Quanzhou, China (R.H.); Department of Pathology, Sir Run Run Shaw Hospital, Hangzhou, China (R.W.); Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China (K.W., S.W.); Shanghai Key Laboratory of MICCAI, Shanghai, China (K.W., S.W.); Xiamen Municipal Clinical Research Center for Medical Imaging, Xiamen, China (J.Z.); and Xiamen Key Clinical Specialty, Xiamen, China (J.Z.)
| | - Jun Hou
- From the Departments of Radiology (C.D., P.Z., Z.S., M.Z., J.Z.), Urology (Y.X., J.G.), and Pathology (J.H.), Zhongshan Hospital, Fudan University, Shanghai, China; Shanghai Institute of Medical Imaging, Shanghai, China (C.D., P.Z., Z.S., M.Z.); Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China (L.Y., F.C.); Departments of Urology (J.L.) and Radiology (J.Z.), Xiamen Branch, Zhongshan Hospital, Fudan University, 668 Jinhu Road, Huli District, Xiamen 361015, China; Department of Urology, Zhangye People's Hospital affiliated to Hexi University, Zhangye, China (J.Y.); Department of Radiology, the First People's Hospital of Lianyungang, Lianyungang, China (X.Z.); Department of Radiology, Quanzhou First Hospital, Fujian Medical University, Quanzhou, China (R.H.); Department of Pathology, Sir Run Run Shaw Hospital, Hangzhou, China (R.W.); Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China (K.W., S.W.); Shanghai Key Laboratory of MICCAI, Shanghai, China (K.W., S.W.); Xiamen Municipal Clinical Research Center for Medical Imaging, Xiamen, China (J.Z.); and Xiamen Key Clinical Specialty, Xiamen, China (J.Z.)
| | - Kang Wang
- From the Departments of Radiology (C.D., P.Z., Z.S., M.Z., J.Z.), Urology (Y.X., J.G.), and Pathology (J.H.), Zhongshan Hospital, Fudan University, Shanghai, China; Shanghai Institute of Medical Imaging, Shanghai, China (C.D., P.Z., Z.S., M.Z.); Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China (L.Y., F.C.); Departments of Urology (J.L.) and Radiology (J.Z.), Xiamen Branch, Zhongshan Hospital, Fudan University, 668 Jinhu Road, Huli District, Xiamen 361015, China; Department of Urology, Zhangye People's Hospital affiliated to Hexi University, Zhangye, China (J.Y.); Department of Radiology, the First People's Hospital of Lianyungang, Lianyungang, China (X.Z.); Department of Radiology, Quanzhou First Hospital, Fujian Medical University, Quanzhou, China (R.H.); Department of Pathology, Sir Run Run Shaw Hospital, Hangzhou, China (R.W.); Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China (K.W., S.W.); Shanghai Key Laboratory of MICCAI, Shanghai, China (K.W., S.W.); Xiamen Municipal Clinical Research Center for Medical Imaging, Xiamen, China (J.Z.); and Xiamen Key Clinical Specialty, Xiamen, China (J.Z.)
| | - Zhang Shi
- From the Departments of Radiology (C.D., P.Z., Z.S., M.Z., J.Z.), Urology (Y.X., J.G.), and Pathology (J.H.), Zhongshan Hospital, Fudan University, Shanghai, China; Shanghai Institute of Medical Imaging, Shanghai, China (C.D., P.Z., Z.S., M.Z.); Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China (L.Y., F.C.); Departments of Urology (J.L.) and Radiology (J.Z.), Xiamen Branch, Zhongshan Hospital, Fudan University, 668 Jinhu Road, Huli District, Xiamen 361015, China; Department of Urology, Zhangye People's Hospital affiliated to Hexi University, Zhangye, China (J.Y.); Department of Radiology, the First People's Hospital of Lianyungang, Lianyungang, China (X.Z.); Department of Radiology, Quanzhou First Hospital, Fujian Medical University, Quanzhou, China (R.H.); Department of Pathology, Sir Run Run Shaw Hospital, Hangzhou, China (R.W.); Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China (K.W., S.W.); Shanghai Key Laboratory of MICCAI, Shanghai, China (K.W., S.W.); Xiamen Municipal Clinical Research Center for Medical Imaging, Xiamen, China (J.Z.); and Xiamen Key Clinical Specialty, Xiamen, China (J.Z.)
| | - Feng Chen
- From the Departments of Radiology (C.D., P.Z., Z.S., M.Z., J.Z.), Urology (Y.X., J.G.), and Pathology (J.H.), Zhongshan Hospital, Fudan University, Shanghai, China; Shanghai Institute of Medical Imaging, Shanghai, China (C.D., P.Z., Z.S., M.Z.); Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China (L.Y., F.C.); Departments of Urology (J.L.) and Radiology (J.Z.), Xiamen Branch, Zhongshan Hospital, Fudan University, 668 Jinhu Road, Huli District, Xiamen 361015, China; Department of Urology, Zhangye People's Hospital affiliated to Hexi University, Zhangye, China (J.Y.); Department of Radiology, the First People's Hospital of Lianyungang, Lianyungang, China (X.Z.); Department of Radiology, Quanzhou First Hospital, Fujian Medical University, Quanzhou, China (R.H.); Department of Pathology, Sir Run Run Shaw Hospital, Hangzhou, China (R.W.); Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China (K.W., S.W.); Shanghai Key Laboratory of MICCAI, Shanghai, China (K.W., S.W.); Xiamen Municipal Clinical Research Center for Medical Imaging, Xiamen, China (J.Z.); and Xiamen Key Clinical Specialty, Xiamen, China (J.Z.)
| | - Jianming Guo
- From the Departments of Radiology (C.D., P.Z., Z.S., M.Z., J.Z.), Urology (Y.X., J.G.), and Pathology (J.H.), Zhongshan Hospital, Fudan University, Shanghai, China; Shanghai Institute of Medical Imaging, Shanghai, China (C.D., P.Z., Z.S., M.Z.); Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China (L.Y., F.C.); Departments of Urology (J.L.) and Radiology (J.Z.), Xiamen Branch, Zhongshan Hospital, Fudan University, 668 Jinhu Road, Huli District, Xiamen 361015, China; Department of Urology, Zhangye People's Hospital affiliated to Hexi University, Zhangye, China (J.Y.); Department of Radiology, the First People's Hospital of Lianyungang, Lianyungang, China (X.Z.); Department of Radiology, Quanzhou First Hospital, Fujian Medical University, Quanzhou, China (R.H.); Department of Pathology, Sir Run Run Shaw Hospital, Hangzhou, China (R.W.); Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China (K.W., S.W.); Shanghai Key Laboratory of MICCAI, Shanghai, China (K.W., S.W.); Xiamen Municipal Clinical Research Center for Medical Imaging, Xiamen, China (J.Z.); and Xiamen Key Clinical Specialty, Xiamen, China (J.Z.)
| | - Mengsu Zeng
- From the Departments of Radiology (C.D., P.Z., Z.S., M.Z., J.Z.), Urology (Y.X., J.G.), and Pathology (J.H.), Zhongshan Hospital, Fudan University, Shanghai, China; Shanghai Institute of Medical Imaging, Shanghai, China (C.D., P.Z., Z.S., M.Z.); Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China (L.Y., F.C.); Departments of Urology (J.L.) and Radiology (J.Z.), Xiamen Branch, Zhongshan Hospital, Fudan University, 668 Jinhu Road, Huli District, Xiamen 361015, China; Department of Urology, Zhangye People's Hospital affiliated to Hexi University, Zhangye, China (J.Y.); Department of Radiology, the First People's Hospital of Lianyungang, Lianyungang, China (X.Z.); Department of Radiology, Quanzhou First Hospital, Fujian Medical University, Quanzhou, China (R.H.); Department of Pathology, Sir Run Run Shaw Hospital, Hangzhou, China (R.W.); Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China (K.W., S.W.); Shanghai Key Laboratory of MICCAI, Shanghai, China (K.W., S.W.); Xiamen Municipal Clinical Research Center for Medical Imaging, Xiamen, China (J.Z.); and Xiamen Key Clinical Specialty, Xiamen, China (J.Z.)
| | - Jianjun Zhou
- From the Departments of Radiology (C.D., P.Z., Z.S., M.Z., J.Z.), Urology (Y.X., J.G.), and Pathology (J.H.), Zhongshan Hospital, Fudan University, Shanghai, China; Shanghai Institute of Medical Imaging, Shanghai, China (C.D., P.Z., Z.S., M.Z.); Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China (L.Y., F.C.); Departments of Urology (J.L.) and Radiology (J.Z.), Xiamen Branch, Zhongshan Hospital, Fudan University, 668 Jinhu Road, Huli District, Xiamen 361015, China; Department of Urology, Zhangye People's Hospital affiliated to Hexi University, Zhangye, China (J.Y.); Department of Radiology, the First People's Hospital of Lianyungang, Lianyungang, China (X.Z.); Department of Radiology, Quanzhou First Hospital, Fujian Medical University, Quanzhou, China (R.H.); Department of Pathology, Sir Run Run Shaw Hospital, Hangzhou, China (R.W.); Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China (K.W., S.W.); Shanghai Key Laboratory of MICCAI, Shanghai, China (K.W., S.W.); Xiamen Municipal Clinical Research Center for Medical Imaging, Xiamen, China (J.Z.); and Xiamen Key Clinical Specialty, Xiamen, China (J.Z.)
| | - Shuo Wang
- From the Departments of Radiology (C.D., P.Z., Z.S., M.Z., J.Z.), Urology (Y.X., J.G.), and Pathology (J.H.), Zhongshan Hospital, Fudan University, Shanghai, China; Shanghai Institute of Medical Imaging, Shanghai, China (C.D., P.Z., Z.S., M.Z.); Department of Radiology, The First Affiliated Hospital, Zhejiang University School of Medicine, Hangzhou, China (L.Y., F.C.); Departments of Urology (J.L.) and Radiology (J.Z.), Xiamen Branch, Zhongshan Hospital, Fudan University, 668 Jinhu Road, Huli District, Xiamen 361015, China; Department of Urology, Zhangye People's Hospital affiliated to Hexi University, Zhangye, China (J.Y.); Department of Radiology, the First People's Hospital of Lianyungang, Lianyungang, China (X.Z.); Department of Radiology, Quanzhou First Hospital, Fujian Medical University, Quanzhou, China (R.H.); Department of Pathology, Sir Run Run Shaw Hospital, Hangzhou, China (R.W.); Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China (K.W., S.W.); Shanghai Key Laboratory of MICCAI, Shanghai, China (K.W., S.W.); Xiamen Municipal Clinical Research Center for Medical Imaging, Xiamen, China (J.Z.); and Xiamen Key Clinical Specialty, Xiamen, China (J.Z.)
| |
Collapse
|
19
|
Chen Z, Yao L, Liu Y, Han X, Gong Z, Luo J, Zhao J, Fang G. Deep learning-aided 3D proxy-bridged region-growing framework for multi-organ segmentation. Sci Rep 2024; 14:9784. [PMID: 38684904 PMCID: PMC11059262 DOI: 10.1038/s41598-024-60668-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/13/2023] [Accepted: 04/25/2024] [Indexed: 05/02/2024] Open
Abstract
Accurate multi-organ segmentation in 3D CT images is imperative for enhancing computer-aided diagnosis and radiotherapy planning. However, current deep learning-based methods for 3D multi-organ segmentation face challenges such as the need for labor-intensive manual pixel-level annotations and high hardware resource demands, especially regarding GPU resources. To address these issues, we propose a 3D proxy-bridged region-growing framework specifically designed for the segmentation of the liver and spleen. Specifically, a key slice is selected from each 3D volume according to the corresponding intensity histogram. Subsequently, a deep learning model is employed to pinpoint the semantic central patch on this key slice, to calculate the growing seed. To counteract the impact of noise, segmentation of the liver and spleen is conducted on superpixel images created through proxy-bridging strategy. The segmentation process is then extended to adjacent slices by applying the same methodology iteratively, culminating in the comprehensive segmentation results. Experimental results demonstrate that the proposed framework accomplishes segmentation of the liver and spleen with an average Dice Similarity Coefficient of approximately 0.93 and a Jaccard Similarity Coefficient of around 0.88. These outcomes substantiate the framework's capability to achieve performance on par with that of deep learning methods, albeit requiring less guidance information and lower GPU resources.
Collapse
Affiliation(s)
- Zhihong Chen
- Institute of Computing Science and Technology, Guangzhou University, Guangzhou, 510006, China
- Guangdong Provincial Key Laboratory of Artificial Intelligence in Medical Image Analysis and Application, Guangzhou, 510080, China
| | - Lisha Yao
- Guangdong Provincial Key Laboratory of Artificial Intelligence in Medical Image Analysis and Application, Guangzhou, 510080, China
- School of Medicine, South China University of Technology, Guangzhou, 510180, China
| | - Yue Liu
- Institute of Computing Science and Technology, Guangzhou University, Guangzhou, 510006, China
- School of Information Engineering, Jiangxi College of Applied Technology, Ganzhou, 341000, China
| | - Xiaorui Han
- Department of Radiology, School of Medicine, Guangzhou First People's Hospital, South China University of Technology, Guangzhou, 510180, China
| | - Zhengze Gong
- Information and Data Centre, School of Medicine, Guangzhou First People's Hospital, South China University of Technology Guangdong, Guangzhou, 510180, China
| | - Jichao Luo
- Institute of Computing Science and Technology, Guangzhou University, Guangzhou, 510006, China
- Guangdong Provincial Key Laboratory of Artificial Intelligence in Medical Image Analysis and Application, Guangzhou, 510080, China
| | - Jietong Zhao
- Institute of Computing Science and Technology, Guangzhou University, Guangzhou, 510006, China
| | - Gang Fang
- Institute of Computing Science and Technology, Guangzhou University, Guangzhou, 510006, China.
- Guangdong Provincial Key Laboratory of Artificial Intelligence in Medical Image Analysis and Application, Guangzhou, 510080, China.
| |
Collapse
|
20
|
Carriero A, Groenhoff L, Vologina E, Basile P, Albera M. Deep Learning in Breast Cancer Imaging: State of the Art and Recent Advancements in Early 2024. Diagnostics (Basel) 2024; 14:848. [PMID: 38667493 PMCID: PMC11048882 DOI: 10.3390/diagnostics14080848] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/29/2024] [Revised: 04/07/2024] [Accepted: 04/17/2024] [Indexed: 04/28/2024] Open
Abstract
The rapid advancement of artificial intelligence (AI) has significantly impacted various aspects of healthcare, particularly in the medical imaging field. This review focuses on recent developments in the application of deep learning (DL) techniques to breast cancer imaging. DL models, a subset of AI algorithms inspired by human brain architecture, have demonstrated remarkable success in analyzing complex medical images, enhancing diagnostic precision, and streamlining workflows. DL models have been applied to breast cancer diagnosis via mammography, ultrasonography, and magnetic resonance imaging. Furthermore, DL-based radiomic approaches may play a role in breast cancer risk assessment, prognosis prediction, and therapeutic response monitoring. Nevertheless, several challenges have limited the widespread adoption of AI techniques in clinical practice, emphasizing the importance of rigorous validation, interpretability, and technical considerations when implementing DL solutions. By examining fundamental concepts in DL techniques applied to medical imaging and synthesizing the latest advancements and trends, this narrative review aims to provide valuable and up-to-date insights for radiologists seeking to harness the power of AI in breast cancer care.
Collapse
Affiliation(s)
| | - Léon Groenhoff
- Radiology Department, Maggiore della Carità Hospital, 28100 Novara, Italy; (A.C.); (E.V.); (P.B.); (M.A.)
| | | | | | | |
Collapse
|
21
|
Alhussaini AJ, Steele JD, Jawli A, Nabi G. Radiomics Machine Learning Analysis of Clear Cell Renal Cell Carcinoma for Tumour Grade Prediction Based on Intra-Tumoural Sub-Region Heterogeneity. Cancers (Basel) 2024; 16:1454. [PMID: 38672536 PMCID: PMC11048006 DOI: 10.3390/cancers16081454] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2024] [Revised: 03/22/2024] [Accepted: 04/03/2024] [Indexed: 04/28/2024] Open
Abstract
BACKGROUND Renal cancers are among the top ten causes of cancer-specific mortality, of which the ccRCC subtype is responsible for most cases. The grading of ccRCC is important in determining tumour aggressiveness and clinical management. OBJECTIVES The objectives of this research were to predict the WHO/ISUP grade of ccRCC pre-operatively and characterise the heterogeneity of tumour sub-regions using radiomics and ML models, including comparison with pre-operative biopsy-determined grading in a sub-group. METHODS Data were obtained from multiple institutions across two countries, including 391 patients with pathologically proven ccRCC. For analysis, the data were separated into four cohorts. Cohorts 1 and 2 included data from the respective institutions from the two countries, cohort 3 was the combined data from both cohort 1 and 2, and cohort 4 was a subset of cohort 1, for which both the biopsy and subsequent histology from resection (partial or total nephrectomy) were available. 3D image segmentation was carried out to derive a voxel of interest (VOI) mask. Radiomics features were then extracted from the contrast-enhanced images, and the data were normalised. The Pearson correlation coefficient and the XGBoost model were used to reduce the dimensionality of the features. Thereafter, 11 ML algorithms were implemented for the purpose of predicting the ccRCC grade and characterising the heterogeneity of sub-regions in the tumours. RESULTS For cohort 1, the 50% tumour core and 25% tumour periphery exhibited the best performance, with an average AUC of 77.9% and 78.6%, respectively. The 50% tumour core presented the highest performance in cohorts 2 and 3, with average AUC values of 87.6% and 76.9%, respectively. With the 25% periphery, cohort 4 showed AUC values of 95.0% and 80.0% for grade prediction when using internal and external validation, respectively, while biopsy histology had an AUC of 31.0% for the classification with the final grade of resection histology as a reference standard. The CatBoost classifier was the best for each of the four cohorts with an average AUC of 80.0%, 86.5%, 77.0% and 90.3% for cohorts 1, 2, 3 and 4 respectively. CONCLUSIONS Radiomics signatures combined with ML have the potential to predict the WHO/ISUP grade of ccRCC with superior performance, when compared to pre-operative biopsy. Moreover, tumour sub-regions contain useful information that should be analysed independently when determining the tumour grade. Therefore, it is possible to distinguish the grade of ccRCC pre-operatively to improve patient care and management.
Collapse
Affiliation(s)
- Abeer J. Alhussaini
- Division of Imaging Sciences and Technology, School of Medicine, Ninewells Hospital, University of Dundee, Dundee DD1 9SY, UK
- Department of Clinical Radiology, Al-Amiri Hospital, Ministry of Health, Sulaibikhat 1300, Kuwait
| | - J. Douglas Steele
- Division of Imaging Sciences and Technology, School of Medicine, Ninewells Hospital, University of Dundee, Dundee DD1 9SY, UK
| | - Adel Jawli
- Division of Imaging Sciences and Technology, School of Medicine, Ninewells Hospital, University of Dundee, Dundee DD1 9SY, UK
- Department of Clinical Radiology, Sheikh Jaber Al-Ahmad Al-Sabah Hospital, Ministry of Health, Sulaibikhat 1300, Kuwait
| | - Ghulam Nabi
- Division of Imaging Sciences and Technology, School of Medicine, Ninewells Hospital, University of Dundee, Dundee DD1 9SY, UK
| |
Collapse
|
22
|
Zheng S, Sun Q, Ye X, Li W, Yu L, Yang C. Multi-scale adversarial learning with difficult region supervision learning models for primary tumor segmentation. Phys Med Biol 2024; 69:085009. [PMID: 38471170 DOI: 10.1088/1361-6560/ad3321] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2023] [Accepted: 03/12/2024] [Indexed: 03/14/2024]
Abstract
Objective.Recently, deep learning techniques have found extensive application in accurate and automated segmentation of tumor regions. However, owing to the variety of tumor shapes, complex types, and unpredictability of spatial distribution, tumor segmentation still faces major challenges. Taking cues from the deep supervision and adversarial learning, we have devised a cascade-based methodology incorporating multi-scale adversarial learning and difficult-region supervision learning in this study to tackle these challenges.Approach.Overall, the method adheres to a coarse-to-fine strategy, first roughly locating the target region, and then refining the target object with multi-stage cascaded binary segmentation which converts complex multi-class segmentation problems into multiple simpler binary segmentation problems. In addition, a multi-scale adversarial learning difficult supervised UNet (MSALDS-UNet) is proposed as our model for fine-segmentation, which applies multiple discriminators along the decoding path of the segmentation network to implement multi-scale adversarial learning, thereby enhancing the accuracy of network segmentation. Meanwhile, in MSALDS-UNet, we introduce a difficult region supervision loss to effectively utilize structural information for segmenting difficult-to-distinguish areas, such as blurry boundary areas.Main results.A thorough validation of three independent public databases (KiTS21, MSD's Brain and Pancreas datasets) shows that our model achieves satisfactory results for tumor segmentation in terms of key evaluation metrics including dice similarity coefficient, Jaccard similarity coefficient, and HD95.Significance.This paper introduces a cascade approach that combines multi-scale adversarial learning and difficult supervision to achieve precise tumor segmentation. It confirms that the combination can improve the segmentation performance, especially for small objects (our codes are publicly availabled onhttps://zhengshenhai.github.io/).
Collapse
Affiliation(s)
- Shenhai Zheng
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China
- Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China
| | - Qiuyu Sun
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China
| | - Xin Ye
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China
| | - Weisheng Li
- College of Computer Science and Technology, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China
- Chongqing Key Laboratory of Image Cognition, Chongqing University of Posts and Telecommunications, Chongqing, People's Republic of China
| | - Lei Yu
- Emergency Department, The Second Affiliated Hospital of Chongqing Medical University, Chongqing, People's Republic of China
| | - Chaohui Yang
- Nanpeng Artificial Intelligence Research Institute, Chongqing, People's Republic of China
| |
Collapse
|
23
|
Conze PH, Andrade-Miranda G, Le Meur Y, Cornec-Le Gall E, Rousseau F. Dual-task kidney MR segmentation with transformers in autosomal-dominant polycystic kidney disease. Comput Med Imaging Graph 2024; 113:102349. [PMID: 38330635 DOI: 10.1016/j.compmedimag.2024.102349] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2023] [Revised: 01/21/2024] [Accepted: 02/01/2024] [Indexed: 02/10/2024]
Abstract
Autosomal-dominant polycystic kidney disease is a prevalent genetic disorder characterized by the development of renal cysts, leading to kidney enlargement and renal failure. Accurate measurement of total kidney volume through polycystic kidney segmentation is crucial to assess disease severity, predict progression and evaluate treatment effects. Traditional manual segmentation suffers from intra- and inter-expert variability, prompting the exploration of automated approaches. In recent years, convolutional neural networks have been employed for polycystic kidney segmentation from magnetic resonance images. However, the use of Transformer-based models, which have shown remarkable performance in a wide range of computer vision and medical image analysis tasks, remains unexplored in this area. With their self-attention mechanism, Transformers excel in capturing global context information, which is crucial for accurate organ delineations. In this paper, we evaluate and compare various convolutional-based, Transformers-based, and hybrid convolutional/Transformers-based networks for polycystic kidney segmentation. Additionally, we propose a dual-task learning scheme, where a common feature extractor is followed by per-kidney decoders, towards better generalizability and efficiency. We extensively evaluate various architectures and learning schemes on a heterogeneous magnetic resonance imaging dataset collected from 112 patients with polycystic kidney disease. Our results highlight the effectiveness of Transformer-based models for polycystic kidney segmentation and the relevancy of exploiting dual-task learning to improve segmentation accuracy and mitigate data scarcity issues. A promising ability in accurately delineating polycystic kidneys is especially shown in the presence of heterogeneous cyst distributions and adjacent cyst-containing organs. This work contribute to the advancement of reliable delineation methods in nephrology, paving the way for a broad spectrum of clinical applications.
Collapse
Affiliation(s)
- Pierre-Henri Conze
- IMT Atlantique, LaTIM UMR 1101, Technopôle Brest-Iroise, 29238 Brest, France; LaTIM UMR 1101, Inserm, IBRBS, 22 rue Camille Desmoulins, 29200 Brest, France.
| | | | - Yannick Le Meur
- Department of Nephrology, University Hospital of Brest, bd Tanguy Prigent, 29200 Brest, France; LBAI UMR 1227, Inserm, 9 rue Félix le Dantec, 29200 Brest, France
| | - Emilie Cornec-Le Gall
- Department of Nephrology, University Hospital of Brest, bd Tanguy Prigent, 29200 Brest, France; UMR 1078, Inserm, IBRBS, 22 rue Camille Desmoulins, 29238 Brest, France
| | - François Rousseau
- IMT Atlantique, LaTIM UMR 1101, Technopôle Brest-Iroise, 29238 Brest, France; LaTIM UMR 1101, Inserm, IBRBS, 22 rue Camille Desmoulins, 29200 Brest, France
| |
Collapse
|
24
|
Sadikine A, Badic B, Tasu JP, Noblet V, Ballet P, Visvikis D, Conze PH. Improving abdominal image segmentation with overcomplete shape priors. Comput Med Imaging Graph 2024; 113:102356. [PMID: 38340573 DOI: 10.1016/j.compmedimag.2024.102356] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2023] [Revised: 12/11/2023] [Accepted: 02/06/2024] [Indexed: 02/12/2024]
Abstract
The extraction of abdominal structures using deep learning has recently experienced a widespread interest in medical image analysis. Automatic abdominal organ and vessel segmentation is highly desirable to guide clinicians in computer-assisted diagnosis, therapy, or surgical planning. Despite a good ability to extract large organs, the capacity of U-Net inspired architectures to automatically delineate smaller structures remains a major issue, especially given the increase in receptive field size as we go deeper into the network. To deal with various abdominal structure sizes while exploiting efficient geometric constraints, we present a novel approach that integrates into deep segmentation shape priors from a semi-overcomplete convolutional auto-encoder (S-OCAE) embedding. Compared to standard convolutional auto-encoders (CAE), it exploits an over-complete branch that projects data onto higher dimensions to better characterize anatomical structures with a small spatial extent. Experiments on abdominal organs and vessel delineation performed on various publicly available datasets highlight the effectiveness of our method compared to state-of-the-art, including U-Net trained without and with shape priors from a traditional CAE. Exploiting a semi-overcomplete convolutional auto-encoder embedding as shape priors improves the ability of deep segmentation models to provide realistic and accurate abdominal structure contours.
Collapse
Affiliation(s)
- Amine Sadikine
- LaTIM UMR 1101, Inserm, Brest, 29200, France; University of Western Brittany, Brest, 29200, France
| | - Bogdan Badic
- LaTIM UMR 1101, Inserm, Brest, 29200, France; University Hospital of Brest, Brest, 29200, France
| | - Jean-Pierre Tasu
- LaTIM UMR 1101, Inserm, Brest, 29200, France; University Hospital of Poitiers, Poitiers, 86000, France
| | | | - Pascal Ballet
- LaTIM UMR 1101, Inserm, Brest, 29200, France; University of Western Brittany, Brest, 29200, France
| | | | - Pierre-Henri Conze
- LaTIM UMR 1101, Inserm, Brest, 29200, France; IMT Atlantique, Brest, 29200, France.
| |
Collapse
|
25
|
Qian B, Chen H, Wang X, Guan Z, Li T, Jin Y, Wu Y, Wen Y, Che H, Kwon G, Kim J, Choi S, Shin S, Krause F, Unterdechler M, Hou J, Feng R, Li Y, El Habib Daho M, Yang D, Wu Q, Zhang P, Yang X, Cai Y, Tan GSW, Cheung CY, Jia W, Li H, Tham YC, Wong TY, Sheng B. DRAC 2022: A public benchmark for diabetic retinopathy analysis on ultra-wide optical coherence tomography angiography images. PATTERNS (NEW YORK, N.Y.) 2024; 5:100929. [PMID: 38487802 PMCID: PMC10935505 DOI: 10.1016/j.patter.2024.100929] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/25/2023] [Revised: 12/09/2023] [Accepted: 01/15/2024] [Indexed: 03/17/2024]
Abstract
We described a challenge named "DRAC - Diabetic Retinopathy Analysis Challenge" in conjunction with the 25th International Conference on Medical Image Computing and Computer Assisted Intervention (MICCAI 2022). Within this challenge, we provided the DRAC datset, an ultra-wide optical coherence tomography angiography (UW-OCTA) dataset (1,103 images), addressing three primary clinical tasks: diabetic retinopathy (DR) lesion segmentation, image quality assessment, and DR grading. The scientific community responded positively to the challenge, with 11, 12, and 13 teams submitting different solutions for these three tasks, respectively. This paper presents a concise summary and analysis of the top-performing solutions and results across all challenge tasks. These solutions could provide practical guidance for developing accurate classification and segmentation models for image quality assessment and DR diagnosis using UW-OCTA images, potentially improving the diagnostic capabilities of healthcare professionals. The dataset has been released to support the development of computer-aided diagnostic systems for DR evaluation.
Collapse
Affiliation(s)
- Bo Qian
- Shanghai Belt and Road International Joint Laboratory for Intelligent Prevention and Treatment of Metabolic Disorders, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai 200240, China
- MOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Hao Chen
- Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong 999077, China
- Department of Chemical and Biological Engineering, The Hong Kong University of Science and Technology, Hong Kong 999077, China
| | - Xiangning Wang
- Shanghai Belt and Road International Joint Laboratory for Intelligent Prevention and Treatment of Metabolic Disorders, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai 200240, China
- Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai 200233, China
| | - Zhouyu Guan
- Shanghai Belt and Road International Joint Laboratory for Intelligent Prevention and Treatment of Metabolic Disorders, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai 200240, China
| | - Tingyao Li
- Shanghai Belt and Road International Joint Laboratory for Intelligent Prevention and Treatment of Metabolic Disorders, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai 200240, China
- MOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Yixiao Jin
- Tsinghua Medicine, Tsinghua University, Beijing 100084, China
| | - Yilan Wu
- Tsinghua Medicine, Tsinghua University, Beijing 100084, China
| | - Yang Wen
- School of Electronic and Information Engineering, Shenzhen University, Shenzhen 518060, China
| | - Haoxuan Che
- Department of Computer Science and Engineering, The Hong Kong University of Science and Technology, Hong Kong 999077, China
| | | | | | - Sungjin Choi
- AI/DX Convergence Business Group, KT, Seongnam 13606, Korea
| | - Seoyoung Shin
- AI/DX Convergence Business Group, KT, Seongnam 13606, Korea
| | - Felix Krause
- Johannes Kepler University Linz, Linz 4040, Austria
| | | | - Junlin Hou
- School of Computer Science, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University, Shanghai 200433, China
| | - Rui Feng
- School of Computer Science, Shanghai Key Laboratory of Intelligent Information Processing, Fudan University, Shanghai 200433, China
- Academy for Engineering and Technology, Fudan University, Shanghai 200433, China
| | - Yihao Li
- LaTIM UMR 1101, INSERM, 29609 Brest, France
- University of Western Brittany, 29238 Brest, France
| | - Mostafa El Habib Daho
- LaTIM UMR 1101, INSERM, 29609 Brest, France
- University of Western Brittany, 29238 Brest, France
| | - Dawei Yang
- Department of Ophthalmology and Visual Sciences, The Chinese University of Hong Kong, Hong Kong 999077, China
| | - Qiang Wu
- Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai 200233, China
| | - Ping Zhang
- Department of Computer Science and Engineering, The Ohio State University, Columbus, OH 43210, USA
- Department of Biomedical Informatics, The Ohio State University, Columbus, OH 43210, USA
- Translational Data Analytics Institute, The Ohio State University, Columbus, OH 43210, USA
| | - Xiaokang Yang
- MOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
| | - Yiyu Cai
- School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore 639798, Singapore
| | - Gavin Siew Wei Tan
- Singapore Eye Research Institute, Singapore National Eye Centre, Singapore 168751, Singapore
| | - Carol Y. Cheung
- Department of Ophthalmology and Visual Sciences, The Chinese University of Hong Kong, Hong Kong 999077, China
| | - Weiping Jia
- Shanghai Belt and Road International Joint Laboratory for Intelligent Prevention and Treatment of Metabolic Disorders, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai 200240, China
| | - Huating Li
- Shanghai Belt and Road International Joint Laboratory for Intelligent Prevention and Treatment of Metabolic Disorders, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai 200240, China
| | - Yih Chung Tham
- Singapore Eye Research Institute, Singapore National Eye Centre, Singapore 168751, Singapore
- Centre for Innovation and Precision Eye Health; and Department of Ophthalmology, Yong Loo Lin School of Medicine, National University of Singapore, Singapore 119228, Singapore
- Ophthalmology and Visual Sciences Academic Clinical Program, Duke-NUS Medical School, Singapore 169857, Singapore
| | - Tien Yin Wong
- Tsinghua Medicine, Tsinghua University, Beijing 100084, China
- Singapore Eye Research Institute, Singapore National Eye Centre, Singapore 168751, Singapore
- School of Clinical Medicine, Beijing Tsinghua Changgung Hospital, Beijing 102218, China
| | - Bin Sheng
- Shanghai Belt and Road International Joint Laboratory for Intelligent Prevention and Treatment of Metabolic Disorders, Department of Computer Science and Engineering, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Department of Endocrinology and Metabolism, Shanghai Sixth People's Hospital Affiliated to Shanghai Jiao Tong University School of Medicine, Shanghai Diabetes Institute, Shanghai Clinical Center for Diabetes, Shanghai 200240, China
- MOE Key Laboratory of AI, School of Electronic, Information, and Electrical Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
| |
Collapse
|
26
|
Zhang W, Yue Y, Pan H, Chen Z, Wang C, Pfister H, Wang W. Marching Windows: Scalable Mesh Generation for Volumetric Data With Multiple Materials. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 2024; 30:1728-1742. [PMID: 36455093 PMCID: PMC10980537 DOI: 10.1109/tvcg.2022.3225526] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/17/2023]
Abstract
Volumetric data abounds in medical imaging and other fields. With the improved imaging quality and the increased resolution, volumetric datasets are getting so large that the existing tools have become inadequate for processing and analyzing the data. Here we consider the problem of computing tetrahedral meshes to represent large volumetric datasets with labeled multiple materials, which are often encountered in medical imaging or microscopy optical slice tomography. Such tetrahedral meshes are a more compact and expressive geometric representation so are in demand for efficient visualization and simulation of the data, which are impossible if the original large volumetric data are used directly due to the large memory requirement. Existing methods for meshing volumetric data are not scalable for handling large datasets due to their sheer demand on excessively large run-time memory or failure to produce a tet-mesh that preserves the multi-material structure of the original volumetric data. In this article we propose a novel approach, called Marching Windows, that uses a moving window and a disk-swap strategy to reduce the run-time memory footprint, devise a new scheme that guarantees to preserve the topological structure of the original dataset, and adopt an error-guided optimization technique to improve both geometric approximation error and mesh quality. Extensive experiments show that our method is capable of processing very large volumetric datasets beyond the capability of the existing methods and producing tetrahedral meshes of high quality.
Collapse
|
27
|
Kovacs DG, Ladefoged CN, Andersen KF, Brittain JM, Christensen CB, Dejanovic D, Hansen NL, Loft A, Petersen JH, Reichkendler M, Andersen FL, Fischer BM. Clinical Evaluation of Deep Learning for Tumor Delineation on 18F-FDG PET/CT of Head and Neck Cancer. J Nucl Med 2024; 65:jnumed.123.266574. [PMID: 38388516 PMCID: PMC10995525 DOI: 10.2967/jnumed.123.266574] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/23/2023] [Revised: 01/29/2024] [Accepted: 01/29/2024] [Indexed: 02/24/2024] Open
Abstract
Artificial intelligence (AI) may decrease 18F-FDG PET/CT-based gross tumor volume (GTV) delineation variability and automate tumor-volume-derived image biomarker extraction. Hence, we aimed to identify and evaluate promising state-of-the-art deep learning methods for head and neck cancer (HNC) PET GTV delineation. Methods: We trained and evaluated deep learning methods using retrospectively included scans of HNC patients referred for radiotherapy between January 2014 and December 2019 (ISRCTN16907234). We used 3 test datasets: an internal set to compare methods, another internal set to compare AI-to-expert variability and expert interobserver variability (IOV), and an external set to compare internal and external AI-to-expert variability. Expert PET GTVs were used as the reference standard. Our benchmark IOV was measured using the PET GTV of 6 experts. The primary outcome was the Dice similarity coefficient (DSC). ANOVA was used to compare methods, a paired t test was used to compare AI-to-expert variability and expert IOV, an unpaired t test was used to compare internal and external AI-to-expert variability, and post hoc Bland-Altman analysis was used to evaluate biomarker agreement. Results: In total, 1,220 18F-FDG PET/CT scans of 1,190 patients (mean age ± SD, 63 ± 10 y; 858 men) were included, and 5 deep learning methods were trained using 5-fold cross-validation (n = 805). The nnU-Net method achieved the highest similarity (DSC, 0.80 [95% CI, 0.77-0.86]; n = 196). We found no evidence of a difference between expert IOV and AI-to-expert variability (DSC, 0.78 for AI vs. 0.82 for experts; mean difference of 0.04 [95% CI, -0.01 to 0.09]; P = 0.12; n = 64). We found no evidence of a difference between the internal and external AI-to-expert variability (DSC, 0.80 internally vs. 0.81 externally; mean difference of 0.004 [95% CI, -0.05 to 0.04]; P = 0.87; n = 125). PET GTV-derived biomarkers of AI were in good agreement with experts. Conclusion: Deep learning can be used to automate 18F-FDG PET/CT tumor-volume-derived imaging biomarkers, and the deep-learning-based volumes have the potential to assist clinical tumor volume delineation in radiation oncology.
Collapse
Affiliation(s)
- David G Kovacs
- Department of Clinical Physiology and Nuclear Medicine, Rigshospitalet, University of Copenhagen, Copenhagen, Denmark;
- Department of Clinical Medicine, Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| | - Claes N Ladefoged
- Department of Clinical Physiology and Nuclear Medicine, Rigshospitalet, University of Copenhagen, Copenhagen, Denmark
- Department of Applied Mathematics and Computer Science, Technical University of Denmark, Lyngby, Denmark
| | - Kim F Andersen
- Department of Clinical Physiology and Nuclear Medicine, Rigshospitalet, University of Copenhagen, Copenhagen, Denmark
| | - Jane M Brittain
- Department of Clinical Physiology and Nuclear Medicine, Rigshospitalet, University of Copenhagen, Copenhagen, Denmark
| | - Charlotte B Christensen
- Department of Clinical Physiology and Nuclear Medicine, Herlev Hospital, University of Copenhagen, Copenhagen, Denmark
| | - Danijela Dejanovic
- Department of Clinical Physiology and Nuclear Medicine, Rigshospitalet, University of Copenhagen, Copenhagen, Denmark
| | - Naja L Hansen
- Department of Clinical Physiology and Nuclear Medicine, Rigshospitalet, University of Copenhagen, Copenhagen, Denmark
| | - Annika Loft
- Department of Clinical Physiology and Nuclear Medicine, Rigshospitalet, University of Copenhagen, Copenhagen, Denmark
| | - Jørgen H Petersen
- Section of Biostatistics, Institute of Public Health, Faculty of Health Sciences, University of Copenhagen, Denmark; and
| | - Michala Reichkendler
- Department of Clinical Physiology and Nuclear Medicine, Rigshospitalet, University of Copenhagen, Copenhagen, Denmark
| | - Flemming L Andersen
- Department of Clinical Physiology and Nuclear Medicine, Rigshospitalet, University of Copenhagen, Copenhagen, Denmark
- Department of Clinical Medicine, Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
| | - Barbara M Fischer
- Department of Clinical Physiology and Nuclear Medicine, Rigshospitalet, University of Copenhagen, Copenhagen, Denmark
- Department of Clinical Medicine, Faculty of Health and Medical Sciences, University of Copenhagen, Copenhagen, Denmark
- PET Centre, School of Biomedical Engineering and Imaging Science, King's College London, London, United Kingdom
| |
Collapse
|
28
|
Li X, Jia L, Lin F, Chai F, Liu T, Zhang W, Wei Z, Xiong W, Li H, Zhang M, Wang Y. Semi-supervised auto-segmentation method for pelvic organ-at-risk in magnetic resonance images based on deep-learning. J Appl Clin Med Phys 2024; 25:e14296. [PMID: 38386963 DOI: 10.1002/acm2.14296] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/20/2023] [Revised: 01/06/2024] [Accepted: 01/23/2024] [Indexed: 02/24/2024] Open
Abstract
BACKGROUND AND PURPOSE In radiotherapy, magnetic resonance (MR) imaging has higher contrast for soft tissues compared to computed tomography (CT) scanning and does not emit radiation. However, manual annotation of the deep learning-based automatic organ-at-risk (OAR) delineation algorithms is expensive, making the collection of large-high-quality annotated datasets a challenge. Therefore, we proposed the low-cost semi-supervised OAR segmentation method using small pelvic MR image annotations. METHODS We trained a deep learning-based segmentation model using 116 sets of MR images from 116 patients. The bladder, femoral heads, rectum, and small intestine were selected as OAR regions. To generate the training set, we utilized a semi-supervised method and ensemble learning techniques. Additionally, we employed a post-processing algorithm to correct the self-annotation data. Both 2D and 3D auto-segmentation networks were evaluated for their performance. Furthermore, we evaluated the performance of semi-supervised method for 50 labeled data and only 10 labeled data. RESULTS The Dice similarity coefficient (DSC) of the bladder, femoral heads, rectum and small intestine between segmentation results and reference masks is 0.954, 0.984, 0.908, 0.852 only using self-annotation and post-processing methods of 2D segmentation model. The DSC of corresponding OARs is 0.871, 0.975, 0.975, 0.783, 0.724 using 3D segmentation network, 0.896, 0.984, 0.890, 0.828 using 2D segmentation network and common supervised method. CONCLUSION The outcomes of our study demonstrate that it is possible to train a multi-OAR segmentation model using small annotation samples and additional unlabeled data. To effectively annotate the dataset, ensemble learning and post-processing methods were employed. Additionally, when dealing with anisotropy and limited sample sizes, the 2D model outperformed the 3D model in terms of performance.
Collapse
Affiliation(s)
- Xianan Li
- Department of Radiation Oncology, Peking University People's Hospital, Beijing, China
| | - Lecheng Jia
- Radiotherapy laboratory, Shenzhen United Imaging Research Institute of Innovative Medical Equipment, Shenzhen, China
- Zhejiang Engineering Research Center for Innovation and Application of Intelligent Radiotherapy Technology, Wenzhou, China
| | - Fengyu Lin
- Radiotherapy laboratory, Shenzhen United Imaging Research Institute of Innovative Medical Equipment, Shenzhen, China
| | - Fan Chai
- Department of Radiology, Peking University People's Hospital, Beijing, China
| | - Tao Liu
- Department of Radiology, Peking University People's Hospital, Beijing, China
| | - Wei Zhang
- Radiotherapy Business Unit, Shanghai United Imaging Healthcare Co., Ltd., Shanghai, China
| | - Ziquan Wei
- Radiotherapy laboratory, Shenzhen United Imaging Research Institute of Innovative Medical Equipment, Shenzhen, China
| | - Weiqi Xiong
- Radiotherapy Business Unit, Shanghai United Imaging Healthcare Co., Ltd., Shanghai, China
| | - Hua Li
- Radiotherapy laboratory, Shenzhen United Imaging Research Institute of Innovative Medical Equipment, Shenzhen, China
| | - Min Zhang
- Department of Radiation Oncology, Peking University People's Hospital, Beijing, China
| | - Yi Wang
- Department of Radiology, Peking University People's Hospital, Beijing, China
| |
Collapse
|
29
|
Zhang M, Ye Z, Yuan E, Lv X, Zhang Y, Tan Y, Xia C, Tang J, Huang J, Li Z. Imaging-based deep learning in kidney diseases: recent progress and future prospects. Insights Imaging 2024; 15:50. [PMID: 38360904 PMCID: PMC10869329 DOI: 10.1186/s13244-024-01636-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/19/2023] [Accepted: 01/27/2024] [Indexed: 02/17/2024] Open
Abstract
Kidney diseases result from various causes, which can generally be divided into neoplastic and non-neoplastic diseases. Deep learning based on medical imaging is an established methodology for further data mining and an evolving field of expertise, which provides the possibility for precise management of kidney diseases. Recently, imaging-based deep learning has been widely applied to many clinical scenarios of kidney diseases including organ segmentation, lesion detection, differential diagnosis, surgical planning, and prognosis prediction, which can provide support for disease diagnosis and management. In this review, we will introduce the basic methodology of imaging-based deep learning and its recent clinical applications in neoplastic and non-neoplastic kidney diseases. Additionally, we further discuss its current challenges and future prospects and conclude that achieving data balance, addressing heterogeneity, and managing data size remain challenges for imaging-based deep learning. Meanwhile, the interpretability of algorithms, ethical risks, and barriers of bias assessment are also issues that require consideration in future development. We hope to provide urologists, nephrologists, and radiologists with clear ideas about imaging-based deep learning and reveal its great potential in clinical practice.Critical relevance statement The wide clinical applications of imaging-based deep learning in kidney diseases can help doctors to diagnose, treat, and manage patients with neoplastic or non-neoplastic renal diseases.Key points• Imaging-based deep learning is widely applied to neoplastic and non-neoplastic renal diseases.• Imaging-based deep learning improves the accuracy of the delineation, diagnosis, and evaluation of kidney diseases.• The small dataset, various lesion sizes, and so on are still challenges for deep learning.
Collapse
Affiliation(s)
- Meng Zhang
- Department of Radiology, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China
- Medical Equipment Innovation Research Center, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China
- Med+X Center for Manufacturing, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China
| | - Zheng Ye
- Department of Radiology, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China
| | - Enyu Yuan
- Department of Radiology, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China
| | - Xinyang Lv
- Department of Radiology, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China
| | - Yiteng Zhang
- Department of Radiology, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China
| | - Yuqi Tan
- Department of Radiology, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China
| | - Chunchao Xia
- Department of Radiology, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China
| | - Jing Tang
- Department of Radiology, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China.
| | - Jin Huang
- Medical Equipment Innovation Research Center, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China.
- Med+X Center for Manufacturing, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China.
| | - Zhenlin Li
- Department of Radiology, West China Hospital, Sichuan University, No. 37 Guoxue Alley, Chengdu, 610041, China.
| |
Collapse
|
30
|
Jiao R, Zhang Y, Ding L, Xue B, Zhang J, Cai R, Jin C. Learning with limited annotations: A survey on deep semi-supervised learning for medical image segmentation. Comput Biol Med 2024; 169:107840. [PMID: 38157773 DOI: 10.1016/j.compbiomed.2023.107840] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2023] [Revised: 10/30/2023] [Accepted: 12/07/2023] [Indexed: 01/03/2024]
Abstract
Medical image segmentation is a fundamental and critical step in many image-guided clinical approaches. Recent success of deep learning-based segmentation methods usually relies on a large amount of labeled data, which is particularly difficult and costly to obtain, especially in the medical imaging domain where only experts can provide reliable and accurate annotations. Semi-supervised learning has emerged as an appealing strategy and been widely applied to medical image segmentation tasks to train deep models with limited annotations. In this paper, we present a comprehensive review of recently proposed semi-supervised learning methods for medical image segmentation and summarize both the technical novelties and empirical results. Furthermore, we analyze and discuss the limitations and several unsolved problems of existing approaches. We hope this review can inspire the research community to explore solutions to this challenge and further advance the field of medical image segmentation.
Collapse
Affiliation(s)
- Rushi Jiao
- School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China; School of Engineering Medicine, Beihang University, Beijing, 100191, China; Shanghai Artificial Intelligence Laboratory, Shanghai, 200232, China.
| | - Yichi Zhang
- School of Data Science, Fudan University, Shanghai, 200433, China; Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, 200433, China.
| | - Le Ding
- School of Biological Science and Medical Engineering, Beihang University, Beijing, 100191, China.
| | - Bingsen Xue
- School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China; Shanghai Artificial Intelligence Laboratory, Shanghai, 200232, China.
| | - Jicong Zhang
- School of Biological Science and Medical Engineering, Beihang University, Beijing, 100191, China; Hefei Innovation Research Institute, Beihang University, Hefei, 230012, China.
| | - Rong Cai
- School of Engineering Medicine, Beihang University, Beijing, 100191, China; Key Laboratory for Biomechanics and Mechanobiology of Ministry of Education, Beihang University, Beijing, 100191, China.
| | - Cheng Jin
- School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China; Shanghai Artificial Intelligence Laboratory, Shanghai, 200232, China; Beijing Anding Hospital, Capital Medical University, Beijing, 100088, China.
| |
Collapse
|
31
|
Mahootiha M, Qadir HA, Bergsland J, Balasingham I. Multimodal deep learning for personalized renal cell carcinoma prognosis: Integrating CT imaging and clinical data. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 244:107978. [PMID: 38113804 DOI: 10.1016/j.cmpb.2023.107978] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/01/2023] [Revised: 12/05/2023] [Accepted: 12/12/2023] [Indexed: 12/21/2023]
Abstract
BACKGROUND AND OBJECTIVE Renal cell carcinoma represents a significant global health challenge with a low survival rate. The aim of this research was to devise a comprehensive deep-learning model capable of predicting survival probabilities in patients with renal cell carcinoma by integrating CT imaging and clinical data and addressing the limitations observed in prior studies. The aim is to facilitate the identification of patients requiring urgent treatment. METHODS The proposed framework comprises three modules: a 3D image feature extractor, clinical variable selection, and survival prediction. Based on the 3D CNN architecture, the feature extractor module predicts the ISUP grade of renal cell carcinoma tumors linked to mortality rates from CT images. Clinical variables are systematically selected using the Spearman score and random forest importance score as criteria. A deep learning-based network, trained with discrete LogisticHazard-based loss, performs the survival prediction. Nine distinct experiments are performed, with varying numbers of clinical variables determined by different thresholds of the Spearman and importance scores. RESULTS Our findings demonstrate that the proposed strategy surpasses the current literature on renal cancer prognosis based on CT scans and clinical factors. The best-performing experiment yielded a concordance index of 0.84 and an area under the curve value of 0.8 on the test cohort, which suggests strong predictive power. CONCLUSIONS The multimodal deep-learning approach developed in this study shows promising results in estimating survival probabilities for renal cell carcinoma patients using CT imaging and clinical data. This may have potential implications in identifying patients who require urgent treatment, potentially improving patient outcomes. The code created for this project is available for the public on: GitHub.
Collapse
Affiliation(s)
- Maryamalsadat Mahootiha
- The Intervention Centre, Oslo University Hospital, Oslo, 0372, Norway; Faculty of Medicine, University of Oslo, Oslo, 0372, Norway.
| | - Hemin Ali Qadir
- The Intervention Centre, Oslo University Hospital, Oslo, 0372, Norway
| | - Jacob Bergsland
- The Intervention Centre, Oslo University Hospital, Oslo, 0372, Norway
| | - Ilangko Balasingham
- The Intervention Centre, Oslo University Hospital, Oslo, 0372, Norway; Department of Electronic Systems, Norwegian University of Science and Technology, Trondheim, Norway
| |
Collapse
|
32
|
Raman AG, Fisher D, Yap F, Oberai A, Duddalwar VA. Radiomics and Artificial Intelligence: Renal Cell Carcinoma. Urol Clin North Am 2024; 51:35-45. [PMID: 37945101 DOI: 10.1016/j.ucl.2023.06.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2023]
Abstract
There is a clinical need for accurate diagnosis and prognostication of kidney cancer using imaging. Radiomics and deep learning methods applied to imaging have shown promise in tasks such as tumor segmentation, classification, staging, and grading, as well as assessment of preoperative scores and correlation with tumor biomarkers. Artificial intelligence is also expected to play a significant role in advancing personalized medicine for the treatment of renal cell carcinoma.
Collapse
Affiliation(s)
- Alex G Raman
- Department of Radiology, University of Southern California, 1500 San Pablo Street, 2nd Floor, Los Angeles, CA 90033, USA; Western University of Health Sciences, 309 East Second Street, Pomona, CA 91766-1854, USA
| | - David Fisher
- Department of Radiology, University of Southern California, 1500 San Pablo Street, 2nd Floor, Los Angeles, CA 90033, USA
| | - Felix Yap
- Radiology Associates, San Luis Obispo, 1310 Las Tablas Road, Templeton, CA 93465, USA
| | - Assad Oberai
- Viterbi School of Engineering, University of Southern California, 3650 McClintock Avenue, Los Angeles, CA 90089, USA
| | - Vinay A Duddalwar
- Department of Radiology, University of Southern California, 1500 San Pablo Street, 2nd Floor, Los Angeles, CA 90033, USA; Viterbi School of Engineering, University of Southern California, 3650 McClintock Avenue, Los Angeles, CA 90089, USA.
| |
Collapse
|
33
|
Yang F, Li X, Duan H, Xu F, Huang Y, Zhang X, Long Y, Zheng Y. MRL-Seg: Overcoming Imbalance in Medical Image Segmentation With Multi-Step Reinforcement Learning. IEEE J Biomed Health Inform 2024; 28:858-869. [PMID: 38032774 DOI: 10.1109/jbhi.2023.3336726] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/02/2023]
Abstract
Medical image segmentation is a critical task for clinical diagnosis and research. However, dealing with highly imbalanced data remains a significant challenge in this domain, where the region of interest (ROI) may exhibit substantial variations across different slices. This presents a significant hurdle to medical image segmentation, as conventional segmentation methods may either overlook the minority class or overly emphasize the majority class, ultimately leading to a decrease in the overall generalization ability of the segmentation results. To overcome this, we propose a novel approach based on multi-step reinforcement learning, which integrates prior knowledge of medical images and pixel-wise segmentation difficulty into the reward function. Our method treats each pixel as an individual agent, utilizing diverse actions to evaluate its relevance for segmentation. To validate the effectiveness of our approach, we conduct experiments on four imbalanced medical datasets, and the results show that our approach surpasses other state-of-the-art methods in highly imbalanced scenarios. These findings hold substantial implications for clinical diagnosis and research.
Collapse
|
34
|
R SSRM, T J. Multi-Scale and Spatial Information Extraction for Kidney Tumor Segmentation: A Contextual Deformable Attention and Edge-Enhanced U-Net. JOURNAL OF IMAGING INFORMATICS IN MEDICINE 2024; 37:151-166. [PMID: 38343255 DOI: 10.1007/s10278-023-00900-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/23/2023] [Revised: 09/04/2023] [Accepted: 09/05/2023] [Indexed: 03/02/2024]
Abstract
Kidney tumor segmentation is a difficult task because of the complex spatial and volumetric information present in medical images. Recent advances in deep convolutional neural networks (DCNNs) have improved tumor segmentation accuracy. However, the practical usability of current CNN-based networks is constrained by their high computational complexity. Additionally, these techniques often struggle to make adaptive modifications based on the structure of the tumors, which can lead to blurred edges in segmentation results. A lightweight architecture called the contextual deformable attention and edge-enhanced U-Net (CDA2E-Net) for high-accuracy pixel-level kidney tumor segmentation is proposed to address these challenges. Rather than using complex deep encoders, the approach includes a lightweight depthwise dilated ShuffleNetV2 (LDS-Net) encoder integrated into the CDA2E-Net framework. The proposed method also contains a multiscale attention feature pyramid pooling (MAF2P) module that improves the ability of multiscale features to adapt to various tumor shapes. Finally, an edge-enhanced loss function is introduced to guide the CDA2E-Net to concentrate on tumor edge information. The CDA2E-Net is evaluated on the KiTS19 and KiTS21 datasets, and the results demonstrate its superiority over existing approaches in terms of Hausdorff distance (HD), intersection over union (IoU), and dice coefficient (DSC) metrics.
Collapse
Affiliation(s)
- Shamija Sherryl R M R
- Department of Electronics & Communication Engineering, Ponjesly College of Engineering, Nagercoil, Tamil Nadu, India.
| | - Jaya T
- Department of Electronics & Communication Engineering, Saveetha Engineering College, Thandalam, India
| |
Collapse
|
35
|
Zhou Y, Lalande A, Chevalier C, Baude J, Aubignac L, Boudet J, Bessieres I. Deep learning application for abdominal organs segmentation on 0.35 T MR-Linac images. Front Oncol 2024; 13:1285924. [PMID: 38260833 PMCID: PMC10800957 DOI: 10.3389/fonc.2023.1285924] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/30/2023] [Accepted: 11/30/2023] [Indexed: 01/24/2024] Open
Abstract
Introduction Linear accelerator (linac) incorporating a magnetic resonance (MR) imaging device providing enhanced soft tissue contrast is particularly suited for abdominal radiation therapy. In particular, accurate segmentation for abdominal tumors and organs at risk (OARs) required for the treatment planning is becoming possible. Currently, this segmentation is performed manually by radiation oncologists. This process is very time consuming and subject to inter and intra operator variabilities. In this work, deep learning based automatic segmentation solutions were investigated for abdominal OARs on 0.35 T MR-images. Methods One hundred and twenty one sets of abdominal MR images and their corresponding ground truth segmentations were collected and used for this work. The OARs of interest included the liver, the kidneys, the spinal cord, the stomach and the duodenum. Several UNet based models have been trained in 2D (the Classical UNet, the ResAttention UNet, the EfficientNet UNet, and the nnUNet). The best model was then trained with a 3D strategy in order to investigate possible improvements. Geometrical metrics such as Dice Similarity Coefficient (DSC), Intersection over Union (IoU), Hausdorff Distance (HD) and analysis of the calculated volumes (thanks to Bland-Altman plot) were performed to evaluate the results. Results The nnUNet trained in 3D mode achieved the best performance, with DSC scores for the liver, the kidneys, the spinal cord, the stomach, and the duodenum of 0.96 ± 0.01, 0.91 ± 0.02, 0.91 ± 0.01, 0.83 ± 0.10, and 0.69 ± 0.15, respectively. The matching IoU scores were 0.92 ± 0.01, 0.84 ± 0.04, 0.84 ± 0.02, 0.54 ± 0.16 and 0.72 ± 0.13. The corresponding HD scores were 13.0 ± 6.0 mm, 16.0 ± 6.6 mm, 3.3 ± 0.7 mm, 35.0 ± 33.0 mm, and 42.0 ± 24.0 mm. The analysis of the calculated volumes followed the same behavior. Discussion Although the segmentation results for the duodenum were not optimal, these findings imply a potential clinical application of the 3D nnUNet model for the segmentation of abdominal OARs for images from 0.35 T MR-Linac.
Collapse
Affiliation(s)
- You Zhou
- Department of Medical Physics, Centre Georges-François Leclerc, Dijon, France
- Institut de Chimie Moléculaire de l’Université de Bourgogne (ICMUB) Laboratory, Centre National de la Recherche Scientifique (CNRS) 6302, University of Burgundy, Dijon, France
| | - Alain Lalande
- Institut de Chimie Moléculaire de l’Université de Bourgogne (ICMUB) Laboratory, Centre National de la Recherche Scientifique (CNRS) 6302, University of Burgundy, Dijon, France
- Medical Imaging Department, University Hospital of Dijon, Dijon, France
| | - Cédric Chevalier
- Department of Radiotherapy, Centre Georges-François Leclerc, Dijon, France
| | - Jérémy Baude
- Department of Radiotherapy, Centre Georges-François Leclerc, Dijon, France
| | - Léone Aubignac
- Department of Medical Physics, Centre Georges-François Leclerc, Dijon, France
| | - Julien Boudet
- Department of Medical Physics, Centre Georges-François Leclerc, Dijon, France
| | - Igor Bessieres
- Department of Medical Physics, Centre Georges-François Leclerc, Dijon, France
| |
Collapse
|
36
|
Xie Y, Zhang J, Liu L, Wang H, Ye Y, Verjans J, Xia Y. ReFs: A hybrid pre-training paradigm for 3D medical image segmentation. Med Image Anal 2024; 91:103023. [PMID: 37956551 DOI: 10.1016/j.media.2023.103023] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/05/2022] [Revised: 10/31/2023] [Accepted: 11/02/2023] [Indexed: 11/15/2023]
Abstract
Self-supervised learning (SSL) has achieved remarkable progress in medical image segmentation. The application of an SSL algorithm often follows a two-stage training process: using unlabeled data to perform label-free representation learning and fine-tuning the pre-trained model on the downstream tasks. One issue of this paradigm is that the SSL step is unaware of the downstream task, which may lead to sub-optimal feature representation for a target task. In this paper, we propose a hybrid pre-training paradigm that is driven by both self-supervised and supervised objectives. To achieve this, a supervised reference task is involved in self-supervised learning, aiming to improve the representation quality. Specifically, we employ the off-the-shelf medical image segmentation task as reference, and encourage learning a representation that (1) incurs low prediction loss on both SSL and reference tasks and (2) leads to a similar gradient when updating the feature extractor from either task. In this way, the reference task pilots SSL in the direction beneficial for the downstream segmentation. To this end, we propose a simple but effective gradient matching method to optimize the model towards a consistent direction, thus improving the compatibility of both SSL and supervised reference tasks. We call this hybrid pre-training paradigm reference-guided self-supervised learning (ReFs), and perform it on a large-scale unlabeled dataset and an additional reference dataset. The experimental results demonstrate its effectiveness on seven downstream medical image segmentation benchmarks.
Collapse
Affiliation(s)
| | - Jianpeng Zhang
- School of Computer Science and Engineering, Northwestern Polytechnical University, Xi'an 710072, China
| | | | - Hu Wang
- University of Adelaide, Australia
| | - Yiwen Ye
- School of Computer Science and Engineering, Northwestern Polytechnical University, Xi'an 710072, China
| | | | - Yong Xia
- School of Computer Science and Engineering, Northwestern Polytechnical University, Xi'an 710072, China.
| |
Collapse
|
37
|
Ruthven M, Peplinski AM, Adams DM, King AP, Miquel ME. Real-time speech MRI datasets with corresponding articulator ground-truth segmentations. Sci Data 2023; 10:860. [PMID: 38042857 PMCID: PMC10693552 DOI: 10.1038/s41597-023-02766-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/28/2023] [Accepted: 11/20/2023] [Indexed: 12/04/2023] Open
Abstract
The use of real-time magnetic resonance imaging (rt-MRI) of speech is increasing in clinical practice and speech science research. Analysis of such images often requires segmentation of articulators and the vocal tract, and the community is turning to deep-learning-based methods to perform this segmentation. While there are publicly available rt-MRI datasets of speech, these do not include ground-truth (GT) segmentations, a key requirement for the development of deep-learning-based segmentation methods. To begin to address this barrier, this work presents rt-MRI speech datasets of five healthy adult volunteers with corresponding GT segmentations and velopharyngeal closure patterns. The images were acquired using standard clinical MRI scanners, coils and sequences to facilitate acquisition of similar images in other centres. The datasets include manually created GT segmentations of six anatomical features including the tongue, soft palate and vocal tract. In addition, this work makes code and instructions to implement a current state-of-the-art deep-learning-based method to segment rt-MRI speech datasets publicly available, thus providing the community and others with a starting point for developing such methods.
Collapse
Affiliation(s)
- Matthieu Ruthven
- Clinical Physics, Barts Health NHS Trust, West Smithfield, London, EC1A 7BE, UK
- School of Biomedical Engineering & Imaging Sciences, King's College London, King's Health Partners, St Thomas' Hospital, London, SE1 7EH, UK
| | | | - David M Adams
- Clinical Physics, Barts Health NHS Trust, West Smithfield, London, EC1A 7BE, UK
| | - Andrew P King
- School of Biomedical Engineering & Imaging Sciences, King's College London, King's Health Partners, St Thomas' Hospital, London, SE1 7EH, UK
| | - Marc Eric Miquel
- Clinical Physics, Barts Health NHS Trust, West Smithfield, London, EC1A 7BE, UK.
- Digital Environment Research Institute (DERI), Empire House, 67-75 New Road, Queen Mary University of London, London, E1 1HH, UK.
- Advanced Cardiovascular Imaging, Barts NIHR BRC, Queen Mary University of London, London, EC1M 6BQ, UK.
| |
Collapse
|
38
|
Yu X, Yang Q, Zhou Y, Cai LY, Gao R, Lee HH, Li T, Bao S, Xu Z, Lasko TA, Abramson RG, Zhang Z, Huo Y, Landman BA, Tang Y. UNesT: Local spatial representation learning with hierarchical transformer for efficient medical segmentation. Med Image Anal 2023; 90:102939. [PMID: 37725868 PMCID: PMC11229077 DOI: 10.1016/j.media.2023.102939] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2022] [Revised: 07/14/2023] [Accepted: 08/16/2023] [Indexed: 09/21/2023]
Abstract
Transformer-based models, capable of learning better global dependencies, have recently demonstrated exceptional representation learning capabilities in computer vision and medical image analysis. Transformer reformats the image into separate patches and realizes global communication via the self-attention mechanism. However, positional information between patches is hard to preserve in such 1D sequences, and loss of it can lead to sub-optimal performance when dealing with large amounts of heterogeneous tissues of various sizes in 3D medical image segmentation. Additionally, current methods are not robust and efficient for heavy-duty medical segmentation tasks such as predicting a large number of tissue classes or modeling globally inter-connected tissue structures. To address such challenges and inspired by the nested hierarchical structures in vision transformer, we proposed a novel 3D medical image segmentation method (UNesT), employing a simplified and faster-converging transformer encoder design that achieves local communication among spatially adjacent patch sequences by aggregating them hierarchically. We extensively validate our method on multiple challenging datasets, consisting of multiple modalities, anatomies, and a wide range of tissue classes, including 133 structures in the brain, 14 organs in the abdomen, 4 hierarchical components in the kidneys, inter-connected kidney tumors and brain tumors. We show that UNesT consistently achieves state-of-the-art performance and evaluate its generalizability and data efficiency. Particularly, the model achieves whole brain segmentation task complete ROI with 133 tissue classes in a single network, outperforming prior state-of-the-art method SLANT27 ensembled with 27 networks. Our model performance increases the mean DSC score of the publicly available Colin and CANDI dataset from 0.7264 to 0.7444 and from 0.6968 to 0.7025, respectively. Code, pre-trained models, and use case pipeline are available at: https://github.com/MASILab/UNesT.
Collapse
Affiliation(s)
- Xin Yu
- Department of Computer Science, Vanderbilt University, Nashville TN, 37212, USA
| | - Qi Yang
- Department of Computer Science, Vanderbilt University, Nashville TN, 37212, USA
| | - Yinchi Zhou
- Department of Computer Science, Vanderbilt University, Nashville TN, 37212, USA
| | - Leon Y Cai
- Department of Biomedical Engineering, Vanderbilt University, Nashville, TN, 37212, USA
| | - Riqiang Gao
- Department of Computer Science, Vanderbilt University, Nashville TN, 37212, USA; Digital Technology and Innovation, Siemens Healthineers, Princeton, NJ, 08540, USA
| | - Ho Hin Lee
- Department of Computer Science, Vanderbilt University, Nashville TN, 37212, USA
| | - Thomas Li
- Department of Biomedical Engineering, Vanderbilt University, Nashville, TN, 37212, USA
| | - Shunxing Bao
- Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, 37212, USA
| | - Zhoubing Xu
- Digital Technology and Innovation, Siemens Healthineers, Princeton, NJ, 08540, USA
| | - Thomas A Lasko
- Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, 37212, USA
| | - Richard G Abramson
- Department of Biomedical Engineering, Vanderbilt University, Nashville, TN, 37212, USA; Annalise-AI, Pty, Ltd, USA
| | | | - Yuankai Huo
- Department of Computer Science, Vanderbilt University, Nashville TN, 37212, USA; Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, 37212, USA
| | - Bennett A Landman
- Department of Computer Science, Vanderbilt University, Nashville TN, 37212, USA; Department of Biomedical Engineering, Vanderbilt University, Nashville, TN, 37212, USA; Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, 37212, USA; Department of Biomedical Informatics, Vanderbilt University Medical Center, Nashville, TN, 37212, USA
| | - Yucheng Tang
- Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN, 37212, USA; Nvidia Corporation, USA.
| |
Collapse
|
39
|
La Barbera G, Rouet L, Boussaid H, Lubet A, Kassir R, Sarnacki S, Gori P, Bloch I. Tubular structures segmentation of pediatric abdominal-visceral ceCT images with renal tumors: Assessment, comparison and improvement. Med Image Anal 2023; 90:102986. [PMID: 37820418 DOI: 10.1016/j.media.2023.102986] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2023] [Revised: 08/23/2023] [Accepted: 09/28/2023] [Indexed: 10/13/2023]
Abstract
Renal tubular structures, such as ureters, arteries and veins, are very important for building a complete digital 3D anatomical model of a patient. However, they can be challenging to segment from ceCT images due to their elongated shape, diameter variation and intra- and inter-patient contrast heterogeneity. This task is even more difficult in pediatric and pathological subjects, due to high inter-subject anatomical variations, potential presence of tumors, small volume of these structures compared to the surrounding, and small available labeled datasets. Given the limited literature on methods dedicated to children, and in order to find inspirational approaches, a complete assessment of state-of-the-art methods for the segmentation of renal tubular structures on ceCT images on adults is presented. Then, these methods are tested and compared on a private pediatric and pathological dataset of 79 abdominal-visceral ceCT images with arteriovenous phase acquisitions. To the best of our knowledge, both assessment and comparison in this specific case are novel. Eventually, we also propose a new loss function which leverages for the first time the use of vesselness functions on the predicted segmentation. We show that the combination of this loss function with state-of-the-art methods improves the topological coherence of the segmented tubular structures.2.
Collapse
Affiliation(s)
- Giammarco La Barbera
- LTCI, Télécom Paris, Institut Polytechnique de Paris, France; IMAG2, Institut Imagine, Université Paris Cité, France.
| | | | - Haithem Boussaid
- Philips Research Paris, Suresnes, France; Technology Innovation Institute, Abu Dhabi, United Arab Emirates
| | - Alexis Lubet
- IMAG2, Institut Imagine, Université Paris Cité, France
| | - Rani Kassir
- IMAG2, Institut Imagine, Université Paris Cité, France; Université Paris Cité, Department of Pediatric Surgery, Hôpital Necker Enfants-Malades, APHP, France
| | - Sabine Sarnacki
- IMAG2, Institut Imagine, Université Paris Cité, France; Université Paris Cité, Department of Pediatric Surgery, Hôpital Necker Enfants-Malades, APHP, France
| | - Pietro Gori
- LTCI, Télécom Paris, Institut Polytechnique de Paris, France
| | - Isabelle Bloch
- LTCI, Télécom Paris, Institut Polytechnique de Paris, France; IMAG2, Institut Imagine, Université Paris Cité, France; Sorbonne Université, CNRS, LIP6, Paris, France
| |
Collapse
|
40
|
Liao W, Luo X, He Y, Dong Y, Li C, Li K, Zhang S, Zhang S, Wang G, Xiao J. Comprehensive Evaluation of a Deep Learning Model for Automatic Organs-at-Risk Segmentation on Heterogeneous Computed Tomography Images for Abdominal Radiation Therapy. Int J Radiat Oncol Biol Phys 2023; 117:994-1006. [PMID: 37244625 DOI: 10.1016/j.ijrobp.2023.05.034] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2022] [Revised: 03/13/2023] [Accepted: 05/18/2023] [Indexed: 05/29/2023]
Abstract
PURPOSE Our purpose was to develop a deep learning model (AbsegNet) that produces accurate contours of 16 organs at risk (OARs) for abdominal malignancies as an essential part of fully automated radiation treatment planning. METHODS AND MATERIALS Three data sets with 544 computed tomography scans were retrospectively collected. Data set 1 was split into 300 training cases and 128 test cases (cohort 1) for AbsegNet. Data set 2, including cohort 2 (n = 24) and cohort 3 (n = 20), were used to validate AbsegNet externally. Data set 3, including cohort 4 (n = 40) and cohort 5 (n = 32), were used to clinically assess the accuracy of AbsegNet-generated contours. Each cohort was from a different center. The Dice similarity coefficient and 95th-percentile Hausdorff distance were calculated to evaluate the delineation quality for each OAR. Clinical accuracy evaluation was classified into 4 levels: no revision, minor revisions (0% < volumetric revision degrees [VRD] ≤ 10%), moderate revisions (10% ≤ VRD < 20%), and major revisions (VRD ≥20%). RESULTS For all OARs, AbsegNet achieved a mean Dice similarity coefficient of 86.73%, 85.65%, and 88.04% in cohorts 1, 2, and 3, respectively, and a mean 95th-percentile Hausdorff distance of 8.92, 10.18, and 12.40 mm, respectively. The performance of AbsegNet outperformed SwinUNETR, DeepLabV3+, Attention-UNet, UNet, and 3D-UNet. When experts evaluated contours from cohorts 4 and 5, 4 OARs (liver, kidney_L, kidney_R, and spleen) of all patients were scored as having no revision, and over 87.5% of patients with contours of the stomach, esophagus, adrenals, or rectum were considered as having no or minor revisions. Only 15.0% of patients with colon and small bowel contours required major revisions. CONCLUSIONS We propose a novel deep-learning model to delineate OARs on diverse data sets. Most contours produced by AbsegNet are accurate and robust and are, therefore, clinically applicable and helpful to facilitate radiation therapy workflow.
Collapse
Affiliation(s)
- Wenjun Liao
- Department of Radiation Oncology, Radiation Oncology Key Laboratory of Sichuan Province, Sichuan Clinical Research Center for Cancer, Sichuan Cancer Hospital & Institute, Sichuan Cancer Center, Affiliated Cancer Hospital of University of Electronic Science and Technology of China, Chengdu, China
| | - Xiangde Luo
- School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Chengdu, China; Shanghai AI Laboratory, Shanghai, China
| | - Yuan He
- Department of Radiation Oncology, The First Affiliated Hospital of USTC, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui, China
| | - Ye Dong
- Department of NanFang PET Center, Nanfang Hospital, Southern Medical University, Guangzhou, China
| | - Churong Li
- Department of Radiation Oncology, Radiation Oncology Key Laboratory of Sichuan Province, Sichuan Clinical Research Center for Cancer, Sichuan Cancer Hospital & Institute, Sichuan Cancer Center, Affiliated Cancer Hospital of University of Electronic Science and Technology of China, Chengdu, China
| | - Kang Li
- West China Biomedical Big Data Center
| | - Shichuan Zhang
- Department of Radiation Oncology, Radiation Oncology Key Laboratory of Sichuan Province, Sichuan Clinical Research Center for Cancer, Sichuan Cancer Hospital & Institute, Sichuan Cancer Center, Affiliated Cancer Hospital of University of Electronic Science and Technology of China, Chengdu, China
| | - Shaoting Zhang
- School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Chengdu, China; Shanghai AI Laboratory, Shanghai, China
| | - Guotai Wang
- School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Chengdu, China; Shanghai AI Laboratory, Shanghai, China
| | - Jianghong Xiao
- Radiotherapy Physics & Technology Center, Department of Radiation Oncology, Cancer Center, West China Hospital, Sichuan University, Chengdu, China.
| |
Collapse
|
41
|
Guo J, Goyal M, Xi Y, Hinojosa L, Haddad G, Albayrak E, Pedrosa I. Style Transfer-assisted Deep Learning Method for Kidney Segmentation at Multiphase MRI. Radiol Artif Intell 2023; 5:e230043. [PMID: 38074795 PMCID: PMC10698598 DOI: 10.1148/ryai.230043] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/12/2023] [Revised: 07/28/2023] [Accepted: 08/30/2023] [Indexed: 02/12/2024]
Abstract
Purpose To develop and validate a semisupervised style transfer-assisted deep learning method for automated segmentation of the kidneys using multiphase contrast-enhanced (MCE) MRI acquisitions. Materials and Methods This retrospective, Health Insurance Portability and Accountability Act-compliant, institutional review board-approved study included 125 patients (mean age, 57.3 years; 67 male, 58 female) with renal masses. Cohort 1 consisted of 102 coronal T2-weighted MRI acquisitions and 27 MCE MRI acquisitions during the corticomedullary phase. Cohort 2 comprised 92 MCE MRI acquisitions (23 acquisitions during four phases each, including precontrast, corticomedullary, early nephrographic, and nephrographic phases). The kidneys were manually segmented on T2-weighted images. A cycle-consistent generative adversarial network (CycleGAN) was trained to generate anatomically coregistered synthetic corticomedullary style images using T2-weighted images as input. Synthetic images for precontrast, early nephrographic, and nephrographic phases were then generated using the synthetic corticomedullary images as input. Mask region-based convolutional neural networks were trained on the four synthetic phase series for kidney segmentation using T2-weighted masks. Segmentation performance was evaluated in a different cohort of 20 originally acquired MCE MRI examinations by using Dice and Jaccard scores. Results The CycleGAN network successfully generated anatomically coregistered synthetic MCE MRI-like datasets from T2-weighted acquisitions. The proposed deep learning approach for kidney segmentation achieved high mean Dice scores in all four phases of the original MCE MRI acquisitions (0.91 for precontrast, 0.92 for corticomedullary, 0.91 for early nephrographic, and 0.93 for nephrographic). Conclusion The proposed deep learning approach achieved high performance in kidney segmentation on different MCE MRI acquisitions.Keywords: Kidney Segmentation, Generative Adversarial Network, CycleGAN, Convolutional Neural Network, Transfer Learning Supplemental material is available for this article. Published under a CC BY 4.0 license.
Collapse
Affiliation(s)
| | | | - Yin Xi
- From the Department of Radiology (J.G., M.G., Y.X., L.H., G.H., E.A.,
I.P.), Department of Urology (I.P.), and Advanced Imaging Research Center
(I.P.), University of Texas Southwestern Medical Center, 2201 Inwood Rd, Suite
202, Dallas, TX 75390-9085
| | - Lauren Hinojosa
- From the Department of Radiology (J.G., M.G., Y.X., L.H., G.H., E.A.,
I.P.), Department of Urology (I.P.), and Advanced Imaging Research Center
(I.P.), University of Texas Southwestern Medical Center, 2201 Inwood Rd, Suite
202, Dallas, TX 75390-9085
| | - Gaelle Haddad
- From the Department of Radiology (J.G., M.G., Y.X., L.H., G.H., E.A.,
I.P.), Department of Urology (I.P.), and Advanced Imaging Research Center
(I.P.), University of Texas Southwestern Medical Center, 2201 Inwood Rd, Suite
202, Dallas, TX 75390-9085
| | - Emin Albayrak
- From the Department of Radiology (J.G., M.G., Y.X., L.H., G.H., E.A.,
I.P.), Department of Urology (I.P.), and Advanced Imaging Research Center
(I.P.), University of Texas Southwestern Medical Center, 2201 Inwood Rd, Suite
202, Dallas, TX 75390-9085
| | - Ivan Pedrosa
- From the Department of Radiology (J.G., M.G., Y.X., L.H., G.H., E.A.,
I.P.), Department of Urology (I.P.), and Advanced Imaging Research Center
(I.P.), University of Texas Southwestern Medical Center, 2201 Inwood Rd, Suite
202, Dallas, TX 75390-9085
| |
Collapse
|
42
|
Dong J, Cheng G, Zhang Y, Peng C, Song Y, Tong R, Lin L, Chen YW. Tailored multi-organ segmentation with model adaptation and ensemble. Comput Biol Med 2023; 166:107467. [PMID: 37725849 DOI: 10.1016/j.compbiomed.2023.107467] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2023] [Revised: 08/10/2023] [Accepted: 09/04/2023] [Indexed: 09/21/2023]
Abstract
Multi-organ segmentation, which identifies and separates different organs in medical images, is a fundamental task in medical image analysis. Recently, the immense success of deep learning motivated its wide adoption in multi-organ segmentation tasks. However, due to expensive labor costs and expertise, the availability of multi-organ annotations is usually limited and hence poses a challenge in obtaining sufficient training data for deep learning-based methods. In this paper, we aim to address this issue by combining off-the-shelf single-organ segmentation models to develop a multi-organ segmentation model on the target dataset, which helps get rid of the dependence on annotated data for multi-organ segmentation. To this end, we propose a novel dual-stage method that consists of a Model Adaptation stage and a Model Ensemble stage. The first stage enhances the generalization of each off-the-shelf segmentation model on the target domain, while the second stage distills and integrates knowledge from multiple adapted single-organ segmentation models. Extensive experiments on four abdomen datasets demonstrate that our proposed method can effectively leverage off-the-shelf single-organ segmentation models to obtain a tailored model for multi-organ segmentation with high accuracy.
Collapse
Affiliation(s)
- Jiahua Dong
- College of Computer Science and Technology, Zhejiang University, Hangzhou, 310027, China
| | - Guohua Cheng
- College of Computer Science and Technology, Zhejiang University, Hangzhou, 310027, China
| | - Yue Zhang
- Center for Medical Imaging, Robotics, Analytic Computing & Learning (MIRACLE), Suzhou Institute for Advanced Research, University of Science and Technology of China, Suzhou, 215163, China; School of Biomedical Engineering, Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, Anhui, 230026, China.
| | - Chengtao Peng
- Department of Electronic Engineering and Information Science, University of Science and Technology of China, Hefei, 230026, China
| | - Yu Song
- Graduate School of Information Science and Engineering, Ritsumeikan University, Shiga, 525-8577, Japan
| | - Ruofeng Tong
- College of Computer Science and Technology, Zhejiang University, Hangzhou, 310027, China
| | - Lanfen Lin
- College of Computer Science and Technology, Zhejiang University, Hangzhou, 310027, China
| | - Yen-Wei Chen
- Graduate School of Information Science and Engineering, Ritsumeikan University, Shiga, 525-8577, Japan
| |
Collapse
|
43
|
Zhou Z, Qian X, Hu J, Chen G, Zhang C, Zhu J, Dai Y. An artificial intelligence-assisted diagnosis modeling software (AIMS) platform based on medical images and machine learning: a development and validation study. Quant Imaging Med Surg 2023; 13:7504-7522. [PMID: 37969634 PMCID: PMC10644131 DOI: 10.21037/qims-23-20] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/15/2023] [Accepted: 06/12/2023] [Indexed: 11/17/2023]
Abstract
Background Supervised machine learning methods [both radiomics and convolutional neural network (CNN)-based deep learning] are usually employed to develop artificial intelligence models with medical images for computer-assisted diagnosis and prognosis of diseases. A classical machine learning-based modeling workflow involves a series of interconnected components and various algorithms, but this makes it challenging, tedious, and labor intensive for radiologists and researchers to build customized models for specific clinical applications if they lack expertise in machine learning methods. Methods We developed a user-friendly artificial intelligence-assisted diagnosis modeling software (AIMS) platform, which supplies standardized machine learning-based modeling workflows for computer-assisted diagnosis and prognosis systems with medical images. In contrast to other existing software platforms, AIMS contains both radiomics and CNN-based deep learning workflows, making it an all-in-one software platform for machine learning-based medical image analysis. The modular design of AIMS allows users to build machine learning models easily, test models comprehensively, and fairly compare the performance of different models in a specific application. The graphical user interface (GUI) enables users to process large numbers of medical images without programming or script addition. Furthermore, AIMS also provides a flexible image processing toolkit (e.g., semiautomatic segmentation, registration, morphological operations) to rapidly create lesion labels for multiphase analysis, multiregion analysis of an individual tumor (e.g., tumor mass and peritumor), and multimodality analysis. Results The functionality and efficiency of AIMS were demonstrated in 3 independent experiments in radiation oncology, where multiphase, multiregion, and multimodality analyses were performed, respectively. For clear cell renal cell carcinoma (ccRCC) Fuhrman grading with multiphase analysis (sample size =187), the area under the curve (AUC) value of the AIMS was 0.776; for ccRCC Fuhrman grading with multiregion analysis (sample size =177), the AUC value of the AIMS was 0.848; for prostate cancer Gleason grading with multimodality analysis (sample size =206), the AUC value of the AIMS was 0.980. Conclusions AIMS provides a user-friendly infrastructure for radiologists and researchers, lowering the barrier to building customized machine learning-based computer-assisted diagnosis models for medical image analysis.
Collapse
Affiliation(s)
- Zhiyong Zhou
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, China
| | - Xusheng Qian
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, China
- School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, China
| | - Jisu Hu
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, China
- School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, China
| | - Guangqiang Chen
- Department of Radiology, The Second Affiliated Hospital of Soochow University, Suzhou, China
| | - Caiyuan Zhang
- Department of Radiology, The Second Affiliated Hospital of Soochow University, Suzhou, China
| | - Jianbing Zhu
- Suzhou Science & Technology Town Hospital, Suzhou Hospital, Affiliated Hospital of Medical School, Nanjing University, Suzhou, China
| | - Yakang Dai
- Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, China
- Suzhou Guoke Kangcheng Medical Technology Co., Ltd., Suzhou, China
| |
Collapse
|
44
|
Rao PK, Chatterjee S, Janardhan M, Nagaraju K, Khan SB, Almusharraf A, Alharbe AI. Optimizing Inference Distribution for Efficient Kidney Tumor Segmentation Using a UNet-PWP Deep-Learning Model with XAI on CT Scan Images. Diagnostics (Basel) 2023; 13:3244. [PMID: 37892065 PMCID: PMC10606269 DOI: 10.3390/diagnostics13203244] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/14/2023] [Revised: 10/10/2023] [Accepted: 10/10/2023] [Indexed: 10/29/2023] Open
Abstract
Kidney tumors represent a significant medical challenge, characterized by their often-asymptomatic nature and the need for early detection to facilitate timely and effective intervention. Although neural networks have shown great promise in disease prediction, their computational demands have limited their practicality in clinical settings. This study introduces a novel methodology, the UNet-PWP architecture, tailored explicitly for kidney tumor segmentation, designed to optimize resource utilization and overcome computational complexity constraints. A key novelty in our approach is the application of adaptive partitioning, which deconstructs the intricate UNet architecture into smaller submodels. This partitioning strategy reduces computational requirements and enhances the model's efficiency in processing kidney tumor images. Additionally, we augment the UNet's depth by incorporating pre-trained weights, therefore significantly boosting its capacity to handle intricate and detailed segmentation tasks. Furthermore, we employ weight-pruning techniques to eliminate redundant zero-weighted parameters, further streamlining the UNet-PWP model without compromising its performance. To rigorously assess the effectiveness of our proposed UNet-PWP model, we conducted a comparative evaluation alongside the DeepLab V3+ model, both trained on the "KiTs 19, 21, and 23" kidney tumor dataset. Our results are optimistic, with the UNet-PWP model achieving an exceptional accuracy rate of 97.01% on both the training and test datasets, surpassing the DeepLab V3+ model in performance. Furthermore, to ensure our model's results are easily understandable and explainable. We included a fusion of the attention and Grad-CAM XAI methods. This approach provides valuable insights into the decision-making process of our model and the regions of interest that affect its predictions. In the medical field, this interpretability aspect is crucial for healthcare professionals to trust and comprehend the model's reasoning.
Collapse
Affiliation(s)
- P. Kiran Rao
- Artificial Intelligence, Department of Computer Science and Engineering, Ravindra College of Engineering for Women, Kurnool 518001, India
- Department of Computer Science and Engineering, Faculty of Engineering, MS Ramaiah University of Applied Sciences, Bengaluru 560058, India;
| | - Subarna Chatterjee
- Department of Computer Science and Engineering, Faculty of Engineering, MS Ramaiah University of Applied Sciences, Bengaluru 560058, India;
| | - M. Janardhan
- Artificial Intelligence, Department of Computer Science and Engineering, G. Pullaiah College of Engineering and Technology, Kurnool 518008, India;
| | - K. Nagaraju
- Department of Computer Science and Engineering, Indian Institute of Information Technology Design and Manufacturing Kurnool, Kurnool 518008, India;
| | - Surbhi Bhatia Khan
- Department of Data Science, School of Science, Engineering and Environment, University of Salford, Salford M5 4WT, UK
- Department of Electrical and Computer Engineering, Lebanese American University, Byblos 13-5053, Lebanon
| | - Ahlam Almusharraf
- Department of Business Administration, College of Business and Administration, Princess Nourah bint Abdulrahman University, P.O. Box 84428, Riyadh 11671, Saudi Arabia;
| | - Abdullah I. Alharbe
- Department of Computer Science, Faculty of Computing and Information Technology, King Abdulaziz University, Rabigh 21911, Saudi Arabia
| |
Collapse
|
45
|
Weng X, Song F, Tang M, Wang K, Zhang Y, Miao Y, Chan LWC, Lei P, Hu Z, Yang F. MDM-U-Net: A novel network for renal cancer structure segmentation. Comput Med Imaging Graph 2023; 109:102301. [PMID: 37738774 DOI: 10.1016/j.compmedimag.2023.102301] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/07/2023] [Revised: 07/27/2023] [Accepted: 09/08/2023] [Indexed: 09/24/2023]
Abstract
Accurate segmentation of the renal cancer structure, including the kidney, renal tumors, veins, and arteries, has great clinical significance, which can assist clinicians in diagnosing and treating renal cancer. For accurate segmentation of the renal cancer structure in contrast-enhanced computed tomography (CT) images, we proposed a novel encoder-decoder structure segmentation network named MDM-U-Net comprising a multi-scale anisotropic convolution block, dual activation attention block, and multi-scale deep supervision mechanism. The multi-scale anisotropic convolution block was used to improve the feature extraction ability of the network, the dual activation attention block as a channel-wise mechanism was used to guide the network to exploit important information, and the multi-scale deep supervision mechanism was used to supervise the layers of the decoder part for improving segmentation performance. In this study, we developed a feasible and generalizable MDM-U-Net model for renal cancer structure segmentation, trained the model from the public KiPA22 dataset, and tested it on the KiPA22 dataset and an in-house dataset. For the KiPA22 dataset, our method ranked first in renal cancer structure segmentation, achieving state-of-the-art (SOTA) performance in terms of 6 of 12 evaluation metrics (3 metrics per structure). For the in-house dataset, our method achieves SOTA performance in terms of 9 of 12 evaluation metrics (3 metrics per structure), demonstrating its superiority and generalization ability over the compared networks in renal structure segmentation from contrast-enhanced CT scans.
Collapse
Affiliation(s)
- Xin Weng
- School of Biology & Engineering (School of Modern Industry for Health and Medicine), Guizhou Medical University, Guiyang, Guizhou, China
| | - Fasong Song
- Department of Radiology, The Affiliated Hospital of Guizhou Medical University, Guiyang, Guizhou, China
| | - Maowen Tang
- Department of Radiology, The Affiliated Hospital of Guizhou Medical University, Guiyang, Guizhou, China
| | - Kansui Wang
- Department of Radiology, The First Affiliated Hospital of Guizhou University of Traditional Chinese Medicine, Guiyang, Guizhou, China
| | - Yusui Zhang
- Department of Radiology, The First Affiliated Hospital of Guizhou University of Traditional Chinese Medicine, Guiyang, Guizhou, China
| | - Yuehong Miao
- School of Biology & Engineering (School of Modern Industry for Health and Medicine), Guizhou Medical University, Guiyang, Guizhou, China
| | - Lawrence Wing-Chi Chan
- Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Hong Kong SAR, China
| | - Pinggui Lei
- Department of Radiology, The Affiliated Hospital of Guizhou Medical University, Guiyang, Guizhou, China; Department of Health Technology and Informatics, The Hong Kong Polytechnic University, Hong Kong SAR, China.
| | - Zuquan Hu
- School of Biology & Engineering (School of Modern Industry for Health and Medicine), Guizhou Medical University, Guiyang, Guizhou, China; Immune Cells and Antibody Engineering Research Center in University of Guizhou Province, Key Laboratory of Biology and Medical Engineering, Guizhou Medical University, Guiyang, Guizhou, China.
| | - Fan Yang
- School of Biology & Engineering (School of Modern Industry for Health and Medicine), Guizhou Medical University, Guiyang, Guizhou, China.
| |
Collapse
|
46
|
Abdallah N, Wood A, Benidir T, Heller N, Isensee F, Tejpaul R, Corrigan D, Suk-Ouichai C, Struyk G, Moore K, Venkatesh N, Ergun O, You A, Campbell R, Remer EM, Haywood S, Krishnamurthi V, Abouassaly R, Campbell S, Papanikolopoulos N, Weight CJ. AI-generated R.E.N.A.L.+ Score Surpasses Human-generated Score in Predicting Renal Oncologic Outcomes. Urology 2023; 180:160-167. [PMID: 37517681 PMCID: PMC10592249 DOI: 10.1016/j.urology.2023.07.017] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2023] [Revised: 07/03/2023] [Accepted: 07/17/2023] [Indexed: 08/01/2023]
Abstract
OBJECTIVE To determine whether we can surpass the traditional R.E.N.A.L. nephrometry score (H-score) prediction ability of pathologic outcomes by creating artificial intelligence (AI)-generated R.E.N.A.L.+ score (AI+ score) with continuous rather than ordinal components. We also assessed the AI+ score components' relative importance with respect to outcome odds. METHODS This is a retrospective study of 300 consecutive patients with preoperative computed tomography scans showing suspected renal cancer at a single institution from 2010 to 2018. H-score was tabulated by three trained medical personnel. Deep neural network approach automatically generated kidney segmentation masks of parenchyma and tumor. Geometric algorithms were used to automatically estimate score components as ordinal and continuous variables. Multivariate logistic regression of continuous R.E.N.A.L. components was used to generate AI+ score. Predictive utility was compared between AI+, AI, and H-scores for variables of interest, and AI+ score components' relative importance was assessed. RESULTS Median age was 60years (interquartile range 51-68), and 40% were female. Median tumor size was 4.2 cm (2.6-6.12), and 92% were malignant, including 27%, 37%, and 23% with high-stage, high-grade, and necrosis, respectively. AI+ score demonstrated superior predictive ability over AI and H-scores for predicting malignancy (area under the curve [AUC] 0.69 vs 0.67 vs 0.64, respectively), high stage (AUC 0.82 vs 0.65 vs 0.71, respectively), high grade (AUC 0.78 vs 0.65 vs 0.65, respectively), pathologic tumor necrosis (AUC 0.81 vs 0.72 vs 0.74, respectively), and partial nephrectomy approach (AUC 0.88 vs 0.74 vs 0.79, respectively). Of AI+ score components, the maximal tumor diameter ("R") was the most important outcomes predictor. CONCLUSION AI+ score was superior to AI-score and H-score in predicting oncologic outcomes. Time-efficient AI+ score can be used at the point of care, surpassing validated clinical scoring systems.
Collapse
Affiliation(s)
- Nour Abdallah
- Glickman Urological and Kidney Institute, Cleveland, OH.
| | - Andrew Wood
- Glickman Urological and Kidney Institute, Cleveland, OH
| | - Tarik Benidir
- Glickman Urological and Kidney Institute, Cleveland, OH
| | - Nicholas Heller
- Department of Computer Science and Engineering, University of Minnesota, Minneapolis, MN
| | - Fabian Isensee
- German Cancer Research Center (DKFZ) Heidelberg, University of Heidelberg, Heidelberg, Germany
| | - Resha Tejpaul
- Department of Computer Science and Engineering, University of Minnesota, Minneapolis, MN
| | - Dillon Corrigan
- Department of Quantitative Health Sciences, Lerner Research Institute, Cleveland, OH
| | | | - Griffin Struyk
- Department of Computer Science and Engineering, University of Minnesota, Minneapolis, MN
| | - Keenan Moore
- Department of Computer Science and Engineering, University of Minnesota, Minneapolis, MN
| | - Nitin Venkatesh
- Department of Computer Science and Engineering, University of Minnesota, Minneapolis, MN
| | | | - Alex You
- Case Western Reserve University, Cleveland, OH
| | | | - Erick M Remer
- Glickman Urological and Kidney Institute, Cleveland, OH; Department of Diagnostic Radiology, Imaging Institute, Cleveland Clinic, Cleveland, OH
| | | | | | | | | | | | | |
Collapse
|
47
|
Xu X, Deng HH, Gateno J, Yan P. Federated Multi-Organ Segmentation With Inconsistent Labels. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:2948-2960. [PMID: 37097793 PMCID: PMC10592562 DOI: 10.1109/tmi.2023.3270140] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/21/2023]
Abstract
Federated learning is an emerging paradigm allowing large-scale decentralized learning without sharing data across different data owners, which helps address the concern of data privacy in medical image analysis. However, the requirement for label consistency across clients by the existing methods largely narrows its application scope. In practice, each clinical site may only annotate certain organs of interest with partial or no overlap with other sites. Incorporating such partially labeled data into a unified federation is an unexplored problem with clinical significance and urgency. This work tackles the challenge by using a novel federated multi-encoding U-Net (Fed-MENU) method for multi-organ segmentation. In our method, a multi-encoding U-Net (MENU-Net) is proposed to extract organ-specific features through different encoding sub-networks. Each sub-network can be seen as an expert of a specific organ and trained for that client. Moreover, to encourage the organ-specific features extracted by different sub-networks to be informative and distinctive, we regularize the training of the MENU-Net by designing an auxiliary generic decoder (AGD). Extensive experiments on six public abdominal CT datasets show that our Fed-MENU method can effectively obtain a federated learning model using the partially labeled datasets with superior performance to other models trained by either localized or centralized learning methods. Source code is publicly available at https://github.com/DIAL-RPI/Fed-MENU.
Collapse
|
48
|
Gottlich HC, Korfiatis P, Gregory AV, Kline TL. AI in the Loop: functionalizing fold performance disagreement to monitor automated medical image segmentation workflows. FRONTIERS IN RADIOLOGY 2023; 3:1223294. [PMID: 37780641 PMCID: PMC10540615 DOI: 10.3389/fradi.2023.1223294] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 05/15/2023] [Accepted: 08/28/2023] [Indexed: 10/03/2023]
Abstract
Introduction Methods that automatically flag poor performing predictions are drastically needed to safely implement machine learning workflows into clinical practice as well as to identify difficult cases during model training. Methods Disagreement between the fivefold cross-validation sub-models was quantified using dice scores between folds and summarized as a surrogate for model confidence. The summarized Interfold Dices were compared with thresholds informed by human interobserver values to determine whether final ensemble model performance should be manually reviewed. Results The method on all tasks efficiently flagged poor segmented images without consulting a reference standard. Using the median Interfold Dice for comparison, substantial dice score improvements after excluding flagged images was noted for the in-domain CT (0.85 ± 0.20 to 0.91 ± 0.08, 8/50 images flagged) and MR (0.76 ± 0.27 to 0.85 ± 0.09, 8/50 images flagged). Most impressively, there were dramatic dice score improvements in the simulated out-of-distribution task where the model was trained on a radical nephrectomy dataset with different contrast phases predicting a partial nephrectomy all cortico-medullary phase dataset (0.67 ± 0.36 to 0.89 ± 0.10, 122/300 images flagged). Discussion Comparing interfold sub-model disagreement against human interobserver values is an effective and efficient way to assess automated predictions when a reference standard is not available. This functionality provides a necessary safeguard to patient care important to safely implement automated medical image segmentation workflows.
Collapse
Affiliation(s)
| | | | | | - Timothy L. Kline
- Department of Radiology, Mayo Clinic, Rochester, MN, United States
- Division of Nephrology and Hypertension, Mayo Clinic, Rochester, MN, United States
| |
Collapse
|
49
|
Chen S, Song D, Chen L, Guo T, Jiang B, Liu A, Pan X, Wang T, Tang H, Chen G, Xue Z, Wang X, Zhang N, Zheng J. Artificial intelligence-based non-invasive tumor segmentation, grade stratification and prognosis prediction for clear-cell renal-cell carcinoma. PRECISION CLINICAL MEDICINE 2023; 6:pbad019. [PMID: 38025974 PMCID: PMC10680020 DOI: 10.1093/pcmedi/pbad019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2023] [Accepted: 08/07/2023] [Indexed: 12/01/2023] Open
Abstract
Due to the complicated histopathological characteristics of clear-cell renal-cell carcinoma (ccRCC), non-invasive prognosis before operative treatment is crucial in selecting the appropriate treatment. A total of 126 345 computerized tomography (CT) images from four independent patient cohorts were included for analysis in this study. We propose a V Bottleneck multi-resolution and focus-organ network (VB-MrFo-Net) using a cascade framework for deep learning analysis. The VB-MrFo-Net achieved better performance than VB-Net in tumor segmentation, with a Dice score of 0.87. The nuclear-grade prediction model performed best in the logistic regression classifier, with area under curve values from 0.782 to 0.746. Survival analysis revealed that our prediction model could significantly distinguish patients with high survival risk, with a hazard ratio (HR) of 2.49 [95% confidence interval (CI): 1.13-5.45, P = 0.023] in the General cohort. Excellent performance had also been verified in the Cancer Genome Atlas cohort, the Clinical Proteomic Tumor Analysis Consortium cohort, and the Kidney Tumor Segmentation Challenge cohort, with HRs of 2.77 (95%CI: 1.58-4.84, P = 0.0019), 3.83 (95%CI: 1.22-11.96, P = 0.029), and 2.80 (95%CI: 1.05-7.47, P = 0.025), respectively. In conclusion, we propose a novel VB-MrFo-Net for the renal tumor segmentation and automatic diagnosis of ccRCC. The risk stratification model could accurately distinguish patients with high tumor grade and high survival risk based on non-invasive CT images before surgical treatments, which could provide practical advice for deciding treatment options.
Collapse
Affiliation(s)
- Siteng Chen
- Department of Urology, Renji Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200001, China
| | - Dandan Song
- Department of Radiology, Shanghai Ninth People's Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200011, China
| | - Lei Chen
- Shanghai United Imaging Intelligence Co., Ltd., Shanghai 201807, China
| | - Tuanjie Guo
- Department of Urology, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200080, China
| | - Beibei Jiang
- Department of Radiology, Erasmus University Medical Center, Rotterdam, P.O. Box 2040, 3000 CA, The Netherlands
| | - Aie Liu
- Shanghai United Imaging Intelligence Co., Ltd., Shanghai 201807, China
| | - Xianpan Pan
- Shanghai United Imaging Intelligence Co., Ltd., Shanghai 201807, China
| | - Tao Wang
- Department of Urology, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200080, China
| | - Heting Tang
- Department of Urology, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200080, China
| | - Guihua Chen
- Department of Urology, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200080, China
| | - Zhong Xue
- Shanghai United Imaging Intelligence Co., Ltd., Shanghai 201807, China
| | - Xiang Wang
- Department of Urology, Shanghai General Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200080, China
| | - Ning Zhang
- Department of Urology, Ruijin Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200025, China
| | - Junhua Zheng
- Department of Urology, Renji Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200001, China
| |
Collapse
|
50
|
Shin SY, Shen TC, Summers RM. Improving segmentation and detection of lesions in CT scans using intensity distribution supervision. Comput Med Imaging Graph 2023; 108:102259. [PMID: 37348281 PMCID: PMC10527342 DOI: 10.1016/j.compmedimag.2023.102259] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/14/2023] [Revised: 05/12/2023] [Accepted: 06/06/2023] [Indexed: 06/24/2023]
Abstract
We propose a method to incorporate the intensity information of a target lesion on CT scans in training segmentation and detection networks. We first build an intensity-based lesion probability (ILP) function from an intensity histogram of the target lesion. It is used to compute the probability of being the lesion for each voxel based on its intensity. Finally, the computed ILP map of each input CT scan is provided as additional supervision for network training, which aims to inform the network about possible lesion locations in terms of intensity values at no additional labeling cost. The method was applied to improve the segmentation of three different lesion types, namely, small bowel carcinoid tumor, kidney tumor, and lung nodule. The effectiveness of the proposed method on a detection task was also investigated. We observed improvements of 41.3% → 47.8%, 74.2% → 76.0%, and 26.4% → 32.7% in segmenting small bowel carcinoid tumor, kidney tumor, and lung nodule, respectively, in terms of per case Dice scores. An improvement of 64.6% → 75.5% was achieved in detecting kidney tumors in terms of average precision. The results of different usages of the ILP map and the effect of varied amount of training data are also presented.
Collapse
Affiliation(s)
- Seung Yeon Shin
- Imaging Biomarkers and Computer-Aided Diagnosis Laboratory, Radiology and Imaging Sciences, Clinical Center, National Institutes of Health, Bethesda, 20892, MD, USA.
| | - Thomas C Shen
- Imaging Biomarkers and Computer-Aided Diagnosis Laboratory, Radiology and Imaging Sciences, Clinical Center, National Institutes of Health, Bethesda, 20892, MD, USA
| | - Ronald M Summers
- Imaging Biomarkers and Computer-Aided Diagnosis Laboratory, Radiology and Imaging Sciences, Clinical Center, National Institutes of Health, Bethesda, 20892, MD, USA
| |
Collapse
|