1
|
Manthe M, Duffner S, Lartizien C. Federated brain tumor segmentation: An extensive benchmark. Med Image Anal 2024; 97:103270. [PMID: 39059241 DOI: 10.1016/j.media.2024.103270] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2023] [Revised: 06/14/2024] [Accepted: 07/08/2024] [Indexed: 07/28/2024]
Abstract
Recently, federated learning has raised increasing interest in the medical image analysis field due to its ability to aggregate multi-center data with privacy-preserving properties. A large amount of federated training schemes have been published, which we categorize into global (one final model), personalized (one model per institution) or hybrid (one model per cluster of institutions) methods. However, their applicability on the recently published Federated Brain Tumor Segmentation 2022 dataset has not been explored yet. We propose an extensive benchmark of federated learning algorithms from all three classes on this task. While standard FedAvg already performs very well, we show that some methods from each category can bring a slight performance improvement and potentially limit the final model(s) bias toward the predominant data distribution of the federation. Moreover, we provide a deeper understanding of the behavior of federated learning on this task through alternative ways of distributing the pooled dataset among institutions, namely an Independent and Identical Distributed (IID) setup, and a limited data setup. Our code is available at (https://github.com/MatthisManthe/Benchmark_FeTS2022).
Collapse
Affiliation(s)
- Matthis Manthe
- INSA Lyon, Universite Claude Bernard Lyon 1, CNRS, Inserm, CREATIS UMR 5220, U1294, F-69621 Lyon, France; INSA Lyon, CNRS, Universite Claude Bernard Lyon 1, Centrale Lyon, Université Lumière Lyon 2, LIRIS, UMR5205, F-69621 Villeurbanne, France.
| | - Stefan Duffner
- INSA Lyon, CNRS, Universite Claude Bernard Lyon 1, Centrale Lyon, Université Lumière Lyon 2, LIRIS, UMR5205, F-69621 Villeurbanne, France
| | - Carole Lartizien
- INSA Lyon, Universite Claude Bernard Lyon 1, CNRS, Inserm, CREATIS UMR 5220, U1294, F-69621 Lyon, France
| |
Collapse
|
2
|
de Boisredon d'Assier MA, Portafaix A, Vorontsov E, Le WT, Kadoury S. Image-level supervision and self-training for transformer-based cross-modality tumor segmentation. Med Image Anal 2024; 97:103287. [PMID: 39111265 DOI: 10.1016/j.media.2024.103287] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2023] [Revised: 06/20/2024] [Accepted: 07/24/2024] [Indexed: 08/30/2024]
Abstract
Deep neural networks are commonly used for automated medical image segmentation, but models will frequently struggle to generalize well across different imaging modalities. This issue is particularly problematic due to the limited availability of annotated data, both in the target as well as the source modality, making it difficult to deploy these models on a larger scale. To overcome these challenges, we propose a new semi-supervised training strategy called MoDATTS. Our approach is designed for accurate cross-modality 3D tumor segmentation on unpaired bi-modal datasets. An image-to-image translation strategy between modalities is used to produce synthetic but annotated images and labels in the desired modality and improve generalization to the unannotated target modality. We also use powerful vision transformer architectures for both image translation (TransUNet) and segmentation (Medformer) tasks and introduce an iterative self-training procedure in the later task to further close the domain gap between modalities, thus also training on unlabeled images in the target modality. MoDATTS additionally allows the possibility to exploit image-level labels with a semi-supervised objective that encourages the model to disentangle tumors from the background. This semi-supervised methodology helps in particular to maintain downstream segmentation performance when pixel-level label scarcity is also present in the source modality dataset, or when the source dataset contains healthy controls. The proposed model achieves superior performance compared to other methods from participating teams in the CrossMoDA 2022 vestibular schwannoma (VS) segmentation challenge, as evidenced by its reported top Dice score of 0.87±0.04 for the VS segmentation. MoDATTS also yields consistent improvements in Dice scores over baselines on a cross-modality adult brain gliomas segmentation task composed of four different contrasts from the BraTS 2020 challenge dataset, where 95% of a target supervised model performance is reached when no target modality annotations are available. We report that 99% and 100% of this maximum performance can be attained if 20% and 50% of the target data is additionally annotated, which further demonstrates that MoDATTS can be leveraged to reduce the annotation burden.
Collapse
Affiliation(s)
| | - Aloys Portafaix
- Polytechnique Montreal, Montreal, QC, Canada; Centre de Recherche du Centre Hospitalier de l'Université de Montréal, Montreal, QC, Canada
| | | | - William Trung Le
- Polytechnique Montreal, Montreal, QC, Canada; Centre de Recherche du Centre Hospitalier de l'Université de Montréal, Montreal, QC, Canada
| | - Samuel Kadoury
- Polytechnique Montreal, Montreal, QC, Canada; Centre de Recherche du Centre Hospitalier de l'Université de Montréal, Montreal, QC, Canada.
| |
Collapse
|
3
|
Han M, Luo X, Xie X, Liao W, Zhang S, Song T, Wang G, Zhang S. DMSPS: Dynamically mixed soft pseudo-label supervision for scribble-supervised medical image segmentation. Med Image Anal 2024; 97:103274. [PMID: 39043109 DOI: 10.1016/j.media.2024.103274] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2023] [Revised: 05/11/2024] [Accepted: 07/09/2024] [Indexed: 07/25/2024]
Abstract
High performance of deep learning on medical image segmentation rely on large-scale pixel-level dense annotations, which poses a substantial burden on medical experts due to the laborious and time-consuming annotation process, particularly for 3D images. To reduce the labeling cost as well as maintain relatively satisfactory segmentation performance, weakly-supervised learning with sparse labels has attained increasing attentions. In this work, we present a scribble-based framework for medical image segmentation, called Dynamically Mixed Soft Pseudo-label Supervision (DMSPS). Concretely, we extend a backbone with an auxiliary decoder to form a dual-branch network to enhance the feature capture capability of the shared encoder. Considering that most pixels do not have labels and hard pseudo-labels tend to be over-confident to result in poor segmentation, we propose to use soft pseudo-labels generated by dynamically mixing the decoders' predictions as auxiliary supervision. To further enhance the model's performance, we adopt a two-stage approach where the sparse scribbles are expanded based on predictions with low uncertainties from the first-stage model, leading to more annotated pixels to train the second-stage model. Experiments on ACDC dataset for cardiac structure segmentation, WORD dataset for 3D abdominal organ segmentation and BraTS2020 dataset for 3D brain tumor segmentation showed that: (1) compared with the baseline, our method improved the average DSC from 50.46% to 89.51%, from 75.46% to 87.56% and from 52.61% to 76.53% on the three datasets, respectively; (2) DMSPS achieved better performance than five state-of-the-art scribble-supervised segmentation methods, and is generalizable to different segmentation backbones. The code is available online at: https://github.com/HiLab-git/DMSPS.
Collapse
Affiliation(s)
- Meng Han
- School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Chengdu, China
| | - Xiangde Luo
- School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Chengdu, China; Shanghai Artificial Intelligence Laboratory, Shanghai, China
| | - Xiangjiang Xie
- School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Chengdu, China
| | - Wenjun Liao
- Department of Radiation Oncology, Sichuan Cancer Hospital and Institute, Sichuan Cancer Center, Chengdu, China; School of Medicine, University of Electronic Science and Technology of China, Chengdu, China
| | - Shichuan Zhang
- Department of Radiation Oncology, Sichuan Cancer Hospital and Institute, Sichuan Cancer Center, Chengdu, China
| | - Tao Song
- SenseTime Research, Shanghai, China
| | - Guotai Wang
- School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Chengdu, China; Shanghai Artificial Intelligence Laboratory, Shanghai, China.
| | - Shaoting Zhang
- School of Mechanical and Electrical Engineering, University of Electronic Science and Technology of China, Chengdu, China; Shanghai Artificial Intelligence Laboratory, Shanghai, China.
| |
Collapse
|
4
|
Zhou X, Zhang Z, Du H, Qiu B. MLMFNet: A multi-level modality fusion network for multi-modal accelerated MRI reconstruction. Magn Reson Imaging 2024; 111:246-255. [PMID: 38663831 DOI: 10.1016/j.mri.2024.04.028] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2023] [Revised: 04/09/2024] [Accepted: 04/19/2024] [Indexed: 06/01/2024]
Abstract
Magnetic resonance imaging produces detailed anatomical and physiological images of the human body that can be used in the clinical diagnosis and treatment of diseases. However, MRI suffers its comparatively longer acquisition time than other imaging methods and is thus vulnerable to motion artifacts, which ultimately lead to likely failed or even wrong diagnosis. In order to perform faster reconstruction, deep learning-based methods along with traditional strategies such as parallel imaging and compressed sensing come into play in recent years in this field. Meanwhile, in order to better analyze the diseases, it is also often necessary to acquire images in the same region of interest under different modalities, which yield images with different contrast levels. However, most of these aforementioned methods tend to use single-modal images for reconstruction, neglecting the correlation and redundancy information embedded in MR images acquired with different modalities. While there are works on multi-modal reconstruction, the information is yet to be efficiently explored. In this paper, we propose an end-to-end neural network called MLMFNet, which helps the reconstruction of the target modality by using information from the auxiliary modality across feature channels and layers. Specifically, this is highlighted by three components: (I) An encoder based on UNet with a single-stream strategy that fuses auxiliary and target modalities; (II) a decoder that tends to multi-level features from all layers of the encoder, and (III) a channel attention module. Quantitative and qualitative analyses are performed on a public brain dataset and knee brain dataset, which show that the proposed method achieves satisfying results in MRI reconstruction within the multi-modal context, and also demonstrate its effectiveness and potential to be used in clinical practice.
Collapse
Affiliation(s)
- Xiuyun Zhou
- Biomedical Engineering Center, University of Science and Technology of China, Hefei, Anhui 230026, China
| | - Zhenxi Zhang
- Biomedical Engineering Center, University of Science and Technology of China, Hefei, Anhui 230026, China
| | - Hongwei Du
- Biomedical Engineering Center, University of Science and Technology of China, Hefei, Anhui 230026, China.
| | - Bensheng Qiu
- Biomedical Engineering Center, University of Science and Technology of China, Hefei, Anhui 230026, China
| |
Collapse
|
5
|
Peper ES, Bastiaansen JAM. Editorial for "Glioma Tumor Grading Using Radiomics on Conventional MRI: A Comparative Study of WHO 2021 and WHO 2016 Classification of Central Nervous Tumors". J Magn Reson Imaging 2024; 60:939-940. [PMID: 38059496 DOI: 10.1002/jmri.29147] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/03/2023] [Accepted: 10/04/2023] [Indexed: 12/08/2023] Open
Affiliation(s)
- Eva S Peper
- Department of Diagnostic, Interventional and Pediatric Radiology (DIPR), Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland
- Translation Imaging Center (TIC), Swiss Institute for Translational and Entrepreneurial Medicine, Bern, Switzerland
| | - Jessica A M Bastiaansen
- Department of Diagnostic, Interventional and Pediatric Radiology (DIPR), Inselspital, Bern University Hospital, University of Bern, Bern, Switzerland
- Translation Imaging Center (TIC), Swiss Institute for Translational and Entrepreneurial Medicine, Bern, Switzerland
| |
Collapse
|
6
|
Moodi F, Khodadadi Shoushtari F, Ghadimi DJ, Valizadeh G, Khormali E, Salari HM, Ohadi MAD, Nilipour Y, Jahanbakhshi A, Rad HS. Glioma Tumor Grading Using Radiomics on Conventional MRI: A Comparative Study of WHO 2021 and WHO 2016 Classification of Central Nervous Tumors. J Magn Reson Imaging 2024; 60:923-938. [PMID: 38031466 DOI: 10.1002/jmri.29146] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/11/2023] [Revised: 11/07/2023] [Accepted: 11/11/2023] [Indexed: 12/01/2023] Open
Abstract
BACKGROUND Glioma grading transformed in World Health Organization (WHO) 2021 CNS tumor classification, integrating molecular markers. However, the impact of this change on radiomics-based machine learning (ML) classifiers remains unexplored. PURPOSE To assess the performance of ML in classifying glioma tumor grades based on various WHO criteria. STUDY TYPE Retrospective. SUBJECTS A neuropathologist regraded gliomas of 237 patients into WHO 2016 and 2021 from 2007 criteria. FIELD STRENGTH/SEQUENCE Multicentric 0.5 to 3 Tesla; pre- and post-contrast T1-weighted, T2-weighted, and fluid-attenuated inversion recovery. ASSESSMENT Radiomic features were selected using random forest-recursive feature elimination. The synthetic minority over-sampling technique (SMOTE) was implemented for data augmentation. Stratified 10-fold cross-validation with and without SMOTE was used to evaluate 11 classifiers for 3-grade (2, 3, and 4; WHO 2016 and 2021) and 2-grade (low and high grade; WHO 2007 and 2021) classification. Additionally, we developed the models on data randomly divided into training and test sets (mixed-data analysis), or data divided based on the centers (independent-data analysis). STATISTICAL TESTS We assessed ML classifiers using sensitivity, specificity, accuracy, and the area under the receiver operating characteristic curve (AUC). Top performances were compared with a t-test and categorical data with the chi-square test using a significance level of P < 0.05. RESULTS In the mixed-data analysis, Stacking Classifier without SMOTE achieved the highest accuracy (0.86) and AUC (0.92) in 3-grade WHO 2021 grouping. The results of WHO 2021 were significantly better than WHO 2016 (P-value<0.0001). In the 2-grade analysis, ML achieved 1.00 in all metrics. In the independent-data analysis, ML classifiers showed strong discrimination between grade 2 and 4, despite lower performance metrics than the mixed analysis. DATA CONCLUSION ML algorithms performed better in glioma tumor grading based on WHO 2021 criteria. Nonetheless, the clinical use of ML classifiers needs further investigation. LEVEL OF EVIDENCE 3 TECHNICAL EFFICACY: Stage 2.
Collapse
Affiliation(s)
- Farzan Moodi
- Quantitative MR Imaging and Spectroscopy Group (QMISG), Tehran University of Medical Sciences, Tehran, Iran
- School of Medicine, Iran University of Medical Sciences, Tehran, Iran
| | | | - Delaram J Ghadimi
- Quantitative MR Imaging and Spectroscopy Group (QMISG), Tehran University of Medical Sciences, Tehran, Iran
- School of Medicine, Shahid Beheshti University of Medical Sciences, Tehran, Iran
| | - Gelareh Valizadeh
- Quantitative MR Imaging and Spectroscopy Group (QMISG), Tehran University of Medical Sciences, Tehran, Iran
| | - Ehsan Khormali
- Student Research Committee, School of Medicine, Mazandaran University of Medical Sciences, Sari, Iran
| | - Hanieh Mobarak Salari
- Quantitative MR Imaging and Spectroscopy Group (QMISG), Tehran University of Medical Sciences, Tehran, Iran
| | - Mohammad Amin Dabbagh Ohadi
- Interdisciplinary Neuroscience Research Program, Tehran University of Medical Sciences, Tehran, Iran
- Departments of Pediatric Neurosurgery Children's Medical Center Hospital, Tehran University of Medical Sciences, Tehran, Iran
| | - Yalda Nilipour
- Pediatric Pathology Research Center, Research Institute of Children's Health, Shahid Beheshti University of Medical Sciences, Tehran, Iran
| | - Amin Jahanbakhshi
- Stem Cell and Regenerative Medicine Research Center, Iran University of Medical Sciences, Tehran, Iran
| | - Hamidreza Saligheh Rad
- Quantitative MR Imaging and Spectroscopy Group (QMISG), Tehran University of Medical Sciences, Tehran, Iran
- Department of Medical Physics and Biomedical Engineering, Tehran University of Medical Sciences, Tehran, Iran
| |
Collapse
|
7
|
Doniselli FM, Pascuzzo R, Mazzi F, Padelli F, Moscatelli M, Akinci D'Antonoli T, Cuocolo R, Aquino D, Cuccarini V, Sconfienza LM. Quality assessment of the MRI-radiomics studies for MGMT promoter methylation prediction in glioma: a systematic review and meta-analysis. Eur Radiol 2024; 34:5802-5815. [PMID: 38308012 DOI: 10.1007/s00330-024-10594-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/23/2023] [Revised: 12/04/2023] [Accepted: 12/31/2023] [Indexed: 02/04/2024]
Abstract
OBJECTIVES To evaluate the methodological quality and diagnostic accuracy of MRI-based radiomic studies predicting O6-methylguanine-DNA methyltransferase (MGMT) promoter methylation status in gliomas. METHODS PubMed Medline, EMBASE, and Web of Science were searched to identify MRI-based radiomic studies on MGMT methylation in gliomas published until December 31, 2022. Three raters evaluated the study methodological quality with Radiomics Quality Score (RQS, 16 components) and Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis Or Diagnosis (TRIPOD, 22 items) scales. Risk of bias and applicability concerns were assessed with QUADAS-2 tool. A meta-analysis was performed to estimate the pooled area under the curve (AUC) and to assess inter-study heterogeneity. RESULTS We included 26 studies, published from 2016. The median RQS total score was 8 out of 36 (22%, range 8-44%). Thirteen studies performed external validation. All studies reported AUC or accuracy, but only 4 (15%) performed calibration and decision curve analysis. No studies performed phantom analysis, cost-effectiveness analysis, and prospective validation. The overall TRIPOD adherence score was between 50% and 70% in 16 studies and below 50% in 10 studies. The pooled AUC was 0.78 (95% CI, 0.73-0.83, I2 = 94.1%) with a high inter-study heterogeneity. Studies with external validation and including only WHO-grade IV gliomas had significantly lower AUC values (0.65; 95% CI, 0.57-0.73, p < 0.01). CONCLUSIONS Study RQS and adherence to TRIPOD guidelines was generally low. Radiomic prediction of MGMT methylation status showed great heterogeneity of results and lower performances in grade IV gliomas, which hinders its current implementation in clinical practice. CLINICAL RELEVANCE STATEMENT MGMT promoter methylation status appears to be variably correlated with MRI radiomic features; radiomic models are not sufficiently robust to be integrated into clinical practice to accurately predict MGMT promoter methylation status in patients with glioma before surgery. KEY POINTS • Adherence to the indications of TRIPOD guidelines was generally low, as was RQS total score. • MGMT promoter methylation status prediction with MRI radiomic features provided heterogeneous diagnostic accuracy results across studies. • Studies that included grade IV glioma only and performed external validation had significantly lower diagnostic accuracy than others.
Collapse
Affiliation(s)
- Fabio M Doniselli
- Neuroradiology Unit, Fondazione IRCCS Istituto Neurologico Carlo Besta, Via Giovanni Celoria 11, 20133, Milan, Italy
- Department of Biomedical Sciences for Health, Università degli Studi di Milano, Via Luigi Mangiagalli 31, 20133, Milan, Italy
| | - Riccardo Pascuzzo
- Neuroradiology Unit, Fondazione IRCCS Istituto Neurologico Carlo Besta, Via Giovanni Celoria 11, 20133, Milan, Italy.
| | - Federica Mazzi
- Neuroradiology Unit, Fondazione IRCCS Istituto Neurologico Carlo Besta, Via Giovanni Celoria 11, 20133, Milan, Italy
| | - Francesco Padelli
- Neuroradiology Unit, Fondazione IRCCS Istituto Neurologico Carlo Besta, Via Giovanni Celoria 11, 20133, Milan, Italy
| | - Marco Moscatelli
- Neuroradiology Unit, Fondazione IRCCS Istituto Neurologico Carlo Besta, Via Giovanni Celoria 11, 20133, Milan, Italy
| | - Tugba Akinci D'Antonoli
- Institute of Radiology and Nuclear Medicine, Cantonal Hospital Baselland, Rheinstrasse 26, 4410, Liestal, Switzerland
| | - Renato Cuocolo
- Department of Medicine, Surgery, and Dentistry, University of Salerno, Via Salvador Allende 43, Baronissi, 84081, Salerno, Italy
| | - Domenico Aquino
- Neuroradiology Unit, Fondazione IRCCS Istituto Neurologico Carlo Besta, Via Giovanni Celoria 11, 20133, Milan, Italy
| | - Valeria Cuccarini
- Neuroradiology Unit, Fondazione IRCCS Istituto Neurologico Carlo Besta, Via Giovanni Celoria 11, 20133, Milan, Italy
| | - Luca Maria Sconfienza
- Department of Biomedical Sciences for Health, Università degli Studi di Milano, Via Luigi Mangiagalli 31, 20133, Milan, Italy
- IRCCS Ospedale Galeazzi-Sant'Ambrogio, Via Cristina Belgioioso 173, 20157, Milan, Italy
| |
Collapse
|
8
|
Su J, Luo Z, Wang C, Lian S, Lin X, Li S. Reconstruct incomplete relation for incomplete modality brain tumor segmentation. Neural Netw 2024; 180:106657. [PMID: 39186839 DOI: 10.1016/j.neunet.2024.106657] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/28/2023] [Revised: 08/14/2024] [Accepted: 08/19/2024] [Indexed: 08/28/2024]
Abstract
Different brain tumor magnetic resonance imaging (MRI) modalities provide diverse tumor-specific information. Previous works have enhanced brain tumor segmentation performance by integrating multiple MRI modalities. However, multi-modal MRI data are often unavailable in clinical practice. An incomplete modality leads to missing tumor-specific information, which degrades the performance of existing models. Various strategies have been proposed to transfer knowledge from a full modality network (teacher) to an incomplete modality one (student) to address this issue. However, they neglect the fact that brain tumor segmentation is a structural prediction problem that requires voxel semantic relations. In this paper, we propose a Reconstruct Incomplete Relation Network (RIRN) that transfers voxel semantic relational knowledge from the teacher to the student. Specifically, we propose two types of voxel relations to incorporate structural knowledge: Class-relative relations (CRR) and Class-agnostic relations (CAR). The CRR groups voxels into different tumor regions and constructs a relation between them. The CAR builds a global relation between all voxel features, complementing the local inter-region relation. Moreover, we use adversarial learning to align the holistic structural prediction between the teacher and the student. Extensive experimentation on both the BraTS 2018 and BraTS 2020 datasets establishes that our method outperforms all state-of-the-art approaches.
Collapse
Affiliation(s)
- Jiawei Su
- School of Computer Engineering, Jimei University, Xiamen, China; The Department of Artificial Intelligence, Xiamen University, Fujian, China
| | - Zhiming Luo
- The Department of Artificial Intelligence, Xiamen University, Fujian, China.
| | - Chengji Wang
- The School of Computer Science, Central China Normal University, Wuhan, China
| | - Sheng Lian
- The College of Computer and Data Science, Fuzhou University, Fujian, China
| | - Xuejuan Lin
- The Department of Traditional Chinese Medicine, Fujian University of Traditional Chinese Medicine, Fujian, China
| | - Shaozi Li
- The Department of Artificial Intelligence, Xiamen University, Fujian, China
| |
Collapse
|
9
|
Kudus K, Wagner MW, Namdar K, Bennett J, Nobre L, Tabori U, Hawkins C, Ertl-Wagner BB, Khalvati F. Beyond hand-crafted features for pretherapeutic molecular status identification of pediatric low-grade gliomas. Sci Rep 2024; 14:19102. [PMID: 39154039 PMCID: PMC11330469 DOI: 10.1038/s41598-024-69870-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/08/2024] [Accepted: 08/09/2024] [Indexed: 08/19/2024] Open
Abstract
The use of targeted agents in the treatment of pediatric low-grade gliomas (pLGGs) relies on the determination of molecular status. It has been shown that genetic alterations in pLGG can be identified non-invasively using MRI-based radiomic features or convolutional neural networks (CNNs). We aimed to build and assess a combined radiomics and CNN non-invasive pLGG molecular status identification model. This retrospective study used the tumor regions, manually segmented from T2-FLAIR MR images, of 336 patients treated for pLGG between 1999 and 2018. We designed a CNN and Random Forest radiomics model, along with a model relying on a combination of CNN and radiomic features, to predict the genetic status of pLGG. Additionally, we investigated whether CNNs could predict radiomic feature values from MR images. The combined model (mean AUC: 0.824) outperformed the radiomics model (0.802) and CNN (0.764). The differences in model performance were statistically significant (p-values < 0.05). The CNN was able to learn predictive radiomic features such as surface-to-volume ratio (average correlation: 0.864), and difference matrix dependence non-uniformity normalized (0.924) well but was unable to learn others such as run-length matrix variance (- 0.017) and non-uniformity normalized (- 0.042). Our results show that a model relying on both CNN and radiomic-based features performs better than either approach separately in differentiating the genetic status of pLGGs, and that CNNs are unable to express all handcrafted features.
Collapse
Affiliation(s)
- Kareem Kudus
- Neurosciences & Mental Health Research Program, The Hospital for Sick Children, Toronto, Canada
- Institute of Medical Science, University of Toronto, Toronto, Canada
| | - Matthias W Wagner
- Department of Diagnostic & Interventional Radiology, The Hospital for Sick Children, Toronto, Canada
- Department of Diagnostic and Interventional Neuroradiology, University Hospital Augsburg, Augsburg, Germany
| | - Khashayar Namdar
- Neurosciences & Mental Health Research Program, The Hospital for Sick Children, Toronto, Canada
- Institute of Medical Science, University of Toronto, Toronto, Canada
| | - Julie Bennett
- Division of Hematology and Oncology, The Hospital for Sick Children, Toronto, Canada
- Division of Medical Oncology and Hematology, Princess Margaret Cancer Centre, Toronto, Canada
- Department of Pediatrics, University of Toronto, Toronto, Canada
| | - Liana Nobre
- Department of Paediatrics, University of Alberta, Edmonton, Canada
- Division of Immunology, Hematology/Oncology and Palliative Care, Stollery Children's Hospital, Edmonton, Canada
| | - Uri Tabori
- Division of Hematology and Oncology, The Hospital for Sick Children, Toronto, Canada
| | - Cynthia Hawkins
- Paediatric Laboratory Medicine, Division of Pathology, The Hospital for Sick Children, Toronto, Canada
| | - Birgit Betina Ertl-Wagner
- Neurosciences & Mental Health Research Program, The Hospital for Sick Children, Toronto, Canada
- Institute of Medical Science, University of Toronto, Toronto, Canada
- Department of Diagnostic & Interventional Radiology, The Hospital for Sick Children, Toronto, Canada
- Department of Medical Imaging, University of Toronto, Toronto, Canada
| | - Farzad Khalvati
- Neurosciences & Mental Health Research Program, The Hospital for Sick Children, Toronto, Canada.
- Institute of Medical Science, University of Toronto, Toronto, Canada.
- Department of Diagnostic & Interventional Radiology, The Hospital for Sick Children, Toronto, Canada.
- Department of Medical Imaging, University of Toronto, Toronto, Canada.
- Department of Computer Science, University of Toronto, Toronto, Canada.
- Department of Mechanical and Industrial Engineering, University of Toronto, Toronto, Canada.
| |
Collapse
|
10
|
Wu X, Zhang S, Zhang Z, He Z, Xu Z, Wang W, Jin Z, You J, Guo Y, Zhang L, Huang W, Wang F, Liu X, Yan D, Cheng J, Yan J, Zhang S, Zhang B. Biologically interpretable multi-task deep learning pipeline predicts molecular alterations, grade, and prognosis in glioma patients. NPJ Precis Oncol 2024; 8:181. [PMID: 39152182 PMCID: PMC11329669 DOI: 10.1038/s41698-024-00670-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/15/2024] [Accepted: 08/01/2024] [Indexed: 08/19/2024] Open
Abstract
Deep learning models have been developed for various predictions in glioma; yet, they were constrained by manual segmentation, task-specific design, or a lack of biological interpretation. Herein, we aimed to develop an end-to-end multi-task deep learning (MDL) pipeline that can simultaneously predict molecular alterations and histological grade (auxiliary tasks), as well as prognosis (primary task) in gliomas. Further, we aimed to provide the biological mechanisms underlying the model's predictions. We collected multiscale data including baseline MRI images from 2776 glioma patients across two private (FAHZU and HPPH, n = 1931) and three public datasets (TCGA, n = 213; UCSF, n = 410; and EGD, n = 222). We trained and internally validated the MDL model using our private datasets, and externally validated it using the three public datasets. We used the model-predicted deep prognosis score (DPS) to stratify patients into low-DPS and high-DPS subtypes. Additionally, a radio-multiomics analysis was conducted to elucidate the biological basis of the DPS. In the external validation cohorts, the MDL model achieved average areas under the curve of 0.892-0.903, 0.710-0.894, and 0.850-0.879 for predicting IDH mutation status, 1p/19q co-deletion status, and tumor grade, respectively. Moreover, the MDL model yielded a C-index of 0.723 in the TCGA and 0.671 in the UCSF for the prediction of overall survival. The DPS exhibits significant correlations with activated oncogenic pathways, immune infiltration patterns, specific protein expression, DNA methylation, tumor mutation burden, and tumor-stroma ratio. Accordingly, our work presents an accurate and biologically meaningful tool for predicting molecular subtypes, tumor grade, and survival outcomes in gliomas, which provides personalized clinical decision-making in a global and non-invasive manner.
Collapse
Affiliation(s)
- Xuewei Wu
- Department of Radiology, The First Affiliated Hospital of Jinan University, Guangzhou, Guangdong, China
| | - Shuaitong Zhang
- School of Medical Technology, Beijing Institute of Technology, Beijing, China
| | - Zhenyu Zhang
- Department of Neurosurgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China
| | - Zicong He
- Department of Radiology, The First Affiliated Hospital of Jinan University, Guangzhou, Guangdong, China
| | - Zexin Xu
- School of Medical Technology, Beijing Institute of Technology, Beijing, China
| | - Weiwei Wang
- Department of Pathology, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, Henan, China
| | - Zhe Jin
- Department of Radiology, The First Affiliated Hospital of Jinan University, Guangzhou, Guangdong, China
| | - Jingjing You
- Department of Radiology, The First Affiliated Hospital of Jinan University, Guangzhou, Guangdong, China
| | - Yang Guo
- Department of Neurosurgery, The Henan Provincial People's Hospital, Zhengzhou, Henan, China
| | - Lu Zhang
- Department of Radiology, The First Affiliated Hospital of Jinan University, Guangzhou, Guangdong, China
| | - Wenhui Huang
- Department of Radiology, The First Affiliated Hospital of Jinan University, Guangzhou, Guangdong, China
| | - Fei Wang
- Department of Radiology, The First Affiliated Hospital of Jinan University, Guangzhou, Guangdong, China
| | - Xianzhi Liu
- Department of Neurosurgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China
| | - Dongming Yan
- Department of Neurosurgery, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China
| | - Jingliang Cheng
- Department of MRI, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China
| | - Jing Yan
- Department of MRI, The First Affiliated Hospital of Zhengzhou University, Zhengzhou, China.
| | - Shuixing Zhang
- Department of Radiology, The First Affiliated Hospital of Jinan University, Guangzhou, Guangdong, China.
| | - Bin Zhang
- Department of Radiology, The First Affiliated Hospital of Jinan University, Guangzhou, Guangdong, China.
| |
Collapse
|
11
|
Javier Gil-Terrón F, Ferri P, Montosa-I-Micó V, Gómez Mahiques M, Lopez-Mateu C, Martí P, García-Gómez JM, Fuster-Garcia E. Exploring the Trade-Off between generalist and specialized Models: A center-based comparative analysis for glioblastoma segmentation. Int J Med Inform 2024; 191:105604. [PMID: 39154600 DOI: 10.1016/j.ijmedinf.2024.105604] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2024] [Revised: 08/08/2024] [Accepted: 08/14/2024] [Indexed: 08/20/2024]
Abstract
INTRODUCTION Inherent variations between inter-center data can undermine the robustness of segmentation models when applied at a specific center (dataset shift). We investigated whether specialized center-specific models are more effective compared to generalist models based on multi-center data, and how center-specific data could enhance the performance of generalist models within a particular center using a fine-tuning transfer learning approach. For this purpose, we studied the dataset shift at center level and conducted a comparative analysis to assess the impact of data source on glioblastoma segmentation models. METHODS & MATERIALS The three key components of dataset shift were studied: prior probability shift-variations in tumor size or tissue distribution among centers; covariate shift-inter-center MRI alterations; and concept shift-different criteria for tumor segmentation. BraTS 2021 dataset was used, which includes 1251 cases from 23 centers. Thereafter, 155 deep-learning models were developed and compared, including 1) generalist models trained with multi-center data, 2) specialized models using only center-specific data, and 3) fine-tuned generalist models using center-specific data. RESULTS The three key components of dataset shift were characterized. The amount of covariate shift was substantial, indicating large variations in MR imaging between different centers. Glioblastoma segmentation models tend to perform best when using data from the application center. Generalist models, trained with over 700 samples, achieved a median Dice score of 88.98%. Specialized models surpassed this with 200 cases, while fine-tuned models outperformed with 50 cases. CONCLUSIONS The influence of dataset shift on model performance is evident. Fine-tuned and specialized models, utilizing data from the evaluated center, outperform generalist models, which rely on data from other centers. These approaches could encourage medical centers to develop customized models for their local use, enhancing the accuracy and reliability of glioblastoma segmentation in a context where dataset shift is inevitable.
Collapse
Affiliation(s)
- F Javier Gil-Terrón
- Biomedical Data Science Laboratory, ITACA Institute, Universitat Politècnica de València, València, Spain
| | - Pablo Ferri
- Biomedical Data Science Laboratory, ITACA Institute, Universitat Politècnica de València, València, Spain
| | - Víctor Montosa-I-Micó
- Biomedical Data Science Laboratory, ITACA Institute, Universitat Politècnica de València, València, Spain
| | - María Gómez Mahiques
- Biomedical Data Science Laboratory, ITACA Institute, Universitat Politècnica de València, València, Spain
| | - Carles Lopez-Mateu
- Biomedical Data Science Laboratory, ITACA Institute, Universitat Politècnica de València, València, Spain
| | - Pau Martí
- Departament d'Enginyeria Industrial i Construcció, Àrea d'Enginyeria Agroforestal, Universitat de les Illes Balears, Palma, Spain
| | - Juan M García-Gómez
- Biomedical Data Science Laboratory, ITACA Institute, Universitat Politècnica de València, València, Spain
| | - Elies Fuster-Garcia
- Biomedical Data Science Laboratory, ITACA Institute, Universitat Politècnica de València, València, Spain
| |
Collapse
|
12
|
Zhuang Y, Liu H, Fang W, Ma G, Sun S, Zhu Y, Zhang X, Ge C, Chen W, Long J, Song E. A 3D hierarchical cross-modality interaction network using transformers and convolutions for brain glioma segmentation in MR images. Med Phys 2024. [PMID: 39137295 DOI: 10.1002/mp.17354] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/12/2024] [Revised: 06/20/2024] [Accepted: 08/02/2024] [Indexed: 08/15/2024] Open
Abstract
BACKGROUND Precise glioma segmentation from multi-parametric magnetic resonance (MR) images is essential for brain glioma diagnosis. However, due to the indistinct boundaries between tumor sub-regions and the heterogeneous appearances of gliomas in volumetric MR scans, designing a reliable and automated glioma segmentation method is still challenging. Although existing 3D Transformer-based or convolution-based segmentation networks have obtained promising results via multi-modal feature fusion strategies or contextual learning methods, they widely lack the capability of hierarchical interactions between different modalities and cannot effectively learn comprehensive feature representations related to all glioma sub-regions. PURPOSE To overcome these problems, in this paper, we propose a 3D hierarchical cross-modality interaction network (HCMINet) using Transformers and convolutions for accurate multi-modal glioma segmentation, which leverages an effective hierarchical cross-modality interaction strategy to sufficiently learn modality-specific and modality-shared knowledge correlated to glioma sub-region segmentation from multi-parametric MR images. METHODS In the HCMINet, we first design a hierarchical cross-modality interaction Transformer (HCMITrans) encoder to hierarchically encode and fuse heterogeneous multi-modal features by Transformer-based intra-modal embeddings and inter-modal interactions in multiple encoding stages, which effectively captures complex cross-modality correlations while modeling global contexts. Then, we collaborate an HCMITrans encoder with a modality-shared convolutional encoder to construct the dual-encoder architecture in the encoding stage, which can learn the abundant contextual information from global and local perspectives. Finally, in the decoding stage, we present a progressive hybrid context fusion (PHCF) decoder to progressively fuse local and global features extracted by the dual-encoder architecture, which utilizes the local-global context fusion (LGCF) module to efficiently alleviate the contextual discrepancy among the decoding features. RESULTS Extensive experiments are conducted on two public and competitive glioma benchmark datasets, including the BraTS2020 dataset with 494 patients and the BraTS2021 dataset with 1251 patients. Results show that our proposed method outperforms existing Transformer-based and CNN-based methods using other multi-modal fusion strategies in our experiments. Specifically, the proposed HCMINet achieves state-of-the-art mean DSC values of 85.33% and 91.09% on the BraTS2020 online validation dataset and the BraTS2021 local testing dataset, respectively. CONCLUSIONS Our proposed method can accurately and automatically segment glioma regions from multi-parametric MR images, which is beneficial for the quantitative analysis of brain gliomas and helpful for reducing the annotation burden of neuroradiologists.
Collapse
Affiliation(s)
- Yuzhou Zhuang
- School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China
| | - Hong Liu
- School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China
| | - Wei Fang
- Wuhan Zhongke Industrial Research Institute of Medical Science Co., Ltd, Wuhan, China
| | - Guangzhi Ma
- School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China
| | - Sisi Sun
- School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China
| | - Yunfeng Zhu
- School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China
| | - Xu Zhang
- Wuhan United Imaging Healthcare Surgical Technology Co., Ltd, Wuhan, China
| | - Chuanbin Ge
- Wuhan United Imaging Healthcare Surgical Technology Co., Ltd, Wuhan, China
| | - Wenyang Chen
- School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China
| | - Jiaosong Long
- School of Art and Design, Hubei University of Technology, Wuhan, China
| | - Enmin Song
- School of Computer Science and Technology, Huazhong University of Science and Technology, Wuhan, China
| |
Collapse
|
13
|
Zhao J, Li S. Evidence modeling for reliability learning and interpretable decision-making under multi-modality medical image segmentation. Comput Med Imaging Graph 2024; 116:102422. [PMID: 39116707 DOI: 10.1016/j.compmedimag.2024.102422] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/17/2024] [Revised: 06/04/2024] [Accepted: 07/30/2024] [Indexed: 08/10/2024]
Abstract
Reliability learning and interpretable decision-making are crucial for multi-modality medical image segmentation. Although many works have attempted multi-modality medical image segmentation, they rarely explore how much reliability is provided by each modality for segmentation. Moreover, the existing approach of decision-making such as the softmax function lacks the interpretability for multi-modality fusion. In this study, we proposed a novel approach named contextual discounted evidential network (CDE-Net) for reliability learning and interpretable decision-making under multi-modality medical image segmentation. Specifically, the CDE-Net first models the semantic evidence by uncertainty measurement using the proposed evidential decision-making module. Then, it leverages the contextual discounted fusion layer to learn the reliability provided by each modality. Finally, a multi-level loss function is deployed for the optimization of evidence modeling and reliability learning. Moreover, this study elaborates on the framework interpretability by discussing the consistency between pixel attribution maps and the learned reliability coefficients. Extensive experiments are conducted on both multi-modality brain and liver datasets. The CDE-Net gains high performance with an average Dice score of 0.914 for brain tumor segmentation and 0.913 for liver tumor segmentation, which proves CDE-Net has great potential to facilitate the interpretation of artificial intelligence-based multi-modality medical image fusion.
Collapse
Affiliation(s)
- Jianfeng Zhao
- School of Biomedical Engineering, Western University, London, ON, Canada
| | - Shuo Li
- Department of Biomedical Engineering, Case Western Reserve University, Cleveland, OH, USA; Department of Computer and Data Science, Case Western Reserve University, Cleveland, OH, USA.
| |
Collapse
|
14
|
De Sutter S, Wuts J, Geens W, Vanbinst AM, Duerinck J, Vandemeulebroucke J. Modality redundancy for MRI-based glioblastoma segmentation. Int J Comput Assist Radiol Surg 2024:10.1007/s11548-024-03238-4. [PMID: 39093499 DOI: 10.1007/s11548-024-03238-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/16/2024] [Accepted: 07/11/2024] [Indexed: 08/04/2024]
Abstract
PURPOSE Automated glioblastoma segmentation from magnetic resonance imaging is generally performed on a four-modality input, including T1, contrast T1, T2 and FLAIR. We hypothesize that information redundancy is present within these image combinations, which can possibly reduce a model's performance. Moreover, for clinical applications, the risk of encountering missing data rises as the number of required input modalities increases. Therefore, this study aimed to explore the relevance and influence of the different modalities used for MRI-based glioblastoma segmentation. METHODS After the training of multiple segmentation models based on nnU-Net and SwinUNETR architectures, differing only in their amount and combinations of input modalities, each model was evaluated with regard to segmentation accuracy and epistemic uncertainty. RESULTS Results show that T1CE-based segmentation (for enhanced tumor and tumor core) and T1CE-FLAIR-based segmentation (for whole tumor and overall segmentation) can reach segmentation accuracies comparable to the full-input version. Notably, the highest segmentation accuracy for nnU-Net was found for a three-input configuration of T1CE-FLAIR-T1, suggesting the confounding effect of redundant input modalities. The SwinUNETR architecture appears to suffer less from this, where said three-input and the full-input model yielded statistically equal results. CONCLUSION The T1CE-FLAIR-based model can therefore be considered as a minimal-input alternative to the full-input configuration. Addition of modalities beyond this does not statistically improve and can even deteriorate accuracy, but does lower the segmentation uncertainty.
Collapse
Affiliation(s)
- Selene De Sutter
- Department of Electronics and Informatics (ETRO), Vrije Universiteit Brussel (VUB), Brussels, Belgium.
| | - Joris Wuts
- Department of Electronics and Informatics (ETRO), Vrije Universiteit Brussel (VUB), Brussels, Belgium
- Department of Radiology and Medical Imaging, Cliniques Universitaires Saint Luc, Université Catholique de Louvain (UCLouvain), Brussels, Belgium
| | - Wietse Geens
- Department of Neurosurgery, Universitair Ziekenhuis Brussel (UZ Brussel), Vrije Universiteit Brussel (VUB), Brussels, Belgium
| | - Anne-Marie Vanbinst
- Department of Radiology, Universitair Ziekenhuis Brussel (UZ Brussel), Vrije Universiteit Brussel (VUB), Brussels, Belgium
| | - Johnny Duerinck
- Department of Neurosurgery, Universitair Ziekenhuis Brussel (UZ Brussel), Vrije Universiteit Brussel (VUB), Brussels, Belgium
| | - Jef Vandemeulebroucke
- Department of Electronics and Informatics (ETRO), Vrije Universiteit Brussel (VUB), Brussels, Belgium
- Department of Radiology, Universitair Ziekenhuis Brussel (UZ Brussel), Vrije Universiteit Brussel (VUB), Brussels, Belgium
- imec, Leuven, Belgium
| |
Collapse
|
15
|
Wan Q, Kim J, Lindsay C, Chen X, Li J, Iorgulescu JB, Huang RY, Zhang C, Reardon D, Young GS, Qin L. Auto-segmentation of Adult-Type Diffuse Gliomas: Comparison of Transfer Learning-Based Convolutional Neural Network Model vs. Radiologists. JOURNAL OF IMAGING INFORMATICS IN MEDICINE 2024; 37:1401-1410. [PMID: 38383806 PMCID: PMC11300742 DOI: 10.1007/s10278-024-01044-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/01/2023] [Revised: 02/03/2024] [Accepted: 02/06/2024] [Indexed: 02/23/2024]
Abstract
Segmentation of glioma is crucial for quantitative brain tumor assessment, to guide therapeutic research and clinical management, but very time-consuming. Fully automated tools for the segmentation of multi-sequence MRI are needed. We developed and pretrained a deep learning (DL) model using publicly available datasets A (n = 210) and B (n = 369) containing FLAIR, T2WI, and contrast-enhanced (CE)-T1WI. This was then fine-tuned with our institutional dataset (n = 197) containing ADC, T2WI, and CE-T1WI, manually annotated by radiologists, and split into training (n = 100) and testing (n = 97) sets. The Dice similarity coefficient (DSC) was used to compare model outputs and manual labels. A third independent radiologist assessed segmentation quality on a semi-quantitative 5-scale score. Differences in DSC between new and recurrent gliomas, and between uni or multifocal gliomas were analyzed using the Mann-Whitney test. Semi-quantitative analyses were compared using the chi-square test. We found that there was good agreement between segmentations from the fine-tuned DL model and ground truth manual segmentations (median DSC: 0.729, std-dev: 0.134). DSC was higher for newly diagnosed (0.807) than recurrent (0.698) (p < 0.001), and higher for unifocal (0.747) than multi-focal (0.613) cases (p = 0.001). Semi-quantitative scores of DL and manual segmentation were not significantly different (mean: 3.567 vs. 3.639; 93.8% vs. 97.9% scoring ≥ 3, p = 0.107). In conclusion, the proposed transfer learning DL performed similarly to human radiologists in glioma segmentation on both structural and ADC sequences. Further improvement in segmenting challenging postoperative and multifocal glioma cases is needed.
Collapse
Affiliation(s)
- Qi Wan
- Department of Imaging, Dana Farber Cancer Institute, Harvard Medical School, Boston, MA, USA
- Department of Radiology, the Key Laboratory of Advanced Interdisciplinary Studies Center, the First Affiliated Hospital of Guangzhou Medical University, Guangzhou, China
| | - Jisoo Kim
- Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Clifford Lindsay
- Image Processing and Analysis Core (iPAC), Department of Radiology, University of Massachusetts Chan Medical School, Worcester, MA, USA
| | - Xin Chen
- School of Medicine, Guangzhou First People's Hospital, South China University of Technology, Guangzhou, Guangdong, China
| | - Jing Li
- Department of Radiology, the Affiliated Cancer Hospital of Zhengzhou University (Henan Cancer Hospital), Zhengzhou, China
| | - J Bryan Iorgulescu
- Molecular Diagnostics Laboratory, Department of Hematopathology, Division of Pathology and Laboratory Medicine, The University of Texas MD Anderson Cancer Center, Houston, USA
| | - Raymond Y Huang
- Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA
| | - Chenxi Zhang
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, China
| | - David Reardon
- Center for Neuro-Oncology, Dana-Farber Cancer Institute, Boston, MA, USA
| | - Geoffrey S Young
- Department of Radiology, Brigham and Women's Hospital, Harvard Medical School, Boston, MA, USA.
| | - Lei Qin
- Department of Imaging, Dana Farber Cancer Institute, Harvard Medical School, Boston, MA, USA.
| |
Collapse
|
16
|
Iglesias G, Talavera E, Troya J, Díaz-Álvarez A, García-Remesal M. Artificial intelligence model for tumoral clinical decision support systems. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 253:108228. [PMID: 38810378 DOI: 10.1016/j.cmpb.2024.108228] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/02/2023] [Revised: 04/21/2024] [Accepted: 05/13/2024] [Indexed: 05/31/2024]
Abstract
BACKGROUND AND OBJECTIVE Comparative diagnostic in brain tumor evaluation makes possible to use the available information of a medical center to compare similar cases when a new patient is evaluated. By leveraging Artificial Intelligence models, the proposed system is able of retrieving the most similar cases of brain tumors for a given query. The primary objective is to enhance the diagnostic process by generating more accurate representations of medical images, with a particular focus on patient-specific normal features and pathologies. A key distinction from previous models lies in its ability to produce enriched image descriptors solely from binary information, eliminating the need for costly and difficult to obtain tumor segmentation. METHODS The proposed model uses Artificial Intelligence to detect patient features to recommend the most similar cases from a database. The system not only suggests similar cases but also balances the representation of healthy and abnormal features in its design. This not only encourages the generalization of its use but also aids clinicians in their decision-making processes. This generalization makes possible for future research in different medical diagnosis areas with almost not any change in the system. RESULTS We conducted a comparative analysis of our approach in relation to similar studies. The proposed architecture obtains a Dice coefficient of 0.474 in both tumoral and healthy regions of the patients, which outperforms previous literature. Our proposed model excels at extracting and combining anatomical and pathological features from brain Magnetic Resonances (MRs), achieving state-of-the-art results while relying on less expensive label information. This substantially reduces the overall cost of the training process. Our findings highlight the significant potential for improving the efficiency and accuracy of comparative diagnostics and the treatment of tumoral pathologies. CONCLUSIONS This paper provides substantial grounds for further exploration of the broader applicability and optimization of the proposed architecture to enhance clinical decision-making. The novel approach presented in this work marks a significant advancement in the field of medical diagnosis, particularly in the context of Artificial Intelligence-assisted image retrieval, and promises to reduce costs and improve the quality of patient care using Artificial Intelligence as a support tool instead of a black box system.
Collapse
Affiliation(s)
- Guillermo Iglesias
- Departamento de Sistemas Informáticos, Escuela Técnica Superior de Ingeniería de Sistemas Informáticos, Universidad Politécnica de Madrid, Spain.
| | - Edgar Talavera
- Departamento de Sistemas Informáticos, Escuela Técnica Superior de Ingeniería de Sistemas Informáticos, Universidad Politécnica de Madrid, Spain.
| | - Jesús Troya
- Infanta Leonor University Hospital. Madrid., Spain
| | - Alberto Díaz-Álvarez
- Departamento de Sistemas Informáticos, Escuela Técnica Superior de Ingeniería de Sistemas Informáticos, Universidad Politécnica de Madrid, Spain.
| | - Miguel García-Remesal
- Biomedical Informatics Group, Departamento de Inteligencia Artificial, Escuela Técnica Superior de Ingenieros Informáticos, Universidad Politécnica de Madrid, Spain.
| |
Collapse
|
17
|
Moshe YH, Buchsweiler Y, Teicher M, Artzi M. Handling Missing MRI Data in Brain Tumors Classification Tasks: Usage of Synthetic Images vs. Duplicate Images and Empty Images. J Magn Reson Imaging 2024; 60:561-573. [PMID: 37864370 DOI: 10.1002/jmri.29072] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/25/2023] [Revised: 09/30/2023] [Accepted: 10/02/2023] [Indexed: 10/22/2023] Open
Abstract
BACKGROUND Deep-learning is widely used for lesion classification. However, in the clinic patient data often has missing images. PURPOSE To evaluate the use of generated, duplicate and empty(black) images for replacing missing MRI data in AI brain tumor classification tasks. STUDY TYPE Retrospective. POPULATION 224 patients (local-dataset; low-grade-glioma (LGG) = 37, high-grade-glioma (HGG) = 187) and 335 patients (public-dataset (BraTS); LGG = 76, HGG = 259). The local-dataset was divided into training (64), validation (16), and internal-test-data (20), while the public-dataset was an independent test-set. FIELD STRENGTH/SEQUENCE T1WI, T1WI+C, T2WI, and FLAIR images (1.5T/3.0T-MR), obtained from different suppliers. ASSESSMENT Three image-to-image translation generative-adversarial-network (Pix2Pix-GAN) models were trained on the local-dataset, to generate T1WI, T2WI, and FLAIR images. The rating-and-preference-judgment assessment was performed by three human-readers (radiologist (MD) and two MRI-technicians). Resnet152 was used for classification, and inference was performed on both datasets, with baseline input, and with missing data replaced by 1) generated images; 2) duplication of existing images; and 3) black images. STATISTICAL TESTS The similarity between the generated and the original images was evaluated using the peak-signal-to-noise-ratio (PSNR) and the structural-similarity-index-measure (SSIM). Classification results were evaluated using accuracy, F1-score and the Kolmogorov-Smirnov test and distance. RESULTS For baseline-state, the classification model reached to accuracy = 0.93,0.82 on the local and public-datasets. For the missing-data methods, high similarity was obtained between the generated and the original images with mean PSNR = 35.65,32.94 and SSIM = 0.87,0.91 on the local and public-datasets; 39% of the generated-images were labeled as real images by the human-readers. The classification model using generated-images to replace missing images produced the highest results with mean accuracy = 0.91,0.82 compared to 0.85,0.79 for duplicated and 0.77,0.68 for use of black images; DATA CONCLUSION: The feasibility for inference classification model on an MRI dataset with missing images using the Pix2pix-GAN generated images, was shown. The stability and generalization ability of the model was demonstrated by producing consistent results on two independent datasets. LEVEL OF EVIDENCE 3 TECHNICAL EFFICACY: Stage 5.
Collapse
Affiliation(s)
- Yael H Moshe
- Sagol Brain Institute, Tel Aviv Sourasky Medical Center, Tel Aviv, Israel
- Department of Mathematics, Bar Ilan University, Ramat Gan, Israel
| | - Yuval Buchsweiler
- Sagol Brain Institute, Tel Aviv Sourasky Medical Center, Tel Aviv, Israel
- The Iby and Aladar Fleischman Faculty of Engineering, Tel Aviv University, Tel Aviv, Israel
| | - Mina Teicher
- Department of Mathematics, Bar Ilan University, Ramat Gan, Israel
- Gonda Brain Research Center, Bar-Ilan University, Ramat Gan, Israel
| | - Moran Artzi
- Sagol Brain Institute, Tel Aviv Sourasky Medical Center, Tel Aviv, Israel
- Faculty of Medicine, Tel Aviv University, Tel Aviv, Israel
- Sagol School of Neuroscience, Tel Aviv University, Tel Aviv, Israel
| |
Collapse
|
18
|
Kundal K, Rao KV, Majumdar A, Kumar N, Kumar R. Comprehensive benchmarking of CNN-based tumor segmentation methods using multimodal MRI data. Comput Biol Med 2024; 178:108799. [PMID: 38925087 DOI: 10.1016/j.compbiomed.2024.108799] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/07/2024] [Revised: 06/12/2024] [Accepted: 06/19/2024] [Indexed: 06/28/2024]
Abstract
Magnetic resonance imaging (MRI) has become an essential and a frontline technique in the detection of brain tumor. However, segmenting tumors manually from scans is laborious and time-consuming. This has led to an increasing trend towards fully automated methods for precise tumor segmentation in MRI scans. Accurate tumor segmentation is crucial for improved diagnosis, treatment, and prognosis. This study benchmarks and evaluates four widely used CNN-based methods for brain tumor segmentation CaPTk, 2DVNet, EnsembleUNets, and ResNet50. Using 1251 multimodal MRI scans from the BraTS2021 dataset, we compared the performance of these methods against a reference dataset of segmented images assisted by radiologists. This comparison was conducted using segmented images directly and further by radiomic features extracted from the segmented images using pyRadiomics. Performance was assessed using the Dice Similarity Coefficient (DSC) and Hausdorff Distance (HD). EnsembleUNets excelled, achieving a DSC of 0.93 and an HD of 18, outperforming the other methods. Further comparative analysis of radiomic features confirmed EnsembleUNets as the most precise segmentation method, surpassing other methods. EnsembleUNets recorded a Concordance Correlation Coefficient (CCC) of 0.79, a Total Deviation Index (TDI) of 1.14, and a Root Mean Square Error (RMSE) of 0.53, underscoring its superior performance. We also performed validation on an independent dataset of 611 samples (UPENN-GBM), which further supported the accuracy of EnsembleUNets, with a DSC of 0.85 and an HD of 17.5. These findings provide valuable insight into the efficacy of EnsembleUNets, supporting informed decisions for accurate brain tumor segmentation.
Collapse
Affiliation(s)
- Kavita Kundal
- Department of Biotechnology, Indian Institute of Technology Hyderabad, Kandi, Telangana, 502284, India
| | - K Venkateswara Rao
- Department of Neurosurgical Oncology, Basavatarakam Indo American Cancer Hospital & Research Institute, Hyderabad, Telangana, 500034, India
| | - Arunabha Majumdar
- Department of Mathematics, Indian Institute of Technology Hyderabad, Kandi, Telangana, 502284, India
| | - Neeraj Kumar
- Department of Biotechnology, Indian Institute of Technology Hyderabad, Kandi, Telangana, 502284, India; Department of Liberal Arts, Indian Institute of Technology Hyderabad, Kandi, Telangana, 502284, India
| | - Rahul Kumar
- Department of Biotechnology, Indian Institute of Technology Hyderabad, Kandi, Telangana, 502284, India.
| |
Collapse
|
19
|
Nerella S, Bandyopadhyay S, Zhang J, Contreras M, Siegel S, Bumin A, Silva B, Sena J, Shickel B, Bihorac A, Khezeli K, Rashidi P. Transformers and large language models in healthcare: A review. Artif Intell Med 2024; 154:102900. [PMID: 38878555 DOI: 10.1016/j.artmed.2024.102900] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2023] [Revised: 05/28/2024] [Accepted: 05/30/2024] [Indexed: 08/09/2024]
Abstract
With Artificial Intelligence (AI) increasingly permeating various aspects of society, including healthcare, the adoption of the Transformers neural network architecture is rapidly changing many applications. Transformer is a type of deep learning architecture initially developed to solve general-purpose Natural Language Processing (NLP) tasks and has subsequently been adapted in many fields, including healthcare. In this survey paper, we provide an overview of how this architecture has been adopted to analyze various forms of healthcare data, including clinical NLP, medical imaging, structured Electronic Health Records (EHR), social media, bio-physiological signals, biomolecular sequences. Furthermore, which have also include the articles that used the transformer architecture for generating surgical instructions and predicting adverse outcomes after surgeries under the umbrella of critical care. Under diverse settings, these models have been used for clinical diagnosis, report generation, data reconstruction, and drug/protein synthesis. Finally, we also discuss the benefits and limitations of using transformers in healthcare and examine issues such as computational cost, model interpretability, fairness, alignment with human values, ethical implications, and environmental impact.
Collapse
Affiliation(s)
- Subhash Nerella
- Department of Biomedical Engineering, University of Florida, Gainesville, United States
| | | | - Jiaqing Zhang
- Department of Electrical and Computer Engineering, University of Florida, Gainesville, United States
| | - Miguel Contreras
- Department of Biomedical Engineering, University of Florida, Gainesville, United States
| | - Scott Siegel
- Department of Biomedical Engineering, University of Florida, Gainesville, United States
| | - Aysegul Bumin
- Department of Computer and Information Science and Engineering, University of Florida, Gainesville, United States
| | - Brandon Silva
- Department of Computer and Information Science and Engineering, University of Florida, Gainesville, United States
| | - Jessica Sena
- Department Of Computer Science, Universidade Federal de Minas Gerais, Belo Horizonte, Brazil
| | - Benjamin Shickel
- Department of Medicine, University of Florida, Gainesville, United States
| | - Azra Bihorac
- Department of Medicine, University of Florida, Gainesville, United States
| | - Kia Khezeli
- Department of Biomedical Engineering, University of Florida, Gainesville, United States
| | - Parisa Rashidi
- Department of Biomedical Engineering, University of Florida, Gainesville, United States.
| |
Collapse
|
20
|
Schmitz E, Guo Y, Wang J. Adaptive fine-tuning based transfer learning for the identification of MGMT promoter methylation status. Biomed Phys Eng Express 2024; 10:055018. [PMID: 39029475 PMCID: PMC11288403 DOI: 10.1088/2057-1976/ad6573] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/26/2024] [Revised: 06/06/2024] [Accepted: 07/19/2024] [Indexed: 07/21/2024]
Abstract
Background.Glioblastoma Multiforme (GBM) is an aggressive form of malignant brain tumor with a generally poor prognosis.O6-methylguanine-DNA methyltransferase (MGMT) promoter methylation has been shown to be a predictive bio-marker for resistance to treatment of GBM, but it is invasive and time-consuming to determine methylation status. There has been effort to predict the MGMT methylation status through analyzing MRI scans using machine learning, which only requires pre-operative scans that are already part of standard-of-care for GBM patients.Purpose.To improve the performance of conventional transfer learning in the identification of MGMT promoter methylation status, we developed a 3D SpotTune network with adaptive fine-tuning capability. Using the pretrained weights of MedicalNet with the SpotTune network, we compared its performance with a randomly initialized network for different combinations of MR modalities.Methods.Using a ResNet50 as the base network, three categories of networks are created: (1) A 3D SpotTune network to process volumetric MR images, (2) a network with randomly initialized weights, and (3) a network pre-trained on MedicalNet. These three networks are trained and evaluated using a public GBM dataset provided by the University of Pennsylvania. The MRI scans from 240 patients are used, with 11 different modalities corresponding to a set of perfusion, diffusion, and structural scans. The performance is evaluated using 5-fold cross validation with a hold-out testing dataset.Results.The SpotTune network showed better performance than the randomly initialized network. The best performing SpotTune model achieved an area under the Receiver Operating Characteristic curve (AUC), average precision of the precision-recall curve (AP), sensitivity, and specificity values of 0.6604, 0.6179, 0.6667, and 0.6061 respectively.Conclusions.SpotTune enables transfer learning to be adaptive to individual patients, resulting in improved performance in predicting MGMT promoter methylation status in GBM using equivalent MRI modalities as compared to a randomly initialized network.
Collapse
Affiliation(s)
- Erich Schmitz
- Advanced Imaging and Informatics for Radiation Therapy (AIRT) and Medical Artificial Intelligence and Automation (MAIA) Laboratory, Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, TX, United States of America
| | - Yunhui Guo
- Department of Computer Science, The University of Texas at Dallas, Richardson, TX, United States of America
| | - Jing Wang
- Advanced Imaging and Informatics for Radiation Therapy (AIRT) and Medical Artificial Intelligence and Automation (MAIA) Laboratory, Department of Radiation Oncology, University of Texas Southwestern Medical Center, Dallas, TX, United States of America
| |
Collapse
|
21
|
Ghadimi DJ, Vahdani AM, Karimi H, Ebrahimi P, Fathi M, Moodi F, Habibzadeh A, Khodadadi Shoushtari F, Valizadeh G, Mobarak Salari H, Saligheh Rad H. Deep Learning-Based Techniques in Glioma Brain Tumor Segmentation Using Multi-Parametric MRI: A Review on Clinical Applications and Future Outlooks. J Magn Reson Imaging 2024. [PMID: 39074952 DOI: 10.1002/jmri.29543] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2024] [Revised: 07/07/2024] [Accepted: 07/08/2024] [Indexed: 07/31/2024] Open
Abstract
This comprehensive review explores the role of deep learning (DL) in glioma segmentation using multiparametric magnetic resonance imaging (MRI) data. The study surveys advanced techniques such as multiparametric MRI for capturing the complex nature of gliomas. It delves into the integration of DL with MRI, focusing on convolutional neural networks (CNNs) and their remarkable capabilities in tumor segmentation. Clinical applications of DL-based segmentation are highlighted, including treatment planning, monitoring treatment response, and distinguishing between tumor progression and pseudo-progression. Furthermore, the review examines the evolution of DL-based segmentation studies, from early CNN models to recent advancements such as attention mechanisms and transformer models. Challenges in data quality, gradient vanishing, and model interpretability are discussed. The review concludes with insights into future research directions, emphasizing the importance of addressing tumor heterogeneity, integrating genomic data, and ensuring responsible deployment of DL-driven healthcare technologies. EVIDENCE LEVEL: N/A TECHNICAL EFFICACY: Stage 2.
Collapse
Affiliation(s)
- Delaram J Ghadimi
- School of Medicine, Shahid Beheshti University of Medical Sciences, Tehran, Iran
| | - Amir M Vahdani
- Image Guided Surgery Lab, Research Center for Biomedical Technologies and Robotics, Advanced Medical Technologies and Equipment Institute, Imam Khomeini Hospital Complex, Tehran University of Medical Sciences, Tehran, Iran
| | - Hanie Karimi
- School of Medicine, Tehran University of Medical Sciences, Tehran, Iran
| | - Pouya Ebrahimi
- Cardiovascular Diseases Research Institute, Tehran Heart Center, Tehran University of Medical Sciences, Tehran, Iran
| | - Mobina Fathi
- School of Medicine, Shahid Beheshti University of Medical Sciences, Tehran, Iran
| | - Farzan Moodi
- School of Medicine, Iran University of Medical Sciences, Tehran, Iran
- Quantitative MR Imaging and Spectroscopy Group (QMISG), Tehran University of Medical Sciences, Tehran, Iran
| | - Adrina Habibzadeh
- Student Research Committee, Fasa University of Medical Sciences, Fasa, Iran
| | | | - Gelareh Valizadeh
- Quantitative MR Imaging and Spectroscopy Group (QMISG), Tehran University of Medical Sciences, Tehran, Iran
| | - Hanieh Mobarak Salari
- Quantitative MR Imaging and Spectroscopy Group (QMISG), Tehran University of Medical Sciences, Tehran, Iran
| | - Hamidreza Saligheh Rad
- Quantitative MR Imaging and Spectroscopy Group (QMISG), Tehran University of Medical Sciences, Tehran, Iran
- Department of Medical Physics and Biomedical Engineering, Tehran University of Medical Sciences, Tehran, Iran
| |
Collapse
|
22
|
Reddy CKK, Reddy PA, Janapati H, Assiri B, Shuaib M, Alam S, Sheneamer A. A fine-tuned vision transformer based enhanced multi-class brain tumor classification using MRI scan imagery. Front Oncol 2024; 14:1400341. [PMID: 39091923 PMCID: PMC11291226 DOI: 10.3389/fonc.2024.1400341] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2024] [Accepted: 06/25/2024] [Indexed: 08/04/2024] Open
Abstract
Brain tumors occur due to the expansion of abnormal cell tissues and can be malignant (cancerous) or benign (not cancerous). Numerous factors such as the position, size, and progression rate are considered while detecting and diagnosing brain tumors. Detecting brain tumors in their initial phases is vital for diagnosis where MRI (magnetic resonance imaging) scans play an important role. Over the years, deep learning models have been extensively used for medical image processing. The current study primarily investigates the novel Fine-Tuned Vision Transformer models (FTVTs)-FTVT-b16, FTVT-b32, FTVT-l16, FTVT-l32-for brain tumor classification, while also comparing them with other established deep learning models such as ResNet50, MobileNet-V2, and EfficientNet - B0. A dataset with 7,023 images (MRI scans) categorized into four different classes, namely, glioma, meningioma, pituitary, and no tumor are used for classification. Further, the study presents a comparative analysis of these models including their accuracies and other evaluation metrics including recall, precision, and F1-score across each class. The deep learning models ResNet-50, EfficientNet-B0, and MobileNet-V2 obtained an accuracy of 96.5%, 95.1%, and 94.9%, respectively. Among all the FTVT models, FTVT-l16 model achieved a remarkable accuracy of 98.70% whereas other FTVT models FTVT-b16, FTVT-b32, and FTVT-132 achieved an accuracy of 98.09%, 96.87%, 98.62%, respectively, hence proving the efficacy and robustness of FTVT's in medical image processing.
Collapse
Affiliation(s)
- C. Kishor Kumar Reddy
- Department of Computer Science and Engineering, Stanley College of Engineering and Technology for Women, Hyderabad, India
| | - Pulakurthi Anaghaa Reddy
- Department of Computer Science and Engineering, Stanley College of Engineering and Technology for Women, Hyderabad, India
| | - Himaja Janapati
- Department of Computer Science and Engineering, Stanley College of Engineering and Technology for Women, Hyderabad, India
| | - Basem Assiri
- Department of Computer Science, College of Engineering and Computer Science, Jazan University, Jazan, Saudi Arabia
| | - Mohammed Shuaib
- Department of Computer Science, College of Engineering and Computer Science, Jazan University, Jazan, Saudi Arabia
| | - Shadab Alam
- Department of Computer Science, College of Engineering and Computer Science, Jazan University, Jazan, Saudi Arabia
| | - Abdullah Sheneamer
- Department of Computer Science, College of Engineering and Computer Science, Jazan University, Jazan, Saudi Arabia
| |
Collapse
|
23
|
Kwon J, Kim J, Park H. Leveraging segmentation-guided spatial feature embedding for overall survival prediction in glioblastoma with multimodal magnetic resonance imaging. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2024; 255:108338. [PMID: 39042996 DOI: 10.1016/j.cmpb.2024.108338] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/17/2024] [Revised: 07/17/2024] [Accepted: 07/17/2024] [Indexed: 07/25/2024]
Abstract
BACKGROUND AND OBJECTIVE Patients with glioblastoma have a five-year relative survival rate of less than 5 %. Thus, accurately predicting the overall survival (OS) of patients with glioblastoma is crucial for effective treatment planning. METHODS To fully leverage the imaging characteristics of glioblastomas, we propose a segmentation-guided regression method for predicting OS of patients with brain tumors using multimodal magnetic resonance imaging. Specifically, a brain tumor segmentation network was first pre-trained without leveraging survival information. Subsequently, the survival regression network was jointly trained with the guidance of brain tumor segmentation, focusing on tumor voxels and suppressing irrelevant backgrounds. RESULTS Our proposed framework, based on the well-known backbone of UNETR++, achieved a Dice score of 0.7910, Spearman correlation of 0.4112, and Harrell's concordance index of 0.6488. The model consistently showed promising results compared with baseline methods on two different datasets (BraTS and UCSF-PDGM). Furthermore, ablation studies on our training configurations demonstrated that both the pre-training segmentation network and contrastive loss significantly improved all metrics for OS prediction. CONCLUSIONS In this study, we propose a joint learning framework based on a pre-trained segmentation backbone for OS prediction by leveraging a brain tumor segmentation map. By utilizing a spatial feature map, our model can operate using a sliding-window approach, which can be adopted by varying the matrix sizes and resolutions of the input images.
Collapse
Affiliation(s)
- Junmo Kwon
- Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, 16419, South Korea; Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon, 16419, South Korea
| | - Jonghun Kim
- Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, 16419, South Korea
| | - Hyunjin Park
- Department of Electrical and Computer Engineering, Sungkyunkwan University, Suwon, 16419, South Korea; Center for Neuroscience Imaging Research, Institute for Basic Science, Suwon, 16419, South Korea.
| |
Collapse
|
24
|
Kumar K, Yeo AU, McIntosh L, Kron T, Wheeler G, Franich RD. Deep Learning Auto-Segmentation Network for Pediatric Computed Tomography Data Sets: Can We Extrapolate From Adults? Int J Radiat Oncol Biol Phys 2024; 119:1297-1306. [PMID: 38246249 DOI: 10.1016/j.ijrobp.2024.01.201] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/27/2023] [Revised: 12/10/2023] [Accepted: 01/07/2024] [Indexed: 01/23/2024]
Abstract
PURPOSE Artificial intelligence (AI)-based auto-segmentation models hold promise for enhanced efficiency and consistency in organ contouring for adaptive radiation therapy and radiation therapy planning. However, their performance on pediatric computed tomography (CT) data and cross-scanner compatibility remain unclear. This study aimed to evaluate the performance of AI-based auto-segmentation models trained on adult CT data when applied to pediatric data sets and explore the improvement in performance gained by including pediatric training data. It also examined their ability to accurately segment CT data acquired from different scanners. METHODS AND MATERIALS Using the nnU-Net framework, segmentation models were trained on data sets of adult, pediatric, and combined CT scans for 7 pelvic/thoracic organs. Each model was trained on 290 to 300 cases per category and organ. Training data sets included a combination of clinical data and several open repositories. The study incorporated a database of 459 pediatric (0-16 years) CT scans and 950 adults (>18 years), ensuring all scans had human expert ground-truth contours of the selected organs. Performance was evaluated based on Dice similarity coefficients (DSC) of the model-generated contours. RESULTS AI models trained exclusively on adult data underperformed on pediatric data, especially for the 0 to 2 age group: mean DSC was below 0.5 for the bladder and spleen. The addition of pediatric training data demonstrated significant improvement for all age groups, achieving a mean DSC of above 0.85 for all organs in every age group. Larger organs like the liver and kidneys maintained consistent performance for all models across age groups. No significant difference emerged in the cross-scanner performance evaluation, suggesting robust cross-scanner generalization. CONCLUSIONS For optimal segmentation across age groups, it is important to include pediatric data in the training of segmentation models. The successful cross-scanner generalization also supports the real-world clinical applicability of these AI models. This study emphasizes the significance of data set diversity in training robust AI systems for medical image interpretation tasks.
Collapse
Affiliation(s)
- Kartik Kumar
- Physical Sciences Department, Peter MacCallum Cancer Centre, Victoria, Australia; School of Science, RMIT University, Melbourne, Victoria, Australia
| | - Adam U Yeo
- Physical Sciences Department, Peter MacCallum Cancer Centre, Victoria, Australia; School of Science, RMIT University, Melbourne, Victoria, Australia; Sir Peter MacCallum Department of Oncology, University of Melbourne, Melbourne, Victoria, Australia
| | - Lachlan McIntosh
- Physical Sciences Department, Peter MacCallum Cancer Centre, Victoria, Australia; School of Science, RMIT University, Melbourne, Victoria, Australia
| | - Tomas Kron
- Physical Sciences Department, Peter MacCallum Cancer Centre, Victoria, Australia; School of Science, RMIT University, Melbourne, Victoria, Australia; Sir Peter MacCallum Department of Oncology, University of Melbourne, Melbourne, Victoria, Australia; Centre for Medical Radiation Physics, University of Wollongong, Wollongong, New South Wales, Australia
| | - Greg Wheeler
- Physical Sciences Department, Peter MacCallum Cancer Centre, Victoria, Australia; Sir Peter MacCallum Department of Oncology, University of Melbourne, Melbourne, Victoria, Australia
| | - Rick D Franich
- Physical Sciences Department, Peter MacCallum Cancer Centre, Victoria, Australia; School of Science, RMIT University, Melbourne, Victoria, Australia.
| |
Collapse
|
25
|
Lorenzo G, Ahmed SR, Hormuth DA, Vaughn B, Kalpathy-Cramer J, Solorio L, Yankeelov TE, Gomez H. Patient-Specific, Mechanistic Models of Tumor Growth Incorporating Artificial Intelligence and Big Data. Annu Rev Biomed Eng 2024; 26:529-560. [PMID: 38594947 DOI: 10.1146/annurev-bioeng-081623-025834] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/11/2024]
Abstract
Despite the remarkable advances in cancer diagnosis, treatment, and management over the past decade, malignant tumors remain a major public health problem. Further progress in combating cancer may be enabled by personalizing the delivery of therapies according to the predicted response for each individual patient. The design of personalized therapies requires the integration of patient-specific information with an appropriate mathematical model of tumor response. A fundamental barrier to realizing this paradigm is the current lack of a rigorous yet practical mathematical theory of tumor initiation, development, invasion, and response to therapy. We begin this review with an overview of different approaches to modeling tumor growth and treatment, including mechanistic as well as data-driven models based on big data and artificial intelligence. We then present illustrative examples of mathematical models manifesting their utility and discuss the limitations of stand-alone mechanistic and data-driven models. We then discuss the potential of mechanistic models for not only predicting but also optimizing response to therapy on a patient-specific basis. We describe current efforts and future possibilities to integrate mechanistic and data-driven models. We conclude by proposing five fundamental challenges that must be addressed to fully realize personalized care for cancer patients driven by computational models.
Collapse
Affiliation(s)
- Guillermo Lorenzo
- Oden Institute for Computational Engineering and Sciences, University of Texas, Austin, Texas, USA
- Department of Civil Engineering and Architecture, University of Pavia, Pavia, Italy
| | - Syed Rakin Ahmed
- Geisel School of Medicine, Dartmouth College, Hanover, New Hampshire, USA
- Health Sciences and Technology, Massachusetts Institute of Technology, Cambridge, Massachusetts, USA
- Harvard Graduate Program in Biophysics, Harvard Medical School, Harvard University, Cambridge, Massachusetts, USA
- Athinoula A. Martinos Center for Biomedical Imaging, Department of Radiology, Massachusetts General Hospital, Boston, Massachusetts, USA
| | - David A Hormuth
- Livestrong Cancer Institutes, University of Texas, Austin, Texas, USA
- Oden Institute for Computational Engineering and Sciences, University of Texas, Austin, Texas, USA
| | - Brenna Vaughn
- Weldon School of Biomedical Engineering, Purdue University, West Lafayette, Indiana, USA;
| | | | - Luis Solorio
- Weldon School of Biomedical Engineering, Purdue University, West Lafayette, Indiana, USA;
| | - Thomas E Yankeelov
- Department of Imaging Physics, MD Anderson Cancer Center, Houston, Texas, USA
- Department of Biomedical Engineering, Department of Oncology, and Department of Diagnostic Medicine, University of Texas, Austin, Texas, USA
- Livestrong Cancer Institutes, University of Texas, Austin, Texas, USA
- Oden Institute for Computational Engineering and Sciences, University of Texas, Austin, Texas, USA
| | - Hector Gomez
- School of Mechanical Engineering and Purdue Center for Cancer Research, Purdue University, West Lafayette, Indiana, USA
- Weldon School of Biomedical Engineering, Purdue University, West Lafayette, Indiana, USA;
| |
Collapse
|
26
|
Chukwujindu E, Faiz H, Ai-Douri S, Faiz K, De Sequeira A. Role of artificial intelligence in brain tumour imaging. Eur J Radiol 2024; 176:111509. [PMID: 38788610 DOI: 10.1016/j.ejrad.2024.111509] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/24/2024] [Revised: 04/29/2024] [Accepted: 05/13/2024] [Indexed: 05/26/2024]
Abstract
Artificial intelligence (AI) is a rapidly evolving field with many neuro-oncology applications. In this review, we discuss how AI can assist in brain tumour imaging, focusing on machine learning (ML) and deep learning (DL) techniques. We describe how AI can help in lesion detection, differential diagnosis, anatomic segmentation, molecular marker identification, prognostication, and pseudo-progression evaluation. We also cover AI applications in non-glioma brain tumours, such as brain metastasis, posterior fossa, and pituitary tumours. We highlight the challenges and limitations of AI implementation in radiology, such as data quality, standardization, and integration. Based on the findings in the aforementioned areas, we conclude that AI can potentially improve the diagnosis and treatment of brain tumours and provide a path towards personalized medicine and better patient outcomes.
Collapse
Affiliation(s)
| | | | | | - Khunsa Faiz
- McMaster University, Department of Radiology, L8S 4L8, Canada.
| | | |
Collapse
|
27
|
Zhou L, Jiang Y, Li W, Hu J, Zheng S. Shape-Scale Co-Awareness Network for 3D Brain Tumor Segmentation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2024; 43:2495-2508. [PMID: 38386578 DOI: 10.1109/tmi.2024.3368531] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/24/2024]
Abstract
The accurate segmentation of brain tumor is significant in clinical practice. Convolutional Neural Network (CNN)-based methods have made great progress in brain tumor segmentation due to powerful local modeling ability. However, brain tumors are frequently pattern-agnostic, i.e. variable in shape, size and location, which can not be effectively matched by traditional CNN-based methods with local and regular receptive fields. To address the above issues, we propose a shape-scale co-awareness network (S2CA-Net) for brain tumor segmentation, which can efficiently learn shape-aware and scale-aware features simultaneously to enhance pattern-agnostic representations. Primarily, three key components are proposed to accomplish the co-awareness of shape and scale. The Local-Global Scale Mixer (LGSM) decouples the extraction of local and global context by adopting the CNN-Former parallel structure, which contributes to obtaining finer hierarchical features. The Multi-level Context Aggregator (MCA) enriches the scale diversity of input patches by modeling global features across multiple receptive fields. The Multi-Scale Attentive Deformable Convolution (MS-ADC) learns the target deformation based on the multiscale inputs, which motivates the network to enforce feature constraints both in terms of scale and shape for optimal feature matching. Overall, LGSM and MCA focus on enhancing the scale-awareness of the network to cope with the size and location variations, while MS-ADC focuses on capturing deformation information for optimal shape matching. Finally, their effective integration prompts the network to perceive variations in shape and scale simultaneously, which can robustly tackle the variations in patterns of brain tumors. The experimental results on BraTS 2019, BraTS 2020, MSD BTS Task and BraTS2023-MEN show that S2CA-Net has superior overall performance in accuracy and efficiency compared to other state-of-the-art methods. Code: https://github.com/jiangyu945/S2CA-Net.
Collapse
|
28
|
Boelders SM, De Baene W, Postma E, Gehring K, Ong LL. Predicting Cognitive Functioning for Patients with a High-Grade Glioma: Evaluating Different Representations of Tumor Location in a Common Space. Neuroinformatics 2024; 22:329-352. [PMID: 38900230 PMCID: PMC11329426 DOI: 10.1007/s12021-024-09671-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 05/31/2024] [Indexed: 06/21/2024]
Abstract
Cognitive functioning is increasingly considered when making treatment decisions for patients with a brain tumor in view of a personalized onco-functional balance. Ideally, one can predict cognitive functioning of individual patients to make treatment decisions considering this balance. To make accurate predictions, an informative representation of tumor location is pivotal, yet comparisons of representations are lacking. Therefore, this study compares brain atlases and principal component analysis (PCA) to represent voxel-wise tumor location. Pre-operative cognitive functioning was predicted for 246 patients with a high-grade glioma across eight cognitive tests while using different representations of voxel-wise tumor location as predictors. Voxel-wise tumor location was represented using 13 different frequently-used population average atlases, 13 randomly generated atlases, and 13 representations based on PCA. ElasticNet predictions were compared between representations and against a model solely using tumor volume. Preoperative cognitive functioning could only partly be predicted from tumor location. Performances of different representations were largely similar. Population average atlases did not result in better predictions compared to random atlases. PCA-based representation did not clearly outperform other representations, although summary metrics indicated that PCA-based representations performed somewhat better in our sample. Representations with more regions or components resulted in less accurate predictions. Population average atlases possibly cannot distinguish between functionally distinct areas when applied to patients with a glioma. This stresses the need to develop and validate methods for individual parcellations in the presence of lesions. Future studies may test if the observed small advantage of PCA-based representations generalizes to other data.
Collapse
Affiliation(s)
- S M Boelders
- Department of Neurosurgery, Elisabeth-TweeSteden Hospital, Tilburg, The Netherlands
- Department of Cognitive Sciences and AI, Tilburg University, Tilburg, The Netherlands
| | - W De Baene
- Department of Cognitive Neuropsychology, Tilburg University Tilburg, Warandelaan 2, P. O. Box 90153, Tilburg, 5000 LE, The Netherlands
| | - E Postma
- Department of Cognitive Sciences and AI, Tilburg University, Tilburg, The Netherlands
| | - K Gehring
- Department of Neurosurgery, Elisabeth-TweeSteden Hospital, Tilburg, The Netherlands.
- Department of Cognitive Neuropsychology, Tilburg University Tilburg, Warandelaan 2, P. O. Box 90153, Tilburg, 5000 LE, The Netherlands.
| | - L L Ong
- Department of Cognitive Sciences and AI, Tilburg University, Tilburg, The Netherlands
| |
Collapse
|
29
|
Hermoza R, Nascimento JC, Carneiro G. Weakly-supervised preclinical tumor localization associated with survival prediction from lung cancer screening Chest X-ray images. Comput Med Imaging Graph 2024; 115:102395. [PMID: 38729092 DOI: 10.1016/j.compmedimag.2024.102395] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/21/2023] [Revised: 04/13/2024] [Accepted: 04/30/2024] [Indexed: 05/12/2024]
Abstract
In this paper, we hypothesize that it is possible to localize image regions of preclinical tumors in a Chest X-ray (CXR) image by a weakly-supervised training of a survival prediction model using a dataset containing CXR images of healthy patients and their time-to-death label. These visual explanations can empower clinicians in early lung cancer detection and increase patient awareness of their susceptibility to the disease. To test this hypothesis, we train a censor-aware multi-class survival prediction deep learning classifier that is robust to imbalanced training, where classes represent quantized number of days for time-to-death prediction. Such multi-class model allows us to use post-hoc interpretability methods, such as Grad-CAM, to localize image regions of preclinical tumors. For the experiments, we propose a new benchmark based on the National Lung Cancer Screening Trial (NLST) dataset to test weakly-supervised preclinical tumor localization and survival prediction models, and results suggest that our proposed method shows state-of-the-art C-index survival prediction and weakly-supervised preclinical tumor localization results. To our knowledge, this constitutes a pioneer approach in the field that is able to produce visual explanations of preclinical events associated with survival prediction results.
Collapse
Affiliation(s)
- Renato Hermoza
- Australian Institute for Machine Learning, The University of Adelaide, Australia.
| | - Jacinto C Nascimento
- Institute for Systems and Robotics (ISR/IST), LARSyS, Instituto Superior Técnico, Universidade de Lisboa, Portugal.
| | - Gustavo Carneiro
- Centre for Vision, Speech and Signal Processing (CVSSP), The University of Surrey, UK.
| |
Collapse
|
30
|
Belton N, Hagos MT, Lawlor A, Curran KM. Towards a unified approach for unsupervised brain MRI Motion Artefact Detection with few shot Anomaly Detection. Comput Med Imaging Graph 2024; 115:102391. [PMID: 38718561 DOI: 10.1016/j.compmedimag.2024.102391] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/07/2023] [Revised: 03/19/2024] [Accepted: 04/26/2024] [Indexed: 06/03/2024]
Abstract
Automated Motion Artefact Detection (MAD) in Magnetic Resonance Imaging (MRI) is a field of study that aims to automatically flag motion artefacts in order to prevent the requirement for a repeat scan. In this paper, we identify and tackle the three current challenges in the field of automated MAD; (1) reliance on fully-supervised training, meaning they require specific examples of Motion Artefacts (MA), (2) inconsistent use of benchmark datasets across different works and use of private datasets for testing and training of newly proposed MAD techniques and (3) a lack of sufficiently large datasets for MRI MAD. To address these challenges, we demonstrate how MAs can be identified by formulating the problem as an unsupervised Anomaly Detection (AD) task. We compare the performance of three State-of-the-Art AD algorithms DeepSVDD, Interpolated Gaussian Descriptor and FewSOME on two open-source Brain MRI datasets on the task of MAD and MA severity classification, with FewSOME achieving a MAD AUC >90% on both datasets and a Spearman Rank Correlation Coefficient of 0.8 on the task of MA severity classification. These models are trained in the few shot setting, meaning large Brain MRI datasets are not required to build robust MAD algorithms. This work also sets a standard protocol for testing MAD algorithms on open-source benchmark datasets. In addition to addressing these challenges, we demonstrate how our proposed 'anomaly-aware' scoring function improves FewSOME's MAD performance in the setting where one and two shots of the anomalous class are available for training. Code available at https://github.com/niamhbelton/Unsupervised-Brain-MRI-Motion-Artefact-Detection/.
Collapse
Affiliation(s)
- Niamh Belton
- Science Foundation Ireland Centre for Research Training in Machine Learning, Ireland; School of Medicine, University College Dublin, Ireland.
| | - Misgina Tsighe Hagos
- Science Foundation Ireland Centre for Research Training in Machine Learning, Ireland; School of Computer Science, University College Dublin, Ireland
| | - Aonghus Lawlor
- School of Computer Science, University College Dublin, Ireland; Insight Centre for Data Analytics, University College Dublin, Dublin, Ireland
| | - Kathleen M Curran
- Science Foundation Ireland Centre for Research Training in Machine Learning, Ireland; School of Medicine, University College Dublin, Ireland
| |
Collapse
|
31
|
Schäfer R, Nicke T, Höfener H, Lange A, Merhof D, Feuerhake F, Schulz V, Lotz J, Kiessling F. Overcoming data scarcity in biomedical imaging with a foundational multi-task model. NATURE COMPUTATIONAL SCIENCE 2024; 4:495-509. [PMID: 39030386 PMCID: PMC11288886 DOI: 10.1038/s43588-024-00662-z] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/21/2023] [Accepted: 06/17/2024] [Indexed: 07/21/2024]
Abstract
Foundational models, pretrained on a large scale, have demonstrated substantial success across non-medical domains. However, training these models typically requires large, comprehensive datasets, which contrasts with the smaller and more specialized datasets common in biomedical imaging. Here we propose a multi-task learning strategy that decouples the number of training tasks from memory requirements. We trained a universal biomedical pretrained model (UMedPT) on a multi-task database including tomographic, microscopic and X-ray images, with various labeling strategies such as classification, segmentation and object detection. The UMedPT foundational model outperformed ImageNet pretraining and previous state-of-the-art models. For classification tasks related to the pretraining database, it maintained its performance with only 1% of the original training data and without fine-tuning. For out-of-domain tasks it required only 50% of the original training data. In an external independent validation, imaging features extracted using UMedPT proved to set a new standard for cross-center transferability.
Collapse
Affiliation(s)
- Raphael Schäfer
- Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany
| | - Till Nicke
- Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany
| | - Henning Höfener
- Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany
| | - Annkristin Lange
- Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany
| | - Dorit Merhof
- Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany
- Institute of Image Analysis and Computer Vision, Faculty of Informatics and Data Science, University of Regensburg, Regensburg, Germany
| | - Friedrich Feuerhake
- Institute for Pathology, Hannover Medical School, Hanover, Germany
- Institute for Neuropathology, Medical Center, University of Freiburg, Freiburg, Germany
| | - Volkmar Schulz
- Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany
- Institute for Experimental Molecular Imaging, RWTH Aachen University, Aachen, Germany
| | - Johannes Lotz
- Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany.
| | - Fabian Kiessling
- Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany.
- Institute for Experimental Molecular Imaging, RWTH Aachen University, Aachen, Germany.
| |
Collapse
|
32
|
Lyu Q, Parreno-Centeno M, Papa JP, Öztürk-Isik E, Booth TC, Costen F. SurvNet: A low-complexity convolutional neural network for survival time classification of patients with glioblastoma. Heliyon 2024; 10:e32870. [PMID: 38988550 PMCID: PMC11234028 DOI: 10.1016/j.heliyon.2024.e32870] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2024] [Revised: 06/10/2024] [Accepted: 06/11/2024] [Indexed: 07/12/2024] Open
Abstract
Background and objective Malignant primary brain tumors cause the greatest number of years of life lost than any other cancer. Grade 4 glioma is particularly devastating: The median survival without any treatment is less than six months and with standard-of-care treatment is only 14.6 months. Accurate identification of the overall survival time of patients with brain tumors is of profound importance in many clinical applications. Automated image analytics with magnetic resonance imaging (MRI) can provide insights into the prognosis of patients with brain tumors. Methods In this paper, We propose SurvNet, a low-complexity deep learning architecture based on the convolutional neural network to classify the overall survival time of patients with brain tumors into long-time and short-time survival cohorts. Through the incorporation of diverse MRI modalities as inputs, we facilitate deep feature extraction at various anatomical sites, thereby augmenting the precision of predictive modeling. We compare SurvNet with the Inception V3, VGG 16 and ensemble CNN models on pre-operative magnetic resonance image datasets. We also analyzed the effect of segmented brain tumors and training data on the system performance. Results Several measures, such as accuracy, precision, and recall, are calculated to examine the perfor-mance of SurvNet on three-fold cross-validation. SurvNet with T1 MRI modality achieved a 62.7 % accuracy, compared with 52.9 % accuracy of the Inception V3 model, 58.5 % accuracy of the VGG 16 model, and 54.9 % of the ensemble CNN model. By increasing the MRI input modalities, SurvNet becomes more accurate and achieves 76.5 % accuracy with four MRI modalities. Combining the segmented data, SurvNet achieved the highest accuracy of 82.4 %. Conclusions The research results show that SurvNet achieves higher metrics such as accuracy and f1-score than the comparisons. Our research also proves that by using multiparametric MRI modalities, SurvNet is able to learn more image features and performs a better classification accuracy. We can conclude that SurvNet with the complete scenario, i.e., segmented data and four MRI modalities, achieved the best accuracy, showing the validity of segmentation information during the survival time prediction process.
Collapse
Affiliation(s)
- Qiyuan Lyu
- Electrical and Electronic Engineering Department, The University of Manchester, Oxford Rd, Manchester M13 9PL, United Kingdom
| | | | | | - Esin Öztürk-Isik
- Institute of Biomedical Engineering, Bogazici University, Istanbul, Turkey
| | - Thomas C. Booth
- Biomedical Engineering and Imaging Sciences, St. Thomas' Hospital, King's College London, London, United Kingdom
- Department of Neuroradiology, King's College Hospital NHS Foundation Trust, London, United Kingdom
| | - Fumie Costen
- Electrical and Electronic Engineering Department, The University of Manchester, Oxford Rd, Manchester M13 9PL, United Kingdom
- Image Processing Research Team, Centre for Advanced Photonics, RIKEN, Saitama, Japan
| |
Collapse
|
33
|
Guo B, Cao N, Zhang R, Yang P. GETNet: Group Normalization Shuffle and Enhanced Channel Self-Attention Network Based on VT-UNet for Brain Tumor Segmentation. Diagnostics (Basel) 2024; 14:1257. [PMID: 38928672 PMCID: PMC11203032 DOI: 10.3390/diagnostics14121257] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/23/2024] [Revised: 06/08/2024] [Accepted: 06/11/2024] [Indexed: 06/28/2024] Open
Abstract
Currently, brain tumors are extremely harmful and prevalent. Deep learning technologies, including CNNs, UNet, and Transformer, have been applied in brain tumor segmentation for many years and have achieved some success. However, traditional CNNs and UNet capture insufficient global information, and Transformer cannot provide sufficient local information. Fusing the global information from Transformer with the local information of convolutions is an important step toward improving brain tumor segmentation. We propose the Group Normalization Shuffle and Enhanced Channel Self-Attention Network (GETNet), a network combining the pure Transformer structure with convolution operations based on VT-UNet, which considers both global and local information. The network includes the proposed group normalization shuffle block (GNS) and enhanced channel self-attention block (ECSA). The GNS is used after the VT Encoder Block and before the downsampling block to improve information extraction. An ECSA module is added to the bottleneck layer to utilize the characteristics of the detailed features in the bottom layer effectively. We also conducted experiments on the BraTS2021 dataset to demonstrate the performance of our network. The Dice coefficient (Dice) score results show that the values for the regions of the whole tumor (WT), tumor core (TC), and enhancing tumor (ET) were 91.77, 86.03, and 83.64, respectively. The results show that the proposed model achieves state-of-the-art performance compared with more than eleven benchmarks.
Collapse
Affiliation(s)
- Bin Guo
- College of Information Science and Engineering, Hohai University, Nanjing 210098, China;
- College of Computer and Information Engineering, Xinjiang Agricultural University, Urumqi 830052, China; (R.Z.); (P.Y.)
| | - Ning Cao
- College of Information Science and Engineering, Hohai University, Nanjing 210098, China;
| | - Ruihao Zhang
- College of Computer and Information Engineering, Xinjiang Agricultural University, Urumqi 830052, China; (R.Z.); (P.Y.)
| | - Peng Yang
- College of Computer and Information Engineering, Xinjiang Agricultural University, Urumqi 830052, China; (R.Z.); (P.Y.)
| |
Collapse
|
34
|
Beser-Robles M, Castellá-Malonda J, Martínez-Gironés PM, Galiana-Bordera A, Ferrer-Lozano J, Ribas-Despuig G, Teruel-Coll R, Cerdá-Alberich L, Martí-Bonmatí L. Deep learning automatic semantic segmentation of glioblastoma multiforme regions on multimodal magnetic resonance images. Int J Comput Assist Radiol Surg 2024:10.1007/s11548-024-03205-z. [PMID: 38849632 DOI: 10.1007/s11548-024-03205-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2023] [Accepted: 05/30/2024] [Indexed: 06/09/2024]
Abstract
OBJECTIVES In patients having naïve glioblastoma multiforme (GBM), this study aims to assess the efficacy of Deep Learning algorithms in automating the segmentation of brain magnetic resonance (MR) images to accurately determine 3D masks for 4 distinct regions: enhanced tumor, peritumoral edema, non-enhanced/necrotic tumor, and total tumor. MATERIAL AND METHODS A 3D U-Net neural network algorithm was developed for semantic segmentation of GBM. The training dataset was manually delineated by a group of expert neuroradiologists on MR images from the Brain Tumor Segmentation Challenge 2021 (BraTS2021) image repository, as ground truth labels for diverse glioma (GBM and low-grade glioma) subregions across four MR sequences (T1w, T1w-contrast enhanced, T2w, and FLAIR) in 1251 patients. The in-house test was performed on 50 GBM patients from our cohort (PerProGlio project). By exploring various hyperparameters, the network's performance was optimized, and the most optimal parameter configuration was identified. The assessment of the optimized network's performance utilized Dice scores, precision, and sensitivity metrics. RESULTS Our adaptation of the 3D U-net with additional residual blocks demonstrated reliable performance on both the BraTS2021 dataset and the in-house PerProGlio cohort, employing only T1w-ce sequences for enhancement and non-enhanced/necrotic tumor models and T1w-ce + T2w + FLAIR for peritumoral edema and total tumor. The mean Dice scores (training and test) were 0.89 and 0.75; 0.75 and 0.64; 0.79 and 0.71; and 0.60 and 0.55, for total tumor, edema, enhanced tumor, and non-enhanced/necrotic tumor, respectively. CONCLUSIONS The results underscore the high precision with which our network can effectively segment GBM tumors and their distinct subregions. The level of accuracy achieved agrees with the coefficients recorded in previous GBM studies. In particular, our approach allows model specialization for each of the different tumor subregions employing only those MR sequences that provide value for segmentation.
Collapse
Affiliation(s)
- Maria Beser-Robles
- Biomedical Imaging Research Group (GIBI230), Instituto de Investigación Sanitaria La Fe, Avenida Fernando Abril Martorell, 106, 46026, Valencia, Spain.
| | | | - Pedro Miguel Martínez-Gironés
- Biomedical Imaging Research Group (GIBI230), Instituto de Investigación Sanitaria La Fe, Avenida Fernando Abril Martorell, 106, 46026, Valencia, Spain
| | - Adrián Galiana-Bordera
- Biomedical Imaging Research Group (GIBI230), Instituto de Investigación Sanitaria La Fe, Avenida Fernando Abril Martorell, 106, 46026, Valencia, Spain
| | - Jaime Ferrer-Lozano
- Department of Pathology, Hospital Universitario y Politécnico de La Fe, Valencia, Spain
| | - Gloria Ribas-Despuig
- Biomedical Imaging Research Group (GIBI230), Instituto de Investigación Sanitaria La Fe, Avenida Fernando Abril Martorell, 106, 46026, Valencia, Spain
| | - Regina Teruel-Coll
- Department of Radiology, Hospital Universitario y Politécnico de La Fe, Valencia, Spain
| | - Leonor Cerdá-Alberich
- Biomedical Imaging Research Group (GIBI230), Instituto de Investigación Sanitaria La Fe, Avenida Fernando Abril Martorell, 106, 46026, Valencia, Spain
| | - Luis Martí-Bonmatí
- Biomedical Imaging Research Group (GIBI230), Instituto de Investigación Sanitaria La Fe, Avenida Fernando Abril Martorell, 106, 46026, Valencia, Spain
- Department of Radiology, Hospital Universitario y Politécnico de La Fe, Valencia, Spain
| |
Collapse
|
35
|
Saran Raj S, Surendiran B, Raja SP. Designing a deep hybridized residual and SE model for MRI image-based brain tumor prediction. JOURNAL OF CLINICAL ULTRASOUND : JCU 2024; 52:588-599. [PMID: 38567722 DOI: 10.1002/jcu.23679] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/06/2024] [Revised: 02/27/2024] [Accepted: 03/19/2024] [Indexed: 06/15/2024]
Abstract
Deep learning techniques have become crucial in the detection of brain tumors but classifying numerous images is time-consuming and error-prone, impacting timely diagnosis. This can hinder the effectiveness of these techniques in detecting brain tumors in a timely manner. To address this limitation, this study introduces a novel brain tumor detection system. The main objective is to overcome the challenges associated with acquiring a large and well-classified dataset. The proposed approach involves generating synthetic Magnetic Resonance Imaging (MRI) images that mimic the patterns commonly found in brain MRI images. The system utilizes a dataset consisting of small images that are unbalanced in terms of class distribution. To enhance the accuracy of tumor detection, two deep learning models are employed. Using a hybrid ResNet+SE model, we capture feature distributions within unbalanced classes, creating a more balanced dataset. The second model, a tailored classifier identifies brain tumors in MRI images. The proposed method has shown promising results, achieving a high detection accuracy of 98.79%. This highlights the potential of the model as an efficient and cost-effective system for brain tumor detection.
Collapse
Affiliation(s)
- S Saran Raj
- Computer Science and Engineering, Vel Tech Rangarajan Dr. Sagunthala R&D Institute of Science and Technology, Chennai, India
| | - B Surendiran
- Department of Computer Science and Engineering, National Institute of Technology Puducherry, Puducherry, India
| | - S P Raja
- School of Computer Science and Engineering, Vellore Institute of Technology, Vellore, Tamilnadu, India
| |
Collapse
|
36
|
Ma S, Mathur P, Ju Z, Lawlor A, Dong R. Model-data-driven adversarial active learning for brain tumor segmentation. Comput Biol Med 2024; 176:108585. [PMID: 38761499 DOI: 10.1016/j.compbiomed.2024.108585] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/28/2023] [Revised: 04/28/2024] [Accepted: 05/08/2024] [Indexed: 05/20/2024]
Abstract
Active learning (AL) attempts to select informative samples in a dataset to minimize the number of required labels while maximizing the performance of the model. Current AL in segmentation tasks is limited to the expansion of popular classification-based methods including entropy, MC-dropout, etc. Meanwhile, most applications in the medical field are simply migrations that fail to consider the nature of medical images, such as high class imbalance, high domain difference, and data scarcity. In this study, we address these challenges and propose a novel AL framework for medical image segmentation task. Our approach introduces a pseudo-label-based filter addressing excessive blank patches in medical abnormalities segmentation tasks, e.g., lesions, and tumors, used before the AL selection. This filter helps reduce resource usage and allows the model to focus on selecting more informative samples. For the sample selection, we propose a novel query strategy that combines both model impact and data stability by employing adversarial attack. Furthermore, we harness the adversarial samples generated during the query process to enhance the robustness of the model. The experimental results verify our framework's effectiveness over various state-of-the-art methods. Our proposed method only needs less than 14% annotated patches in 3D brain MRI multiple sclerosis (MS) segmentation tasks and 20% for Low-Grade Glioma (LGG) tumor segmentation to achieve competitive results with full supervision. These promising outcomes not only improve performance but alleviate the time burden associated with expert annotation, thereby facilitating further advancements in the field of medical image segmentation. Our code is available at https://github.com/HelenMa9998/adversarial_active_learning.
Collapse
Affiliation(s)
- Siteng Ma
- School of Computer Science, University College Dublin, Dublin, D04 V1W8, Ireland.
| | - Prateek Mathur
- School of Computer Science, University College Dublin, Dublin, D04 V1W8, Ireland.
| | - Zheng Ju
- School of Computer Science, University College Dublin, Dublin, D04 V1W8, Ireland.
| | - Aonghus Lawlor
- School of Computer Science, University College Dublin, Dublin, D04 V1W8, Ireland.
| | - Ruihai Dong
- School of Computer Science, University College Dublin, Dublin, D04 V1W8, Ireland.
| |
Collapse
|
37
|
Lu X, Liang X, Liu W, Miao X, Guan X. ReeGAN: MRI image edge-preserving synthesis based on GANs trained with misaligned data. Med Biol Eng Comput 2024; 62:1851-1868. [PMID: 38396277 DOI: 10.1007/s11517-024-03035-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/20/2023] [Accepted: 01/27/2024] [Indexed: 02/25/2024]
Abstract
As a crucial medical examination technique, different modalities of magnetic resonance imaging (MRI) complement each other, offering multi-angle and multi-dimensional insights into the body's internal information. Therefore, research on MRI cross-modality conversion is of great significance, and many innovative techniques have been explored. However, most methods are trained on well-aligned data, and the impact of misaligned data has not received sufficient attention. Additionally, many methods focus on transforming the entire image and ignore crucial edge information. To address these challenges, we propose a generative adversarial network based on multi-feature fusion, which effectively preserves edge information while training on noisy data. Notably, we consider images with limited range random transformations as noisy labels and use an additional small auxiliary registration network to help the generator adapt to the noise distribution. Moreover, we inject auxiliary edge information to improve the quality of synthesized target modality images. Our goal is to find the best solution for cross-modality conversion. Comprehensive experiments and ablation studies demonstrate the effectiveness of the proposed method.
Collapse
Affiliation(s)
- Xiangjiang Lu
- Guangxi Key Lab of Multi-Source Information Mining & Security, School of Computer Science and Engineering & School of Software, Guangxi Normal University, Guilin, 541004, China.
| | - Xiaoshuang Liang
- Guangxi Key Lab of Multi-Source Information Mining & Security, School of Computer Science and Engineering & School of Software, Guangxi Normal University, Guilin, 541004, China
| | - Wenjing Liu
- Guangxi Key Lab of Multi-Source Information Mining & Security, School of Computer Science and Engineering & School of Software, Guangxi Normal University, Guilin, 541004, China
| | - Xiuxia Miao
- Guangxi Key Lab of Multi-Source Information Mining & Security, School of Computer Science and Engineering & School of Software, Guangxi Normal University, Guilin, 541004, China
| | - Xianglong Guan
- Guangxi Key Lab of Multi-Source Information Mining & Security, School of Computer Science and Engineering & School of Software, Guangxi Normal University, Guilin, 541004, China
| |
Collapse
|
38
|
Zoghby MM, Erickson BJ, Conte GM. Generative Adversarial Networks for Brain MRI Synthesis: Impact of Training Set Size on Clinical Application. JOURNAL OF IMAGING INFORMATICS IN MEDICINE 2024; 37:1228-1238. [PMID: 38366293 PMCID: PMC11169340 DOI: 10.1007/s10278-024-00976-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/12/2023] [Revised: 11/27/2023] [Accepted: 11/27/2023] [Indexed: 02/18/2024]
Abstract
We evaluated the impact of training set size on generative adversarial networks (GANs) to synthesize brain MRI sequences. We compared three sets of GANs trained to generate pre-contrast T1 (gT1) from post-contrast T1 and FLAIR (gFLAIR) from T2. The baseline models were trained on 135 cases; for this study, we used the same model architecture but a larger cohort of 1251 cases and two stopping rules, an early checkpoint (early models) and one after 50 epochs (late models). We tested all models on an independent dataset of 485 newly diagnosed gliomas. We compared the generated MRIs with the original ones using the structural similarity index (SSI) and mean squared error (MSE). We simulated scenarios where either the original T1, FLAIR, or both were missing and used their synthesized version as inputs for a segmentation model with the original post-contrast T1 and T2. We compared the segmentations using the dice similarity coefficient (DSC) for the contrast-enhancing area, non-enhancing area, and the whole lesion. For the baseline, early, and late models on the test set, for the gT1, median SSI was .957, .918, and .947; median MSE was .006, .014, and .008. For the gFLAIR, median SSI was .924, .908, and .915; median MSE was .016, .016, and .019. The range DSC was .625-.955, .420-.952, and .610-.954. Overall, GANs trained on a relatively small cohort performed similarly to those trained on a cohort ten times larger, making them a viable option for rare diseases or institutions with limited resources.
Collapse
Affiliation(s)
- M M Zoghby
- Department of Radiology, Mayo Clinic, Rochester, MN, USA
| | - B J Erickson
- Department of Radiology, Mayo Clinic, Rochester, MN, USA
| | - G M Conte
- Department of Radiology, Mayo Clinic, Rochester, MN, USA.
| |
Collapse
|
39
|
Xue H, Lu H, Wang Y, Li N, Wang G. MCE: Medical Cognition Embedded in 3D MRI feature extraction for advancing glioma staging. PLoS One 2024; 19:e0304419. [PMID: 38820482 PMCID: PMC11142489 DOI: 10.1371/journal.pone.0304419] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/14/2024] [Accepted: 05/12/2024] [Indexed: 06/02/2024] Open
Abstract
In recent years, various data-driven algorithms have been applied to the classification and staging of brain glioma MRI detection. However, the restricted availability of brain glioma MRI data in purely data-driven deep learning algorithms has presented challenges in extracting high-quality features and capturing their complex patterns. Moreover, the analysis methods designed for 2D data necessitate the selection of ideal tumor image slices, which does not align with practical clinical scenarios. Our research proposes an novel brain glioma staging model, Medical Cognition Embedded (MCE) model for 3D data. This model embeds knowledge characteristics into data-driven approaches to enhance the quality of feature extraction. Approach includes the following key components: (1) Deep feature extraction, drawing upon the imaging technical characteristics of different MRI sequences, has led to the design of two methods at both the algorithmic and strategic levels to mimic the learning process of real image interpretation by medical professionals during film reading; (2) We conduct an extensive Radiomics feature extraction, capturing relevant features such as texture, morphology, and grayscale distribution; (3) By referencing key points in radiological diagnosis, Radiomics feature experimental results, and the imaging characteristics of various MRI sequences, we manually create diagnostic features (Diag-Features). The efficacy of proposed methodology is rigorously evaluated on the publicly available BraTS2018 and BraTS2020 datasets. Comparing it to most well-known purely data-driven models, our method achieved higher accuracy, recall, and precision, reaching 96.14%, 93.4%, 97.06%, and 97.57%, 92.80%, 95.96%, respectively.
Collapse
Affiliation(s)
- Han Xue
- School of Computer Science and Engineering, Changchun University of Technology, Changchun, Jilin, China
- Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun, Jilin, China
- Jilin Provincial Smart Health Joint Innovation Laboratory for the New Generation of AI, Changchun, Jilin, China
| | - Huimin Lu
- School of Computer Science and Engineering, Changchun University of Technology, Changchun, Jilin, China
- Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun, Jilin, China
- Jilin Provincial Smart Health Joint Innovation Laboratory for the New Generation of AI, Changchun, Jilin, China
| | - Yilong Wang
- Jilin Provincial Smart Health Joint Innovation Laboratory for the New Generation of AI, Changchun, Jilin, China
- The First Hospital of Jilin University, Changchun, Jilin, China
- School of Mathematics and Statistics, Changchun University of Technology, Changchun, Jilin, China
| | - Niya Li
- Jilin Provincial Smart Health Joint Innovation Laboratory for the New Generation of AI, Changchun, Jilin, China
| | - Guizeng Wang
- School of Computer Science and Engineering, Changchun University of Technology, Changchun, Jilin, China
- Key Laboratory of Symbolic Computation and Knowledge Engineering of Ministry of Education, Jilin University, Changchun, Jilin, China
- Jilin Provincial Smart Health Joint Innovation Laboratory for the New Generation of AI, Changchun, Jilin, China
| |
Collapse
|
40
|
Chen B, Sun Q, Han Y, Liu B, Zhang J, Zhang Q. Adaptive cascaded transformer U-Net for MRI brain tumor segmentation. Phys Med Biol 2024; 69:115036. [PMID: 38636503 DOI: 10.1088/1361-6560/ad4081] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2023] [Accepted: 04/18/2024] [Indexed: 04/20/2024]
Abstract
Objective.Brain tumor segmentation on magnetic resonance imaging (MRI) plays an important role in assisting the diagnosis and treatment of cancer patients. Recently, cascaded U-Net models have achieved excellent performance via conducting coarse-to-fine segmentation of MRI brain tumors. However, they are still restricted by obvious global and local differences among various brain tumors, which are difficult to solve with conventional convolutions.Approach.To address the issue, this study proposes a novel Adaptive Cascaded Transformer U-Net (ACTransU-Net) for MRI brain tumor segmentation, which simultaneously integrates Transformer and dynamic convolution into a single cascaded U-Net architecture to adaptively capture global information and local details of brain tumors. ACTransU-Net first cascades two 3D U-Nets into a two-stage network to segment brain tumors from coarse to fine. Subsequently, it integrates omni-dimensional dynamic convolution modules into the second-stage shallow encoder and decoder, thereby enhancing the local detail representation of various brain tumors through dynamically adjusting convolution kernel parameters. Moreover, 3D Swin-Transformer modules are introduced into the second-stage deep encoder and decoder to capture image long-range dependencies, which helps adapt the global representation of brain tumors.Main results.Extensive experiment results evaluated on the public BraTS 2020 and BraTS 2021 brain tumor data sets demonstrate the effectiveness of ACTransU-Net, with average DSC of 84.96% and 91.37%, and HD95 of 10.81 and 7.31 mm, proving competitiveness with the state-of-the-art methods.Significance.The proposed method focuses on adaptively capturing both global information and local details of brain tumors, aiding physicians in their accurate diagnosis. In addition, it has the potential to extend ACTransU-Net for segmenting other types of lesions. The source code is available at:https://github.com/chenbn266/ACTransUnet.
Collapse
Affiliation(s)
- Bonian Chen
- School of Computer Science and Engineering, Dalian Minzu University, Dalian, 116600, People's Republic of China
| | - Qiule Sun
- Faculty of Electronic and Information Engineering, Dalian University of Technology, Dalian, 116024, People's Republic of China
| | - Yutong Han
- School of Computer Science and Engineering, Dalian Minzu University, Dalian, 116600, People's Republic of China
| | - Bin Liu
- International School of Information Science and Engineering (DUT-RUISE), Dalian University of Technology, Dalian, 116620, People's Republic of China
| | - Jianxin Zhang
- School of Computer Science and Engineering, Dalian Minzu University, Dalian, 116600, People's Republic of China
| | - Qiang Zhang
- Faculty of Electronic and Information Engineering, Dalian University of Technology, Dalian, 116024, People's Republic of China
| |
Collapse
|
41
|
Liu S, Wang H, Li S, Zhang C. Mixture-of-experts and semantic-guided network for brain tumor segmentation with missing MRI modalities. Med Biol Eng Comput 2024:10.1007/s11517-024-03130-y. [PMID: 38789839 DOI: 10.1007/s11517-024-03130-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2024] [Accepted: 05/14/2024] [Indexed: 05/26/2024]
Abstract
Accurate brain tumor segmentation with multi-modal MRI images is crucial, but missing modalities in clinical practice often reduce accuracy. The aim of this study is to propose a mixture-of-experts and semantic-guided network to tackle the issue of missing modalities in brain tumor segmentation. We introduce a transformer-based encoder with novel mixture-of-experts blocks. In each block, four modality experts aim for modality-specific feature learning. Learnable modality embeddings are employed to alleviate the negative effect of missing modalities. We also introduce a decoder guided by semantic information, designed to pay higher attention to various tumor regions. Finally, we conduct extensive comparison experiments with other models as well as ablation experiments to validate the performance of the proposed model on the BraTS2018 dataset. The proposed model can accurately segment brain tumor sub-regions even with missing modalities. It achieves an average Dice score of 0.81 for the whole tumor, 0.66 for the tumor core, and 0.52 for the enhanced tumor across the 15 modality combinations, achieving top or near-top results in most cases, while also exhibiting a lower computational cost. Our mixture-of-experts and sematic-guided network achieves accurate and reliable brain tumor segmentation results with missing modalities, indicating its significant potential for clinical applications. Our source code is already available at https://github.com/MaggieLSY/MESG-Net .
Collapse
Affiliation(s)
- Siyu Liu
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, 200032, China
- Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention, Shanghai, 200032, China
| | - Haoran Wang
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, 200032, China
- Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention, Shanghai, 200032, China
| | - Shiman Li
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, 200032, China
- Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention, Shanghai, 200032, China
| | - Chenxi Zhang
- Digital Medical Research Center, School of Basic Medical Sciences, Fudan University, Shanghai, 200032, China.
- Shanghai Key Lab of Medical Image Computing and Computer Assisted Intervention, Shanghai, 200032, China.
| |
Collapse
|
42
|
Wang J, Yin L. CNN-based glioma detection in MRI: A deep learning approach. Technol Health Care 2024:THC240158. [PMID: 39031408 DOI: 10.3233/thc-240158] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/22/2024]
Abstract
BACKGROUND More than a million people are affected by brain tumors each year; high-grade gliomas (HGGs) and low-grade gliomas (LGGs) present serious diagnostic and treatment hurdles, resulting in shortened life expectancies. Glioma segmentation is still a significant difficulty in clinical settings, despite improvements in Magnetic Resonance Imaging (MRI) and diagnostic tools. Convolutional neural networks (CNNs) have seen recent advancements that offer promise for increasing segmentation accuracy, addressing the pressing need for improved diagnostic and therapeutic approaches. OBJECTIVE The study intended to develop an automated glioma segmentation algorithm using CNN to accurately identify tumor components in MRI images. The goal was to match the accuracy of experienced radiologists with commercial instruments, hence improving diagnostic precision and quantification. METHODS 285 MRI scans of high-grade gliomas (HGGs) and low-grade gliomas (LGGs) were analyzed in the study. T1-weighted sequences were utilised for segmentation both pre-and post-contrast agent administration, along with T2-weighted sequences (with and without Fluid Attenuation by Inversion Recovery [FAIRE]). The segmentation performance was assessed with a U-Net network, renowned for its efficacy in medical image segmentation. DICE coefficients were computed for the tumour core with contrast enhancement, the entire tumour, and the tumour nucleus without contrast enhancement. RESULTS The U-Net network produced DICE values of 0.7331 for the tumour core with contrast enhancement, 0.8624 for the total tumour, and 0.7267 for the tumour nucleus without contrast enhancement. The results align with previous studies, demonstrating segmentation accuracy on par with professional radiologists and commercially accessible segmentation tools. CONCLUSION The study developed a CNN-based automated segmentation system for gliomas, achieving high accuracy in recognising glioma components in MRI images. The results confirm the ability of CNNs to enhance the accuracy of brain tumour diagnoses, suggesting a promising avenue for future research in medical imaging and diagnostics. This advancement is expected to improve diagnostic processes for clinicians and patients by providing more precise and quantitative results.
Collapse
|
43
|
Kazerooni AF, Khalili N, Liu X, Haldar D, Jiang Z, Anwar SM, Albrecht J, Adewole M, Anazodo U, Anderson H, Bagheri S, Baid U, Bergquist T, Borja AJ, Calabrese E, Chung V, Conte GM, Dako F, Eddy J, Ezhov I, Familiar A, Farahani K, Haldar S, Iglesias JE, Janas A, Johansen E, Jones BV, Kofler F, LaBella D, Lai HA, Van Leemput K, Li HB, Maleki N, McAllister AS, Meier Z, Menze B, Moawad AW, Nandolia KK, Pavaine J, Piraud M, Poussaint T, Prabhu SP, Reitman Z, Rodriguez A, Rudie JD, Shaikh IS, Shah LM, Sheth N, Shinohara RT, Tu W, Viswanathan K, Wang C, Ware JB, Wiestler B, Wiggins W, Zapaishchykova A, Aboian M, Bornhorst M, de Blank P, Deutsch M, Fouladi M, Hoffman L, Kann B, Lazow M, Mikael L, Nabavizadeh A, Packer R, Resnick A, Rood B, Vossough A, Bakas S, Linguraru MG. The Brain Tumor Segmentation (BraTS) Challenge 2023: Focus on Pediatrics (CBTN-CONNECT-DIPGR-ASNR-MICCAI BraTS-PEDs). ARXIV 2024:arXiv:2305.17033v7. [PMID: 37292481 PMCID: PMC10246083] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
Pediatric tumors of the central nervous system are the most common cause of cancer-related death in children. The five-year survival rate for high-grade gliomas in children is less than 20%. Due to their rarity, the diagnosis of these entities is often delayed, their treatment is mainly based on historic treatment concepts, and clinical trials require multi-institutional collaborations. The MICCAI Brain Tumor Segmentation (BraTS) Challenge is a landmark community benchmark event with a successful history of 12 years of resource creation for the segmentation and analysis of adult glioma. Here we present the CBTN-CONNECT-DIPGR-ASNR-MICCAI BraTS-PEDs 2023 challenge, which represents the first BraTS challenge focused on pediatric brain tumors with data acquired across multiple international consortia dedicated to pediatric neuro-oncology and clinical trials. The BraTS-PEDs 2023 challenge focuses on benchmarking the development of volumentric segmentation algorithms for pediatric brain glioma through standardized quantitative performance evaluation metrics utilized across the BraTS 2023 cluster of challenges. Models gaining knowledge from the BraTS-PEDs multi-parametric structural MRI (mpMRI) training data will be evaluated on separate validation and unseen test mpMRI dataof high-grade pediatric glioma. The CBTN-CONNECT-DIPGR-ASNR-MICCAI BraTS-PEDs 2023 challenge brings together clinicians and AI/imaging scientists to lead to faster development of automated segmentation techniques that could benefit clinical trials, and ultimately the care of children with brain tumors.
Collapse
Affiliation(s)
- Anahita Fathi Kazerooni
- Center for Data-Driven Discovery in Biomedicine (D3b), Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Neurosurgery, University of Pennsylvania, Philadelphia, PA, USA
- Center for AI & Data Science for Integrated Diagnostics (AI2D) and Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA
| | - Nastaran Khalili
- Center for Data-Driven Discovery in Biomedicine (D3b), Children’s Hospital of Philadelphia, Philadelphia, PA, USA
| | - Xinyang Liu
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children’s National Hospital, Washington DC, USA
| | - Debanjan Haldar
- Center for Data-Driven Discovery in Biomedicine (D3b), Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Neurosurgery, Thomas Jefferson University Hospital, PA, USA
| | - Zhifan Jiang
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children’s National Hospital, Washington DC, USA
| | - Syed Muhammed Anwar
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children’s National Hospital, Washington DC, USA
| | | | - Maruf Adewole
- Medical Artificial Intelligence (MAI) Lab, Crestview Radiology, Lagos, Nigeria
| | - Udunna Anazodo
- Montreal Neurological Institute (MNI), McGill University, Montreal, QC, Canada
| | - Hannah Anderson
- Center for Data-Driven Discovery in Biomedicine (D3b), Children’s Hospital of Philadelphia, Philadelphia, PA, USA
| | - Sina Bagheri
- Center for Data-Driven Discovery in Biomedicine (D3b), Children’s Hospital of Philadelphia, Philadelphia, PA, USA
| | - Ujjwal Baid
- Center for AI & Data Science for Integrated Diagnostics (AI2D) and Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
- Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | | | - Austin J. Borja
- Department of Neurosurgery at the University of Southern California, CA, USA
| | - Evan Calabrese
- Department of Radiology, Duke University Medical Center, USA
| | | | | | - Farouk Dako
- Center for Global Health, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | | | - Ivan Ezhov
- Department of Informatics, Technical University Munich, Germany
- TranslaTUM - Central Institute for Translational Cancer Research, Technical University of Munich, Germany
| | - Ariana Familiar
- Center for Data-Driven Discovery in Biomedicine (D3b), Children’s Hospital of Philadelphia, Philadelphia, PA, USA
| | - Keyvan Farahani
- Cancer Imaging Program, National Cancer Institute, National Institutes of Health, Bethesda, MD, USA
| | - Shuvanjan Haldar
- Center for Data-Driven Discovery in Biomedicine (D3b), Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Biomedical Engineering Rutgers University, New Brunswick, NJ, USA
| | - Juan Eugenio Iglesias
- Athinoula A Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Boston, MA, USA
| | | | - Elaine Johansen
- PrecisionFDA, U.S. Food and Drug Administration, Silver Spring, MD, USA
| | | | | | - Dominic LaBella
- Department of Radiation Oncology, Duke University Medical Center, USA
| | - Hollie Anne Lai
- Department of Radiology, Children’s Health Orange County, CA, USA
| | - Koen Van Leemput
- Department of Applied Mathematics and Computer Science, Technical University of Denmark, Denmark
| | - Hongwei Bran Li
- Athinoula A Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Boston, MA, USA
| | | | | | | | - Bjoern Menze
- Biomedical Image Analysis & Machine Learning, Department of Quantitative Biomedicine, University of Zurich, Switzerland
- Department of Neuroradiology, Technical University of Munich, Munich, Germany
| | | | - Khanak K Nandolia
- Department of Diagnostic and Interventional Radiology, All India Institute of Medical Sciences, Rishikesh, India
| | - Julija Pavaine
- Department of Paediatric Radiology, Royal Manchester Children’s Hospital, Manchester University Hospitals NHS Foundation Trust, University of Manchester, Manchester, UK
- Division of Informatics, Imaging & Data Sciences, School of Health Sciences, Faculty of Biology, Medicine and Health, University of Manchester, Manchester Academic Health Science Centre, Manchester, UK
| | | | - Tina Poussaint
- Dana-Farber Brigham Cancer Center & Boston Children’s Hospital, Boston, MA, USA
| | - Sanjay P Prabhu
- Division of Neuroradiology, Department of Radiology, Boston Children’s Hospital, Harvard Medical School, Boston, Massachusetts, USA
| | - Zachary Reitman
- Department of Radiation Oncology, Duke University Medical Center, USA
| | - Andres Rodriguez
- Department of Diagnostic & Interventional Imaging, Neuroradiology section. The University of Texas Health Sciences Center at Houston
| | | | | | | | - Nakul Sheth
- Department of Radiology, The Children’s Hospital of Philadelphia, PA, USA
| | - Russel Taki Shinohara
- Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania, Philadelphia, USA
| | - Wenxin Tu
- College of Arts and Sciences, University of Pennsylvania, PA, USA
| | - Karthik Viswanathan
- Center for Data-Driven Discovery in Biomedicine (D3b), Children’s Hospital of Philadelphia, Philadelphia, PA, USA
| | - Chunhao Wang
- Department of Radiation Oncology, Duke University Medical Center, USA
| | - Jeffrey B Ware
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Benedikt Wiestler
- Department of Neuroradiology, Technical University of Munich, Munich, Germany
| | - Walter Wiggins
- Department of Radiology, Duke University Medical Center, USA
| | - Anna Zapaishchykova
- Dana-Farber Brigham Cancer Center & Boston Children’s Hospital, Boston, MA, USA
| | | | - Miriam Bornhorst
- Brain Tumor Institute, Children’s National Hospital, Washington, DC, USA
| | - Peter de Blank
- Brain Tumor Center, Cincinnati Children’s Hospital, Cincinnati, OH, USA
| | - Michelle Deutsch
- Pediatric Neuro-Oncology Program, Nationwide Children’s Hospital, Columbus, OH, US
| | - Maryam Fouladi
- Pediatric Neuro-Oncology Program, Nationwide Children’s Hospital, Columbus, OH, US
| | - Lindsey Hoffman
- Center for Cancer and Blood Disorders, Phoenix Children’s Hospital, Phoenix, AZ, US
| | - Benjamin Kann
- Dana-Farber Brigham Cancer Center & Boston Children’s Hospital, Boston, MA, USA
| | - Margot Lazow
- Pediatric Neuro-Oncology Program, Nationwide Children’s Hospital, Columbus, OH, US
| | - Leonie Mikael
- Pediatric Neuro-Oncology Program, Nationwide Children’s Hospital, Columbus, OH, US
| | - Ali Nabavizadeh
- Center for Data-Driven Discovery in Biomedicine (D3b), Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Roger Packer
- Brain Tumor Institute, Children’s National Hospital, Washington, DC, USA
| | - Adam Resnick
- Center for Data-Driven Discovery in Biomedicine (D3b), Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Neurosurgery, University of Pennsylvania, Philadelphia, PA, USA
| | - Brian Rood
- Center for Cancer and Blood Disorders, Children’s National Hospital, Washington, DC, USA
| | - Arastoo Vossough
- Center for Data-Driven Discovery in Biomedicine (D3b), Children’s Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Radiology, The Children’s Hospital of Philadelphia, PA, USA
| | - Spyridon Bakas
- Center for AI & Data Science for Integrated Diagnostics (AI2D) and Center for Biomedical Image Computing and Analytics (CBICA), University of Pennsylvania, Philadelphia, PA, USA
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
- Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Marius George Linguraru
- Sheikh Zayed Institute for Pediatric Surgical Innovation, Children’s National Hospital, Washington DC, USA
- Departments of Radiology and Pediatrics, George Washington University School of Medicine and Health Sciences, Washington, DC, USA
| |
Collapse
|
44
|
Luque L, Skogen K, MacIntosh BJ, Emblem KE, Larsson C, Bouget D, Helland RH, Reinertsen I, Solheim O, Schellhorn T, Vardal J, Mireles EEM, Vik-Mo EO, Bjørnerud A. Standardized evaluation of the extent of resection in glioblastoma with automated early post-operative segmentation. FRONTIERS IN RADIOLOGY 2024; 4:1357341. [PMID: 38840717 PMCID: PMC11150796 DOI: 10.3389/fradi.2024.1357341] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/17/2023] [Accepted: 05/06/2024] [Indexed: 06/07/2024]
Abstract
Standard treatment of patients with glioblastoma includes surgical resection of the tumor. The extent of resection (EOR) achieved during surgery significantly impacts prognosis and is used to stratify patients in clinical trials. In this study, we developed a U-Net-based deep-learning model to segment contrast-enhancing tumor on post-operative MRI exams taken within 72 h of resection surgery and used these segmentations to classify the EOR as either maximal or submaximal. The model was trained on 122 multiparametric MRI scans from our institution and achieved a mean Dice score of 0.52 ± 0.03 on an external dataset (n = 248), a performance -on par with the interrater agreement between expert annotators as reported in literature. We obtained an EOR classification precision/recall of 0.72/0.78 on the internal test dataset (n = 462) and 0.90/0.87 on the external dataset. Furthermore, Kaplan-Meier curves were used to compare the overall survival between patients with maximal and submaximal resection in the internal test dataset, as determined by either clinicians or the model. There was no significant difference between the survival predictions using the model's and clinical EOR classification. We find that the proposed segmentation model is capable of reliably classifying the EOR of glioblastoma tumors on early post-operative MRI scans. Moreover, we show that stratification of patients based on the model's predictions offers at least the same prognostic value as when done by clinicians.
Collapse
Affiliation(s)
- Lidia Luque
- Computational Radiology and Artificial Intelligence (CRAI), Department of Physics and Computational Radiology, Clinic for Radiology and Nuclear Medicine, Oslo University Hospital, Oslo, Norway
- Department of Physics, University of Oslo, Oslo, Norway
- Department of Physics and Computational Radiology, Clinic for Radiology and Nuclear Medicine, Oslo University Hospital, Oslo, Norway
| | - Karoline Skogen
- Department of Radiology, Clinic for Radiology and Nuclear Medicine, Oslo University Hospital, Oslo, Norway
| | - Bradley J. MacIntosh
- Computational Radiology and Artificial Intelligence (CRAI), Department of Physics and Computational Radiology, Clinic for Radiology and Nuclear Medicine, Oslo University Hospital, Oslo, Norway
- Department of Physics and Computational Radiology, Clinic for Radiology and Nuclear Medicine, Oslo University Hospital, Oslo, Norway
- Department of Medical Biophysics, University of Toronto, Toronto, ON, Canada
- Sandra E Black Centre for Brain Resilience and Recovery, Sunnybrook Research Institute, Toronto, ON, Canada
| | - Kyrre E. Emblem
- Department of Physics and Computational Radiology, Clinic for Radiology and Nuclear Medicine, Oslo University Hospital, Oslo, Norway
- Institute of Clinical Medicine, Faculty of Medicine, University of Oslo, Oslo, Norway
| | - Christopher Larsson
- Institute of Clinical Medicine, Faculty of Medicine, University of Oslo, Oslo, Norway
- Department of Neurosurgery, Oslo University Hospital, Oslo, Norway
| | - David Bouget
- Department of Health Research, SINTEF Digital, Trondheim, Norway
| | - Ragnhild Holden Helland
- Department of Health Research, SINTEF Digital, Trondheim, Norway
- Department of Circulation and Medical Imaging, Norwegian University of Science and Technology (NTNU), Trondheim, Norway
| | - Ingerid Reinertsen
- Department of Health Research, SINTEF Digital, Trondheim, Norway
- Department of Circulation and Medical Imaging, Norwegian University of Science and Technology (NTNU), Trondheim, Norway
| | - Ole Solheim
- Department of Neurosurgery, St. Olavs Hospital, Trondheim University Hospital, Trondheim, Norway
- Department of Neuromedicine and Movement Science, Norwegian University of Science and Technology (NTNU), Trondheim, Norway
| | - Till Schellhorn
- Computational Radiology and Artificial Intelligence (CRAI), Department of Physics and Computational Radiology, Clinic for Radiology and Nuclear Medicine, Oslo University Hospital, Oslo, Norway
- Department of Radiology, Clinic for Radiology and Nuclear Medicine, Oslo University Hospital, Oslo, Norway
| | - Jonas Vardal
- Department of Radiology, Clinic for Radiology and Nuclear Medicine, Oslo University Hospital, Oslo, Norway
| | - Eduardo E. M. Mireles
- Institute of Clinical Medicine, Faculty of Medicine, University of Oslo, Oslo, Norway
- Department of Neurosurgery, Oslo University Hospital, Oslo, Norway
| | - Einar O. Vik-Mo
- Institute of Clinical Medicine, Faculty of Medicine, University of Oslo, Oslo, Norway
- Department of Neurosurgery, Oslo University Hospital, Oslo, Norway
| | - Atle Bjørnerud
- Computational Radiology and Artificial Intelligence (CRAI), Department of Physics and Computational Radiology, Clinic for Radiology and Nuclear Medicine, Oslo University Hospital, Oslo, Norway
- Department of Physics, University of Oslo, Oslo, Norway
- Department of Physics and Computational Radiology, Clinic for Radiology and Nuclear Medicine, Oslo University Hospital, Oslo, Norway
- Center for Lifespan Changes in Brain and Cognition, University of Oslo, Oslo, Norway
| |
Collapse
|
45
|
Li L, Yu J, Li Y, Wei J, Fan R, Wu D, Ye Y. Multi-sequence generative adversarial network: better generation for enhanced magnetic resonance imaging images. Front Comput Neurosci 2024; 18:1365238. [PMID: 38841427 PMCID: PMC11151883 DOI: 10.3389/fncom.2024.1365238] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2024] [Accepted: 03/27/2024] [Indexed: 06/07/2024] Open
Abstract
Introduction MRI is one of the commonly used diagnostic methods in clinical practice, especially in brain diseases. There are many sequences in MRI, but T1CE images can only be obtained by using contrast agents. Many patients (such as cancer patients) must undergo alignment of multiple MRI sequences for diagnosis, especially the contrast-enhanced magnetic resonance sequence. However, some patients such as pregnant women, children, etc. find it difficult to use contrast agents to obtain enhanced sequences, and contrast agents have many adverse reactions, which can pose a significant risk. With the continuous development of deep learning, the emergence of generative adversarial networks makes it possible to extract features from one type of image to generate another type of image. Methods We propose a generative adversarial network model with multimodal inputs and end-to-end decoding based on the pix2pix model. For the pix2pix model, we used four evaluation metrics: NMSE, RMSE, SSIM, and PNSR to assess the effectiveness of our generated model. Results Through statistical analysis, we compared our proposed new model with pix2pix and found significant differences between the two. Our model outperformed pix2pix, with higher SSIM and PNSR, lower NMSE and RMSE. We also found that the input of T1W images and T2W images had better effects than other combinations, providing new ideas for subsequent work on generating magnetic resonance enhancement sequence images. By using our model, it is possible to generate magnetic resonance enhanced sequence images based on magnetic resonance non-enhanced sequence images. Discussion This has significant implications as it can greatly reduce the use of contrast agents to protect populations such as pregnant women and children who are contraindicated for contrast agents. Additionally, contrast agents are relatively expensive, and this generation method may bring about substantial economic benefits.
Collapse
Affiliation(s)
- Leizi Li
- South China Normal University-Panyu Central Hospital Joint Laboratory of Basic and Translational Medical Research, Guangzhou Panyu Central Hospital, Guangzhou, China
- Guangzhou Key Laboratory of Subtropical Biodiversity and Biomonitoring and Guangdong Provincial Engineering Technology Research Center for Drug and Food Biological Resources Processing and Comprehensive Utilization, School of Life Sciences, South China Normal University, Guangzhou, China
| | - Jingchun Yu
- Guangzhou Key Laboratory of Subtropical Biodiversity and Biomonitoring and Guangdong Provincial Engineering Technology Research Center for Drug and Food Biological Resources Processing and Comprehensive Utilization, School of Life Sciences, South China Normal University, Guangzhou, China
| | - Yijin Li
- Guangzhou Key Laboratory of Subtropical Biodiversity and Biomonitoring and Guangdong Provincial Engineering Technology Research Center for Drug and Food Biological Resources Processing and Comprehensive Utilization, School of Life Sciences, South China Normal University, Guangzhou, China
| | - Jinbo Wei
- South China Normal University-Panyu Central Hospital Joint Laboratory of Basic and Translational Medical Research, Guangzhou Panyu Central Hospital, Guangzhou, China
| | - Ruifang Fan
- Guangzhou Key Laboratory of Subtropical Biodiversity and Biomonitoring and Guangdong Provincial Engineering Technology Research Center for Drug and Food Biological Resources Processing and Comprehensive Utilization, School of Life Sciences, South China Normal University, Guangzhou, China
| | - Dieen Wu
- South China Normal University-Panyu Central Hospital Joint Laboratory of Basic and Translational Medical Research, Guangzhou Panyu Central Hospital, Guangzhou, China
| | - Yufeng Ye
- South China Normal University-Panyu Central Hospital Joint Laboratory of Basic and Translational Medical Research, Guangzhou Panyu Central Hospital, Guangzhou, China
- Medical Imaging Institute of Panyu, Guangzhou, China
| |
Collapse
|
46
|
LaBella D, Khanna O, McBurney-Lin S, Mclean R, Nedelec P, Rashid AS, Tahon NH, Altes T, Baid U, Bhalerao R, Dhemesh Y, Floyd S, Godfrey D, Hilal F, Janas A, Kazerooni A, Kent C, Kirkpatrick J, Kofler F, Leu K, Maleki N, Menze B, Pajot M, Reitman ZJ, Rudie JD, Saluja R, Velichko Y, Wang C, Warman PI, Sollmann N, Diffley D, Nandolia KK, Warren DI, Hussain A, Fehringer JP, Bronstein Y, Deptula L, Stein EG, Taherzadeh M, Portela de Oliveira E, Haughey A, Kontzialis M, Saba L, Turner B, Brüßeler MMT, Ansari S, Gkampenis A, Weiss DM, Mansour A, Shawali IH, Yordanov N, Stein JM, Hourani R, Moshebah MY, Abouelatta AM, Rizvi T, Willms K, Martin DC, Okar A, D'Anna G, Taha A, Sharifi Y, Faghani S, Kite D, Pinho M, Haider MA, Alonso-Basanta M, Villanueva-Meyer J, Rauschecker AM, Nada A, Aboian M, Flanders A, Bakas S, Calabrese E. A multi-institutional meningioma MRI dataset for automated multi-sequence image segmentation. Sci Data 2024; 11:496. [PMID: 38750041 PMCID: PMC11096318 DOI: 10.1038/s41597-024-03350-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/20/2024] [Accepted: 05/07/2024] [Indexed: 05/18/2024] Open
Abstract
Meningiomas are the most common primary intracranial tumors and can be associated with significant morbidity and mortality. Radiologists, neurosurgeons, neuro-oncologists, and radiation oncologists rely on brain MRI for diagnosis, treatment planning, and longitudinal treatment monitoring. However, automated, objective, and quantitative tools for non-invasive assessment of meningiomas on multi-sequence MR images are not available. Here we present the BraTS Pre-operative Meningioma Dataset, as the largest multi-institutional expert annotated multilabel meningioma multi-sequence MR image dataset to date. This dataset includes 1,141 multi-sequence MR images from six sites, each with four structural MRI sequences (T2-, T2/FLAIR-, pre-contrast T1-, and post-contrast T1-weighted) accompanied by expert manually refined segmentations of three distinct meningioma sub-compartments: enhancing tumor, non-enhancing tumor, and surrounding non-enhancing T2/FLAIR hyperintensity. Basic demographic data are provided including age at time of initial imaging, sex, and CNS WHO grade. The goal of releasing this dataset is to facilitate the development of automated computational methods for meningioma segmentation and expedite their incorporation into clinical practice, ultimately targeting improvement in the care of meningioma patients.
Collapse
Affiliation(s)
- Dominic LaBella
- Department of Radiation Oncology, Duke University Medical Center, Durham, NC, USA
| | - Omaditya Khanna
- Department of Neurosurgery, Thomas Jefferson University, Philadelphia, PA, USA
| | - Shan McBurney-Lin
- Center for Intelligent Imaging (ci2), Department of Radiology & Biomedical Imaging, University of California San Francisco (UCSF), San Francisco, CA, USA
| | | | - Pierre Nedelec
- Center for Intelligent Imaging (ci2), Department of Radiology & Biomedical Imaging, University of California San Francisco (UCSF), San Francisco, CA, USA
| | - Arif S Rashid
- Department of Radiation Oncology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | | | | | - Ujjwal Baid
- Division of Computational Pathology, Department of Pathology and Laboratory Medicine, School of Medicine, Indiana University, Indianapolis, IN, USA
| | - Radhika Bhalerao
- Center for Intelligent Imaging (ci2), Department of Radiology & Biomedical Imaging, University of California San Francisco (UCSF), San Francisco, CA, USA
| | | | - Scott Floyd
- Department of Radiation Oncology, Duke University Medical Center, Durham, NC, USA
| | - Devon Godfrey
- Department of Radiation Oncology, Duke University Medical Center, Durham, NC, USA
| | | | | | - Anahita Kazerooni
- Center for Data-Driven Discovery in Biomedicine (D3b), The Children's Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Neurosurgery, The Children's Hospital of Philadelphia, Philadelphia, PA, USA
- Department of Neurosurgery, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Collin Kent
- Department of Radiation Oncology, Duke University Medical Center, Durham, NC, USA
| | - John Kirkpatrick
- Department of Radiation Oncology, Duke University Medical Center, Durham, NC, USA
| | - Florian Kofler
- Helmholtz AI, Helmholtz Munich, Neuherberg, Germany
- Department of Computer Science, TUM School of Computation, Information and Technology, Technical University of Munich, Munich, Germany
- TranslaTUM - Central Institute for Translational Cancer Research, Tech nical University of Munich, Munich, Germany
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Kevin Leu
- Center for Intelligent Imaging (ci2), Department of Radiology & Biomedical Imaging, University of California San Francisco (UCSF), San Francisco, CA, USA
| | | | | | - Maxence Pajot
- Center for Intelligent Imaging (ci2), Department of Radiology & Biomedical Imaging, University of California San Francisco (UCSF), San Francisco, CA, USA
| | - Zachary J Reitman
- Department of Radiation Oncology, Duke University Medical Center, Durham, NC, USA
| | - Jeffrey D Rudie
- Department of Radiology, University of California San Diego, San Diego, CA, USA
| | - Rachit Saluja
- Department of Radiology, Cornell University, Ithaca, NY, USA
| | - Yury Velichko
- Department of Radiology, Northwestern University, Evanston, IL, USA
| | - Chunhao Wang
- Department of Radiation Oncology, Duke University Medical Center, Durham, NC, USA
| | - Pranav I Warman
- Duke University Medical Center, School of Medicine, Durham, NC, USA
| | - Nico Sollmann
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
- Department of Diagnostic and Interventional Radiology, University Hospital Ulm, Ulm, Germany
- TUM-Neuroimaging Center, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | | | - Khanak K Nandolia
- Department of Diagnostic and Interventional Radiology, All India Institute of Medical Sciences, Rishikesh, India
| | - Daniel I Warren
- Department of Neuroradiology, Washington University, St. Louis, MO, USA
| | - Ali Hussain
- University of Rochester Medical Center, Rochester, NY, USA
| | - John Pascal Fehringer
- Faculty of Medicine, Jena University Hospital, Friedrich Schiller University Jena, Jena, Germany
| | | | - Lisa Deptula
- Ross University School of Medicine, Bridgetown, Barbados
| | - Evan G Stein
- Department of Radiology, New York University Grossman School of Medicine, New York, NY, USA
| | | | | | - Aoife Haughey
- Department of Neuroradiology, JDMI, University of Toronto, Toronto, TO, Canada
| | | | - Luca Saba
- Department of Radiology, Azienda Ospedaliero Universitaria of Cagliari-Polo di Monserrato, Cagliari, Italy
| | | | | | | | | | - David Maximilian Weiss
- Department of Neuroradiology, University Hospital Essen, Essen, North Rhine-Westphalia, Germany
| | | | - Islam H Shawali
- Department of Radiology, Kasr Alainy, Cairo University, Cairo, Egypt
| | - Nikolay Yordanov
- Faculty of Medicine, Medical University of Sofia, Sofia, Bulgaria
| | - Joel M Stein
- Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Roula Hourani
- Department of Radiology, American University of Beirut Medical center, Beirut, Lebanon
| | | | | | - Tanvir Rizvi
- Department of Radiology and Medical Imaging, University of Virginia Health, Charlottesville, VA, USA
| | | | - Dann C Martin
- Department of Radiology and Radiologic Sciences, Vanderbilt University Medical Center, Nashville, TN, USA
| | - Abdullah Okar
- Faculty of Medicine, Hamburg University, Hamburg, Germany
| | - Gennaro D'Anna
- Neuroimaging Unit, ASST Ovest Milanese, Legnano, Milan, Italy
| | - Ahmed Taha
- University of Manitoba, Winnipeg, Manitoba, Canada
| | - Yasaman Sharifi
- Department of Radiology, School of Medicine, Iran University of Medical Sciences, Tehran, Iran
| | - Shahriar Faghani
- Radiology Informatics Lab, Department of Radiology, Mayo Clinic, Rochester, MN, USA
| | - Dominic Kite
- Department of Radiology, University Hospitals Bristol and Weston NHS Foundation Trust, Bristol, United Kingdom
| | - Marco Pinho
- Department of Radiology, University of Texas Southwestern Medical Center, Dallas, TX, USA
| | | | - Michelle Alonso-Basanta
- Department of Radiation Oncology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Javier Villanueva-Meyer
- Center for Intelligent Imaging (ci2), Department of Radiology & Biomedical Imaging, University of California San Francisco (UCSF), San Francisco, CA, USA
| | - Andreas M Rauschecker
- Center for Intelligent Imaging (ci2), Department of Radiology & Biomedical Imaging, University of California San Francisco (UCSF), San Francisco, CA, USA
| | - Ayman Nada
- University of Missouri, Columbia, MO, USA
| | - Mariam Aboian
- Department of Radiology, Children's Hospital of Philadelphia (CHOP), Philadelphia, PA, USA
| | - Adam Flanders
- Department of Radiology, Thomas Jefferson University, Philadelphia, PA, USA
| | - Spyridon Bakas
- Division of Computational Pathology, Department of Pathology and Laboratory Medicine, School of Medicine, Indiana University, Indianapolis, IN, USA
- Department of Neurological Surgery, School of Medicine, Indiana University, Indianapolis, IN, USA
- Department of Radiology and Imaging Sciences, School of Medicine, Indiana University, Indianapolis, IN, USA
| | - Evan Calabrese
- Department of Radiology, Duke University Medical Center, Durham, NC, USA.
| |
Collapse
|
47
|
Koçak B, Yüzkan S, Mutlu S, Karagülle M, Kala A, Kadıoğlu M, Solak S, Sunman Ş, Temiz ZH, Ganiyusufoğlu AK. Influence of image preprocessing on the segmentation-based reproducibility of radiomic features: in vivo experiments on discretization and resampling parameters. Diagn Interv Radiol 2024; 30:152-162. [PMID: 38073244 PMCID: PMC11095065 DOI: 10.4274/dir.2023.232543] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2023] [Accepted: 11/14/2023] [Indexed: 05/15/2024]
Abstract
PURPOSE To systematically investigate the impact of image preprocessing parameters on the segmentation-based reproducibility of magnetic resonance imaging (MRI) radiomic features. METHODS The MRI scans of 50 patients were included from the multi-institutional Brain Tumor Segmentation 2021 public glioma dataset. Whole tumor volumes were manually segmented by two independent readers, with the participation of eight readers. Radiomic features were extracted from two sequences: T2-weighted (T2) and contrast-enhanced T1-weighted (T1ce). Two methods were considered for discretization: bin count (i.e., relative discretization) and bin width (i.e., absolute discretization). Ten discretization (five for each method) and five resampling parameters were varied while other parameters were fixed. The intraclass correlation coefficient (ICC) was used for reliability analysis based on two commonly used cut-off values (0.75 and 0.90). RESULTS Image preprocessing parameters had a significant impact on the segmentation-based reproducibility of radiomic features. The bin width method yielded more reproducible features than the bin count method. In discretization experiments using the bin width on both sequences, according to the ICC cut-off values of 0.75 and 0.90, the rate of reproducible features ranged from 70% to 84% and from 35% to 57%, respectively, with an increasing percentage trend as parameter values decreased (from 84 to 5 for T2; 100 to 6 for T1ce). In the resampling experiments, these ranged from 53% to 74% and from 10% to 20%, respectively, with an increasing percentage trend from lower to higher parameter values (physical voxel size; from 1 x 1 x 1 to 2 x 2 x 2 mm3). CONCLUSION The segmentation-based reproducibility of radiomic features appears to be substantially influenced by discretization and resampling parameters. Our findings indicate that the bin width method should be used for discretization and lower bin width and higher resampling values should be used to allow more reproducible features.
Collapse
Affiliation(s)
- Burak Koçak
- University of Health Sciences, Başakşehir Çam and Sakura City Hospital, Clinic of Radiology, İstanbul, Türkiye
| | - Sabahattin Yüzkan
- University of Health Sciences, Başakşehir Çam and Sakura City Hospital, Clinic of Radiology, İstanbul, Türkiye
| | - Samet Mutlu
- University of Health Sciences, Başakşehir Çam and Sakura City Hospital, Clinic of Radiology, İstanbul, Türkiye
| | - Mehmet Karagülle
- University of Health Sciences, Başakşehir Çam and Sakura City Hospital, Clinic of Radiology, İstanbul, Türkiye
| | - Ahmet Kala
- University of Health Sciences, Başakşehir Çam and Sakura City Hospital, Clinic of Radiology, İstanbul, Türkiye
| | - Mehmet Kadıoğlu
- University of Health Sciences, Başakşehir Çam and Sakura City Hospital, Clinic of Radiology, İstanbul, Türkiye
| | - Sıla Solak
- University of Health Sciences, Başakşehir Çam and Sakura City Hospital, Clinic of Radiology, İstanbul, Türkiye
| | - Şeyma Sunman
- University of Health Sciences, Başakşehir Çam and Sakura City Hospital, Clinic of Radiology, İstanbul, Türkiye
| | - Zişan Hayriye Temiz
- University of Health Sciences, Başakşehir Çam and Sakura City Hospital, Clinic of Radiology, İstanbul, Türkiye
| | - Ali Kürşad Ganiyusufoğlu
- University of Health Sciences, Başakşehir Çam and Sakura City Hospital, Clinic of Radiology, İstanbul, Türkiye
| |
Collapse
|
48
|
Xie Y, Zaccagna F, Rundo L, Testa C, Zhu R, Tonon C, Lodi R, Manners DN. IMPA-Net: Interpretable Multi-Part Attention Network for Trustworthy Brain Tumor Classification from MRI. Diagnostics (Basel) 2024; 14:997. [PMID: 38786294 PMCID: PMC11119919 DOI: 10.3390/diagnostics14100997] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/18/2024] [Revised: 05/08/2024] [Accepted: 05/09/2024] [Indexed: 05/25/2024] Open
Abstract
Deep learning (DL) networks have shown attractive performance in medical image processing tasks such as brain tumor classification. However, they are often criticized as mysterious "black boxes". The opaqueness of the model and the reasoning process make it difficult for health workers to decide whether to trust the prediction outcomes. In this study, we develop an interpretable multi-part attention network (IMPA-Net) for brain tumor classification to enhance the interpretability and trustworthiness of classification outcomes. The proposed model not only predicts the tumor grade but also provides a global explanation for the model interpretability and a local explanation as justification for the proffered prediction. Global explanation is represented as a group of feature patterns that the model learns to distinguish high-grade glioma (HGG) and low-grade glioma (LGG) classes. Local explanation interprets the reasoning process of an individual prediction by calculating the similarity between the prototypical parts of the image and a group of pre-learned task-related features. Experiments conducted on the BraTS2017 dataset demonstrate that IMPA-Net is a verifiable model for the classification task. A percentage of 86% of feature patterns were assessed by two radiologists to be valid for representing task-relevant medical features. The model shows a classification accuracy of 92.12%, of which 81.17% were evaluated as trustworthy based on local explanations. Our interpretable model is a trustworthy model that can be used for decision aids for glioma classification. Compared with black-box CNNs, it allows health workers and patients to understand the reasoning process and trust the prediction outcomes.
Collapse
Affiliation(s)
- Yuting Xie
- Department of Biomedical and Neuromotor Sciences, University of Bologna, 40126 Bologna, Italy; (Y.X.); (C.T.); (R.L.)
- Functional and Molecular Neuroimaging Unit, IRCCS Istituto delle Scienze Neurologiche di Bologna, Bellaria Hospital, 40139 Bologna, Italy
| | - Fulvio Zaccagna
- Department of Imaging, Cambridge University Hospitals NHS Foundation Trust, Cambridge Biomedical Campus, Cambridge CB2 0SL, UK;
- Department of Radiology, University of Cambridge, Cambridge CB2 0QQ, UK
| | - Leonardo Rundo
- Department of Information and Electrical Engineering and Applied Mathematics, University of Salerno, 84084 Fisciano, Italy;
| | - Claudia Testa
- INFN Bologna Division, Viale C. Berti Pichat, 6/2, 40127 Bologna, Italy
- Department of Physics and Astronomy, University of Bologna, 40127 Bologna, Italy
| | - Ruifeng Zhu
- Department of Engineering “Enzo Ferrari”, University of Modena and Reggio Emilia, 41125 Modena, Italy;
| | - Caterina Tonon
- Department of Biomedical and Neuromotor Sciences, University of Bologna, 40126 Bologna, Italy; (Y.X.); (C.T.); (R.L.)
- Functional and Molecular Neuroimaging Unit, IRCCS Istituto delle Scienze Neurologiche di Bologna, Bellaria Hospital, 40139 Bologna, Italy
| | - Raffaele Lodi
- Department of Biomedical and Neuromotor Sciences, University of Bologna, 40126 Bologna, Italy; (Y.X.); (C.T.); (R.L.)
- Functional and Molecular Neuroimaging Unit, IRCCS Istituto delle Scienze Neurologiche di Bologna, Bellaria Hospital, 40139 Bologna, Italy
| | - David Neil Manners
- Functional and Molecular Neuroimaging Unit, IRCCS Istituto delle Scienze Neurologiche di Bologna, Bellaria Hospital, 40139 Bologna, Italy
- Department for Life Quality Studies, University of Bologna, 40126 Bologna, Italy
| |
Collapse
|
49
|
Santarossa M, Beyer TT, Scharf ABA, Tatli A, von der Burchard C, Nazarenus J, Roider JB, Koch R. When Two Eyes Don't Suffice-Learning Difficult Hyperfluorescence Segmentations in Retinal Fundus Autofluorescence Images via Ensemble Learning. J Imaging 2024; 10:116. [PMID: 38786570 PMCID: PMC11122615 DOI: 10.3390/jimaging10050116] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/16/2024] [Revised: 05/03/2024] [Accepted: 05/06/2024] [Indexed: 05/25/2024] Open
Abstract
Hyperfluorescence (HF) and reduced autofluorescence (RA) are important biomarkers in fundus autofluorescence images (FAF) for the assessment of health of the retinal pigment epithelium (RPE), an important indicator of disease progression in geographic atrophy (GA) or central serous chorioretinopathy (CSCR). Autofluorescence images have been annotated by human raters, but distinguishing biomarkers (whether signals are increased or decreased) from the normal background proves challenging, with borders being particularly open to interpretation. Consequently, significant variations emerge among different graders, and even within the same grader during repeated annotations. Tests on in-house FAF data show that even highly skilled medical experts, despite previously discussing and settling on precise annotation guidelines, reach a pair-wise agreement measured in a Dice score of no more than 63-80% for HF segmentations and only 14-52% for RA. The data further show that the agreement of our primary annotation expert with herself is a 72% Dice score for HF and 51% for RA. Given these numbers, the task of automated HF and RA segmentation cannot simply be refined to the improvement in a segmentation score. Instead, we propose the use of a segmentation ensemble. Learning from images with a single annotation, the ensemble reaches expert-like performance with an agreement of a 64-81% Dice score for HF and 21-41% for RA with all our experts. In addition, utilizing the mean predictions of the ensemble networks and their variance, we devise ternary segmentations where FAF image areas are labeled either as confident background, confident HF, or potential HF, ensuring that predictions are reliable where they are confident (97% Precision), while detecting all instances of HF (99% Recall) annotated by all experts.
Collapse
Affiliation(s)
- Monty Santarossa
- Department of Computer Science, Kiel University, 24118 Kiel, Germany; (T.T.B.); (J.N.); (R.K.)
| | - Tebbo Tassilo Beyer
- Department of Computer Science, Kiel University, 24118 Kiel, Germany; (T.T.B.); (J.N.); (R.K.)
| | | | - Ayse Tatli
- Department of Ophthalmology, Kiel University, 24118 Kiel, Germany; (A.B.A.S.); (A.T.); (C.v.d.B.); (J.B.R.)
| | - Claus von der Burchard
- Department of Ophthalmology, Kiel University, 24118 Kiel, Germany; (A.B.A.S.); (A.T.); (C.v.d.B.); (J.B.R.)
| | - Jakob Nazarenus
- Department of Computer Science, Kiel University, 24118 Kiel, Germany; (T.T.B.); (J.N.); (R.K.)
| | - Johann Baptist Roider
- Department of Ophthalmology, Kiel University, 24118 Kiel, Germany; (A.B.A.S.); (A.T.); (C.v.d.B.); (J.B.R.)
| | - Reinhard Koch
- Department of Computer Science, Kiel University, 24118 Kiel, Germany; (T.T.B.); (J.N.); (R.K.)
| |
Collapse
|
50
|
Dalmaz O, Mirza MU, Elmas G, Ozbey M, Dar SUH, Ceyani E, Oguz KK, Avestimehr S, Çukur T. One model to unite them all: Personalized federated learning of multi-contrast MRI synthesis. Med Image Anal 2024; 94:103121. [PMID: 38402791 DOI: 10.1016/j.media.2024.103121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/26/2023] [Revised: 02/20/2024] [Accepted: 02/21/2024] [Indexed: 02/27/2024]
Abstract
Curation of large, diverse MRI datasets via multi-institutional collaborations can help improve learning of generalizable synthesis models that reliably translate source- onto target-contrast images. To facilitate collaborations, federated learning (FL) adopts decentralized model training while mitigating privacy concerns by avoiding sharing of imaging data. However, conventional FL methods can be impaired by the inherent heterogeneity in the data distribution, with domain shifts evident within and across imaging sites. Here we introduce the first personalized FL method for MRI Synthesis (pFLSynth) that improves reliability against data heterogeneity via model specialization to individual sites and synthesis tasks (i.e., source-target contrasts). To do this, pFLSynth leverages an adversarial model equipped with novel personalization blocks that control the statistics of generated feature maps across the spatial/channel dimensions, given latent variables specific to sites and tasks. To further promote communication efficiency and site specialization, partial network aggregation is employed over later generator stages while earlier generator stages and the discriminator are trained locally. As such, pFLSynth enables multi-task training of multi-site synthesis models with high generalization performance across sites and tasks. Comprehensive experiments demonstrate the superior performance and reliability of pFLSynth in MRI synthesis against prior federated methods.
Collapse
Affiliation(s)
- Onat Dalmaz
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Muhammad U Mirza
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Gokberk Elmas
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Muzaffer Ozbey
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Salman U H Dar
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey
| | - Emir Ceyani
- Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA 90089, USA
| | - Kader K Oguz
- Department of Radiology, University of California, Davis Medical Center, Sacramento, CA 95817, USA
| | - Salman Avestimehr
- Department of Electrical and Computer Engineering, University of Southern California, Los Angeles, CA 90089, USA
| | - Tolga Çukur
- Department of Electrical and Electronics Engineering, Bilkent University, Ankara 06800, Turkey; National Magnetic Resonance Research Center (UMRAM), Bilkent University, Ankara 06800, Turkey; Neuroscience Program, Bilkent University, Ankara 06800, Turkey.
| |
Collapse
|