1
|
Liu Z, Kainth K, Zhou A, Deyer TW, Fayad ZA, Greenspan H, Mei X. A review of self-supervised, generative, and few-shot deep learning methods for data-limited magnetic resonance imaging segmentation. NMR IN BIOMEDICINE 2024; 37:e5143. [PMID: 38523402 DOI: 10.1002/nbm.5143] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/01/2023] [Revised: 02/15/2024] [Accepted: 02/16/2024] [Indexed: 03/26/2024]
Abstract
Magnetic resonance imaging (MRI) is a ubiquitous medical imaging technology with applications in disease diagnostics, intervention, and treatment planning. Accurate MRI segmentation is critical for diagnosing abnormalities, monitoring diseases, and deciding on a course of treatment. With the advent of advanced deep learning frameworks, fully automated and accurate MRI segmentation is advancing. Traditional supervised deep learning techniques have advanced tremendously, reaching clinical-level accuracy in the field of segmentation. However, these algorithms still require a large amount of annotated data, which is oftentimes unavailable or impractical. One way to circumvent this issue is to utilize algorithms that exploit a limited amount of labeled data. This paper aims to review such state-of-the-art algorithms that use a limited number of annotated samples. We explain the fundamental principles of self-supervised learning, generative models, few-shot learning, and semi-supervised learning and summarize their applications in cardiac, abdomen, and brain MRI segmentation. Throughout this review, we highlight algorithms that can be employed based on the quantity of annotated data available. We also present a comprehensive list of notable publicly available MRI segmentation datasets. To conclude, we discuss possible future directions of the field-including emerging algorithms, such as contrastive language-image pretraining, and potential combinations across the methods discussed-that can further increase the efficacy of image segmentation with limited labels.
Collapse
Affiliation(s)
- Zelong Liu
- BioMedical Engineering and Imaging Institute, Icahn School of Medicine at Mount Sinai, New York, New York, USA
| | - Komal Kainth
- BioMedical Engineering and Imaging Institute, Icahn School of Medicine at Mount Sinai, New York, New York, USA
| | - Alexander Zhou
- BioMedical Engineering and Imaging Institute, Icahn School of Medicine at Mount Sinai, New York, New York, USA
| | - Timothy W Deyer
- East River Medical Imaging, New York, New York, USA
- Department of Radiology, Cornell Medicine, New York, New York, USA
| | - Zahi A Fayad
- BioMedical Engineering and Imaging Institute, Icahn School of Medicine at Mount Sinai, New York, New York, USA
- Department of Diagnostic, Molecular, and Interventional Radiology, Icahn School of Medicine at Mount Sinai, New York, New York, USA
| | - Hayit Greenspan
- BioMedical Engineering and Imaging Institute, Icahn School of Medicine at Mount Sinai, New York, New York, USA
- Department of Diagnostic, Molecular, and Interventional Radiology, Icahn School of Medicine at Mount Sinai, New York, New York, USA
| | - Xueyan Mei
- BioMedical Engineering and Imaging Institute, Icahn School of Medicine at Mount Sinai, New York, New York, USA
- Department of Diagnostic, Molecular, and Interventional Radiology, Icahn School of Medicine at Mount Sinai, New York, New York, USA
| |
Collapse
|
2
|
Shephard AJ, Bashir RMS, Mahmood H, Jahanifar M, Minhas F, Raza SEA, McCombe KD, Craig SG, James J, Brooks J, Nankivell P, Mehanna H, Khurram SA, Rajpoot NM. A fully automated and explainable algorithm for predicting malignant transformation in oral epithelial dysplasia. NPJ Precis Oncol 2024; 8:137. [PMID: 38942998 PMCID: PMC11213925 DOI: 10.1038/s41698-024-00624-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/27/2023] [Accepted: 05/29/2024] [Indexed: 06/30/2024] Open
Abstract
Oral epithelial dysplasia (OED) is a premalignant histopathological diagnosis given to lesions of the oral cavity. Its grading suffers from significant inter-/intra-observer variability, and does not reliably predict malignancy progression, potentially leading to suboptimal treatment decisions. To address this, we developed an artificial intelligence (AI) algorithm, that assigns an Oral Malignant Transformation (OMT) risk score based on the Haematoxylin and Eosin (H&E) stained whole slide images (WSIs). Our AI pipeline leverages an in-house segmentation model to detect and segment both nuclei and epithelium. Subsequently, a shallow neural network utilises interpretable morphological and spatial features, emulating histological markers, to predict progression. We conducted internal cross-validation on our development cohort (Sheffield; n = 193 cases) and independent validation on two external cohorts (Birmingham and Belfast; n = 89 cases). On external validation, the proposed OMTscore achieved an AUROC = 0.75 (Recall = 0.92) in predicting OED progression, outperforming other grading systems (Binary: AUROC = 0.72, Recall = 0.85). Survival analyses showed the prognostic value of our OMTscore (C-index = 0.60, p = 0.02), compared to WHO (C-index = 0.64, p = 0.003) and binary grades (C-index = 0.65, p < 0.001). Nuclear analyses elucidated the presence of peri-epithelial and intra-epithelial lymphocytes in highly predictive patches of transforming cases (p < 0.001). This is the first study to propose a completely automated, explainable, and externally validated algorithm for predicting OED transformation. Our algorithm shows comparable-to-human-level performance, offering a promising solution to the challenges of grading OED in routine clinical practice.
Collapse
Affiliation(s)
- Adam J Shephard
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, UK
| | | | - Hanya Mahmood
- School of Clinical Dentistry, University of Sheffield, Sheffield, UK
| | - Mostafa Jahanifar
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, UK
| | - Fayyaz Minhas
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, UK
| | - Shan E Ahmed Raza
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, UK
| | - Kris D McCombe
- Precision Medicine Centre, Patrick G. Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, UK
| | - Stephanie G Craig
- Precision Medicine Centre, Patrick G. Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, UK
| | - Jacqueline James
- Precision Medicine Centre, Patrick G. Johnston Centre for Cancer Research, Queen's University Belfast, Belfast, UK
| | - Jill Brooks
- Institute of Head and Neck Studies and Education, Institute of Cancer and Genomic Sciences, University of Birmingham, Birmingham, UK
| | - Paul Nankivell
- Institute of Head and Neck Studies and Education, Institute of Cancer and Genomic Sciences, University of Birmingham, Birmingham, UK
| | - Hisham Mehanna
- Institute of Head and Neck Studies and Education, Institute of Cancer and Genomic Sciences, University of Birmingham, Birmingham, UK
| | - Syed Ali Khurram
- School of Clinical Dentistry, University of Sheffield, Sheffield, UK
| | - Nasir M Rajpoot
- Tissue Image Analytics Centre, Department of Computer Science, University of Warwick, Coventry, UK.
| |
Collapse
|
3
|
Bai L, Wang D, Wang H, Barnett M, Cabezas M, Cai W, Calamante F, Kyle K, Liu D, Ly L, Nguyen A, Shieh CC, Sullivan R, Zhan G, Ouyang W, Wang C. Improving multiple sclerosis lesion segmentation across clinical sites: A federated learning approach with noise-resilient training. Artif Intell Med 2024; 152:102872. [PMID: 38701636 DOI: 10.1016/j.artmed.2024.102872] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/26/2023] [Revised: 03/28/2024] [Accepted: 04/15/2024] [Indexed: 05/05/2024]
Abstract
Accurately measuring the evolution of Multiple Sclerosis (MS) with magnetic resonance imaging (MRI) critically informs understanding of disease progression and helps to direct therapeutic strategy. Deep learning models have shown promise for automatically segmenting MS lesions, but the scarcity of accurately annotated data hinders progress in this area. Obtaining sufficient data from a single clinical site is challenging and does not address the heterogeneous need for model robustness. Conversely, the collection of data from multiple sites introduces data privacy concerns and potential label noise due to varying annotation standards. To address this dilemma, we explore the use of the federated learning framework while considering label noise. Our approach enables collaboration among multiple clinical sites without compromising data privacy under a federated learning paradigm that incorporates a noise-robust training strategy based on label correction. Specifically, we introduce a Decoupled Hard Label Correction (DHLC) strategy that considers the imbalanced distribution and fuzzy boundaries of MS lesions, enabling the correction of false annotations based on prediction confidence. We also introduce a Centrally Enhanced Label Correction (CELC) strategy, which leverages the aggregated central model as a correction teacher for all sites, enhancing the reliability of the correction process. Extensive experiments conducted on two multi-site datasets demonstrate the effectiveness and robustness of our proposed methods, indicating their potential for clinical applications in multi-site collaborations to train better deep learning models with lower cost in data collection and annotation.
Collapse
Affiliation(s)
- Lei Bai
- Brain and Mind Centre, The University of Sydney, NSW 2050, Australia; School of Electrical and Information Engineering, The University of Sydney, NSW 2006, Australia
| | - Dongang Wang
- Brain and Mind Centre, The University of Sydney, NSW 2050, Australia; Sydney Neuroimaging Analysis Centre, 94 Mallett Street, NSW 2050, Australia.
| | - Hengrui Wang
- Sydney Neuroimaging Analysis Centre, 94 Mallett Street, NSW 2050, Australia
| | - Michael Barnett
- Brain and Mind Centre, The University of Sydney, NSW 2050, Australia; Sydney Neuroimaging Analysis Centre, 94 Mallett Street, NSW 2050, Australia; Royal Prince Alfred Hospital, NSW, 2050, Australia
| | - Mariano Cabezas
- Brain and Mind Centre, The University of Sydney, NSW 2050, Australia
| | - Weidong Cai
- Brain and Mind Centre, The University of Sydney, NSW 2050, Australia; School of Computer Science, The University of Sydney, NSW 2006, Australia
| | - Fernando Calamante
- Brain and Mind Centre, The University of Sydney, NSW 2050, Australia; School of Biomedical Engineering, The University of Sydney, NSW 2006, Australia; Sydney Imaging, The University of Sydney, NSW 2006, Australia
| | - Kain Kyle
- Brain and Mind Centre, The University of Sydney, NSW 2050, Australia; Sydney Neuroimaging Analysis Centre, 94 Mallett Street, NSW 2050, Australia
| | - Dongnan Liu
- Brain and Mind Centre, The University of Sydney, NSW 2050, Australia; School of Computer Science, The University of Sydney, NSW 2006, Australia
| | - Linda Ly
- Sydney Neuroimaging Analysis Centre, 94 Mallett Street, NSW 2050, Australia
| | - Aria Nguyen
- Sydney Neuroimaging Analysis Centre, 94 Mallett Street, NSW 2050, Australia
| | - Chun-Chien Shieh
- Sydney Neuroimaging Analysis Centre, 94 Mallett Street, NSW 2050, Australia
| | - Ryan Sullivan
- School of Biomedical Engineering, The University of Sydney, NSW 2006, Australia; Australian Imaging Service, NSW 2006, Australia
| | - Geng Zhan
- Brain and Mind Centre, The University of Sydney, NSW 2050, Australia; Sydney Neuroimaging Analysis Centre, 94 Mallett Street, NSW 2050, Australia
| | - Wanli Ouyang
- School of Electrical and Information Engineering, The University of Sydney, NSW 2006, Australia
| | - Chenyu Wang
- Brain and Mind Centre, The University of Sydney, NSW 2050, Australia; Sydney Neuroimaging Analysis Centre, 94 Mallett Street, NSW 2050, Australia.
| |
Collapse
|
4
|
Wiltgen T, McGinnis J, Schlaeger S, Kofler F, Voon C, Berthele A, Bischl D, Grundl L, Will N, Metz M, Schinz D, Sepp D, Prucker P, Schmitz-Koep B, Zimmer C, Menze B, Rueckert D, Hemmer B, Kirschke J, Mühlau M, Wiestler B. LST-AI: A deep learning ensemble for accurate MS lesion segmentation. Neuroimage Clin 2024; 42:103611. [PMID: 38703470 PMCID: PMC11088188 DOI: 10.1016/j.nicl.2024.103611] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/08/2024] [Revised: 04/19/2024] [Accepted: 04/23/2024] [Indexed: 05/06/2024]
Abstract
Automated segmentation of brain white matter lesions is crucial for both clinical assessment and scientific research in multiple sclerosis (MS). Over a decade ago, we introduced an engineered lesion segmentation tool, LST. While recent lesion segmentation approaches have leveraged artificial intelligence (AI), they often remain proprietary and difficult to adopt. As an open-source tool, we present LST-AI, an advanced deep learning-based extension of LST that consists of an ensemble of three 3D U-Nets. LST-AI explicitly addresses the imbalance between white matter (WM) lesions and non-lesioned WM. It employs a composite loss function incorporating binary cross-entropy and Tversky loss to improve segmentation of the highly heterogeneous MS lesions. We train the network ensemble on 491 MS pairs of T1-weighted and FLAIR images, collected in-house from a 3T MRI scanner, and expert neuroradiologists manually segmented the utilized lesion maps for training. LST-AI also includes a lesion location annotation tool, labeling lesions as periventricular, infratentorial, and juxtacortical according to the 2017 McDonald criteria, and, additionally, as subcortical. We conduct evaluations on 103 test cases consisting of publicly available data using the Anima segmentation validation tools and compare LST-AI with several publicly available lesion segmentation models. Our empirical analysis shows that LST-AI achieves superior performance compared to existing methods. Its Dice and F1 scores exceeded 0.62, outperforming LST, SAMSEG (Sequence Adaptive Multimodal SEGmentation), and the popular nnUNet framework, which all scored below 0.56. Notably, LST-AI demonstrated exceptional performance on the MSSEG-1 challenge dataset, an international WM lesion segmentation challenge, with a Dice score of 0.65 and an F1 score of 0.63-surpassing all other competing models at the time of the challenge. With increasing lesion volume, the lesion detection rate rapidly increased with a detection rate of >75% for lesions with a volume between 10 mm3 and 100 mm3. Given its higher segmentation performance, we recommend that research groups currently using LST transition to LST-AI. To facilitate broad adoption, we are releasing LST-AI as an open-source model, available as a command-line tool, dockerized container, or Python script, enabling diverse applications across multiple platforms.
Collapse
Affiliation(s)
- Tun Wiltgen
- Department of Neurology, School of Medicine, Technical University of Munich, Munich, Germany; TUM-Neuroimaging Center, School of Medicine, Technical University of Munich, Munich, Germany
| | - Julian McGinnis
- Department of Neurology, School of Medicine, Technical University of Munich, Munich, Germany; TUM-Neuroimaging Center, School of Medicine, Technical University of Munich, Munich, Germany; Department of Computer Science, Institute for AI in Medicine, Technical University of Munich, Munich, Germany
| | - Sarah Schlaeger
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany
| | - Florian Kofler
- Department of Computer Science, Institute for AI in Medicine, Technical University of Munich, Munich, Germany; Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany; TranslaTUM, Central Institute for Translational Cancer Research of the Technical University of Munich, Munich, Germany; Helmholtz AI, Helmholtz Munich, Neuherberg, Germany
| | - CuiCi Voon
- Department of Neurology, School of Medicine, Technical University of Munich, Munich, Germany; TUM-Neuroimaging Center, School of Medicine, Technical University of Munich, Munich, Germany
| | - Achim Berthele
- Department of Neurology, School of Medicine, Technical University of Munich, Munich, Germany
| | - Daria Bischl
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany
| | - Lioba Grundl
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany
| | - Nikolaus Will
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany
| | - Marie Metz
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany
| | - David Schinz
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany; Institute of Radiology, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Dominik Sepp
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany
| | - Philipp Prucker
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany
| | - Benita Schmitz-Koep
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany
| | - Claus Zimmer
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany
| | - Bjoern Menze
- Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland
| | - Daniel Rueckert
- Department of Computer Science, Institute for AI in Medicine, Technical University of Munich, Munich, Germany; Department of Computing, Imperial College London, London, United Kingdom
| | - Bernhard Hemmer
- Department of Neurology, School of Medicine, Technical University of Munich, Munich, Germany; Munich Cluster for Systems Neurology (SyNergy), Munich, Germany
| | - Jan Kirschke
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany
| | - Mark Mühlau
- Department of Neurology, School of Medicine, Technical University of Munich, Munich, Germany; TUM-Neuroimaging Center, School of Medicine, Technical University of Munich, Munich, Germany.
| | - Benedikt Wiestler
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Technical University of Munich, Munich, Germany; TranslaTUM, Central Institute for Translational Cancer Research of the Technical University of Munich, Munich, Germany; AI for Image-Guided Diagnosis and Therapy, School of Medicine, Technical University of Munich, Munich, Germany
| |
Collapse
|
5
|
Wiltgen T, McGinnis J, Schlaeger S, Kofler F, Voon C, Berthele A, Bischl D, Grundl L, Will N, Metz M, Schinz D, Sepp D, Prucker P, Schmitz-Koep B, Zimmer C, Menze B, Rueckert D, Hemmer B, Kirschke J, Mühlau M, Wiestler B. LST-AI: a Deep Learning Ensemble for Accurate MS Lesion Segmentation. MEDRXIV : THE PREPRINT SERVER FOR HEALTH SCIENCES 2024:2023.11.23.23298966. [PMID: 38045345 PMCID: PMC10690346 DOI: 10.1101/2023.11.23.23298966] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/05/2023]
Abstract
Automated segmentation of brain white matter lesions is crucial for both clinical assessment and scientific research in multiple sclerosis (MS). Over a decade ago, we introduced an engineered lesion segmentation tool, LST. While recent lesion segmentation approaches have leveraged artificial intelligence (AI), they often remain proprietary and difficult to adopt. As an open-source tool, we present LST-AI, an advanced deep learning-based extension of LST that consists of an ensemble of three 3D-UNets. LST-AI explicitly addresses the imbalance between white matter (WM) lesions and non-lesioned WM. It employs a composite loss function incorporating binary cross-entropy and Tversky loss to improve segmentation of the highly heterogeneous MS lesions. We train the network ensemble on 491 MS pairs of T1w and FLAIR images, collected in-house from a 3T MRI scanner, and expert neuroradiologists manually segmented the utilized lesion maps for training. LST-AI additionally includes a lesion location annotation tool, labeling lesion location according to the 2017 McDonald criteria (periventricular, infratentorial, juxtacortical, subcortical). We conduct evaluations on 103 test cases consisting of publicly available data using the Anima segmentation validation tools and compare LST-AI with several publicly available lesion segmentation models. Our empirical analysis shows that LST-AI achieves superior performance compared to existing methods. Its Dice and F1 scores exceeded 0.62, outperforming LST, SAMSEG (Sequence Adaptive Multimodal SEGmentation), and the popular nnUNet framework, which all scored below 0.56. Notably, LST-AI demonstrated exceptional performance on the MSSEG-1 challenge dataset, an international WM lesion segmentation challenge, with a Dice score of 0.65 and an F1 score of 0.63-surpassing all other competing models at the time of the challenge. With increasing lesion volume, the lesion detection rate rapidly increased with a detection rate of >75% for lesions with a volume between 10mm3 and 100mm3. Given its higher segmentation performance, we recommend that research groups currently using LST transition to LST-AI. To facilitate broad adoption, we are releasing LST-AI as an open-source model, available as a command-line tool, dockerized container, or Python script, enabling diverse applications across multiple platforms.
Collapse
Affiliation(s)
- Tun Wiltgen
- Department of Neurology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
- TUM-Neuroimaging Center, School of Medicine, Technical University of Munich, Munich, Germany
| | - Julian McGinnis
- Department of Neurology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
- TUM-Neuroimaging Center, School of Medicine, Technical University of Munich, Munich, Germany
- Department of Computer Science, Institute for AI in Medicine, Technical University of Munich, Munich, Germany
| | - Sarah Schlaeger
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Florian Kofler
- Department of Computer Science, Institute for AI in Medicine, Technical University of Munich, Munich, Germany
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
- TranslaTUM, Center for Translational Cancer Research, Munich, Germany
- Helmholtz AI, Helmholtz Munich, Neuherberg, Germany
| | - CuiCi Voon
- Department of Neurology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
- TUM-Neuroimaging Center, School of Medicine, Technical University of Munich, Munich, Germany
| | - Achim Berthele
- Department of Neurology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Daria Bischl
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Lioba Grundl
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Nikolaus Will
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Marie Metz
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - David Schinz
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
- Institute of Radiology, University Hospital Erlangen, Friedrich-Alexander-Universität Erlangen-Nürnberg, Erlangen, Germany
| | - Dominik Sepp
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Philipp Prucker
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Benita Schmitz-Koep
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Claus Zimmer
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Bjoern Menze
- Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland
| | - Daniel Rueckert
- Department of Computer Science, Institute for AI in Medicine, Technical University of Munich, Munich, Germany
- Department of Computing, Imperial College London, London, United Kingdom
| | - Bernhard Hemmer
- Department of Neurology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
- Munich Cluster for Systems Neurology (SyNergy), Munich, Germany
| | - Jan Kirschke
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
| | - Mark Mühlau
- Department of Neurology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
- TUM-Neuroimaging Center, School of Medicine, Technical University of Munich, Munich, Germany
| | - Benedikt Wiestler
- Department of Diagnostic and Interventional Neuroradiology, School of Medicine, Klinikum rechts der Isar, Technical University of Munich, Munich, Germany
- TranslaTUM, Center for Translational Cancer Research, Munich, Germany
| |
Collapse
|
6
|
Posselt C, Avci MY, Yigitsoy M, Schuenke P, Kolbitsch C, Schaeffter T, Remmele S. Simulation of acquisition shifts in T2 weighted fluid-attenuated inversion recovery magnetic resonance images to stress test artificial intelligence segmentation networks. J Med Imaging (Bellingham) 2024; 11:024013. [PMID: 38666039 PMCID: PMC11042016 DOI: 10.1117/1.jmi.11.2.024013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/10/2023] [Revised: 03/01/2024] [Accepted: 03/29/2024] [Indexed: 04/28/2024] Open
Abstract
Purpose To provide a simulation framework for routine neuroimaging test data, which allows for "stress testing" of deep segmentation networks against acquisition shifts that commonly occur in clinical practice for T2 weighted (T2w) fluid-attenuated inversion recovery magnetic resonance imaging protocols. Approach The approach simulates "acquisition shift derivatives" of MR images based on MR signal equations. Experiments comprise the validation of the simulated images by real MR scans and example stress tests on state-of-the-art multiple sclerosis lesion segmentation networks to explore a generic model function to describe the F1 score in dependence of the contrast-affecting sequence parameters echo time (TE) and inversion time (TI). Results The differences between real and simulated images range up to 19% in gray and white matter for extreme parameter settings. For the segmentation networks under test, the F1 score dependency on TE and TI can be well described by quadratic model functions (R 2 > 0.9 ). The coefficients of the model functions indicate that changes of TE have more influence on the model performance than TI. Conclusions We show that these deviations are in the range of values as may be caused by erroneous or individual differences in relaxation times as described by literature. The coefficients of the F1 model function allow for a quantitative comparison of the influences of TE and TI. Limitations arise mainly from tissues with a low baseline signal (like cerebrospinal fluid) and when the protocol contains contrast-affecting measures that cannot be modeled due to missing information in the DICOM header.
Collapse
Affiliation(s)
- Christiane Posselt
- University of Applied Sciences, Faculty of Electrical and Industrial Engineering, Landshut, Germany
| | | | | | - Patrick Schuenke
- Physikalisch‐Technische Bundesanstalt (PTB), Braunschweig and Berlin, Germany
| | - Christoph Kolbitsch
- Physikalisch‐Technische Bundesanstalt (PTB), Braunschweig and Berlin, Germany
| | - Tobias Schaeffter
- Physikalisch‐Technische Bundesanstalt (PTB), Braunschweig and Berlin, Germany
- Technical University of Berlin, Department of Medical Engineering, Berlin, Germany
| | - Stefanie Remmele
- University of Applied Sciences, Faculty of Electrical and Industrial Engineering, Landshut, Germany
| |
Collapse
|
7
|
Choi S, Lake S, Harrison DM. Evaluation of the Blood-Brain Barrier, Demyelination, and Neurodegeneration in Paramagnetic Rim Lesions in Multiple Sclerosis on 7 Tesla MRI. J Magn Reson Imaging 2024; 59:941-951. [PMID: 37276054 PMCID: PMC10754232 DOI: 10.1002/jmri.28847] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2023] [Revised: 05/22/2023] [Accepted: 05/23/2023] [Indexed: 06/07/2023] Open
Abstract
BACKGROUND Paramagnetic rim lesions (PRLs) are associated with chronic inflammation in multiple sclerosis (MS). 7-Tesla (7T) magnetic resonance imaging (MRI) can evaluate the integrity of the blood-brain barrier (BBB) in addition to the tissue myelination status and cell loss. PURPOSE To use MRI metrics to investigate underlying physiology and clinical importance of PRLs. STUDY TYPE Prospective. SUBJECTS Thirty-six participants (mean-age 47, 23 females, 13 males) of mixed MS subtypes. FIELD STRENGTH/SEQUENCE 7T, MP2RAGE, MULTI-ECHO 3D-GRE, FLAIR. ASSESSMENT Lesion heterogeneity; longitudinal changes in lesion counts; comparison of T1, R2*, and χ; association between baseline lesion types and disease progression (2-3 annual MRI visits with additional years of annual clinical follow-up). STATISTICAL TESTS Two-sample t-test, Wilcoxon Rank-Sum test, Pearson's chi-square test, two-group comparison with linear-mixed-effect model, mixed-effect ANOVA, logistic regression. P-values <0.05 were considered significant. RESULTS A total of 58.3% of participants had at least one PRL at baseline. Higher male proportion in PRL+ group was found. Average change in PRL count was 0.20 (SD = 2.82) for PRLs and 0.00 (SD = 0.82) for mottled lesions. Mean and median pre-/post-contrast T1 were longer in PRL+ than in PRL-. No differences in mean χ were seen for lesions grouped by PRL (P = 0.310, pre-contrast; 0.086, post-contrast) or PRL/M presence (P = 0.234, pre-contrast; 0.163, post-contrast). Median χ were less negative in PRL+ and PRL/M+ than in PRL- and PRL/M-. Mean and median pre-/post-contrast R2* were slower in PRL+ compared to PRL-. Mean and median pre-/post-contrast R2* were slower in PRL/M+ than in PRL/M-. PRL presence at baseline was associated with confirmed EDSS Plus progression (OR 3.75 [1.22-7.59]) and PRL/M+ at baseline with confirmed EDSS Plus progression (OR 3.63 [1.14-7.43]). DATA CONCLUSION Evidence of BBB breakdown in PRLs was not seen. Quantitative metrics confirmed prior results suggesting greater demyelination, cell loss, and possibly disruption of tissue anisotropy in PRLs. EVIDENCE LEVEL 2 TECHNICAL EFFICACY: Stage 2.
Collapse
Affiliation(s)
- Seongjin Choi
- Department of Neurology, University of Maryland School of Medicine, Baltimore Maryland
| | - Sarah Lake
- Hasbro Children’s Hospital, Brown University
| | - Daniel M. Harrison
- Department of Neurology, University of Maryland School of Medicine, Baltimore Maryland
- Department of Neurology, Baltimore Veterans Affairs Medical Center, Baltimore, Maryland
| |
Collapse
|
8
|
Carass A, Greenman D, Dewey BE, Calabresi PA, Prince JL, Pham DL. Image harmonization improves consistency of intra-rater delineations of MS lesions in heterogeneous MRI. NEUROIMAGE. REPORTS 2024; 4:100195. [PMID: 38370461 PMCID: PMC10871705 DOI: 10.1016/j.ynirp.2024.100195] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/20/2024]
Abstract
Clinical magnetic resonance images (MRIs) lack a standard intensity scale due to differences in scanner hardware and the pulse sequences used to acquire the images. When MRIs are used for quantification, as in the evaluation of white matter lesions (WMLs) in multiple sclerosis, this lack of intensity standardization becomes a critical problem affecting both the staging and tracking of the disease and its treatment. This paper presents a study of harmonization on WML segmentation consistency, which is evaluated using an object detection classification scheme that incorporates manual delineations from both the original and harmonized MRIs. A cohort of ten people scanned on two different imaging platforms was studied. An expert rater, blinded to the image source, manually delineated WMLs on images from both scanners before and after harmonization. It was found that there is closer agreement in both global and per-lesion WML volume and spatial distribution after harmonization, demonstrating the importance of image harmonization prior to the creation of manual delineations. These results could lead to better truth models in both the development and evaluation of automated lesion segmentation algorithms.
Collapse
Affiliation(s)
- Aaron Carass
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Danielle Greenman
- Center for Neuroscience and Regenerative Medicine, The Henry M. Jackson Foundation for the Advancement of Military Medicine, Bethesda, MD 20817, USA
| | - Blake E. Dewey
- Department of Neurology, Johns Hopkins School of Medicine, Baltimore, MD 21287, USA
| | - Peter A. Calabresi
- Department of Neurology, Johns Hopkins School of Medicine, Baltimore, MD 21287, USA
| | - Jerry L. Prince
- Department of Electrical and Computer Engineering, Johns Hopkins University, Baltimore, MD 21218, USA
| | - Dzung L. Pham
- Department of Radiology, Uniformed Services University of the Health Sciences, Bethesda, MD 20814, USA
| |
Collapse
|
9
|
Jiao R, Zhang Y, Ding L, Xue B, Zhang J, Cai R, Jin C. Learning with limited annotations: A survey on deep semi-supervised learning for medical image segmentation. Comput Biol Med 2024; 169:107840. [PMID: 38157773 DOI: 10.1016/j.compbiomed.2023.107840] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2023] [Revised: 10/30/2023] [Accepted: 12/07/2023] [Indexed: 01/03/2024]
Abstract
Medical image segmentation is a fundamental and critical step in many image-guided clinical approaches. Recent success of deep learning-based segmentation methods usually relies on a large amount of labeled data, which is particularly difficult and costly to obtain, especially in the medical imaging domain where only experts can provide reliable and accurate annotations. Semi-supervised learning has emerged as an appealing strategy and been widely applied to medical image segmentation tasks to train deep models with limited annotations. In this paper, we present a comprehensive review of recently proposed semi-supervised learning methods for medical image segmentation and summarize both the technical novelties and empirical results. Furthermore, we analyze and discuss the limitations and several unsolved problems of existing approaches. We hope this review can inspire the research community to explore solutions to this challenge and further advance the field of medical image segmentation.
Collapse
Affiliation(s)
- Rushi Jiao
- School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China; School of Engineering Medicine, Beihang University, Beijing, 100191, China; Shanghai Artificial Intelligence Laboratory, Shanghai, 200232, China.
| | - Yichi Zhang
- School of Data Science, Fudan University, Shanghai, 200433, China; Artificial Intelligence Innovation and Incubation Institute, Fudan University, Shanghai, 200433, China.
| | - Le Ding
- School of Biological Science and Medical Engineering, Beihang University, Beijing, 100191, China.
| | - Bingsen Xue
- School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China; Shanghai Artificial Intelligence Laboratory, Shanghai, 200232, China.
| | - Jicong Zhang
- School of Biological Science and Medical Engineering, Beihang University, Beijing, 100191, China; Hefei Innovation Research Institute, Beihang University, Hefei, 230012, China.
| | - Rong Cai
- School of Engineering Medicine, Beihang University, Beijing, 100191, China; Key Laboratory for Biomechanics and Mechanobiology of Ministry of Education, Beihang University, Beijing, 100191, China.
| | - Cheng Jin
- School of Biomedical Engineering, Shanghai Jiao Tong University, Shanghai, 200240, China; Shanghai Artificial Intelligence Laboratory, Shanghai, 200232, China; Beijing Anding Hospital, Capital Medical University, Beijing, 100088, China.
| |
Collapse
|
10
|
Huang L, Zhao Z, An L, Gong Y, Wang Y, Yang Q, Wang Z, Hu G, Wang Y, Guo C. 2.5D transfer deep learning model for segmentation of contrast-enhancing lesions on brain magnetic resonance imaging of multiple sclerosis and neuromyelitis optica spectrum disorder. Quant Imaging Med Surg 2024; 14:273-290. [PMID: 38223040 PMCID: PMC10784073 DOI: 10.21037/qims-23-846] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/10/2023] [Accepted: 10/18/2023] [Indexed: 01/16/2024]
Abstract
Background Multiple sclerosis (MS) and neuromyelitis optica spectrum disorder (NMOSD) are the two mimic autoimmune diseases of the central nervous system, which are rare in East Asia. Quantitative detection of contrast-enhancing lesions (CELs) on contrast-enhancing T1-weighted magnetic resonance (MR) images is of great significance for assessing the disease activity of MS and NMOSD. However, it is challenging to develop automatic segmentation algorithms due to the lack of data. In this work, we present an automatic segmentation model of CELs based on Fully Convolutional with Attention DenseNet (FCA-DenseNet) and transfer learning strategy to address the challenge of CEL quantification in small-scale datasets. Methods A transfer learning approach was employed in this study, whereby pretraining was conducted using 77 MS subjects from the open access datasets (MICCAI 2016, MICCAI 2017, ISBI 2015) for white matter hyperintensity segmentation, followed by fine-tuning using 24 MS and NMOSD subjects from the local dataset for CEL segmentation. The proposed FCA-DenseNet combined the Fully Convolutional DenseNet and Convolutional Block Attention Module in order to improve the learning capability. A 2.5D data slicing strategy was used to process complex 3D MR images. U-Net, ResUNet, TransUNet, and Attention-UNet are used as comparison models to FCA-DenseNet. Dice similarity coefficient (DSC), positive predictive value (PPV), true positive rate (TPR), and volume difference (VD) are used as evaluation metrics to evaluate the performances of different models. Results FCA-DenseNet outperforms all other models in terms of all evaluation metrics, with a DSC of 0.661±0.187, PPV of 0.719±0.201, TPR of 0.680±0.254, and VD of 0.388±0.334. Transfer learning strategy has achieved success in building segmentation models on a small-scale local dataset where traditional deep learning approaches fail to train effectively. Conclusions The improved FCA-DenseNet, combined with transfer learning strategy and 2.5D data slicing strategy, has successfully addressed the challenges in constructing deep learning models on small-scale datasets, making it conducive to clinical quantification of brain CELs and diagnosis of MS and NMOSD.
Collapse
Affiliation(s)
- Lan Huang
- Key Laboratory of Symbol Computation and Knowledge Engineering of Ministry of Education, College of Computer Science and Technology, Jilin University, Changchun, China
| | - Ziqi Zhao
- Key Laboratory of Symbol Computation and Knowledge Engineering of Ministry of Education, College of Computer Science and Technology, Jilin University, Changchun, China
| | - Liying An
- Department of Radiology, the First Hospital of Jilin University, Changchun, China
| | - Yingchun Gong
- Key Laboratory of Symbol Computation and Knowledge Engineering of Ministry of Education, College of Computer Science and Technology, Jilin University, Changchun, China
| | - Yao Wang
- Key Laboratory of Symbol Computation and Knowledge Engineering of Ministry of Education, College of Computer Science and Technology, Jilin University, Changchun, China
| | - Qixing Yang
- Key Laboratory of Symbol Computation and Knowledge Engineering of Ministry of Education, College of Computer Science and Technology, Jilin University, Changchun, China
| | - Zhuo Wang
- Department of Radiology, the First Hospital of Jilin University, Changchun, China
| | - Geli Hu
- Clinical and Technical Support, Philips Healthcare, Beijing, China
| | - Yan Wang
- Key Laboratory of Symbol Computation and Knowledge Engineering of Ministry of Education, College of Computer Science and Technology, Jilin University, Changchun, China
| | - Chunjie Guo
- Department of Radiology, the First Hospital of Jilin University, Changchun, China
| |
Collapse
|
11
|
Raab F, Malloni W, Wein S, Greenlee MW, Lang EW. Investigation of an efficient multi-modal convolutional neural network for multiple sclerosis lesion detection. Sci Rep 2023; 13:21154. [PMID: 38036638 PMCID: PMC10689724 DOI: 10.1038/s41598-023-48578-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2023] [Accepted: 11/28/2023] [Indexed: 12/02/2023] Open
Abstract
In this study, an automated 2D machine learning approach for fast and precise segmentation of MS lesions from multi-modal magnetic resonance images (mmMRI) is presented. The method is based on an U-Net like convolutional neural network (CNN) for automated 2D slice-based-segmentation of brain MRI volumes. The individual modalities are encoded in separate downsampling branches without weight sharing, to leverage the specific features. Skip connections input feature maps to multi-scale feature fusion (MSFF) blocks at every decoder stage of the network. Those are followed by multi-scale feature upsampling (MSFU) blocks which use the information about lesion shape and location. The CNN is evaluated on two publicly available datasets: The ISBI 2015 longitudinal MS lesion segmentation challenge dataset containing 19 subjects and the MICCAI 2016 MSSEG challenge dataset containing 15 subjects from various scanners. The proposed multi-input 2D architecture is among the top performing approaches in the ISBI challenge, to which open-access papers are available, is able to outperform state-of-the-art 3D approaches without additional post-processing, can be adapted to other scanners quickly, is robust against scanner variability and can be deployed for inference even on a standard laptop without a dedicated GPU.
Collapse
Affiliation(s)
- Florian Raab
- Computational Intelligence and Machine Learning Group, University of Regensburg, 93051, Regensburg, Germany.
| | - Wilhelm Malloni
- Experimental Psychology, University of Regensburg, Regensburg, 93051, Germany
| | - Simon Wein
- Computational Intelligence and Machine Learning Group, University of Regensburg, 93051, Regensburg, Germany
- Experimental Psychology, University of Regensburg, Regensburg, 93051, Germany
| | - Mark W Greenlee
- Experimental Psychology, University of Regensburg, Regensburg, 93051, Germany
| | - Elmar W Lang
- Computational Intelligence and Machine Learning Group, University of Regensburg, 93051, Regensburg, Germany
| |
Collapse
|
12
|
Wahlig SG, Nedelec P, Weiss DA, Rudie JD, Sugrue LP, Rauschecker AM. 3D U-Net for automated detection of multiple sclerosis lesions: utility of transfer learning from other pathologies. Front Neurosci 2023; 17:1188336. [PMID: 37965219 PMCID: PMC10641790 DOI: 10.3389/fnins.2023.1188336] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/17/2023] [Accepted: 09/26/2023] [Indexed: 11/16/2023] Open
Abstract
Background and purpose Deep learning algorithms for segmentation of multiple sclerosis (MS) plaques generally require training on large datasets. This manuscript evaluates the effect of transfer learning from segmentation of another pathology to facilitate use of smaller MS-specific training datasets. That is, a model trained for detection of one type of pathology was re-trained to identify MS lesions and active demyelination. Materials and methods In this retrospective study using MRI exams from 149 patients spanning 4/18/2014 to 7/8/2021, 3D convolutional neural networks were trained with a variable number of manually-segmented MS studies. Models were trained for FLAIR lesion segmentation at a single timepoint, new FLAIR lesion segmentation comparing two timepoints, and enhancing (actively demyelinating) lesion segmentation on T1 post-contrast imaging. Models were trained either de-novo or fine-tuned with transfer learning applied to a pre-existing model initially trained on non-MS data. Performance was evaluated with lesionwise sensitivity and positive predictive value (PPV). Results For single timepoint FLAIR lesion segmentation with 10 training studies, a fine-tuned model demonstrated improved performance [lesionwise sensitivity 0.55 ± 0.02 (mean ± standard error), PPV 0.66 ± 0.02] compared to a de-novo model (sensitivity 0.49 ± 0.02, p = 0.001; PPV 0.32 ± 0.02, p < 0.001). For new lesion segmentation with 30 training studies and their prior comparisons, a fine-tuned model demonstrated similar sensitivity (0.49 ± 0.05) and significantly improved PPV (0.60 ± 0.05) compared to a de-novo model (sensitivity 0.51 ± 0.04, p = 0.437; PPV 0.43 ± 0.04, p = 0.002). For enhancement segmentation with 20 training studies, a fine-tuned model demonstrated significantly improved overall performance (sensitivity 0.74 ± 0.06, PPV 0.69 ± 0.05) compared to a de-novo model (sensitivity 0.44 ± 0.09, p = 0.001; PPV 0.37 ± 0.05, p = 0.001). Conclusion By fine-tuning models trained for other disease pathologies with MS-specific data, competitive models identifying existing MS plaques, new MS plaques, and active demyelination can be built with substantially smaller datasets than would otherwise be required to train new models.
Collapse
Affiliation(s)
- Stephen G. Wahlig
- Center for Intelligent Imaging (ci), Department of Radiology and Biomedical Imaging, University of California, San Francisco, San Francisco, CA, United States
| | - Pierre Nedelec
- Center for Intelligent Imaging (ci), Department of Radiology and Biomedical Imaging, University of California, San Francisco, San Francisco, CA, United States
| | - David A. Weiss
- Center for Intelligent Imaging (ci), Department of Radiology and Biomedical Imaging, University of California, San Francisco, San Francisco, CA, United States
| | - Jeffrey D. Rudie
- Center for Intelligent Imaging (ci), Department of Radiology and Biomedical Imaging, University of California, San Francisco, San Francisco, CA, United States
- Department of Radiology, University of California, San Diego, San Diego, CA, United States
| | - Leo P. Sugrue
- Center for Intelligent Imaging (ci), Department of Radiology and Biomedical Imaging, University of California, San Francisco, San Francisco, CA, United States
| | - Andreas M. Rauschecker
- Center for Intelligent Imaging (ci), Department of Radiology and Biomedical Imaging, University of California, San Francisco, San Francisco, CA, United States
| |
Collapse
|
13
|
Wagner DT, Tilmans L, Peng K, Niedermeier M, Rohl M, Ryan S, Yadav D, Takacs N, Garcia-Fraley K, Koso M, Dikici E, Prevedello LM, Nguyen XV. Artificial Intelligence in Neuroradiology: A Review of Current Topics and Competition Challenges. Diagnostics (Basel) 2023; 13:2670. [PMID: 37627929 PMCID: PMC10453240 DOI: 10.3390/diagnostics13162670] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/16/2023] [Revised: 08/07/2023] [Accepted: 08/09/2023] [Indexed: 08/27/2023] Open
Abstract
There is an expanding body of literature that describes the application of deep learning and other machine learning and artificial intelligence methods with potential relevance to neuroradiology practice. In this article, we performed a literature review to identify recent developments on the topics of artificial intelligence in neuroradiology, with particular emphasis on large datasets and large-scale algorithm assessments, such as those used in imaging AI competition challenges. Numerous applications relevant to ischemic stroke, intracranial hemorrhage, brain tumors, demyelinating disease, and neurodegenerative/neurocognitive disorders were discussed. The potential applications of these methods to spinal fractures, scoliosis grading, head and neck oncology, and vascular imaging were also reviewed. The AI applications examined perform a variety of tasks, including localization, segmentation, longitudinal monitoring, diagnostic classification, and prognostication. While research on this topic is ongoing, several applications have been cleared for clinical use and have the potential to augment the accuracy or efficiency of neuroradiologists.
Collapse
Affiliation(s)
- Daniel T. Wagner
- Department of Radiology, The Ohio State University Wexner Medical Center, Columbus, OH 43210, USA (L.M.P.)
| | - Luke Tilmans
- Department of Radiology, The Ohio State University Wexner Medical Center, Columbus, OH 43210, USA (L.M.P.)
| | - Kevin Peng
- College of Medicine, The Ohio State University, Columbus, OH 43210, USA
| | | | - Matt Rohl
- College of Arts and Sciences, The Ohio State University, Columbus, OH 43210, USA
| | - Sean Ryan
- Department of Radiology, The Ohio State University Wexner Medical Center, Columbus, OH 43210, USA (L.M.P.)
| | - Divya Yadav
- College of Medicine, The Ohio State University, Columbus, OH 43210, USA
| | - Noah Takacs
- College of Medicine, The Ohio State University, Columbus, OH 43210, USA
| | - Krystle Garcia-Fraley
- Department of Radiology, The Ohio State University Wexner Medical Center, Columbus, OH 43210, USA (L.M.P.)
| | - Mensur Koso
- College of Medicine, The Ohio State University, Columbus, OH 43210, USA
| | - Engin Dikici
- Department of Radiology, The Ohio State University Wexner Medical Center, Columbus, OH 43210, USA (L.M.P.)
| | - Luciano M. Prevedello
- Department of Radiology, The Ohio State University Wexner Medical Center, Columbus, OH 43210, USA (L.M.P.)
| | - Xuan V. Nguyen
- Department of Radiology, The Ohio State University Wexner Medical Center, Columbus, OH 43210, USA (L.M.P.)
| |
Collapse
|
14
|
Spagnolo F, Depeursinge A, Schädelin S, Akbulut A, Müller H, Barakovic M, Melie-Garcia L, Bach Cuadra M, Granziera C. How far MS lesion detection and segmentation are integrated into the clinical workflow? A systematic review. Neuroimage Clin 2023; 39:103491. [PMID: 37659189 PMCID: PMC10480555 DOI: 10.1016/j.nicl.2023.103491] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/02/2023] [Accepted: 08/04/2023] [Indexed: 09/04/2023]
Abstract
INTRODUCTION Over the past few years, the deep learning community has developed and validated a plethora of tools for lesion detection and segmentation in Multiple Sclerosis (MS). However, there is an important gap between validating models technically and clinically. To this end, a six-step framework necessary for the development, validation, and integration of quantitative tools in the clinic was recently proposed under the name of the Quantitative Neuroradiology Initiative (QNI). AIMS Investigate to what extent automatic tools in MS fulfill the QNI framework necessary to integrate automated detection and segmentation into the clinical neuroradiology workflow. METHODS Adopting the systematic Cochrane literature review methodology, we screened and summarised published scientific articles that perform automatic MS lesions detection and segmentation. We categorised the retrieved studies based on their degree of fulfillment of QNI's six-steps, which include a tool's technical assessment, clinical validation, and integration. RESULTS We found 156 studies; 146/156 (94%) fullfilled the first QNI step, 155/156 (99%) the second, 8/156 (5%) the third, 3/156 (2%) the fourth, 5/156 (3%) the fifth and only one the sixth. CONCLUSIONS To date, little has been done to evaluate the clinical performance and the integration in the clinical workflow of available methods for MS lesion detection/segmentation. In addition, the socio-economic effects and the impact on patients' management of such tools remain almost unexplored.
Collapse
Affiliation(s)
- Federico Spagnolo
- Translational Imaging in Neurology (ThINK) Basel, Department of Biomedical Engineering, Faculty of Medicine, University Hospital Basel and University of Basel, Basel, Switzerland; Department of Neurology, University Hospital Basel, Basel, Switzerland; Research Center for Clinical Neuroimmunology and Neuroscience Basel (RC2NB), University Hospital Basel and University of Basel, Basel, Switzerland; MedGIFT, Institute of Informatics, School of Management, HES-SO Valais-Wallis University of Applied Sciences and Arts Western Switzerland, Sierre, Switzerland
| | - Adrien Depeursinge
- MedGIFT, Institute of Informatics, School of Management, HES-SO Valais-Wallis University of Applied Sciences and Arts Western Switzerland, Sierre, Switzerland; Nuclear Medicine and Molecular Imaging Department, Lausanne University Hospital (CHUV) and University of Lausanne, Lausanne, Switzerland
| | - Sabine Schädelin
- Translational Imaging in Neurology (ThINK) Basel, Department of Biomedical Engineering, Faculty of Medicine, University Hospital Basel and University of Basel, Basel, Switzerland; Clinical Trial Unit, Department of Clinical Research, University Hospital Basel, University of Basel, Basel, Switzerland
| | - Aysenur Akbulut
- Translational Imaging in Neurology (ThINK) Basel, Department of Biomedical Engineering, Faculty of Medicine, University Hospital Basel and University of Basel, Basel, Switzerland; Ankara University School of Medicine, Ankara, Turkey
| | - Henning Müller
- MedGIFT, Institute of Informatics, School of Management, HES-SO Valais-Wallis University of Applied Sciences and Arts Western Switzerland, Sierre, Switzerland; The Sense Research and Innovation Center, Lausanne and Sion, Switzerland
| | - Muhamed Barakovic
- Translational Imaging in Neurology (ThINK) Basel, Department of Biomedical Engineering, Faculty of Medicine, University Hospital Basel and University of Basel, Basel, Switzerland; Department of Neurology, University Hospital Basel, Basel, Switzerland; Research Center for Clinical Neuroimmunology and Neuroscience Basel (RC2NB), University Hospital Basel and University of Basel, Basel, Switzerland
| | - Lester Melie-Garcia
- Translational Imaging in Neurology (ThINK) Basel, Department of Biomedical Engineering, Faculty of Medicine, University Hospital Basel and University of Basel, Basel, Switzerland; Department of Neurology, University Hospital Basel, Basel, Switzerland; Research Center for Clinical Neuroimmunology and Neuroscience Basel (RC2NB), University Hospital Basel and University of Basel, Basel, Switzerland
| | - Meritxell Bach Cuadra
- CIBM Center for Biomedical Imaging, Lausanne, Switzerland; Radiology Department, Lausanne University Hospital (CHUV) and University of Lausanne, Lausanne, Switzerland
| | - Cristina Granziera
- Translational Imaging in Neurology (ThINK) Basel, Department of Biomedical Engineering, Faculty of Medicine, University Hospital Basel and University of Basel, Basel, Switzerland; Department of Neurology, University Hospital Basel, Basel, Switzerland; Research Center for Clinical Neuroimmunology and Neuroscience Basel (RC2NB), University Hospital Basel and University of Basel, Basel, Switzerland.
| |
Collapse
|
15
|
Kazerouni A, Aghdam EK, Heidari M, Azad R, Fayyaz M, Hacihaliloglu I, Merhof D. Diffusion models in medical imaging: A comprehensive survey. Med Image Anal 2023; 88:102846. [PMID: 37295311 DOI: 10.1016/j.media.2023.102846] [Citation(s) in RCA: 29] [Impact Index Per Article: 29.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/16/2022] [Revised: 05/12/2023] [Accepted: 05/16/2023] [Indexed: 06/12/2023]
Abstract
Denoising diffusion models, a class of generative models, have garnered immense interest lately in various deep-learning problems. A diffusion probabilistic model defines a forward diffusion stage where the input data is gradually perturbed over several steps by adding Gaussian noise and then learns to reverse the diffusion process to retrieve the desired noise-free data from noisy data samples. Diffusion models are widely appreciated for their strong mode coverage and quality of the generated samples in spite of their known computational burdens. Capitalizing on the advances in computer vision, the field of medical imaging has also observed a growing interest in diffusion models. With the aim of helping the researcher navigate this profusion, this survey intends to provide a comprehensive overview of diffusion models in the discipline of medical imaging. Specifically, we start with an introduction to the solid theoretical foundation and fundamental concepts behind diffusion models and the three generic diffusion modeling frameworks, namely, diffusion probabilistic models, noise-conditioned score networks, and stochastic differential equations. Then, we provide a systematic taxonomy of diffusion models in the medical domain and propose a multi-perspective categorization based on their application, imaging modality, organ of interest, and algorithms. To this end, we cover extensive applications of diffusion models in the medical domain, including image-to-image translation, reconstruction, registration, classification, segmentation, denoising, 2/3D generation, anomaly detection, and other medically-related challenges. Furthermore, we emphasize the practical use case of some selected approaches, and then we discuss the limitations of the diffusion models in the medical domain and propose several directions to fulfill the demands of this field. Finally, we gather the overviewed studies with their available open-source implementations at our GitHub.1 We aim to update the relevant latest papers within it regularly.
Collapse
Affiliation(s)
- Amirhossein Kazerouni
- School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran
| | | | - Moein Heidari
- School of Electrical Engineering, Iran University of Science and Technology, Tehran, Iran
| | - Reza Azad
- Faculty of Electrical Engineering and Information Technology, RWTH Aachen University, Aachen, Germany
| | | | - Ilker Hacihaliloglu
- Department of Radiology, University of British Columbia, Vancouver, Canada; Department of Medicine, University of British Columbia, Vancouver, Canada
| | - Dorit Merhof
- Faculty of Informatics and Data Science, University of Regensburg, Regensburg, Germany; Fraunhofer Institute for Digital Medicine MEVIS, Bremen, Germany.
| |
Collapse
|
16
|
Bao R, Song Y, Bates SV, Weiss RJ, Foster AN, Cobos CJ, Sotardi S, Zhang Y, Gollub RL, Grant PE, Ou Y. BOston Neonatal Brain Injury Dataset for Hypoxic Ischemic Encephalopathy (BONBID-HIE): Part I. MRI and Manual Lesion Annotation. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.06.30.546841. [PMID: 37461570 PMCID: PMC10350009 DOI: 10.1101/2023.06.30.546841] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 07/25/2023]
Abstract
Hypoxic ischemic encephalopathy (HIE) is a brain injury that occurs in 1 ~ 5/1000 term neonates. Accurate identification and segmentation of HIE-related lesions in neonatal brain magnetic resonance images (MRIs) is the first step toward predicting prognosis, identifying high-risk patients, and evaluating treatment effects. It will lead to a more accurate estimation of prognosis, a better understanding of neurological symptoms, and a timely prediction of response to therapy. We release the first public dataset containing neonatal brain diffusion MRI and expert annotation of lesions from 133 patients diagnosed with HIE. HIE-related lesions in brain MRI are often diffuse (i.e., multi-focal), and small (over half the patients in our data having lesions occupying <1% of brain volume). Segmentation for HIE MRI data is remarkably different from, and arguably more challenging than, other segmentation tasks such as brain tumors with focal and relatively large lesions. We hope that this dataset can help fuel the development of MRI lesion segmentation methods for HIE and small diffuse lesions in general.
Collapse
Affiliation(s)
- Rina Bao
- Boston Children’s Hospital, Boston, MA, USA
- Harvard Medical School, Boston, MA, USA
| | | | | | | | - Anna N. Foster
- Boston Children’s Hospital, Boston, MA, USA
- Massachusetts General Hospital, Boston, MA, USA
| | | | | | - Yue Zhang
- Boston Children’s Hospital, Boston, MA, USA
| | - Randy L. Gollub
- Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, MA, USA
- Department of Psychiatry, Massachusetts General Hospital, Boston, MA, USA
| | - P. Ellen Grant
- Boston Children’s Hospital, Boston, MA, USA
- Harvard Medical School, Boston, MA, USA
| | - Yangming Ou
- Boston Children’s Hospital, Boston, MA, USA
- Harvard Medical School, Boston, MA, USA
| |
Collapse
|
17
|
Homssi M, Sweeney EM, Demmon E, Mannheim W, Sakirsky M, Wang Y, Gauthier SA, Gupta A, Nguyen TD. Evaluation of the Statistical Detection of Change Algorithm for Screening Patients with MS with New Lesion Activity on Longitudinal Brain MRI. AJNR Am J Neuroradiol 2023; 44:649-655. [PMID: 37142431 PMCID: PMC10249703 DOI: 10.3174/ajnr.a7858] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/03/2023] [Accepted: 04/03/2023] [Indexed: 05/06/2023]
Abstract
BACKGROUND AND PURPOSE Identification of new MS lesions on longitudinal MR imaging by human readers is time-consuming and prone to error. Our objective was to evaluate the improvement in the performance of subject-level detection by readers when assisted by the automated statistical detection of change algorithm. MATERIALS AND METHODS A total of 200 patients with MS with a mean interscan interval of 13.2 (SD, 2.4) months were included. Statistical detection of change was applied to the baseline and follow-up FLAIR images to detect potential new lesions for confirmation by readers (Reader + statistical detection of change method). This method was compared with readers operating in the clinical workflow (Reader method) for a subject-level detection of new lesions. RESULTS Reader + statistical detection of change found 30 subjects (15.0%) with at least 1 new lesion, while Reader detected 16 subjects (8.0%). As a subject-level screening tool, statistical detection of change achieved a perfect sensitivity of 1.00 (95% CI, 0.88-1.00) and a moderate specificity of 0.67 (95% CI, 0.59-0.74). The agreement on a subject level was 0.91 (95% CI, 0.87-0.95) between Reader + statistical detection of change and Reader, and 0.72 (95% CI, 0.66-0.78) between Reader + statistical detection of change and statistical detection of change. CONCLUSIONS The statistical detection of change algorithm can serve as a time-saving screening tool to assist human readers in verifying 3D FLAIR images of patients with MS with suspected new lesions. Our promising results warrant further evaluation of statistical detection of change in prospective multireader clinical studies.
Collapse
Affiliation(s)
- M Homssi
- From the Department of Radiology (M.H., Y.W., A.G., T.D.N.)
| | - E M Sweeney
- Penn Statistics in Imaging and Visualization Endeavor (PennSIVE) Center, Department of Biostatistics, Epidemiology, and Informatics (E.M.S.), University of Pennsylvania, Philadelphia, Pennsylvania
| | - E Demmon
- Department of Neurology (E.D., W.M., M.S., S.A.G.)
| | - W Mannheim
- Department of Neurology (E.D., W.M., M.S., S.A.G.)
| | - M Sakirsky
- Department of Neurology (E.D., W.M., M.S., S.A.G.)
| | - Y Wang
- From the Department of Radiology (M.H., Y.W., A.G., T.D.N.)
| | - S A Gauthier
- Department of Neurology (E.D., W.M., M.S., S.A.G.)
- The Feil Family Brain & Mind Institute (S.A.G.), Weill Cornell Medicine, New York, New York
| | - A Gupta
- From the Department of Radiology (M.H., Y.W., A.G., T.D.N.)
| | - T D Nguyen
- From the Department of Radiology (M.H., Y.W., A.G., T.D.N.)
| |
Collapse
|
18
|
Zhang L, Tanno R, Xu M, Huang Y, Bronik K, Jin C, Jacob J, Zheng Y, Shao L, Ciccarelli O, Barkhof F, Alexander DC. Learning from multiple annotators for medical image segmentation. PATTERN RECOGNITION 2023; 138:None. [PMID: 37781685 PMCID: PMC10533416 DOI: 10.1016/j.patcog.2023.109400] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 05/10/2022] [Revised: 12/18/2022] [Accepted: 02/05/2023] [Indexed: 10/03/2023]
Abstract
Supervised machine learning methods have been widely developed for segmentation tasks in recent years. However, the quality of labels has high impact on the predictive performance of these algorithms. This issue is particularly acute in the medical image domain, where both the cost of annotation and the inter-observer variability are high. Different human experts contribute estimates of the "actual" segmentation labels in a typical label acquisition process, influenced by their personal biases and competency levels. The performance of automatic segmentation algorithms is limited when these noisy labels are used as the expert consensus label. In this work, we use two coupled CNNs to jointly learn, from purely noisy observations alone, the reliability of individual annotators and the expert consensus label distributions. The separation of the two is achieved by maximally describing the annotator's "unreliable behavior" (we call it "maximally unreliable") while achieving high fidelity with the noisy training data. We first create a toy segmentation dataset using MNIST and investigate the properties of the proposed algorithm. We then use three public medical imaging segmentation datasets to demonstrate our method's efficacy, including both simulated (where necessary) and real-world annotations: 1) ISBI2015 (multiple-sclerosis lesions); 2) BraTS (brain tumors); 3) LIDC-IDRI (lung abnormalities). Finally, we create a real-world multiple sclerosis lesion dataset (QSMSC at UCL: Queen Square Multiple Sclerosis Center at UCL, UK) with manual segmentations from 4 different annotators (3 radiologists with different level skills and 1 expert to generate the expert consensus label). In all datasets, our method consistently outperforms competing methods and relevant baselines, especially when the number of annotations is small and the amount of disagreement is large. The studies also reveal that the system is capable of capturing the complicated spatial characteristics of annotators' mistakes.
Collapse
Affiliation(s)
- Le Zhang
- Queen Square Institute of Neurology, Faculty of Brain Sciences, University College London, London, WC1B 5EH, United Kingdom
- Centre for Medical Image Computing, Department of Computer Science, University College London, London, WC1E 6BT, United Kingdom
| | - Ryutaro Tanno
- Healthcare Intelligence, Microsoft Research, Cambridge, CB1 2FB, United Kingdom
| | - Moucheng Xu
- Centre for Medical Image Computing, Department of Computer Science, University College London, London, WC1E 6BT, United Kingdom
| | | | - Kevin Bronik
- Queen Square Institute of Neurology, Faculty of Brain Sciences, University College London, London, WC1B 5EH, United Kingdom
| | - Chen Jin
- Centre for Medical Image Computing, Department of Computer Science, University College London, London, WC1E 6BT, United Kingdom
| | - Joseph Jacob
- Centre for Medical Image Computing, Department of Computer Science, University College London, London, WC1E 6BT, United Kingdom
- UCL Respiratory, University College London, London, WC1E 6JF, United Kingdom
| | | | - Ling Shao
- Inception Institute of Artificial Intelligence, Abu Dhabi, United Arab Emirates
| | - Olga Ciccarelli
- Queen Square Institute of Neurology, Faculty of Brain Sciences, University College London, London, WC1B 5EH, United Kingdom
| | - Frederik Barkhof
- Queen Square Institute of Neurology, Faculty of Brain Sciences, University College London, London, WC1B 5EH, United Kingdom
- Amsterdam UMC, Vrije Universiteit Amsterdam, Department of Radiology and Nuclear Medicine, Amsterdam, Netherlands
| | - Daniel C. Alexander
- Centre for Medical Image Computing, Department of Computer Science, University College London, London, WC1E 6BT, United Kingdom
| |
Collapse
|
19
|
Hindsholm AM, Andersen FL, Cramer SP, Simonsen HJ, Askløf MG, Magyari M, Madsen PN, Hansen AE, Sellebjerg F, Larsson HBW, Langkilde AR, Frederiksen JL, Højgaard L, Ladefoged CN, Lindberg U. Scanner agnostic large-scale evaluation of MS lesion delineation tool for clinical MRI. Front Neurosci 2023; 17:1177540. [PMID: 37274207 PMCID: PMC10235534 DOI: 10.3389/fnins.2023.1177540] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2023] [Accepted: 05/05/2023] [Indexed: 06/06/2023] Open
Abstract
Introduction Patients with MS are MRI scanned continuously throughout their disease course resulting in a large manual workload for radiologists which includes lesion detection and size estimation. Though many models for automatic lesion segmentation have been published, few are used broadly in clinic today, as there is a lack of testing on clinical datasets. By collecting a large, heterogeneous training dataset directly from our MS clinic we aim to present a model which is robust to different scanner protocols and artefacts and which only uses MRI modalities present in routine clinical examinations. Methods We retrospectively included 746 patients from routine examinations at our MS clinic. The inclusion criteria included acquisition at one of seven different scanners and an MRI protocol including 2D or 3D T2-w FLAIR, T2-w and T1-w images. Reference lesion masks on the training (n = 571) and validation (n = 70) datasets were generated using a preliminary segmentation model and subsequent manual correction. The test dataset (n = 100) was manually delineated. Our segmentation model https://github.com/CAAI/AIMS/ was based on the popular nnU-Net, which has won several biomedical segmentation challenges. We tested our model against the published segmentation models HD-MS-Lesions, which is also based on nnU-Net, trained with a more homogenous patient cohort. We furthermore tested model robustness to data from unseen scanners by performing a leave-one-scanner-out experiment. Results We found that our model was able to segment MS white matter lesions with a performance comparable to literature: DSC = 0.68, precision = 0.90, recall = 0.70, f1 = 0.78. Furthermore, the model outperformed HD-MS-Lesions in all metrics except precision = 0.96. In the leave-one-scanner-out experiment there was no significant change in performance (p < 0.05) between any of the models which were only trained on part of the dataset and the full segmentation model. Conclusion In conclusion we have seen, that by including a large, heterogeneous dataset emulating clinical reality, we have trained a segmentation model which maintains a high segmentation performance while being robust to data from unseen scanners. This broadens the applicability of the model in clinic and paves the way for clinical implementation.
Collapse
Affiliation(s)
- Amalie Monberg Hindsholm
- Department of Clinical Physiology and Nuclear Medicine, Copenhagen University Hospital–Rigshospitalet, Copenhagen, Denmark
| | - Flemming Littrup Andersen
- Department of Clinical Physiology and Nuclear Medicine, Copenhagen University Hospital–Rigshospitalet, Copenhagen, Denmark
| | - Stig Præstekjær Cramer
- Department of Clinical Physiology and Nuclear Medicine, Copenhagen University Hospital–Rigshospitalet, Copenhagen, Denmark
| | - Helle Juhl Simonsen
- Department of Clinical Physiology and Nuclear Medicine, Copenhagen University Hospital–Rigshospitalet, Copenhagen, Denmark
| | - Mathias Gæde Askløf
- Department of Clinical Physiology and Nuclear Medicine, Copenhagen University Hospital–Rigshospitalet, Copenhagen, Denmark
| | - Melinda Magyari
- Department of Neurology, Copenhagen University Hospital–Rigshospitalet, Copenhagen, Denmark
| | - Poul Nørgaard Madsen
- Center for IT and Medical Technology, Copenhagen University Hospital–Rigshospitalet, Copenhagen, Denmark
| | - Adam Espe Hansen
- Department of Radiology, Copenhagen University Hospital–Rigshospitalet, Copenhagen, Denmark
- Department of Clinical Medicine, University of Copenhagen, Copenhagen, Denmark
| | - Finn Sellebjerg
- Department of Neurology, Copenhagen University Hospital–Rigshospitalet, Copenhagen, Denmark
- Department of Clinical Medicine, University of Copenhagen, Copenhagen, Denmark
| | - Henrik Bo Wiberg Larsson
- Department of Clinical Physiology and Nuclear Medicine, Copenhagen University Hospital–Rigshospitalet, Copenhagen, Denmark
- Department of Clinical Medicine, University of Copenhagen, Copenhagen, Denmark
| | | | - Jette Lautrup Frederiksen
- Department of Neurology, Copenhagen University Hospital–Rigshospitalet, Copenhagen, Denmark
- Department of Clinical Medicine, University of Copenhagen, Copenhagen, Denmark
| | - Liselotte Højgaard
- Department of Clinical Physiology and Nuclear Medicine, Copenhagen University Hospital–Rigshospitalet, Copenhagen, Denmark
- Department of Clinical Medicine, University of Copenhagen, Copenhagen, Denmark
| | - Claes Nøhr Ladefoged
- Department of Clinical Physiology and Nuclear Medicine, Copenhagen University Hospital–Rigshospitalet, Copenhagen, Denmark
| | - Ulrich Lindberg
- Department of Clinical Physiology and Nuclear Medicine, Copenhagen University Hospital–Rigshospitalet, Copenhagen, Denmark
| |
Collapse
|
20
|
Liu D, Cabezas M, Wang D, Tang Z, Bai L, Zhan G, Luo Y, Kyle K, Ly L, Yu J, Shieh CC, Nguyen A, Kandasamy Karuppiah E, Sullivan R, Calamante F, Barnett M, Ouyang W, Cai W, Wang C. Multiple sclerosis lesion segmentation: revisiting weighting mechanisms for federated learning. Front Neurosci 2023; 17:1167612. [PMID: 37274196 PMCID: PMC10232857 DOI: 10.3389/fnins.2023.1167612] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2023] [Accepted: 04/24/2023] [Indexed: 06/06/2023] Open
Abstract
Background and introduction Federated learning (FL) has been widely employed for medical image analysis to facilitate multi-client collaborative learning without sharing raw data. Despite great success, FL's applications remain suboptimal in neuroimage analysis tasks such as lesion segmentation in multiple sclerosis (MS), due to variance in lesion characteristics imparted by different scanners and acquisition parameters. Methods In this work, we propose the first FL MS lesion segmentation framework via two effective re-weighting mechanisms. Specifically, a learnable weight is assigned to each local node during the aggregation process, based on its segmentation performance. In addition, the segmentation loss function in each client is also re-weighted according to the lesion volume for the data during training. Results The proposed method has been validated on two FL MS segmentation scenarios using public and clinical datasets. Specifically, the case-wise and voxel-wise Dice score of the proposed method under the first public dataset is 65.20 and 74.30, respectively. On the second in-house dataset, the case-wise and voxel-wise Dice score is 53.66, and 62.31, respectively. Discussions and conclusions The Comparison experiments on two FL MS segmentation scenarios using public and clinical datasets have demonstrated the effectiveness of the proposed method by significantly outperforming other FL methods. Furthermore, the segmentation performance of FL incorporating our proposed aggregation mechanism can achieve comparable performance to that from centralized training with all the raw data.
Collapse
Affiliation(s)
- Dongnan Liu
- School of Computer Science, The University of Sydney, Sydney, NSW, Australia
- Brain and Mind Centre, The University of Sydney, Sydney, NSW, Australia
| | - Mariano Cabezas
- Brain and Mind Centre, The University of Sydney, Sydney, NSW, Australia
| | - Dongang Wang
- Brain and Mind Centre, The University of Sydney, Sydney, NSW, Australia
- Sydney Neuroimaging Analysis Centre, Camperdown, NSW, Australia
| | - Zihao Tang
- School of Computer Science, The University of Sydney, Sydney, NSW, Australia
- Brain and Mind Centre, The University of Sydney, Sydney, NSW, Australia
| | - Lei Bai
- Brain and Mind Centre, The University of Sydney, Sydney, NSW, Australia
- School of Electrical and Information Engineering, The University of Sydney, Sydney, NSW, Australia
| | - Geng Zhan
- Brain and Mind Centre, The University of Sydney, Sydney, NSW, Australia
- Sydney Neuroimaging Analysis Centre, Camperdown, NSW, Australia
| | - Yuling Luo
- Brain and Mind Centre, The University of Sydney, Sydney, NSW, Australia
- Sydney Neuroimaging Analysis Centre, Camperdown, NSW, Australia
| | - Kain Kyle
- Brain and Mind Centre, The University of Sydney, Sydney, NSW, Australia
- Sydney Neuroimaging Analysis Centre, Camperdown, NSW, Australia
| | - Linda Ly
- Brain and Mind Centre, The University of Sydney, Sydney, NSW, Australia
- Sydney Neuroimaging Analysis Centre, Camperdown, NSW, Australia
| | - James Yu
- Brain and Mind Centre, The University of Sydney, Sydney, NSW, Australia
- Sydney Neuroimaging Analysis Centre, Camperdown, NSW, Australia
| | - Chun-Chien Shieh
- Brain and Mind Centre, The University of Sydney, Sydney, NSW, Australia
- Sydney Neuroimaging Analysis Centre, Camperdown, NSW, Australia
| | - Aria Nguyen
- Brain and Mind Centre, The University of Sydney, Sydney, NSW, Australia
- Sydney Neuroimaging Analysis Centre, Camperdown, NSW, Australia
| | | | - Ryan Sullivan
- School of Biomedical Engineering, The University of Sydney, Sydney, NSW, Australia
| | - Fernando Calamante
- Brain and Mind Centre, The University of Sydney, Sydney, NSW, Australia
- School of Biomedical Engineering, The University of Sydney, Sydney, NSW, Australia
- Sydney Imaging, The University of Sydney, Sydney, NSW, Australia
| | - Michael Barnett
- Brain and Mind Centre, The University of Sydney, Sydney, NSW, Australia
- Sydney Neuroimaging Analysis Centre, Camperdown, NSW, Australia
| | - Wanli Ouyang
- School of Electrical and Information Engineering, The University of Sydney, Sydney, NSW, Australia
| | - Weidong Cai
- School of Computer Science, The University of Sydney, Sydney, NSW, Australia
| | - Chenyu Wang
- Brain and Mind Centre, The University of Sydney, Sydney, NSW, Australia
- Sydney Neuroimaging Analysis Centre, Camperdown, NSW, Australia
| |
Collapse
|
21
|
Rondinella A, Crispino E, Guarnera F, Giudice O, Ortis A, Russo G, Di Lorenzo C, Maimone D, Pappalardo F, Battiato S. Boosting multiple sclerosis lesion segmentation through attention mechanism. Comput Biol Med 2023; 161:107021. [PMID: 37216775 DOI: 10.1016/j.compbiomed.2023.107021] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2023] [Revised: 04/11/2023] [Accepted: 05/05/2023] [Indexed: 05/24/2023]
Abstract
Magnetic resonance imaging is a fundamental tool to reach a diagnosis of multiple sclerosis and monitoring its progression. Although several attempts have been made to segment multiple sclerosis lesions using artificial intelligence, fully automated analysis is not yet available. State-of-the-art methods rely on slight variations in segmentation architectures (e.g. U-Net, etc.). However, recent research has demonstrated how exploiting temporal-aware features and attention mechanisms can provide a significant boost to traditional architectures. This paper proposes a framework that exploits an augmented U-Net architecture with a convolutional long short-term memory layer and attention mechanism which is able to segment and quantify multiple sclerosis lesions detected in magnetic resonance images. Quantitative and qualitative evaluation on challenging examples demonstrated how the method outperforms previous state-of-the-art approaches, reporting an overall Dice score of 89% and also demonstrating robustness and generalization ability on never seen new test samples of a new dedicated under construction dataset.
Collapse
Affiliation(s)
- Alessia Rondinella
- Department of Mathematics and Computer Science, University of Catania, Viale Andrea Doria 6, Catania, 95125, Italy.
| | - Elena Crispino
- Department of Biomedical and Biotechnological Sciences, University of Catania, Via Santa Sofia 97, Catania, 95125, Italy
| | - Francesco Guarnera
- Department of Mathematics and Computer Science, University of Catania, Viale Andrea Doria 6, Catania, 95125, Italy
| | - Oliver Giudice
- Department of Mathematics and Computer Science, University of Catania, Viale Andrea Doria 6, Catania, 95125, Italy
| | - Alessandro Ortis
- Department of Mathematics and Computer Science, University of Catania, Viale Andrea Doria 6, Catania, 95125, Italy
| | - Giulia Russo
- Department of Drug and Health Sciences, University of Catania, Viale Andrea Doria 6, Catania, 95125, Italy
| | - Clara Di Lorenzo
- UOC Radiologia, ARNAS Garibaldi, P.zza S. Maria di Gesù, Catania, 95124, Italy
| | - Davide Maimone
- Centro Sclerosi Multipla, UOC Neurologia, ARNAS Garibaldi, P.zza S. Maria di Gesù, Catania, 95124, Italy
| | - Francesco Pappalardo
- Department of Drug and Health Sciences, University of Catania, Viale Andrea Doria 6, Catania, 95125, Italy
| | - Sebastiano Battiato
- Department of Mathematics and Computer Science, University of Catania, Viale Andrea Doria 6, Catania, 95125, Italy
| |
Collapse
|
22
|
Wang M, Qi S, Wu Y, Sun Y, Chang R, Pang H, Qian W. CE-NC-VesselSegNet: Supervised by contrast-enhanced CT images but utilized to segment pulmonary vessels from non-contrast-enhanced CT images. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2022.104565] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/01/2023]
|
23
|
Wang R, Bashyam V, Yang Z, Yu F, Tassopoulou V, Chintapalli SS, Skampardoni I, Sreepada LP, Sahoo D, Nikita K, Abdulkadir A, Wen J, Davatzikos C. Applications of generative adversarial networks in neuroimaging and clinical neuroscience. Neuroimage 2023; 269:119898. [PMID: 36702211 PMCID: PMC9992336 DOI: 10.1016/j.neuroimage.2023.119898] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/15/2022] [Revised: 12/16/2022] [Accepted: 01/21/2023] [Indexed: 01/25/2023] Open
Abstract
Generative adversarial networks (GANs) are one powerful type of deep learning models that have been successfully utilized in numerous fields. They belong to the broader family of generative methods, which learn to generate realistic data with a probabilistic model by learning distributions from real samples. In the clinical context, GANs have shown enhanced capabilities in capturing spatially complex, nonlinear, and potentially subtle disease effects compared to traditional generative methods. This review critically appraises the existing literature on the applications of GANs in imaging studies of various neurological conditions, including Alzheimer's disease, brain tumors, brain aging, and multiple sclerosis. We provide an intuitive explanation of various GAN methods for each application and further discuss the main challenges, open questions, and promising future directions of leveraging GANs in neuroimaging. We aim to bridge the gap between advanced deep learning methods and neurology research by highlighting how GANs can be leveraged to support clinical decision making and contribute to a better understanding of the structural and functional patterns of brain diseases.
Collapse
Affiliation(s)
- Rongguang Wang
- Center for AI and Data Science for Integrated Diagnostics, University of Pennsylvania, Philadelphia, USA; Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, USA.
| | - Vishnu Bashyam
- Center for AI and Data Science for Integrated Diagnostics, University of Pennsylvania, Philadelphia, USA; Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, USA
| | - Zhijian Yang
- Center for AI and Data Science for Integrated Diagnostics, University of Pennsylvania, Philadelphia, USA; Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, USA
| | - Fanyang Yu
- Center for AI and Data Science for Integrated Diagnostics, University of Pennsylvania, Philadelphia, USA; Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, USA
| | - Vasiliki Tassopoulou
- Center for AI and Data Science for Integrated Diagnostics, University of Pennsylvania, Philadelphia, USA; Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, USA
| | - Sai Spandana Chintapalli
- Center for AI and Data Science for Integrated Diagnostics, University of Pennsylvania, Philadelphia, USA; Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, USA
| | - Ioanna Skampardoni
- Center for AI and Data Science for Integrated Diagnostics, University of Pennsylvania, Philadelphia, USA; Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, USA; School of Electrical and Computer Engineering, National Technical University of Athens, Athens, Greece
| | - Lasya P Sreepada
- Center for AI and Data Science for Integrated Diagnostics, University of Pennsylvania, Philadelphia, USA; Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, USA
| | - Dushyant Sahoo
- Center for AI and Data Science for Integrated Diagnostics, University of Pennsylvania, Philadelphia, USA; Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, USA
| | - Konstantina Nikita
- School of Electrical and Computer Engineering, National Technical University of Athens, Athens, Greece
| | - Ahmed Abdulkadir
- Center for AI and Data Science for Integrated Diagnostics, University of Pennsylvania, Philadelphia, USA; Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, USA; Department of Clinical Neurosciences, Lausanne University Hospital and University of Lausanne, Lausanne, Switzerland
| | - Junhao Wen
- Center for AI and Data Science for Integrated Diagnostics, University of Pennsylvania, Philadelphia, USA; Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, USA
| | - Christos Davatzikos
- Center for AI and Data Science for Integrated Diagnostics, University of Pennsylvania, Philadelphia, USA; Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, USA; Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, USA.
| |
Collapse
|
24
|
Ma T, Wang AQ, Dalca AV, Sabuncu MR. Hyper-convolutions via implicit kernels for medical image analysis. Med Image Anal 2023; 86:102796. [PMID: 36948069 DOI: 10.1016/j.media.2023.102796] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2022] [Revised: 08/08/2022] [Accepted: 03/06/2023] [Indexed: 03/18/2023]
Abstract
The convolutional neural network (CNN) is one of the most commonly used architectures for computer vision tasks. The key building block of a CNN is the convolutional kernel that aggregates information from the pixel neighborhood and shares weights across all pixels. A standard CNN's capacity, and thus its performance, is directly related to the number of learnable kernel weights, which is determined by the number of channels and the kernel size (support). In this paper, we present the hyper-convolution, a novel building block that implicitly encodes the convolutional kernel using spatial coordinates. Unlike a regular convolutional kernel, whose weights are independently learned, hyper-convolution kernel weights are correlated through an encoder that maps spatial coordinates to their corresponding values. Hyper-convolutions decouple kernel size from the total number of learnable parameters, enabling a more flexible architecture design. We demonstrate in our experiments that replacing regular convolutions with hyper-convolutions can improve performance with less parameters, and increase robustness against noise. We provide our code here: https://github.com/tym002/Hyper-Convolution.
Collapse
Affiliation(s)
- Tianyu Ma
- School of Electrical and Computer Engineering, Cornell University, Ithaca, NY, USA; Cornell Tech, NYC, NY, USA; Department of Radiology, Weill Cornell Medical School, NYC, NY, USA
| | - Alan Q Wang
- School of Electrical and Computer Engineering, Cornell University, Ithaca, NY, USA; Cornell Tech, NYC, NY, USA; Department of Radiology, Weill Cornell Medical School, NYC, NY, USA
| | - Adrian V Dalca
- Computer Science and Artificial Intelligence Lab, Massachusetts Institute of Technology, Cambridge, MA, USA; A.A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Boston, MA, USA
| | - Mert R Sabuncu
- School of Electrical and Computer Engineering, Cornell University, Ithaca, NY, USA; Cornell Tech, NYC, NY, USA; Department of Radiology, Weill Cornell Medical School, NYC, NY, USA.
| |
Collapse
|
25
|
Krishnan AP, Song Z, Clayton D, Jia X, de Crespigny A, Carano RAD. Multi-arm U-Net with dense input and skip connectivity for T2 lesion segmentation in clinical trials of multiple sclerosis. Sci Rep 2023; 13:4102. [PMID: 36914715 PMCID: PMC10011580 DOI: 10.1038/s41598-023-31207-5] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2022] [Accepted: 03/08/2023] [Indexed: 03/16/2023] Open
Abstract
T2 lesion quantification plays a crucial role in monitoring disease progression and evaluating treatment response in multiple sclerosis (MS). We developed a 3D, multi-arm U-Net for T2 lesion segmentation, which was trained on a large, multicenter clinical trial dataset of relapsing MS. We investigated its generalization to other relapsing and primary progressive MS clinical trial datasets, and to an external dataset from the MICCAI 2016 MS lesion segmentation challenge. Additionally, we assessed the model's ability to reproduce the separation of T2 lesion volumes between treatment and control arms; and the association of baseline T2 lesion volumes with clinical disability scores compared with manual lesion annotations. The trained model achieved a mean dice coefficient of ≥ 0.66 and a lesion detection sensitivity of ≥ 0.72 across the internal test datasets. On the external test dataset, the model achieved a mean dice coefficient of 0.62, which is comparable to 0.59 from the best model in the challenge, and a lesion detection sensitivity of 0.68. Lesion detection performance was reduced for smaller lesions (≤ 30 μL, 3-10 voxels). The model successfully maintained the separation of the longitudinal changes in T2 lesion volumes between the treatment and control arms. Such tools could facilitate semi-automated MS lesion quantification; and reduce rater burden in clinical trials.
Collapse
Affiliation(s)
- Anitha Priya Krishnan
- Data Analytics and Imaging, Pharma Personalized Healthcare, Genentech Inc., 600 E Grand Ave., South San Francisco, CA, 94080, USA.
| | - Zhuang Song
- Data Analytics and Imaging, Pharma Personalized Healthcare, Genentech Inc., 600 E Grand Ave., South San Francisco, CA, 94080, USA
| | - David Clayton
- Clinical Imaging Group, gRED, Genentech Inc., South San Francisco, CA, USA
| | - Xiaoming Jia
- Translational Medicine OMNI - Biomarker Development, Genentech Inc., South San Francisco, CA, USA
| | - Alex de Crespigny
- Clinical Imaging Group, gRED, Genentech Inc., South San Francisco, CA, USA
| | - Richard A D Carano
- Data Analytics and Imaging, Pharma Personalized Healthcare, Genentech Inc., 600 E Grand Ave., South San Francisco, CA, 94080, USA
| |
Collapse
|
26
|
Huang F, Xia P, Vardhanabhuti V, Hui SK, Lau KK, Ka-Fung Mak H, Cao P. Semisupervised white matter hyperintensities segmentation on MRI. Hum Brain Mapp 2023; 44:1344-1358. [PMID: 36214210 PMCID: PMC9921214 DOI: 10.1002/hbm.26109] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/03/2021] [Revised: 08/25/2022] [Accepted: 09/07/2022] [Indexed: 11/10/2022] Open
Abstract
This study proposed a semisupervised loss function named level-set loss (LSLoss) for cerebral white matter hyperintensities (WMHs) segmentation on fluid-attenuated inversion recovery images. The training procedure did not require manually labeled WMH masks. Our image preprocessing steps included biased field correction, skull stripping, and white matter segmentation. With the proposed LSLoss, we trained a V-Net using the MRI images from both local and public databases. Local databases were the small vessel disease cohort (HKU-SVD, n = 360) and the multiple sclerosis cohort (HKU-MS, n = 20) from our institutional imaging center. Public databases were the Medical Image Computing Computer-assisted Intervention (MICCAI) WMH challenge database (MICCAI-WMH, n = 60) and the normal control cohort of the Alzheimer's Disease Neuroimaging Initiative database (ADNI-CN, n = 15). We achieved an overall dice similarity coefficient (DSC) of 0.81 on the HKU-SVD testing set (n = 20), DSC = 0.77 on the HKU-MS testing set (n = 5), and DSC = 0.78 on MICCAI-WMH testing set (n = 30). The segmentation results obtained by our semisupervised V-Net were comparable with the supervised methods and outperformed the unsupervised methods in the literature.
Collapse
Affiliation(s)
- Fan Huang
- Department of Diagnostic Radiology, LKS Faculty of Medicine, The University of Hong Kong, Hong Kong, China
| | - Peng Xia
- Department of Diagnostic Radiology, LKS Faculty of Medicine, The University of Hong Kong, Hong Kong, China
| | - Varut Vardhanabhuti
- Department of Diagnostic Radiology, LKS Faculty of Medicine, The University of Hong Kong, Hong Kong, China
| | - Sai-Kam Hui
- Department of Rehabilitation Science, The Hong Kong Polytechnic University, Hong Kong, China
| | - Kui-Kai Lau
- Department of Medicine, LKS Faculty of Medicine, The University of Hong Kong, Hong Kong, China.,The State Key Laboratory of Brain and Cognitive Sciences, The University of Hong Kong, Hong Kong, China
| | - Henry Ka-Fung Mak
- Department of Diagnostic Radiology, LKS Faculty of Medicine, The University of Hong Kong, Hong Kong, China
| | - Peng Cao
- Department of Diagnostic Radiology, LKS Faculty of Medicine, The University of Hong Kong, Hong Kong, China
| |
Collapse
|
27
|
Sarica B, Seker DZ, Bayram B. A dense residual U-net for multiple sclerosis lesions segmentation from multi-sequence 3D MR images. Int J Med Inform 2023; 170:104965. [PMID: 36580821 DOI: 10.1016/j.ijmedinf.2022.104965] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/06/2022] [Accepted: 12/08/2022] [Indexed: 12/28/2022]
Abstract
Multiple Sclerosis (MS) is an autoimmune disease that causes brain and spinal cord lesions, which magnetic resonance imaging (MRI) can detect and characterize. Recently, deep learning methods have achieved remarkable results in the automated segmentation of MS lesions from MRI data. Hence, this study proposes a novel dense residual U-Net model that combines attention gate (AG), efficient channel attention (ECA), and Atrous Spatial Pyramid Pooling (ASPP) to enhance the performance of the automatic MS lesion segmentation using 3D MRI sequences. First, convolution layers in each block of the U-Net architecture are replaced by residual blocks and connected densely. Then, AGs are exploited to capture salient features passed through the skip connections. The ECA module is appended at the end of each residual block and each downsampling block of U-Net. Later, the bottleneck of U-Net is replaced with the ASSP module to extract multi-scale contextual information. Furthermore, 3D MR images of Fluid Attenuated Inversion Recovery (FLAIR), T1-weighted (T1-w), and T2-weighted (T2-w) are exploited jointly to perform better MS lesion segmentation. The proposed model is validated on the publicly available ISBI2015 and MSSEG2016 challenge datasets. This model produced an ISBI score of 92.75, a mean Dice score of 66.88%, a mean positive predictive value (PPV) of 86.50%, and a mean lesion-wise true positive rate (LTPR) of 60.64% on the ISBI2015 testing set. Also, it achieved a mean Dice score of 67.27%, a mean PPV of 65.19%, and a mean sensitivity of 74.40% on the MSSEG2016 testing set. The results show that the proposed model performs better than the results of some experts and some of the other state-of-the-art methods realized related to this particular subject. Specifically, the best Dice score and the best LTPR are obtained on the ISBI2015 testing set by using the proposed model to segment MS lesions.
Collapse
Affiliation(s)
- Beytullah Sarica
- Istanbul Technical University, Graduate School, Department of Applied Informatics, Istanbul, 34469, Turkey.
| | - Dursun Zafer Seker
- Istanbul Technical University, Civil Engineering Faculty, Department of Geomatics Engineering, Istanbul, 34469, Turkey.
| | - Bulent Bayram
- Yildiz Technical University, Civil Engineering Faculty, Department of Geomatics Engineering, Istanbul, 34220, Turkey.
| |
Collapse
|
28
|
Lin D, Wang Z, Li H, Zhang H, Deng L, Ren H, Sun S, Zheng F, Zhou J, Wang M. Automated Measurement of Pancreatic Fat Deposition on Dixon MRI Using nnU-Net. J Magn Reson Imaging 2023; 57:296-307. [PMID: 35635494 DOI: 10.1002/jmri.28275] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2022] [Revised: 05/12/2022] [Accepted: 05/13/2022] [Indexed: 02/04/2023] Open
Abstract
BACKGROUND Pancreatic fat accumulation may cause or aggravate the process of acute pancreatitis, β-cell dysfunction, T2DM disease, and even be associated with pancreatic tumors. The pathophysiology of fatty pancreas remains overlooked and lacks effective imaging diagnostics. PURPOSE To automatically measure the distribution of pancreatic fat deposition on Dixon MRI in multicenter/population datasets using nnU-Net models. STUDY TYPE Retrospective. POPULATION A total of 176 obese/nonobese subjects (90 males, 86 females; mean age, 27.2 ± 19.7) were enrolled, including a training set (N = 132) and a testing set (N = 44). FIELD STRENGTH/SEQUENCE A 3 T and 1.5 T/gradient echo T1 dual-echo Dixon. ASSESSMENT The segmentation results of four types of nnU-Net models were compared using dice similarity coefficient (DSC), positive predicted value (PPV), and sensitivity. The ground truth was the manual delineation by two radiologists according to in-phase (IP) and opposed-phase (OP) images. STATISTICAL TESTS The group difference of segmentation results of four models were assessed by the Kruskal-Wallis H test with Dunn-Bonferroni comparisons. The interobserver agreement of pancreatic fat fraction measurements across three observers and test-retest reliability of human and machine were assessed by intragroup correlation coefficient (ICC). P < 0.05 was considered statistically significant. RESULTS The three-dimensional (3D) dual-contrast model had significantly improved performance than 2D dual-contrast (DSC/sensitivity) and 3D one-contrast (IP) models (DSC/PPV/sensitivity) and had less errors than 3D one-contrast (OP) model according to higher DSC and PPV (not significant), with a mean DSC of 0.9158, PPV of 0.9105 and sensitivity of 0.9232 in the testing set. The test-retest ICC of this model was above 0.900 in all pancreatic regions, exceeded human. DATA CONCLUSION 3D Dual-contrast nnU-Net aided segmentation of pancreas on Dixon images appears to be adaptable to multicenter/population datasets. It fully automates the assessment of pancreatic fat distribution and has high reliability. EVIDENCE LEVEL 3 TECHNICAL EFFICACY: Stage 3.
Collapse
Affiliation(s)
- Dingyi Lin
- College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, Zhejiang, China
| | - Ziyan Wang
- College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, Zhejiang, China
| | - Hong Li
- School of Medicine, Sir Run Run Shaw Hospital, Department of Endocrinology, Zhejiang University, Hangzhou, Zhejiang, China
| | - Hongxi Zhang
- School of Medicine, Children's Hospital Binjiang Campus, Department of Radiology, Zhejiang University, Hangzhou, Zhejiang, China
| | - Liping Deng
- School of Medicine, Sir Run Run Shaw Hospital, Department of Radiology, Zhejiang University, Hangzhou, Zhejiang, China
| | - Hong Ren
- School of Medicine, Sir Run Run Shaw Hospital, Department of Radiology, Zhejiang University, Hangzhou, Zhejiang, China
| | - Shuiya Sun
- School of Medicine, Sir Run Run Shaw Hospital, Department of Endocrinology, Zhejiang University, Hangzhou, Zhejiang, China
| | - Fenping Zheng
- School of Medicine, Sir Run Run Shaw Hospital, Department of Endocrinology, Zhejiang University, Hangzhou, Zhejiang, China
| | - Jiaqiang Zhou
- School of Medicine, Sir Run Run Shaw Hospital, Department of Endocrinology, Zhejiang University, Hangzhou, Zhejiang, China
| | - Min Wang
- College of Biomedical Engineering and Instrument Science, Zhejiang University, Hangzhou, Zhejiang, China.,School of Medicine, Sir Run Run Shaw Hospital, Department of Endocrinology, Zhejiang University, Hangzhou, Zhejiang, China
| |
Collapse
|
29
|
Mendelsohn Z, Pemberton HG, Gray J, Goodkin O, Carrasco FP, Scheel M, Nawabi J, Barkhof F. Commercial volumetric MRI reporting tools in multiple sclerosis: a systematic review of the evidence. Neuroradiology 2023; 65:5-24. [PMID: 36331588 PMCID: PMC9816195 DOI: 10.1007/s00234-022-03074-w] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/13/2022] [Accepted: 09/29/2022] [Indexed: 11/06/2022]
Abstract
PURPOSE MRI is integral to the diagnosis of multiple sclerosis (MS) and is important for clinical prognostication. Quantitative volumetric reporting tools (QReports) can improve the accuracy and objectivity of MRI-based assessments. Several QReports are commercially available; however, validation can be difficult to establish and does not currently follow a common pathway. To aid evidence-based clinical decision-making, we performed a systematic review of commercial QReports for use in MS including technical details and published reports of validation and in-use evaluation. METHODS We categorized studies into three types of testing: technical validation, for example, comparison to manual segmentation, clinical validation by clinicians or interpretation of results alongside clinician-rated variables, and in-use evaluation, such as health economic assessment. RESULTS We identified 10 companies, which provide MS lesion and brain segmentation and volume quantification, and 38 relevant publications. Tools received regulatory approval between 2006 and 2020, contextualize results to normative reference populations, ranging from 620 to 8000 subjects, and require T1- and T2-FLAIR-weighted input sequences for longitudinal assessment of whole-brain volume and lesions. In MS, six QReports provided evidence of technical validation, four companies have conducted clinical validation by correlating results with clinical variables, only one has tested their QReport by clinician end-users, and one has performed a simulated in-use socioeconomic evaluation. CONCLUSION We conclude that there is limited evidence in the literature regarding clinical validation and in-use evaluation of commercial MS QReports with a particular lack of clinician end-user testing. Our systematic review provides clinicians and institutions with the available evidence when considering adopting a quantitative reporting tool for MS.
Collapse
Affiliation(s)
- Zoe Mendelsohn
- grid.83440.3b0000000121901201Neuroradiological Academic Unit, UCL Queen Square Institute of Neurology, University College London, London, UK ,grid.83440.3b0000000121901201Department of Medical Physics and Bioengineering, Centre for Medical Image Computing (CMIC), University College London, London, UK ,grid.83440.3b0000000121901201Department of Neuroinflammation, Queen Square Multiple Sclerosis Centre, UCL Institute of Neurology, University College London, London, UK ,grid.6363.00000 0001 2218 4662Department of Neuroradiology, Charité School of Medicine and University Hospital Berlin, Berlin, Germany ,grid.6363.00000 0001 2218 4662Department of Radiology, Charité School of Medicine and University Hospital Berlin, Berlin, Germany
| | - Hugh G. Pemberton
- grid.83440.3b0000000121901201Neuroradiological Academic Unit, UCL Queen Square Institute of Neurology, University College London, London, UK ,grid.83440.3b0000000121901201Department of Medical Physics and Bioengineering, Centre for Medical Image Computing (CMIC), University College London, London, UK ,grid.420685.d0000 0001 1940 6527GE Healthcare, Amersham, UK
| | - James Gray
- grid.416626.10000 0004 0391 2793Stepping Hill Hospital, NHS Foundation Trust, Stockport, UK
| | - Olivia Goodkin
- grid.83440.3b0000000121901201Neuroradiological Academic Unit, UCL Queen Square Institute of Neurology, University College London, London, UK ,grid.83440.3b0000000121901201Department of Medical Physics and Bioengineering, Centre for Medical Image Computing (CMIC), University College London, London, UK ,grid.83440.3b0000000121901201Department of Neuroinflammation, Queen Square Multiple Sclerosis Centre, UCL Institute of Neurology, University College London, London, UK
| | - Ferran Prados Carrasco
- grid.83440.3b0000000121901201Department of Medical Physics and Bioengineering, Centre for Medical Image Computing (CMIC), University College London, London, UK ,grid.83440.3b0000000121901201Department of Neuroinflammation, Queen Square Multiple Sclerosis Centre, UCL Institute of Neurology, University College London, London, UK ,grid.36083.3e0000 0001 2171 6620E-Health Centre, Universitat Oberta de Catalunya, Barcelona, Spain
| | - Michael Scheel
- grid.6363.00000 0001 2218 4662Department of Neuroradiology, Charité School of Medicine and University Hospital Berlin, Berlin, Germany
| | - Jawed Nawabi
- grid.6363.00000 0001 2218 4662Department of Radiology, Charité School of Medicine and University Hospital Berlin, Berlin, Germany ,grid.484013.a0000 0004 6879 971XBerlin Institute of Health at Charité – Universitätsmedizin Berlin, BIH Biomedical Innovation Academy, BIH Charité Digital Clinician Scientist Program, Berlin, Germany
| | - Frederik Barkhof
- grid.83440.3b0000000121901201Neuroradiological Academic Unit, UCL Queen Square Institute of Neurology, University College London, London, UK ,grid.83440.3b0000000121901201Department of Medical Physics and Bioengineering, Centre for Medical Image Computing (CMIC), University College London, London, UK ,grid.83440.3b0000000121901201Department of Neuroinflammation, Queen Square Multiple Sclerosis Centre, UCL Institute of Neurology, University College London, London, UK ,grid.12380.380000 0004 1754 9227Radiology & Nuclear Medicine, Amsterdam University Medical Centers, Vrije Universiteit, Amsterdam, The Netherlands
| |
Collapse
|
30
|
Ouyang J, Zhao Q, Adeli E, Zaharchuk G, Pohl KM. Self-supervised learning of neighborhood embedding for longitudinal MRI. Med Image Anal 2022; 82:102571. [PMID: 36115098 PMCID: PMC10168684 DOI: 10.1016/j.media.2022.102571] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/04/2022] [Revised: 07/11/2022] [Accepted: 08/11/2022] [Indexed: 11/19/2022]
Abstract
In recent years, several deep learning models recommend first to represent Magnetic Resonance Imaging (MRI) as latent features before performing a downstream task of interest (such as classification or regression). The performance of the downstream task generally improves when these latent representations are explicitly associated with factors of interest. For example, we derived such a representation for capturing brain aging by applying self-supervised learning to longitudinal MRIs and then used the resulting encoding to automatically identify diseases accelerating the aging of the brain. We now propose a refinement of this representation by replacing the linear modeling of brain aging with one that is consistent in local neighborhoods in the latent space. Called Longitudinal Neighborhood Embedding (LNE), we derive an encoding so that neighborhoods are age-consistent (i.e., brain MRIs of different subjects with similar brain ages are in close proximity of each other) and progression-consistent, i.e., the latent space is defined by a smooth trajectory field where each trajectory captures changes in brain ages between a pair of MRIs extracted from a longitudinal sequence. To make the problem computationally tractable, we further propose a strategy for mini-batch sampling so that the resulting local neighborhoods accurately approximate the ones that would be defined based on the whole cohort. We evaluate LNE on three different downstream tasks: (1) to predict chronological age from T1-w MRI of 274 healthy subjects participating in a study at SRI International; (2) to distinguish Normal Control (NC) from Alzheimer's Disease (AD) and stable Mild Cognitive Impairment (sMCI) from progressive Mild Cognitive Impairment (pMCI) based on T1-w MRI of 632 participants of the Alzheimer's Disease Neuroimaging Initiative (ADNI); and (3) to distinguish no-to-low from moderate-to-heavy alcohol drinkers based on fractional anisotropy derived from diffusion tensor MRIs of 764 adolescents recruited by the National Consortium on Alcohol and NeuroDevelopment in Adolescence (NCANDA). Across the three data sets, the visualization of the smooth trajectory vector fields and superior accuracy on downstream tasks demonstrate the strength of the proposed method over existing self-supervised methods in extracting information related to brain aging, which could help study the impact of substance use and neurodegenerative disorders. The code is available at https://github.com/ouyangjiahong/longitudinal-neighbourhood-embedding.
Collapse
Affiliation(s)
- Jiahong Ouyang
- Department of Electrical Engineering, Stanford University, Stanford, United States of America
| | - Qingyu Zhao
- Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, United States of America
| | - Ehsan Adeli
- Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, United States of America
| | - Greg Zaharchuk
- Department of Radiology, Stanford University, Stanford, United States of America
| | - Kilian M Pohl
- Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, United States of America; Center for Health Sciences, SRI International, Menlo Park, United States of America.
| |
Collapse
|
31
|
Basaran BD, Matthews PM, Bai W. New lesion segmentation for multiple sclerosis brain images with imaging and lesion-aware augmentation. Front Neurosci 2022; 16:1007453. [PMID: 36340756 PMCID: PMC9633994 DOI: 10.3389/fnins.2022.1007453] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/30/2022] [Accepted: 10/06/2022] [Indexed: 09/04/2023] Open
Abstract
Multiple sclerosis (MS) is an inflammatory and demyelinating neurological disease of the central nervous system. Image-based biomarkers, such as lesions defined on magnetic resonance imaging (MRI), play an important role in MS diagnosis and patient monitoring. The detection of newly formed lesions provides crucial information for assessing disease progression and treatment outcome. Here, we propose a deep learning-based pipeline for new MS lesion detection and segmentation, which is built upon the nnU-Net framework. In addition to conventional data augmentation, we employ imaging and lesion-aware data augmentation methods, axial subsampling and CarveMix, to generate diverse samples and improve segmentation performance. The proposed pipeline is evaluated on the MICCAI 2021 MS new lesion segmentation challenge (MSSEG-2) dataset. It achieves an average Dice score of 0.510 and F 1 score of 0.552 on cases with new lesions, and an average false positive lesion number n FP of 0.036 and false positive lesion volume V FP of 0.192 mm 3 on cases with no new lesions. Our method outperforms other participating methods in the challenge and several state-of-the-art network architectures.
Collapse
Affiliation(s)
- Berke Doga Basaran
- Department of Computing, Imperial College London, London, United Kingdom
- Data Science Institute, Imperial College London, London, United Kingdom
| | - Paul M. Matthews
- Department of Brain Sciences, Imperial College London, London, United Kingdom
- UK Dementia Research Institute, Imperial College London, London, United Kingdom
| | - Wenjia Bai
- Department of Computing, Imperial College London, London, United Kingdom
- Data Science Institute, Imperial College London, London, United Kingdom
- Department of Brain Sciences, Imperial College London, London, United Kingdom
| |
Collapse
|
32
|
Ouyang J, Zhao Q, Adeli E, Zaharchuk G, Pohl KM. Disentangling Normal Aging From Severity of Disease via Weak Supervision on Longitudinal MRI. IEEE TRANSACTIONS ON MEDICAL IMAGING 2022; 41:2558-2569. [PMID: 35404811 PMCID: PMC9578549 DOI: 10.1109/tmi.2022.3166131] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
The continuous progression of neurological diseases are often categorized into conditions according to their severity. To relate the severity to changes in brain morphometry, there is a growing interest in replacing these categories with a continuous severity scale that longitudinal MRIs are mapped onto via deep learning algorithms. However, existing methods based on supervised learning require large numbers of samples and those that do not, such as self-supervised models, fail to clearly separate the disease effect from normal aging. Here, we propose to explicitly disentangle those two factors via weak-supervision. In other words, training is based on longitudinal MRIs being labelled either normal or diseased so that the training data can be augmented with samples from disease categories that are not of primary interest to the analysis. We do so by encouraging trajectories of controls to be fully encoded by the direction associated with brain aging. Furthermore, an orthogonal direction linked to disease severity captures the residual component from normal aging in the diseased cohort. Hence, the proposed method quantifies disease severity and its progression speed in individuals without knowing their condition. We apply the proposed method on data from the Alzheimer's Disease Neuroimaging Initiative (ADNI, N =632 ). We then show that the model properly disentangled normal aging from the severity of cognitive impairment by plotting the resulting disentangled factors of each subject and generating simulated MRIs for a given chronological age and condition. Moreover, our representation obtains higher balanced accuracy when used for two downstream classification tasks compared to other pre-training approaches. The code for our weak-supervised approach is available at https://github.com/ouyangjiahong/longitudinal-direction-disentangle.
Collapse
|
33
|
La Rosa F, Wynen M, Al-Louzi O, Beck ES, Huelnhagen T, Maggi P, Thiran JP, Kober T, Shinohara RT, Sati P, Reich DS, Granziera C, Absinta M, Bach Cuadra M. Cortical lesions, central vein sign, and paramagnetic rim lesions in multiple sclerosis: Emerging machine learning techniques and future avenues. Neuroimage Clin 2022; 36:103205. [PMID: 36201950 PMCID: PMC9668629 DOI: 10.1016/j.nicl.2022.103205] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/13/2021] [Revised: 09/09/2022] [Accepted: 09/16/2022] [Indexed: 12/14/2022]
Abstract
The current diagnostic criteria for multiple sclerosis (MS) lack specificity, and this may lead to misdiagnosis, which remains an issue in present-day clinical practice. In addition, conventional biomarkers only moderately correlate with MS disease progression. Recently, some MS lesional imaging biomarkers such as cortical lesions (CL), the central vein sign (CVS), and paramagnetic rim lesions (PRL), visible in specialized magnetic resonance imaging (MRI) sequences, have shown higher specificity in differential diagnosis. Moreover, studies have shown that CL and PRL are potential prognostic biomarkers, the former correlating with cognitive impairments and the latter with early disability progression. As machine learning-based methods have achieved extraordinary performance in the assessment of conventional imaging biomarkers, such as white matter lesion segmentation, several automated or semi-automated methods have been proposed as well for CL, PRL, and CVS. In the present review, we first introduce these MS biomarkers and their imaging methods. Subsequently, we describe the corresponding machine learning-based methods that were proposed to tackle these clinical questions, putting them into context with respect to the challenges they are facing, including non-standardized MRI protocols, limited datasets, and moderate inter-rater variability. We conclude by presenting the current limitations that prevent their broader deployment and suggesting future research directions.
Collapse
Key Words
- ms, multiple sclerosis
- mri, magnetic resonance imaging
- dl, deep learning
- ml, machine learning
- cl, cortical lesions
- prl, paramagnetic rim lesions
- cvs, central vein sign
- wml, white matter lesions
- flair, fluid-attenuated inversion recovery
- mprage, magnetization prepared rapid gradient-echo
- gm, gray matter
- wm, white matter
- psir, phase-sensitive inversion recovery
- dir, double inversion recovery
- mp2rage, magnetization-prepared 2 rapid gradient echoes
- sels, slowly evolving/expanding lesions
- cnn, convolutional neural network
- xai, explainable ai
- pv, partial volume
Collapse
Affiliation(s)
- Francesco La Rosa
- Signal Processing Laboratory (LTS5), Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland; CIBM Center for Biomedical Imaging, Switzerland; Department of Neurology, Icahn School of Medicine at Mount Sinai, New York, NY, USA.
| | - Maxence Wynen
- CIBM Center for Biomedical Imaging, Switzerland; ICTeam, UCLouvain, Louvain-la-Neuve, Belgium; Louvain Inflammation Imaging Lab (NIL), Institute of Neuroscience (IoNS), UCLouvain, Brussels, Belgium; Radiology Department, Lausanne University and University Hospital, Switzerland
| | - Omar Al-Louzi
- Translational Neuroradiology Section, National Institute of Neurological Disorders and Stroke, National Institutes of Health, Bethesda, MD, USA; Department of Neurology, Cedars-Sinai Medical Center, Los Angeles, CA, USA
| | - Erin S Beck
- Department of Neurology, Icahn School of Medicine at Mount Sinai, New York, NY, USA; Translational Neuroradiology Section, National Institute of Neurological Disorders and Stroke, National Institutes of Health, Bethesda, MD, USA
| | - Till Huelnhagen
- Signal Processing Laboratory (LTS5), Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland; Radiology Department, Lausanne University and University Hospital, Switzerland; Advanced Clinical Imaging Technology, Siemens Healthcare AG, Lausanne, Switzerland
| | - Pietro Maggi
- Louvain Inflammation Imaging Lab (NIL), Institute of Neuroscience (IoNS), UCLouvain, Brussels, Belgium; Department of Neurology, Cliniques universitaires Saint-Luc, Université catholique de Louvain, Brussels, Belgium; Department of Neurology, CHUV, Lausanne, Switzerland
| | - Jean-Philippe Thiran
- Signal Processing Laboratory (LTS5), Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland; CIBM Center for Biomedical Imaging, Switzerland; Radiology Department, Lausanne University and University Hospital, Switzerland
| | - Tobias Kober
- Signal Processing Laboratory (LTS5), Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland; Radiology Department, Lausanne University and University Hospital, Switzerland; Advanced Clinical Imaging Technology, Siemens Healthcare AG, Lausanne, Switzerland
| | - Russell T Shinohara
- Center for Biomedical Image Computing and Analysis (CBICA), Department of Radiology, University of Pennsylvania, Philadelphia, PA, USA; Penn Statistics in Imaging and Visualization Endeavor (PennSIVE), Center for Clinical Epidemiology and Biostatistics, University of Pennsylvania, Philadelphia, PA, USA; Department of Biostatistics, Epidemiology, and Informatics, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, USA
| | - Pascal Sati
- Translational Neuroradiology Section, National Institute of Neurological Disorders and Stroke, National Institutes of Health, Bethesda, MD, USA; Department of Neurology, Cedars-Sinai Medical Center, Los Angeles, CA, USA
| | - Daniel S Reich
- Translational Neuroradiology Section, National Institute of Neurological Disorders and Stroke, National Institutes of Health, Bethesda, MD, USA
| | - Cristina Granziera
- Translational Imaging in Neurology (ThINk) Basel, Department of Biomedical Engineering, Faculty of Medicine, University Hospital Basel and University of Basel, Switzerland; Neurologic Clinic and Policlinic, MS Center and Research Center for Clinical Neuroimmunology and Neuroscience Basel (RC2NB), University Hospital Basel and University of Basel, Basel, Switzerland
| | - Martina Absinta
- IRCCS San Raffaele Hospital and Vita-Salute San Raffaele University, Milan, Italy; Department of Neurology, Johns Hopkins University School of Medicine, Baltimore, MD, USA
| | - Meritxell Bach Cuadra
- CIBM Center for Biomedical Imaging, Switzerland; Radiology Department, Lausanne University and University Hospital, Switzerland
| |
Collapse
|
34
|
Andresen J, Uzunova H, Ehrhardt J, Kepp T, Handels H. Image registration and appearance adaptation in non-correspondent image regions for new MS lesions detection. Front Neurosci 2022; 16:981523. [PMID: 36161180 PMCID: PMC9490269 DOI: 10.3389/fnins.2022.981523] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2022] [Accepted: 08/18/2022] [Indexed: 11/25/2022] Open
Abstract
Manual detection of newly formed lesions in multiple sclerosis is an important but tedious and difficult task. Several approaches for automating the detection of new lesions have recently been proposed, but they tend to either overestimate the actual amount of new lesions or to miss many lesions. In this paper, an image registration convolutional neural network (CNN) that adapts the baseline image to the follow-up image by spatial deformations and simulation of new lesions is proposed. Simultaneously, segmentations of new lesions are generated, which are shown to reliably estimate the real new lesion load and to separate stable and progressive patients. Several applications of the proposed network emerge: image registration, detection and segmentation of new lesions, and modeling of new MS lesions. The modeled lesions offer the possibility to investigate the intensity profile of new lesions.
Collapse
Affiliation(s)
- Julia Andresen
- Institute of Medical Informatics, University of Lübeck, Lübeck, Germany
- *Correspondence: Julia Andresen
| | - Hristina Uzunova
- German Research Center for Artificial Intelligence, Lübeck, Germany
| | - Jan Ehrhardt
- Institute of Medical Informatics, University of Lübeck, Lübeck, Germany
- German Research Center for Artificial Intelligence, Lübeck, Germany
| | - Timo Kepp
- Institute of Medical Informatics, University of Lübeck, Lübeck, Germany
| | - Heinz Handels
- Institute of Medical Informatics, University of Lübeck, Lübeck, Germany
- German Research Center for Artificial Intelligence, Lübeck, Germany
| |
Collapse
|
35
|
Hitziger S, Ling WX, Fritz T, D'Albis T, Lemke A, Grilo J. Triplanar U-Net with lesion-wise voting for the segmentation of new lesions on longitudinal MRI studies. Front Neurosci 2022; 16:964250. [PMID: 36033604 PMCID: PMC9412001 DOI: 10.3389/fnins.2022.964250] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/08/2022] [Accepted: 07/22/2022] [Indexed: 11/13/2022] Open
Abstract
We present a deep learning method for the segmentation of new lesions in longitudinal FLAIR MRI sequences acquired at two different time points. In our approach, the 3D volumes are processed slice-wise across the coronal, axial, and sagittal planes and the predictions from the three orientations are merged using an optimized voting strategy. Our method achieved best F1 score (0.541) among all participating methods in the MICCAI 2021 challenge Multiple sclerosis new lesions segmentation (MSSEG-2). Moreover, we show that our method is on par with the challenge's expert neuroradiologists: on an unbiased ground truth, our method achieves results comparable to those of the four experts in terms of detection (F1 score) and segmentation accuracy (Dice score).
Collapse
|
36
|
Diaz-Hurtado M, Martínez-Heras E, Solana E, Casas-Roma J, Llufriu S, Kanber B, Prados F. Recent advances in the longitudinal segmentation of multiple sclerosis lesions on magnetic resonance imaging: a review. Neuroradiology 2022; 64:2103-2117. [PMID: 35864180 DOI: 10.1007/s00234-022-03019-3] [Citation(s) in RCA: 7] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/29/2022] [Accepted: 07/12/2022] [Indexed: 01/18/2023]
Abstract
Multiple sclerosis (MS) is a chronic autoimmune disease characterized by demyelinating lesions that are often visible on magnetic resonance imaging (MRI). Segmentation of these lesions can provide imaging biomarkers of disease burden that can help monitor disease progression and the imaging response to treatment. Manual delineation of MRI lesions is tedious and prone to subjective bias, while automated lesion segmentation methods offer objectivity and speed, the latter being particularly important when analysing large datasets. Lesion segmentation can be broadly categorised into two groups: cross-sectional methods, which use imaging data acquired at a single time-point to characterise MRI lesions; and longitudinal methods, which use imaging data from the same subject acquired at two or more different time-points to characterise lesions over time. The main objective of longitudinal segmentation approaches is to more accurately detect the presence of new MS lesions and the growth or remission of existing lesions, which may be effective biomarkers of disease progression and treatment response. This paper reviews articles on longitudinal MS lesion segmentation methods published over the past 10 years. These are divided into traditional machine learning methods and deep learning techniques. PubMed articles using longitudinal information and comparing fully automatic two time point segmentations in any step of the process were selected. Nineteen articles were reviewed. There is an increasing number of deep learning techniques for longitudinal MS lesion segmentation that are promising to help better understand disease progression.
Collapse
Affiliation(s)
| | - Eloy Martínez-Heras
- Center of Neuroimmunology, Laboratory of Advanced Imaging in Neuroimmunological Diseases, Hospital Clinic Barcelona, Institut d'Investigacions Biomèdiques August Pi I Sunyer (IDIBAPS), Universitat de Barcelona, Barcelona, Spain
| | - Elisabeth Solana
- Center of Neuroimmunology, Laboratory of Advanced Imaging in Neuroimmunological Diseases, Hospital Clinic Barcelona, Institut d'Investigacions Biomèdiques August Pi I Sunyer (IDIBAPS), Universitat de Barcelona, Barcelona, Spain
| | - Jordi Casas-Roma
- E-Health Center, Universitat Oberta de Catalunya, Barcelona, Spain
| | - Sara Llufriu
- Center of Neuroimmunology, Laboratory of Advanced Imaging in Neuroimmunological Diseases, Hospital Clinic Barcelona, Institut d'Investigacions Biomèdiques August Pi I Sunyer (IDIBAPS), Universitat de Barcelona, Barcelona, Spain
| | - Baris Kanber
- Centre for Medical Image Computing, Department of Medical Physics and Biomedical Engineering, University College London, London, UK.,National Institute for Health Research Biomedical Research Centre, University College London, London, UK.,Queen Square MS Centre, Department of Neuroinflammation, Faculty of Brain Sciences, UCL Institute of Neurology, University College London, London, UK
| | - Ferran Prados
- E-Health Center, Universitat Oberta de Catalunya, Barcelona, Spain.,Centre for Medical Image Computing, Department of Medical Physics and Biomedical Engineering, University College London, London, UK.,National Institute for Health Research Biomedical Research Centre, University College London, London, UK.,Queen Square MS Centre, Department of Neuroinflammation, Faculty of Brain Sciences, UCL Institute of Neurology, University College London, London, UK
| |
Collapse
|
37
|
Dimov AV, Nguyen TD, Gillen KM, Marcille M, Spincemaille P, Pitt D, Gauthier SA, Wang Y. Susceptibility source separation from gradient echo data using magnitude decay modeling. J Neuroimaging 2022; 32:852-859. [PMID: 35668022 DOI: 10.1111/jon.13014] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/13/2022] [Revised: 05/17/2022] [Accepted: 05/19/2022] [Indexed: 11/29/2022] Open
Abstract
BACKGROUND AND PURPOSE The objective is to demonstrate feasibility of separating magnetic sources in quantitative susceptibility mapping (QSM) by incorporating magnitude decay rates R 2 ∗ $R_2^{\rm{*}}$ in gradient echo (GRE) MRI. METHODS Magnetic susceptibility source separation was developed using R 2 ∗ $R_2^{\rm{*}}$ and compared with a prior method using R 2 ' = R 2 ∗ - R 2 ${R^{\prime}_2} = R_2^* - {R_2}$ that required an additional sequence to measure the transverse relaxation rate R2 . Both susceptibility separation methods were compared in multiple sclerosis (MS) patients (n = 17). Susceptibility values of negative sources estimated with R 2 ∗ $R_2^{\rm{*}}$ -based source separation in a set of enhancing MS lesions (n = 44) were correlated against longitudinal myelin water fraction (MWF) changes. RESULTS In in vivo data, linear regression of the estimated χ + ${\chi}^{+}$ and χ - ${\chi}^{-}$ susceptibility values between the R 2 ∗ $R_2^*$ - and the R 2 ' ${R^{\prime}_2}$ -based separation methods performed across 182 segmented lesions revealed correlation coefficient r = .96 and slope close .99. Correlation analysis in enhancing lesions revealed a significant positive association between the χ - ${\chi}^{-}$ increase at 1-year post-onset relative to 0 year and the MWF increase at 1 year relative to 0 year (β = -0.144, 95% confidence interval: [-0.199, -0.1], p = .0008) and good agreement between R 2 ' ${R^{\prime}_2}$ and R 2 ∗ $R_2^*$ methods (r = .79, slope = .95). CONCLUSIONS Separation of magnetic sources based solely on GRE complex data is feasible by combining magnitude decay rate modeling and phase-based QSM and χ - ${\chi}^{-}$ change may serve as a biomarker for myelin recovery or damage in acute MS lesions.
Collapse
Affiliation(s)
- Alexey V Dimov
- Department of Radiology, Weill Cornell Medicine, New York, New York, USA
| | - Thanh D Nguyen
- Department of Radiology, Weill Cornell Medicine, New York, New York, USA
| | - Kelly M Gillen
- Department of Radiology, Weill Cornell Medicine, New York, New York, USA
| | - Melanie Marcille
- Department of Neurology, Weill Cornell Medicine, New York, New York, USA
| | | | - David Pitt
- Department of Neurology, Yale School of Medicine, New Haven, Connecticut, USA
| | - Susan A Gauthier
- Department of Neurology, Weill Cornell Medicine, New York, New York, USA
| | - Yi Wang
- Department of Radiology, Weill Cornell Medicine, New York, New York, USA
| |
Collapse
|
38
|
Role of artificial intelligence in MS clinical practice. Neuroimage Clin 2022; 35:103065. [PMID: 35661470 PMCID: PMC9163993 DOI: 10.1016/j.nicl.2022.103065] [Citation(s) in RCA: 22] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/23/2021] [Revised: 05/04/2022] [Accepted: 05/26/2022] [Indexed: 11/24/2022]
Abstract
For medical applications, machine learning (including deep learning) are the most commonly used artificial intelligence (AI) approaches. It can improve multiple sclerosis (MS) diagnosis, prognostication and treatment monitoring. Thanks to AI, MRI and cognitive phenotypes of MS patients were identified. AI can shorten MRI protocols for MS, allowing the application of advanced techniques. It can reduce the human effort for MRI analysis, especially for lesion segmentation.
Machine learning (ML) and its subset, deep learning (DL), are branches of artificial intelligence (AI) showing promising findings in the medical field, especially when applied to imaging data. Given the substantial role of MRI in the diagnosis and management of patients with multiple sclerosis (MS), this disease is an ideal candidate for the application of AI techniques. In this narrative review, we are going to discuss the potential applications of AI for MS clinical practice, together with their limitations. Among their several advantages, ML algorithms are able to automate repetitive tasks, to analyze more data in less time and to achieve higher accuracy and reproducibility than the human counterpart. To date, these algorithms have been applied to MS diagnosis, prognosis, disease and treatment monitoring. Other fields of application have been improvement of MRI protocols as well as automated lesion and tissue segmentation. However, several challenges remain, including a better understanding of the information selected by AI algorithms, appropriate multicenter and longitudinal validations of results and practical aspects regarding hardware and software integration. Finally, one cannot overemphasize the paramount importance of human supervision, in order to optimize the use and take full advantage of the potential of AI approaches.
Collapse
|
39
|
Balwant M. A Review on Convolutional Neural Networks for Brain Tumor Segmentation: Methods, Datasets, Libraries, and Future Directions. Ing Rech Biomed 2022. [DOI: 10.1016/j.irbm.2022.05.002] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
|
40
|
Hashemi M, Akhbari M, Jutten C. Delve into Multiple Sclerosis (MS) lesion exploration: A modified attention U-Net for MS lesion segmentation in Brain MRI. Comput Biol Med 2022; 145:105402. [PMID: 35344864 DOI: 10.1016/j.compbiomed.2022.105402] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/24/2021] [Revised: 02/18/2022] [Accepted: 03/10/2022] [Indexed: 12/27/2022]
Abstract
Multiple Sclerosis (MS) is a Central Nervous System (CNS) disease that Magnetic Resonance Imaging (MRI) system can detect and segment its lesions. Artificial Neural Networks (ANNs) recently reached a noticeable performance in finding MS lesions from MRI. U-Net and Attention U-Net are two of the most successful ANNs in the field of MS lesion segmentation. In this work, we proposed a framework to segment MS lesions in Fluid-Attenuated Inversion Recovery (FLAIR) and T2 MRI images by modified U-Net and modified Attention U-Net. For this purpose, we developed some extra preprocessing on MRI scans, made modifications in the loss function of U-Net and Attention U-Net, and proposed using the union of FLAIR and T2 predictions to reach a better performance. Results show that the union of FLAIR and T2 predicted masks by the modified Attention U-Net reaches the performance of 82.30% in terms of Dice Similarity Coefficient (DSC) in the test dataset, which is a considerable improvement compared to the previous works.
Collapse
Affiliation(s)
| | - Mahsa Akhbari
- Islamic Azad University of Science and Research Branch, Tehran, Iran.
| | - Christian Jutten
- GIPSA-Lab, Grenoble, and Institut Universitaire de France, France.
| |
Collapse
|
41
|
Thakur SP, Schindler MK, Bilello M, Bakas S. Clinically Deployed Computational Assessment of Multiple Sclerosis Lesions. Front Med (Lausanne) 2022; 9:797586. [PMID: 35372431 PMCID: PMC8968446 DOI: 10.3389/fmed.2022.797586] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/18/2021] [Accepted: 02/17/2022] [Indexed: 02/05/2023] Open
Abstract
Multiple Sclerosis (MS) is a demyelinating disease of the central nervous system that affects nearly 1 million adults in the United States. Magnetic Resonance Imaging (MRI) plays a vital role in diagnosis and treatment monitoring in MS patients. In particular, follow-up MRI with T2-FLAIR images of the brain, depicting white matter lesions, is the mainstay for monitoring disease activity and making treatment decisions. In this article, we present a computational approach that has been deployed and integrated into a real-world routine clinical workflow, focusing on two tasks: (a) detecting new disease activity in MS patients, and (b) determining the necessity for injecting Gadolinium Based Contract Agents (GBCAs). This computer-aided detection (CAD) software has been utilized for the former task on more than 19, 000 patients over the course of 10 years, while its added function of identifying patients who need GBCA injection, has been operative for the past 3 years, with > 85% sensitivity. The benefits of this approach are summarized in: (1) offering a reproducible and accurate clinical assessment of MS lesion patients, (2) reducing the adverse effects of GBCAs (and the deposition of GBCAs to the patient's brain) by identifying the patients who may benefit from injection, and (3) reducing healthcare costs, patients' discomfort, and caregivers' workload.
Collapse
Affiliation(s)
- Siddhesh P. Thakur
- Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, PA, United States,Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States,Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States
| | - Matthew K. Schindler
- Department of Neurology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States
| | - Michel Bilello
- Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, PA, United States,Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States,Michel Bilello
| | - Spyridon Bakas
- Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, PA, United States,Department of Radiology, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States,Department of Pathology and Laboratory Medicine, Perelman School of Medicine, University of Pennsylvania, Philadelphia, PA, United States,*Correspondence: Spyridon Bakas
| |
Collapse
|
42
|
Abstract
Machine learning techniques used in computer-aided medical image analysis usually suffer from the domain shift problem caused by different distributions between source/reference data and target data. As a promising solution, domain adaptation has attracted considerable attention in recent years. The aim of this paper is to survey the recent advances of domain adaptation methods in medical image analysis. We first present the motivation of introducing domain adaptation techniques to tackle domain heterogeneity issues for medical image analysis. Then we provide a review of recent domain adaptation models in various medical image analysis tasks. We categorize the existing methods into shallow and deep models, and each of them is further divided into supervised, semi-supervised and unsupervised methods. We also provide a brief summary of the benchmark medical image datasets that support current domain adaptation research. This survey will enable researchers to gain a better understanding of the current status, challenges and future directions of this energetic research field.
Collapse
|
43
|
Ma Y, Zhang C, Cabezas M, Song Y, Tang Z, Liu D, Cai W, Barnett M, Wang C. Multiple Sclerosis Lesion Analysis in Brain Magnetic Resonance Images: Techniques and Clinical Applications. IEEE J Biomed Health Inform 2022; 26:2680-2692. [PMID: 35171783 DOI: 10.1109/jbhi.2022.3151741] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
Multiple sclerosis (MS) is a chronic inflammatory and degenerative disease of the central nervous system, characterized by the appearance of focal lesions in the white and gray matter that topographically correlate with an individual patients neurological symptoms and signs. Magnetic resonance imaging (MRI) provides detailed in-vivo structural information, permitting the quantification and categorization of MS lesions that critically inform disease management. Traditionally, MS lesions have been manually annotated on 2D MRI slices, a process that is inefficient and prone to inter-/intra-observer errors. Recently, automated statistical imaging analysis techniques have been proposed to detect and segment MS lesions based on MRI voxel intensity. However, their effectiveness is limited by the heterogeneity of both MRI data acquisition techniques and the appearance of MS lesions. By learning complex lesion representations directly from images, deep learning techniques have achieved remarkable breakthroughs in the MS lesion segmentation task. Here, we provide a comprehensive review of state-of-the-art automatic statistical and deep-learning MS segmentation methods and discuss current and future clinical applications. Further, we review technical strategies, such as domain adaptation, to enhance MS lesion segmentation in real-world clinical settings.
Collapse
|
44
|
Vedmurthy P, Pinto ALR, Lin DDM, Comi AM, Ou Y. Study protocol: retrospectively mining multisite clinical data to presymptomatically predict seizure onset for individual patients with Sturge-Weber. BMJ Open 2022; 12:e053103. [PMID: 35121603 PMCID: PMC8819809 DOI: 10.1136/bmjopen-2021-053103] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 11/11/2022] Open
Abstract
INTRODUCTION Secondary analysis of hospital-hosted clinical data can save time and cost compared with prospective clinical trials for neuroimaging biomarker development. We present such a study for Sturge-Weber syndrome (SWS), a rare neurovascular disorder that affects 1 in 20 000-50 000 newborns. Children with SWS are at risk for developing neurocognitive deficit by school age. A critical period for early intervention is before 2 years of age, but early diagnostic and prognostic biomarkers are lacking. We aim to retrospectively mine clinical data for SWS at two national centres to develop presymptomatic biomarkers. METHODS AND ANALYSIS We will retrospectively collect clinical, MRI and neurocognitive outcome data for patients with SWS who underwent brain MRI before 2 years of age at two national SWS care centres. Expert review of clinical records and MRI quality control will be used to refine the cohort. The merged multisite data will be used to develop algorithms for abnormality detection, lesion-symptom mapping to identify neural substrate and machine learning to predict individual outcomes (presence or absence of seizures) by 2 years of age. Presymptomatic treatment in 0-2 years and before seizure onset may delay or prevent the onset of seizures by 2 years of age, and thereby improve neurocognitive outcomes. The proposed work, if successful, will be one of the largest and most comprehensive multisite databases for the presymptomatic phase of this rare disease. ETHICS AND DISSEMINATION This study involves human participants and was approved by Boston Children's Hospital Institutional Review Board: IRB-P00014482 and IRB-P00025916 Johns Hopkins School of Medicine Institutional Review Board: NA_00043846. Participants gave informed consent to participate in the study before taking part. The Institutional Review Boards at Kennedy Krieger Institute and Boston Children's Hospital approval have been obtained at each site to retrospectively study this data. Results will be disseminated by presentations, publication and sharing of algorithms generated.
Collapse
Affiliation(s)
- Pooja Vedmurthy
- Department of Neurology and Developmental Medicine, Hugo Moser Research Institute, Baltimore, Maryland, USA
- Department of Neurology and Pediatrics, Kennedy Krieger Institute, Baltimore, MD, USA
| | - Anna L R Pinto
- Department of Neurology, Division of Epilepsy, Harvard Medical School, Boston, Massachusetts, USA
| | - Doris D M Lin
- Neuroradiology, Johns Hopkins School of Medicine, Baltimore, Maryland, USA
| | - Anne M Comi
- Department of Neurology and Developmental Medicine, Hugo Moser Research Institute, Baltimore, Maryland, USA
- Department of Neurology and Pediatrics, Kennedy Krieger Institute, Baltimore, MD, USA
- Department of Neurology and Pediatrics, Johns Hopkins School of Medicine, Baltimore, Maryland, USA
| | - Yangming Ou
- Fetal-Neonatal Neuroimaging and Developmental Science Center, Boston Children's Hospital, Boston, Massachusetts, USA
- Computational Health Informatics Program, Boston Children's Hospital, Boston, Massachusetts, USA
- Department of Radiology, Boston Children's Hospital; Harvard Medical School, Boston, MA, USA
| |
Collapse
|
45
|
Lesion Volume Quantification Using Two Convolutional Neural Networks in MRIs of Multiple Sclerosis Patients. Diagnostics (Basel) 2022; 12:diagnostics12020230. [PMID: 35204321 PMCID: PMC8870921 DOI: 10.3390/diagnostics12020230] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/30/2021] [Revised: 12/23/2021] [Accepted: 01/14/2022] [Indexed: 01/18/2023] Open
Abstract
Background: Multiple sclerosis (MS) is a neurologic disease of the central nervous system which affects almost three million people worldwide. MS is characterized by a demyelination process that leads to brain lesions, allowing these affected areas to be visualized with magnetic resonance imaging (MRI). Deep learning techniques, especially computational algorithms based on convolutional neural networks (CNNs), have become a frequently used algorithm that performs feature self-learning and enables segmentation of structures in the image useful for quantitative analysis of MRIs, including quantitative analysis of MS. To obtain quantitative information about lesion volume, it is important to perform proper image preprocessing and accurate segmentation. Therefore, we propose a method for volumetric quantification of lesions on MRIs of MS patients using automatic segmentation of the brain and lesions by two CNNs. Methods: We used CNNs at two different moments: the first to perform brain extraction, and the second for lesion segmentation. This study includes four independent MRI datasets: one for training the brain segmentation models, two for training the lesion segmentation model, and one for testing. Results: The proposed brain detection architecture using binary cross-entropy as the loss function achieved a 0.9786 Dice coefficient, 0.9969 accuracy, 0.9851 precision, 0.9851 sensitivity, and 0.9985 specificity. In the second proposed framework for brain lesion segmentation, we obtained a 0.8893 Dice coefficient, 0.9996 accuracy, 0.9376 precision, 0.8609 sensitivity, and 0.9999 specificity. After quantifying the lesion volume of all patients from the test group using our proposed method, we obtained a mean value of 17,582 mm3. Conclusions: We concluded that the proposed algorithm achieved accurate lesion detection and segmentation with reproducibility corresponding to state-of-the-art software tools and manual segmentation. We believe that this quantification method can add value to treatment monitoring and routine clinical evaluation of MS patients.
Collapse
|
46
|
Sadeghibakhi M, Pourreza H, Mahyar H. Multiple Sclerosis Lesions Segmentation Using Attention-Based CNNs in FLAIR Images. IEEE JOURNAL OF TRANSLATIONAL ENGINEERING IN HEALTH AND MEDICINE 2022; 10:1800411. [PMID: 35711337 PMCID: PMC9191687 DOI: 10.1109/jtehm.2022.3172025] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/05/2022] [Revised: 03/05/2022] [Accepted: 04/08/2022] [Indexed: 11/17/2022]
Abstract
Objective: Multiple Sclerosis (MS) is an autoimmune and demyelinating disease that leads to lesions in the central nervous system. This disease can be tracked and diagnosed using Magnetic Resonance Imaging (MRI). A multitude of multimodality automatic biomedical approaches are used to segment lesions that are not beneficial for patients in terms of cost, time, and usability. The authors of the present paper propose a method employing just one modality (FLAIR image) to segment MS lesions accurately. Methods: A patch-based Convolutional Neural Network (CNN) is designed, inspired by 3D-ResNet and spatial-channel attention module, to segment MS lesions. The proposed method consists of three stages: (1) the Contrast-Limited Adaptive Histogram Equalization (CLAHE) is applied to the original images and concatenated to the extracted edges to create 4D images; (2) the patches of size \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{upgreek}
\usepackage{mathrsfs}
\setlength{\oddsidemargin}{-69pt}
\begin{document}
}{}$80\times 80\times 80\times2$
\end{document} are randomly selected from the 4D images; and (3) the extracted patches are passed into an attention-based CNN which is used to segment the lesions. Finally, the proposed method was compared to previous studies of the same dataset. Results: The current study evaluates the model with a test set of ISIB challenge data. Experimental results illustrate that the proposed approach significantly surpasses existing methods of Dice similarity and Absolute Volume Difference while the proposed method uses just one modality (FLAIR) to segment the lesions. Conclusion: The authors have introduced an automated approach to segment the lesions, which is based on, at most, two modalities as an input. The proposed architecture comprises convolution, deconvolution, and an SCA-VoxRes module as an attention module. The results show, that the proposed method outperforms well compared to other methods.
Collapse
Affiliation(s)
- Mehdi Sadeghibakhi
- Department of Computer Engineering, Faculty of Engineering, MV Laboratory, Ferdowsi University of Mashhad, Mashhad, Iran
| | - Hamidreza Pourreza
- Department of Computer Engineering, Faculty of Engineering, MV Laboratory, Ferdowsi University of Mashhad, Mashhad, Iran
| | - Hamidreza Mahyar
- Faculty of Engineering, W Booth School of Engineering Practice and Technology, McMaster University, Hamilton, ON, Canada
| |
Collapse
|
47
|
QSMRim-Net: Imbalance-aware learning for identification of chronic active multiple sclerosis lesions on quantitative susceptibility maps. Neuroimage Clin 2022; 34:102979. [PMID: 35247730 PMCID: PMC8892132 DOI: 10.1016/j.nicl.2022.102979] [Citation(s) in RCA: 11] [Impact Index Per Article: 5.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2022] [Revised: 02/27/2022] [Accepted: 02/28/2022] [Indexed: 11/23/2022]
Abstract
BACKGROUND AND PURPOSE Chronic active multiple sclerosis (MS) lesions are characterized by a paramagnetic rim at the edge of the lesion and are associated with increased disability in patients. Quantitative susceptibility mapping (QSM) is an MRI technique that is sensitive to chronic active lesions, termed rim + lesions on the QSM. We present QSMRim-Net, a data imbalance-aware deep neural network that fuses lesion-level radiomic and convolutional image features for automated identification of rim + lesions on QSM. METHODS QSM and T2-weighted-Fluid-Attenuated Inversion Recovery (T2-FLAIR) MRI of the brain were collected at 3 T for 172 MS patients. Rim + lesions were manually annotated by two human experts, followed by consensus from a third expert, for a total of 177 rim + and 3986 rim negative (rim-) lesions. Our automated rim + detection algorithm, QSMRim-Net, consists of a two-branch feature extraction network and a synthetic minority oversampling network to classify rim + lesions. The first network branch is for image feature extraction from the QSM and T2-FLAIR, and the second network branch is a fully connected network for QSM lesion-level radiomic feature extraction. The oversampling network is designed to increase classification performance with imbalanced data. RESULTS On a lesion-level, in a five-fold cross validation framework, the proposed QSMRim-Net detected rim + lesions with a partial area under the receiver operating characteristic curve (pROC AUC) of 0.760, where clinically relevant false positive rates of less than 0.1 were considered. The method attained an area under the precision recall curve (PR AUC) of 0.704. QSMRim-Net out-performed other state-of-the-art methods applied to the QSM on both pROC AUC and PR AUC. On a subject-level, comparing the predicted rim + lesion count and the human expert annotated count, QSMRim-Net achieved the lowest mean square error of 0.98 and the highest correlation of 0.89 (95% CI: 0.86, 0.92). CONCLUSION This study develops a novel automated deep neural network for rim + MS lesion identification using T2-FLAIR and QSM images.
Collapse
|
48
|
MTANS: Multi-Scale Mean Teacher Combined Adversarial Network with Shape-Aware Embedding for Semi-Supervised Brain Lesion Segmentation. Neuroimage 2021; 244:118568. [PMID: 34508895 DOI: 10.1016/j.neuroimage.2021.118568] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2021] [Accepted: 09/07/2021] [Indexed: 11/23/2022] Open
Abstract
The annotation of brain lesion images is a key step in clinical diagnosis and treatment of a wide spectrum of brain diseases. In recent years, segmentation methods based on deep learning have gained unprecedented popularity, leveraging a large amount of data with high-quality voxel-level annotations. However, due to the limited time clinicians can provide for the cumbersome task of manual image segmentation, semi-supervised medical image segmentation methods present an alternative solution as they require only a few labeled samples for training. In this paper, we propose a novel semi-supervised segmentation framework that combines improved mean teacher and adversarial network. Specifically, our framework consists of (i) a student model and a teacher model for segmenting the target and generating the signed distance maps of object surfaces, and (ii) a discriminator network for extracting hierarchical features and distinguishing the signed distance maps of labeled and unlabeled data. Besides, based on two different adversarial learning processes, a multi-scale feature consistency loss derived from the student and teacher models is proposed, and a shape-aware embedding scheme is integrated into our framework. We evaluated the proposed method on the public brain lesion datasets from ISBI 2015, ISLES 2015, and BRATS 2018 for the multiple sclerosis lesion, ischemic stroke lesion, and brain tumor segmentation respectively. Experiments demonstrate that our method can effectively leverage unlabeled data while outperforming the supervised baseline and other state-of-the-art semi-supervised methods trained with the same labeled data. The proposed framework is suitable for joint training of limited labeled data and additional unlabeled data, which is expected to reduce the effort of obtaining annotated images.
Collapse
|
49
|
Multiple sclerosis lesions segmentation from multiple experts: The MICCAI 2016 challenge dataset. Neuroimage 2021; 244:118589. [PMID: 34563682 DOI: 10.1016/j.neuroimage.2021.118589] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2021] [Revised: 09/03/2021] [Accepted: 09/16/2021] [Indexed: 11/23/2022] Open
Abstract
MRI plays a crucial role in multiple sclerosis diagnostic and patient follow-up. In particular, the delineation of T2-FLAIR hyperintense lesions is crucial although mostly performed manually - a tedious task. Many methods have thus been proposed to automate this task. However, sufficiently large datasets with a thorough expert manual segmentation are still lacking to evaluate these methods. We present a unique dataset for MS lesions segmentation evaluation. It consists of 53 patients acquired on 4 different scanners with a harmonized protocol. Hyperintense lesions on FLAIR were manually delineated on each patient by 7 experts with control on T2 sequence, and gathered in a consensus segmentation for evaluation. We provide raw and preprocessed data and a split of the dataset into training and testing data, the latter including data from a scanner not present in the training dataset. We strongly believe that this dataset will become a reference in MS lesions segmentation evaluation, allowing to evaluate many aspects: evaluation of performance on unseen scanner, comparison to individual experts performance, comparison to other challengers who already used this dataset, etc.
Collapse
|
50
|
Kamraoui RA, Ta VT, Tourdias T, Mansencal B, Manjon JV, Coupé P. DeepLesionBrain: Towards a broader deep-learning generalization for multiple sclerosis lesion segmentation. Med Image Anal 2021; 76:102312. [PMID: 34894571 DOI: 10.1016/j.media.2021.102312] [Citation(s) in RCA: 17] [Impact Index Per Article: 5.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/14/2020] [Revised: 10/18/2021] [Accepted: 11/22/2021] [Indexed: 11/29/2022]
Abstract
Recently, segmentation methods based on Convolutional Neural Networks (CNNs) showed promising performance in automatic Multiple Sclerosis (MS) lesions segmentation. These techniques have even outperformed human experts in controlled evaluation conditions such as Longitudinal MS Lesion Segmentation Challenge (ISBI Challenge). However, state-of-the-art approaches trained to perform well on highly-controlled datasets fail to generalize on clinical data from unseen datasets. Instead of proposing another improvement of the segmentation accuracy, we propose a novel method robust to domain shift and performing well on unseen datasets, called DeepLesionBrain (DLB). This generalization property results from three main contributions. First, DLB is based on a large group of compact 3D CNNs. This spatially distributed strategy aims to produce a robust prediction despite the risk of generalization failure of some individual networks. Second, we propose a hierarchical specialization learning (HSL) by pre-training a generic network over the whole brain, before using its weights as initialization to locally specialized networks. By this end, DLB learns both generic features extracted at global image level and specific features extracted at local image level. Finally, DLB includes a new image quality data augmentation to reduce dependency to training data specificity (e.g., acquisition protocol). DLB generalization was validated in cross-dataset experiments on MSSEG'16, ISBI challenge, and in-house datasets. During experiments, DLB showed higher segmentation accuracy, better segmentation consistency and greater generalization performance compared to state-of-the-art methods. Therefore, DLB offers a robust framework well-suited for clinical practice.
Collapse
Affiliation(s)
| | - Vinh-Thong Ta
- Univ. Bordeaux, Bordeaux INP, CNRS, LaBRI, UMR5800, PICTURA, Talence F-33400, France
| | - Thomas Tourdias
- Service de Neuroimagerie Diagnostique et Thérapeutique, Univ. Bordeaux, Bordeaux F-33000, France; Univ. Bordeaux, INSERM, Neurocentre Magendie, U1215, Bordeaux F-3300, France
| | - Boris Mansencal
- Univ. Bordeaux, Bordeaux INP, CNRS, LaBRI, UMR5800, PICTURA, Talence F-33400, France
| | - José V Manjon
- ITACA, Universitat Politécnica de Valéncia, Valencia 46022, Spain
| | - Pierrick Coupé
- Univ. Bordeaux, Bordeaux INP, CNRS, LaBRI, UMR5800, PICTURA, Talence F-33400, France
| |
Collapse
|