1
|
Svanera M, Savardi M, Signoroni A, Benini S, Muckli L. Fighting the scanner effect in brain MRI segmentation with a progressive level-of-detail network trained on multi-site data. Med Image Anal 2024; 93:103090. [PMID: 38241763 DOI: 10.1016/j.media.2024.103090] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2023] [Revised: 10/30/2023] [Accepted: 01/12/2024] [Indexed: 01/21/2024]
Abstract
Many clinical and research studies of the human brain require accurate structural MRI segmentation. While traditional atlas-based methods can be applied to volumes from any acquisition site, recent deep learning algorithms ensure high accuracy only when tested on data from the same sites exploited in training (i.e., internal data). Performance degradation experienced on external data (i.e., unseen volumes from unseen sites) is due to the inter-site variability in intensity distributions, and to unique artefacts caused by different MR scanner models and acquisition parameters. To mitigate this site-dependency, often referred to as the scanner effect, we propose LOD-Brain, a 3D convolutional neural network with progressive levels-of-detail (LOD), able to segment brain data from any site. Coarser network levels are responsible for learning a robust anatomical prior helpful in identifying brain structures and their locations, while finer levels refine the model to handle site-specific intensity distributions and anatomical variations. We ensure robustness across sites by training the model on an unprecedentedly rich dataset aggregating data from open repositories: almost 27,000 T1w volumes from around 160 acquisition sites, at 1.5 - 3T, from a population spanning from 8 to 90 years old. Extensive tests demonstrate that LOD-Brain produces state-of-the-art results, with no significant difference in performance between internal and external sites, and robust to challenging anatomical variations. Its portability paves the way for large-scale applications across different healthcare institutions, patient populations, and imaging technology manufacturers. Code, model, and demo are available on the project website.
Collapse
Affiliation(s)
- Michele Svanera
- Center for Cognitive Neuroimaging at the School of Psychology & Neuroscience, University of Glasgow, UK.
| | - Mattia Savardi
- Department of Medical and Surgical Specialties, Radiological Sciences, and Public Health, University of Brescia, Italy
| | - Alberto Signoroni
- Department of Medical and Surgical Specialties, Radiological Sciences, and Public Health, University of Brescia, Italy
| | - Sergio Benini
- Department of Information Engineering, University of Brescia, Italy
| | - Lars Muckli
- Center for Cognitive Neuroimaging at the School of Psychology & Neuroscience, University of Glasgow, UK
| |
Collapse
|
2
|
A survey of feature detection methods for localisation of plain sections of axial brain magnetic resonance imaging. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2023.104611] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/24/2023]
|
3
|
Chen X, Peng Y, Guo Y, Sun J, Li D, Cui J. MLRD-Net: 3D multiscale local cross-channel residual denoising network for MRI-based brain tumor segmentation. Med Biol Eng Comput 2022; 60:3377-3395. [DOI: 10.1007/s11517-022-02673-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/26/2021] [Accepted: 09/17/2022] [Indexed: 11/11/2022]
|
4
|
Zhang D, Han J, Cheng G, Yang MH. Weakly Supervised Object Localization and Detection: A Survey. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2022; 44:5866-5885. [PMID: 33877967 DOI: 10.1109/tpami.2021.3074313] [Citation(s) in RCA: 31] [Impact Index Per Article: 15.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
As an emerging and challenging problem in the computer vision community, weakly supervised object localization and detection plays an important role for developing new generation computer vision systems and has received significant attention in the past decade. As methods have been proposed, a comprehensive survey of these topics is of great importance. In this work, we review (1) classic models, (2) approaches with feature representations from off-the-shelf deep networks, (3) approaches solely based on deep learning, and (4) publicly available datasets and standard evaluation metrics that are widely used in this field. We also discuss the key challenges in this field, development history of this field, advantages/disadvantages of the methods in each category, the relationships between methods in different categories, applications of the weakly supervised object localization and detection methods, and potential future directions to further promote the development of this research field.
Collapse
|
5
|
Zhao YX, Zhang YM, Song M, Liu CL. Adaptable Global Network for Whole-Brain Segmentation with Symmetry Consistency Loss. Cognit Comput 2022. [DOI: 10.1007/s12559-022-10011-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/03/2022]
|
6
|
Ranjbar S, Singleton KW, Curtin L, Rickertsen CR, Paulson LE, Hu LS, Mitchell JR, Swanson KR. Weakly Supervised Skull Stripping of Magnetic Resonance Imaging of Brain Tumor Patients. FRONTIERS IN NEUROIMAGING 2022; 1:832512. [PMID: 37555156 PMCID: PMC10406204 DOI: 10.3389/fnimg.2022.832512] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/09/2021] [Accepted: 02/21/2022] [Indexed: 08/10/2023]
Abstract
Automatic brain tumor segmentation is particularly challenging on magnetic resonance imaging (MRI) with marked pathologies, such as brain tumors, which usually cause large displacement, abnormal appearance, and deformation of brain tissue. Despite an abundance of previous literature on learning-based methodologies for MRI segmentation, few works have focused on tackling MRI skull stripping of brain tumor patient data. This gap in literature can be associated with the lack of publicly available data (due to concerns about patient identification) and the labor-intensive nature of generating ground truth labels for model training. In this retrospective study, we assessed the performance of Dense-Vnet in skull stripping brain tumor patient MRI trained on our large multi-institutional brain tumor patient dataset. Our data included pretreatment MRI of 668 patients from our in-house institutional review board-approved multi-institutional brain tumor repository. Because of the absence of ground truth, we used imperfect automatically generated training labels using SPM12 software. We trained the network using common MRI sequences in oncology: T1-weighted with gadolinium contrast, T2-weighted fluid-attenuated inversion recovery, or both. We measured model performance against 30 independent brain tumor test cases with available manual brain masks. All images were harmonized for voxel spacing and volumetric dimensions before model training. Model training was performed using the modularly structured deep learning platform NiftyNet that is tailored toward simplifying medical image analysis. Our proposed approach showed the success of a weakly supervised deep learning approach in MRI brain extraction even in the presence of pathology. Our best model achieved an average Dice score, sensitivity, and specificity of, respectively, 94.5, 96.4, and 98.5% on the multi-institutional independent brain tumor test set. To further contextualize our results within existing literature on healthy brain segmentation, we tested the model against healthy subjects from the benchmark LBPA40 dataset. For this dataset, the model achieved an average Dice score, sensitivity, and specificity of 96.2, 96.6, and 99.2%, which are, although comparable to other publications, slightly lower than the performance of models trained on healthy patients. We associate this drop in performance with the use of brain tumor data for model training and its influence on brain appearance.
Collapse
Affiliation(s)
- Sara Ranjbar
- Mathematical NeuroOncology Lab, Department of Neurosurgery, Mayo Clinic, Phoenix, AZ, United States
| | - Kyle W. Singleton
- Mathematical NeuroOncology Lab, Department of Neurosurgery, Mayo Clinic, Phoenix, AZ, United States
| | - Lee Curtin
- Mathematical NeuroOncology Lab, Department of Neurosurgery, Mayo Clinic, Phoenix, AZ, United States
| | - Cassandra R. Rickertsen
- Mathematical NeuroOncology Lab, Department of Neurosurgery, Mayo Clinic, Phoenix, AZ, United States
| | - Lisa E. Paulson
- Mathematical NeuroOncology Lab, Department of Neurosurgery, Mayo Clinic, Phoenix, AZ, United States
| | - Leland S. Hu
- Mathematical NeuroOncology Lab, Department of Neurosurgery, Mayo Clinic, Phoenix, AZ, United States
- Department of Diagnostic Imaging and Interventional Radiology, Mayo Clinic, Phoenix, AZ, United States
| | - Joseph Ross Mitchell
- Department of Medicine, Faculty of Medicine & Dentistry and the Alberta Machine Intelligence Institute, University of Alberta, Edmonton, AB, Canada
- Provincial Clinical Excellence Portfolio, Alberta Health Services, Edmonton, AB, Canada
| | - Kristin R. Swanson
- Mathematical NeuroOncology Lab, Department of Neurosurgery, Mayo Clinic, Phoenix, AZ, United States
| |
Collapse
|
7
|
Çelik G, Talu MF. A new 3D MRI segmentation method based on Generative Adversarial Network and Atrous Convolution. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2021.103155] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/18/2023]
|
8
|
Spindler M, Thiel CM. Quantitative magnetic resonance imaging for segmentation and white matter extraction of the hypothalamus. J Neurosci Res 2021; 100:564-577. [PMID: 34850453 DOI: 10.1002/jnr.24988] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/08/2021] [Revised: 10/05/2021] [Accepted: 10/12/2021] [Indexed: 11/09/2022]
Abstract
Since the hypothalamus is involved in many neuroendocrine, metabolic, and affective disorders, detailed hypothalamic imaging has become of major interest to better characterize disease-induced tissue damages and abnormalities. Still, image contrast of conventional anatomical magnetic resonance imaging lacks morphological detail, thus complicating complete and precise segmentation of the hypothalamus. The hypothalamus' position lateral to the third ventricle and close proximity to white matter tracts including the optic tract, fornix, and mammillothalamic tract display one of the remaining shortcomings of hypothalamic segmentation, as reliable exclusion of white matter is not yet possible. Recent studies found that quantitative magnetic resonance imaging (qMRI), a method to create maps of different standardized tissue contents, improved segmentation of cortical and subcortical brain regions. So far, this has not been tested for the hypothalamus. Therefore, in this study, we investigated the usability of qMRI and diffusion MRI for the purpose of detailed and reproducible manual segmentation of the hypothalamus and data-driven white matter extraction and compared our results to recent state-of-the-art segmentations. Our results show that qMRI presents good contrast for delineation of the hypothalamus and white matter, and that the properties of these images differ between subunits, such that they can be used to reliably exclude white matter from hypothalamic tissue. We propose that qMRI poses a useful addition to detailed hypothalamic segmentation and volumetry.
Collapse
Affiliation(s)
- Melanie Spindler
- Biological Psychology, Department of Psychology, School of Medicine and Health Sciences, Carl von Ossietzky Universität Oldenburg, Oldenburg, Germany
| | - Christiane M Thiel
- Biological Psychology, Department of Psychology, School of Medicine and Health Sciences, Carl von Ossietzky Universität Oldenburg, Oldenburg, Germany.,Cluster of Excellence "Hearing4all", Carl von Ossietzky Universität Oldenburg, Oldenburg, Germany.,Research Centre Neurosensory Science, Carl von Ossietzky Universität Oldenburg, Oldenburg, Germany
| |
Collapse
|
9
|
Svanera M, Benini S, Bontempi D, Muckli L. CEREBRUM-7T: Fast and Fully Volumetric Brain Segmentation of 7 Tesla MR Volumes. Hum Brain Mapp 2021; 42:5563-5580. [PMID: 34598307 PMCID: PMC8559470 DOI: 10.1002/hbm.25636] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2021] [Revised: 06/14/2021] [Accepted: 08/09/2021] [Indexed: 01/16/2023] Open
Abstract
Ultra-high-field magnetic resonance imaging (MRI) enables sub-millimetre resolution imaging of the human brain, allowing the study of functional circuits of cortical layers at the meso-scale. An essential step in many functional and structural neuroimaging studies is segmentation, the operation of partitioning the MR images in anatomical structures. Despite recent efforts in brain imaging analysis, the literature lacks in accurate and fast methods for segmenting 7-tesla (7T) brain MRI. We here present CEREBRUM-7T, an optimised end-to-end convolutional neural network, which allows fully automatic segmentation of a whole 7T T1w MRI brain volume at once, without partitioning the volume, pre-processing, nor aligning it to an atlas. The trained model is able to produce accurate multi-structure segmentation masks on six different classes plus background in only a few seconds. The experimental part, a combination of objective numerical evaluations and subjective analysis, confirms that the proposed solution outperforms the training labels it was trained on and is suitable for neuroimaging studies, such as layer functional MRI studies. Taking advantage of a fine-tuning operation on a reduced set of volumes, we also show how it is possible to effectively apply CEREBRUM-7T to different sites data. Furthermore, we release the code, 7T data, and other materials, including the training labels and the Turing test.
Collapse
Affiliation(s)
- Michele Svanera
- Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, UK
| | - Sergio Benini
- Department of Information Engineering, University of Brescia, Brescia, Italy
| | - Dennis Bontempi
- Department of Information Engineering, University of Brescia, Brescia, Italy
| | - Lars Muckli
- Institute of Neuroscience and Psychology, University of Glasgow, Glasgow, UK
| |
Collapse
|
10
|
Ribeiro FL, Bollmann S, Puckett AM. Predicting the retinotopic organization of human visual cortex from anatomy using geometric deep learning. Neuroimage 2021; 244:118624. [PMID: 34607019 DOI: 10.1016/j.neuroimage.2021.118624] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2021] [Revised: 09/13/2021] [Accepted: 09/27/2021] [Indexed: 10/20/2022] Open
Abstract
Whether it be in a single neuron or a more complex biological system like the human brain, form and function are often directly related. The functional organization of human visual cortex, for instance, is tightly coupled with the underlying anatomy with cortical shape having been shown to be a useful predictor of the retinotopic organization in early visual cortex. Although the current state-of-the-art in predicting retinotopic maps is able to account for gross individual differences, such models are unable to account for any idiosyncratic differences in the structure-function relationship from anatomical information alone due to their initial assumption of a template. Here we developed a geometric deep learning model capable of exploiting the actual structure of the cortex to learn the complex relationship between brain function and anatomy in human visual cortex such that more realistic and idiosyncratic maps could be predicted. We show that our neural network was not only able to predict the functional organization throughout the visual cortical hierarchy, but that it was also able to predict nuanced variations across individuals. Although we demonstrate its utility for modeling the relationship between structure and function in human visual cortex, our approach is flexible and well-suited for a range of other applications involving data structured in non-Euclidean spaces.
Collapse
Affiliation(s)
- Fernanda L Ribeiro
- School of Psychology, The University of Queensland, Saint Lucia, Brisbane, QLD 4072, Australia; Queensland Brain Institute, The University of Queensland, Brisbane, QLD 4072, Australia.
| | - Steffen Bollmann
- School of Information Technology and Electrical Engineering, The University of Queensland, Brisbane, QLD 4072, Australia
| | - Alexander M Puckett
- School of Psychology, The University of Queensland, Saint Lucia, Brisbane, QLD 4072, Australia; Queensland Brain Institute, The University of Queensland, Brisbane, QLD 4072, Australia
| |
Collapse
|
11
|
Thalamus Atrophy in the Peri-Pregnancy Period in Clinically Stable Multiple Sclerosis Patients: Preliminary Results. Brain Sci 2021; 11:brainsci11101270. [PMID: 34679335 PMCID: PMC8534211 DOI: 10.3390/brainsci11101270] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/20/2021] [Revised: 09/14/2021] [Accepted: 09/23/2021] [Indexed: 11/22/2022] Open
Abstract
Radiological activity in the post-partum period in MS patients is a well-known phenomenon, but there is no data concerning the influence of pregnancy on regional brain atrophy. The aim of this article was to investigate local brain atrophy in the peri-pregnancy period (PPP) in patients with MS. Thalamic volume (TV); corpus callosum volume (CCV) and classical MRI activity (new gadolinium enhancing lesions (Gd+), new T2 lesions, T1 lesions volume (T1LV) and T2 lesions volume (T2LV)) were analyzed in 12 clinically stable women with relapsing–remitting MS and with MRI performed in the PPP. We showed that there was a significant decrease in TV (p = 0.021) in the PPP. We also observed a significant increase in the T1 lesion volume (p = 0.028), new gadolinium-enhanced and new T2 lesions (in 46% and 77% of the scans, respectively) in the post-partum period. Our results suggest that the PPP in MS may be associated not only with classical MRI activity but, also, with regional brain atrophy.
Collapse
|
12
|
Weiss DA, Saluja R, Xie L, Gee JC, Sugrue LP, Pradhan A, Nick Bryan R, Rauschecker AM, Rudie JD. Automated multiclass tissue segmentation of clinical brain MRIs with lesions. NEUROIMAGE-CLINICAL 2021; 31:102769. [PMID: 34333270 PMCID: PMC8346689 DOI: 10.1016/j.nicl.2021.102769] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 03/15/2021] [Revised: 06/29/2021] [Accepted: 07/20/2021] [Indexed: 12/21/2022]
Abstract
A U-Net incorporating spatial prior information can successfully segment 6 brain tissue types. The U-Net was able to segment gray and white matter in the presence of lesions. The U-Net surpassed the performance of its source algorithm in an external dataset. Segmentations were produced in a hundredth of the time of its predecessor algorithm.
Delineation and quantification of normal and abnormal brain tissues on Magnetic Resonance Images is fundamental to the diagnosis and longitudinal assessment of neurological diseases. Here we sought to develop a convolutional neural network for automated multiclass tissue segmentation of brain MRIs that was robust at typical clinical resolutions and in the presence of a variety of lesions. We trained a 3D U-Net for full brain multiclass tissue segmentation from a prior atlas-based segmentation method on an internal dataset that consisted of 558 clinical T1-weighted brain MRIs (453/52/53; training/validation/test) of patients with one of 50 different diagnostic entities (n = 362) or with a normal brain MRI (n = 196). We then used transfer learning to refine our model on an external dataset that consisted of 7 patients with hand-labeled tissue types. We evaluated the tissue-wise and intra-lesion performance with different loss functions and spatial prior information in the validation set and applied the best performing model to the internal and external test sets. The network achieved an average overall Dice score of 0.87 and volume similarity of 0.97 in the internal test set. Further, the network achieved a median intra-lesion tissue segmentation accuracy of 0.85 inside lesions within white matter and 0.61 inside lesions within gray matter. After transfer learning, the network achieved an average overall Dice score of 0.77 and volume similarity of 0.96 in the external dataset compared to human raters. The network had equivalent or better performance than the original atlas-based method on which it was trained across all metrics and produced segmentations in a hundredth of the time. We anticipate that this pipeline will be a useful tool for clinical decision support and quantitative analysis of clinical brain MRIs in the presence of lesions.
Collapse
Affiliation(s)
- David A Weiss
- University of Pennsylvania, United States; University of California, San Francisco, United States.
| | | | - Long Xie
- University of Pennsylvania, United States
| | | | - Leo P Sugrue
- University of California, San Francisco, United States
| | | | | | | | | |
Collapse
|
13
|
Signoroni A, Savardi M, Benini S, Adami N, Leonardi R, Gibellini P, Vaccher F, Ravanelli M, Borghesi A, Maroldi R, Farina D. BS-Net: Learning COVID-19 pneumonia severity on a large chest X-ray dataset. Med Image Anal 2021; 71:102046. [PMID: 33862337 PMCID: PMC8010334 DOI: 10.1016/j.media.2021.102046] [Citation(s) in RCA: 53] [Impact Index Per Article: 17.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/06/2020] [Revised: 02/04/2021] [Accepted: 03/17/2021] [Indexed: 12/22/2022]
Abstract
In this work we design an end-to-end deep learning architecture for predicting, on Chest X-rays images (CXR), a multi-regional score conveying the degree of lung compromise in COVID-19 patients. Such semi-quantitative scoring system, namely Brixia score, is applied in serial monitoring of such patients, showing significant prognostic value, in one of the hospitals that experienced one of the highest pandemic peaks in Italy. To solve such a challenging visual task, we adopt a weakly supervised learning strategy structured to handle different tasks (segmentation, spatial alignment, and score estimation) trained with a "from-the-part-to-the-whole" procedure involving different datasets. In particular, we exploit a clinical dataset of almost 5,000 CXR annotated images collected in the same hospital. Our BS-Net demonstrates self-attentive behavior and a high degree of accuracy in all processing stages. Through inter-rater agreement tests and a gold standard comparison, we show that our solution outperforms single human annotators in rating accuracy and consistency, thus supporting the possibility of using this tool in contexts of computer-assisted monitoring. Highly resolved (super-pixel level) explainability maps are also generated, with an original technique, to visually help the understanding of the network activity on the lung areas. We also consider other scores proposed in literature and provide a comparison with a recently proposed non-specific approach. We eventually test the performance robustness of our model on an assorted public COVID-19 dataset, for which we also provide Brixia score annotations, observing good direct generalization and fine-tuning capabilities that highlight the portability of BS-Net in other clinical settings. The CXR dataset along with the source code and the trained model are publicly released for research purposes.
Collapse
Affiliation(s)
- Alberto Signoroni
- Department of Information Engineering, University of Brescia, Brescia, Italy.
| | - Mattia Savardi
- Department of Information Engineering, University of Brescia, Brescia, Italy
| | - Sergio Benini
- Department of Information Engineering, University of Brescia, Brescia, Italy
| | - Nicola Adami
- Department of Information Engineering, University of Brescia, Brescia, Italy
| | - Riccardo Leonardi
- Department of Information Engineering, University of Brescia, Brescia, Italy
| | - Paolo Gibellini
- Department of Medical and Surgical Specialties, Radiological Sciences, and Public Health, University of Brescia, Brescia, Italy
| | - Filippo Vaccher
- Department of Medical and Surgical Specialties, Radiological Sciences, and Public Health, University of Brescia, Brescia, Italy
| | - Marco Ravanelli
- Department of Medical and Surgical Specialties, Radiological Sciences, and Public Health, University of Brescia, Brescia, Italy
| | - Andrea Borghesi
- Department of Medical and Surgical Specialties, Radiological Sciences, and Public Health, University of Brescia, Brescia, Italy
| | - Roberto Maroldi
- Department of Medical and Surgical Specialties, Radiological Sciences, and Public Health, University of Brescia, Brescia, Italy
| | - Davide Farina
- Department of Medical and Surgical Specialties, Radiological Sciences, and Public Health, University of Brescia, Brescia, Italy
| |
Collapse
|
14
|
Lv D, Wang Y, Wang S, Zhang Q, Qi W, Li Y, Sun L. A Cascade-SEME network for COVID-19 detection in chest x-ray images. Med Phys 2021; 48:2337-2353. [PMID: 33778966 PMCID: PMC8250185 DOI: 10.1002/mp.14711] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2020] [Revised: 12/30/2020] [Accepted: 12/31/2020] [Indexed: 11/12/2022] Open
Abstract
PURPOSE The worldwide spread of the SARS-CoV-2 virus poses unprecedented challenges to medical resources and infection prevention and control measures around the world. In this case, a rapid and effective detection method for COVID-19 can not only relieve the pressure of the medical system but find and isolate patients in time, to a certain extent, slow down the development of the epidemic. In this paper, we propose a method that can quickly and accurately diagnose whether pneumonia is viral pneumonia, and classify viral pneumonia in a fine-grained way to diagnose COVID-19. METHODS We proposed a Cascade Squeeze-Excitation and Moment Exchange (Cascade-SEME) framework that can effectively detect COVID-19 cases by evaluating the chest x-ray images, where SE is the structure we designed in the network which has attention mechanism, and ME is a method for image enhancement from feature dimension. The framework integrates a model for a coarse level detection of virus cases among other forms of lung infection, and a model for fine-grained categorisation of pneumonia types identifying COVID-19 cases. In addition, a Regional Learning approach is proposed to mitigate the impact of non-lesion features on network training. The network output is also visualised, highlighting the likely areas of lesion, to assist experts' assessment and diagnosis of COVID-19. RESULTS Three datasets were used: a set of Chest x-ray Images for Classification with bacterial pneumonia, viral pneumonia and normal chest x-rays, a COVID chest x-ray dataset with COVID-19, and a Lung Segmentation dataset containing 1000 chest x-rays with masks in the lung region. We evaluated all the models on the test set. The results shows the proposed SEME structure significantly improves the performance of the models: in the task of pneumonia infection type diagnosis, the sensitivity, specificity, accuracy and F1 score of ResNet50 with SEME structure are significantly improved in each category, and the accuracy and AUC of the whole test set are also enhanced; in the detection task of COVID-19, the evaluation results shows that when SEME structure was added to the task, the sensitivities, specificities, accuracy and F1 scores of ResNet50 and DenseNet169 are improved. Although the sensitivities and specificities are not significantly promoted, SEME well balanced these two significant indicators. Regional learning also plays an important role. Experiments show that Regional Learning can effectively correct the impact of non-lesion features on the network, which can be seen in the Grad-CAM method. CONCLUSIONS Experiments show that after the application of SEME structure in the network, the performance of SEME-ResNet50 and SEME-DenseNet169 in both two datasets show a clear enhancement. And the proposed regional learning method effectively directs the network's attention to focus on relevant pathological regions in the lung radiograph, ensuring the performance of the proposed framework even when a small training set is used. The visual interpretation step using Grad-CAM finds that the region of attention on radiographs of different types of pneumonia are located in different regions of the lungs.
Collapse
Affiliation(s)
- Dailin Lv
- The Key Lab of RF Circuits and Systems of Ministry of Education, Microelectronics CAD Center, Hangzhou Dianzi University, Hangzhou, 310018, China
| | - Yaqi Wang
- College of Media Engineering, Communication University of Zhejiang, Hangzhou, 310018, China
| | - Shuai Wang
- Imaging Biomarkers and Computer-Aided Diagnosis Laboratory, National Institutes of Health Clinical Center, Bethesda, MD, 20892, USA
| | - Qianni Zhang
- School of Electronic Engineering and Computer Science, Queen Mary University of London, London, E1 4NS, UK
| | - Wuteng Qi
- The Key Lab of RF Circuits and Systems of Ministry of Education, Microelectronics CAD Center, Hangzhou Dianzi University, Hangzhou, 310018, China
| | - Yunxiang Li
- The Key Lab of RF Circuits and Systems of Ministry of Education, Microelectronics CAD Center, Hangzhou Dianzi University, Hangzhou, 310018, China
| | - Lingling Sun
- The Key Lab of RF Circuits and Systems of Ministry of Education, Microelectronics CAD Center, Hangzhou Dianzi University, Hangzhou, 310018, China
| |
Collapse
|
15
|
Differentiable neural architecture search for optimal spatial/temporal brain function network decomposition. Med Image Anal 2021; 69:101974. [PMID: 33588118 DOI: 10.1016/j.media.2021.101974] [Citation(s) in RCA: 13] [Impact Index Per Article: 4.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2020] [Revised: 01/03/2021] [Accepted: 01/11/2021] [Indexed: 12/29/2022]
Abstract
It has been a key topic to decompose the brain's spatial/temporal function networks from 4D functional magnetic resonance imaging (fMRI) data. With the advantages of robust and meaningful brain pattern extraction, deep neural networks have been shown to be more powerful and flexible in fMRI data modeling than other traditional methods. However, the challenge of designing neural network architecture for high-dimensional and complex fMRI data has also been realized recently. In this paper, we propose a new spatial/temporal differentiable neural architecture search algorithm (ST-DARTS) for optimal brain network decomposition. The core idea of ST-DARTS is to optimize the inner cell structure of the vanilla recurrent neural network (RNN) in order to effectively decompose spatial/temporal brain function networks from fMRI data. Based on the evaluations on all seven fMRI tasks in human connectome project (HCP) dataset, the ST-DARTS model is shown to perform promisingly, both spatially (i.e., it can recognize the most stimuli-correlated spatial brain network activation that is very similar to the benchmark) and temporally (i.e., its temporal activity is highly positively correlated with the task-design). To further improve the efficiency of ST-DARTS model, we introduce a flexible early-stopping mechanism, named as ST-DARTS+, which further improves experimental results significantly. To our best knowledge, the proposed ST-DARTS and ST-DARTS+ models are among the early efforts in optimally decomposing spatial/temporal brain function networks from fMRI data with neural architecture search strategy and they demonstrate great promise.
Collapse
|