1
|
Li Y, El Habib Daho M, Conze PH, Zeghlache R, Le Boité H, Tadayoni R, Cochener B, Lamard M, Quellec G. A review of deep learning-based information fusion techniques for multimodal medical image classification. Comput Biol Med 2024; 177:108635. [PMID: 38796881 DOI: 10.1016/j.compbiomed.2024.108635] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2023] [Revised: 03/18/2024] [Accepted: 05/18/2024] [Indexed: 05/29/2024]
Abstract
Multimodal medical imaging plays a pivotal role in clinical diagnosis and research, as it combines information from various imaging modalities to provide a more comprehensive understanding of the underlying pathology. Recently, deep learning-based multimodal fusion techniques have emerged as powerful tools for improving medical image classification. This review offers a thorough analysis of the developments in deep learning-based multimodal fusion for medical classification tasks. We explore the complementary relationships among prevalent clinical modalities and outline three main fusion schemes for multimodal classification networks: input fusion, intermediate fusion (encompassing single-level fusion, hierarchical fusion, and attention-based fusion), and output fusion. By evaluating the performance of these fusion techniques, we provide insight into the suitability of different network architectures for various multimodal fusion scenarios and application domains. Furthermore, we delve into challenges related to network architecture selection, handling incomplete multimodal data management, and the potential limitations of multimodal fusion. Finally, we spotlight the promising future of Transformer-based multimodal fusion techniques and give recommendations for future research in this rapidly evolving field.
Collapse
Affiliation(s)
- Yihao Li
- LaTIM UMR 1101, Inserm, Brest, France; University of Western Brittany, Brest, France
| | - Mostafa El Habib Daho
- LaTIM UMR 1101, Inserm, Brest, France; University of Western Brittany, Brest, France.
| | | | - Rachid Zeghlache
- LaTIM UMR 1101, Inserm, Brest, France; University of Western Brittany, Brest, France
| | - Hugo Le Boité
- Sorbonne University, Paris, France; Ophthalmology Department, Lariboisière Hospital, AP-HP, Paris, France
| | - Ramin Tadayoni
- Ophthalmology Department, Lariboisière Hospital, AP-HP, Paris, France; Paris Cité University, Paris, France
| | - Béatrice Cochener
- LaTIM UMR 1101, Inserm, Brest, France; University of Western Brittany, Brest, France; Ophthalmology Department, CHRU Brest, Brest, France
| | - Mathieu Lamard
- LaTIM UMR 1101, Inserm, Brest, France; University of Western Brittany, Brest, France
| | | |
Collapse
|
2
|
Mahdy S, Abuelmakarem HS. Alzheimer's disease progression detection based on optical fluence rate measurements using alternative laser wavelengths. INTERNATIONAL JOURNAL FOR NUMERICAL METHODS IN BIOMEDICAL ENGINEERING 2024; 40:e3816. [PMID: 38523567 DOI: 10.1002/cnm.3816] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/19/2023] [Revised: 01/14/2024] [Accepted: 03/11/2024] [Indexed: 03/26/2024]
Abstract
Alzheimer's disease (AD) levels have increased globally, which is considered the sixth reason for deaths. So, a requirement exists for economic and quantitative methods to follow up the gradual progression of AD. The current study presents a simulation for a non-irradiated, safe, wearable, and noninvasive mobile approach for detecting the progression of Alzheimer's brain atrophy using the optical diffusion technique and for investigating the difference between the normal and the diseased brain. The virtual study was accomplished using COMSOL Multiphysics. The simulated head is implemented as the following: scalp, skull, cerebrospinal fluid, gray matter, and white matter. The optical properties of the heterogeneous tissue are observed using the fluence rate after irradiating the head with different wavelengths (630, 700, 810, 915, and 1000 nm) of lasers. Two assessment techniques were applied to evaluate the brain atrophy measurements; the first technique was an array of photodetectors, which were lined at the head posterior, while a matrix of photodetectors was applied over the head surface in the second technique. The results show that the surface photodetectors approach differentiates the normal from AD brains without measuring the brain atrophy percentages by applying 630 nm. The array of photodetectors distinguishes normal from AD brains without detecting the brain atrophy percentages when the wavelengths 630, 700, and 810 nm were applied. The line detector at 1000 nm evaluates the brain atrophy percentages with AD. The future explores applying those techniques in vivo and analyzing the information by the spectrometer for extensively safer early detection of neural disorders.
Collapse
Affiliation(s)
- Shimaa Mahdy
- Department of Electrical Engineering, Egyptian Academy for Engineering and Advanced Technology (EAE&AT), Affiliated to the Ministry of Military Production, El-Nahda, Al Salam First, Egypt
| | - Hala S Abuelmakarem
- SBME Department, The Higher Institute of Engineering, El Shrouk Academy, Cairo, Egypt
| |
Collapse
|
3
|
Hussein R, Shin D, Zhao MY, Guo J, Davidzon G, Steinberg G, Moseley M, Zaharchuk G. Turning brain MRI into diagnostic PET: 15O-water PET CBF synthesis from multi-contrast MRI via attention-based encoder-decoder networks. Med Image Anal 2024; 93:103072. [PMID: 38176356 PMCID: PMC10922206 DOI: 10.1016/j.media.2023.103072] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/15/2022] [Revised: 12/20/2023] [Accepted: 12/20/2023] [Indexed: 01/06/2024]
Abstract
Accurate quantification of cerebral blood flow (CBF) is essential for the diagnosis and assessment of a wide range of neurological diseases. Positron emission tomography (PET) with radiolabeled water (15O-water) is the gold-standard for the measurement of CBF in humans, however, it is not widely available due to its prohibitive costs and the use of short-lived radiopharmaceutical tracers that require onsite cyclotron production. Magnetic resonance imaging (MRI), in contrast, is more accessible and does not involve ionizing radiation. This study presents a convolutional encoder-decoder network with attention mechanisms to predict the gold-standard 15O-water PET CBF from multi-contrast MRI scans, thus eliminating the need for radioactive tracers. The model was trained and validated using 5-fold cross-validation in a group of 126 subjects consisting of healthy controls and cerebrovascular disease patients, all of whom underwent simultaneous 15O-water PET/MRI. The results demonstrate that the model can successfully synthesize high-quality PET CBF measurements (with an average SSIM of 0.924 and PSNR of 38.8 dB) and is more accurate compared to concurrent and previous PET synthesis methods. We also demonstrate the clinical significance of the proposed algorithm by evaluating the agreement for identifying the vascular territories with impaired CBF. Such methods may enable more widespread and accurate CBF evaluation in larger cohorts who cannot undergo PET imaging due to radiation concerns, lack of access, or logistic challenges.
Collapse
Affiliation(s)
- Ramy Hussein
- Radiological Sciences Laboratory, Department of Radiology, Stanford University, Stanford, CA 94305, USA.
| | - David Shin
- Global MR Applications & Workflow, GE Healthcare, Menlo Park, CA 94025, USA
| | - Moss Y Zhao
- Radiological Sciences Laboratory, Department of Radiology, Stanford University, Stanford, CA 94305, USA; Stanford Cardiovascular Institute, Stanford University, Stanford, CA 94305, USA
| | - Jia Guo
- Department of Bioengineering, University of California, Riverside, CA 92521, USA
| | - Guido Davidzon
- Division of Nuclear Medicine, Department of Radiology, Stanford University, Stanford, CA 94305, USA
| | - Gary Steinberg
- Department of Neurosurgery, Stanford University, Stanford, CA 94304, USA
| | - Michael Moseley
- Radiological Sciences Laboratory, Department of Radiology, Stanford University, Stanford, CA 94305, USA
| | - Greg Zaharchuk
- Radiological Sciences Laboratory, Department of Radiology, Stanford University, Stanford, CA 94305, USA
| |
Collapse
|
4
|
Vega F, Addeh A, Ganesh A, Smith EE, MacDonald ME. Image Translation for Estimating Two-Dimensional Axial Amyloid-Beta PET From Structural MRI. J Magn Reson Imaging 2024; 59:1021-1031. [PMID: 37921361 DOI: 10.1002/jmri.29070] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2023] [Revised: 09/29/2023] [Accepted: 10/02/2023] [Indexed: 11/04/2023] Open
Abstract
BACKGROUND Amyloid-beta and brain atrophy are hallmarks for Alzheimer's Disease that can be targeted with positron emission tomography (PET) and MRI, respectively. MRI is cheaper, less-invasive, and more available than PET. There is a known relationship between amyloid-beta and brain atrophy, meaning PET images could be inferred from MRI. PURPOSE To build an image translation model using a Conditional Generative Adversarial Network able to synthesize Amyloid-beta PET images from structural MRI. STUDY TYPE Retrospective. POPULATION Eight hundred eighty-two adults (348 males/534 females) with different stages of cognitive decline (control, mild cognitive impairment, moderate cognitive impairment, and severe cognitive impairment). Five hundred fifty-two subjects for model training and 331 for testing (80%:20%). FIELD STRENGTH/SEQUENCE 3 T, T1-weighted structural (T1w). ASSESSMENT The testing cohort was used to evaluate the performance of the model using the Structural Similarity Index Measure (SSIM) and Peak Signal-to-Noise Ratio (PSNR), comparing the likeness of the overall synthetic PET images created from structural MRI with the overall true PET images. SSIM was computed in the overall image to include the luminance, contrast, and structural similarity components. Experienced observers reviewed the images for quality, performance and tried to determine if they could tell the difference between real and synthetic images. STATISTICAL TESTS Pixel wise Pearson correlation was significant, and had an R2 greater than 0.96 in example images. From blinded readings, a Pearson Chi-squared test showed that there was no significant difference between the real and synthetic images by the observers (P = 0.68). RESULTS A high degree of likeness across the evaluation set, which had a mean SSIM = 0.905 and PSNR = 2.685. The two observers were not able to determine the difference between the real and synthetic images, with accuracies of 54% and 46%, respectively. CONCLUSION Amyloid-beta PET images can be synthesized from structural MRI with a high degree of similarity to the real PET images. EVIDENCE LEVEL 3 TECHNICAL EFFICACY: Stage 1.
Collapse
Affiliation(s)
- Fernando Vega
- Department of Biomedical, University of Calgary, Calgary, Alberta, Canada
- Department of Electrical and Software Engineering, University of Calgary, Calgary, Alberta, Canada
- Department of Radiology, University of Calgary, Calgary, Alberta, Canada
- Hotchkiss Brain Institute, University of Calgary, Calgary, Alberta, Canada
| | - Abdoljalil Addeh
- Department of Biomedical, University of Calgary, Calgary, Alberta, Canada
- Department of Electrical and Software Engineering, University of Calgary, Calgary, Alberta, Canada
- Department of Radiology, University of Calgary, Calgary, Alberta, Canada
- Hotchkiss Brain Institute, University of Calgary, Calgary, Alberta, Canada
| | - Aravind Ganesh
- Department of Radiology, University of Calgary, Calgary, Alberta, Canada
- Hotchkiss Brain Institute, University of Calgary, Calgary, Alberta, Canada
- Department of Clinical Neuroscience, University of Calgary, Calgary, Alberta, Canada
| | - Eric E Smith
- Department of Radiology, University of Calgary, Calgary, Alberta, Canada
- Hotchkiss Brain Institute, University of Calgary, Calgary, Alberta, Canada
- Department of Clinical Neuroscience, University of Calgary, Calgary, Alberta, Canada
| | - M Ethan MacDonald
- Department of Biomedical, University of Calgary, Calgary, Alberta, Canada
- Department of Electrical and Software Engineering, University of Calgary, Calgary, Alberta, Canada
- Department of Radiology, University of Calgary, Calgary, Alberta, Canada
- Hotchkiss Brain Institute, University of Calgary, Calgary, Alberta, Canada
| |
Collapse
|
5
|
Li W, Liu J, Wang S, Feng C. MTFN: multi-temporal feature fusing network with co-attention for DCE-MRI synthesis. BMC Med Imaging 2024; 24:47. [PMID: 38373915 PMCID: PMC10875895 DOI: 10.1186/s12880-024-01201-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/08/2022] [Accepted: 01/15/2024] [Indexed: 02/21/2024] Open
Abstract
BACKGROUND Dynamic Contrast Enhanced Magnetic Resonance Imaging (DCE-MRI) plays an important role in the diagnosis and treatment of breast cancer. However, obtaining complete eight temporal images of DCE-MRI requires a long scanning time, which causes patients' discomfort in the scanning process. Therefore, to reduce the time, the multi temporal feature fusing neural network with Co-attention (MTFN) is proposed to generate the eighth temporal images of DCE-MRI, which enables the acquisition of DCE-MRI images without scanning. In order to reduce the time, multi-temporal feature fusion cooperative attention mechanism neural network (MTFN) is proposed to generate the eighth temporal images of DCE-MRI, which enables DCE-MRI image acquisition without scanning. METHODS In this paper, we propose multi temporal feature fusing neural network with Co-attention (MTFN) for DCE-MRI Synthesis, in which the Co-attention module can fully fuse the features of the first and third temporal image to obtain the hybrid features. The Co-attention explore long-range dependencies, not just relationships between pixels. Therefore, the hybrid features are more helpful to generate the eighth temporal images. RESULTS We conduct experiments on the private breast DCE-MRI dataset from hospitals and the multi modal Brain Tumor Segmentation Challenge2018 dataset (BraTs2018). Compared with existing methods, the experimental results of our method show the improvement and our method can generate more realistic images. In the meanwhile, we also use synthetic images to classify the molecular typing of breast cancer that the accuracy on the original eighth time-series images and the generated images are 89.53% and 92.46%, which have been improved by about 3%, and the classification results verify the practicability of the synthetic images. CONCLUSIONS The results of subjective evaluation and objective image quality evaluation indicators show the effectiveness of our method, which can obtain comprehensive and useful information. The improvement of classification accuracy proves that the images generated by our method are practical.
Collapse
Affiliation(s)
- Wei Li
- Key Laboratory of Intelligent Computing in Medical Image MIIC, Northeastern University, Shenyang, China
| | - Jiaye Liu
- School of Computer Science and Engineering, Northeastern University, Shenyang, China
| | - Shanshan Wang
- School of Computer Science and Engineering, Northeastern University, Shenyang, China.
| | - Chaolu Feng
- Key Laboratory of Intelligent Computing in Medical Image MIIC, Northeastern University, Shenyang, China
| |
Collapse
|
6
|
Dayarathna S, Islam KT, Uribe S, Yang G, Hayat M, Chen Z. Deep learning based synthesis of MRI, CT and PET: Review and analysis. Med Image Anal 2024; 92:103046. [PMID: 38052145 DOI: 10.1016/j.media.2023.103046] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2023] [Revised: 11/14/2023] [Accepted: 11/29/2023] [Indexed: 12/07/2023]
Abstract
Medical image synthesis represents a critical area of research in clinical decision-making, aiming to overcome the challenges associated with acquiring multiple image modalities for an accurate clinical workflow. This approach proves beneficial in estimating an image of a desired modality from a given source modality among the most common medical imaging contrasts, such as Computed Tomography (CT), Magnetic Resonance Imaging (MRI), and Positron Emission Tomography (PET). However, translating between two image modalities presents difficulties due to the complex and non-linear domain mappings. Deep learning-based generative modelling has exhibited superior performance in synthetic image contrast applications compared to conventional image synthesis methods. This survey comprehensively reviews deep learning-based medical imaging translation from 2018 to 2023 on pseudo-CT, synthetic MR, and synthetic PET. We provide an overview of synthetic contrasts in medical imaging and the most frequently employed deep learning networks for medical image synthesis. Additionally, we conduct a detailed analysis of each synthesis method, focusing on their diverse model designs based on input domains and network architectures. We also analyse novel network architectures, ranging from conventional CNNs to the recent Transformer and Diffusion models. This analysis includes comparing loss functions, available datasets and anatomical regions, and image quality assessments and performance in other downstream tasks. Finally, we discuss the challenges and identify solutions within the literature, suggesting possible future directions. We hope that the insights offered in this survey paper will serve as a valuable roadmap for researchers in the field of medical image synthesis.
Collapse
Affiliation(s)
- Sanuwani Dayarathna
- Department of Data Science and AI, Faculty of Information Technology, Monash University, Clayton VIC 3800, Australia.
| | | | - Sergio Uribe
- Department of Medical Imaging and Radiation Sciences, Faculty of Medicine, Monash University, Clayton VIC 3800, Australia
| | - Guang Yang
- Bioengineering Department and Imperial-X, Imperial College London, W12 7SL, United Kingdom
| | - Munawar Hayat
- Department of Data Science and AI, Faculty of Information Technology, Monash University, Clayton VIC 3800, Australia
| | - Zhaolin Chen
- Department of Data Science and AI, Faculty of Information Technology, Monash University, Clayton VIC 3800, Australia; Monash Biomedical Imaging, Clayton VIC 3800, Australia
| |
Collapse
|
7
|
Wang D, Jiang C, He J, Teng Y, Qin H, Liu J, Yang X. M 3S-Net: multi-modality multi-branch multi-self-attention network with structure-promoting loss for low-dose PET/CT enhancement. Phys Med Biol 2024; 69:025001. [PMID: 38086073 DOI: 10.1088/1361-6560/ad14c5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/17/2023] [Accepted: 12/12/2023] [Indexed: 01/05/2024]
Abstract
Objective.PET (Positron Emission Tomography) inherently involves radiotracer injections and long scanning time, which raises concerns about the risk of radiation exposure and patient comfort. Reductions in radiotracer dosage and acquisition time can lower the potential risk and improve patient comfort, respectively, but both will also reduce photon counts and hence degrade the image quality. Therefore, it is of interest to improve the quality of low-dose PET images.Approach.A supervised multi-modality deep learning model, named M3S-Net, was proposed to generate standard-dose PET images (60 s per bed position) from low-dose ones (10 s per bed position) and the corresponding CT images. Specifically, we designed a multi-branch convolutional neural network with multi-self-attention mechanisms, which first extracted features from PET and CT images in two separate branches and then fused the features to generate the final generated PET images. Moreover, a novel multi-modality structure-promoting term was proposed in the loss function to learn the anatomical information contained in CT images.Main results.We conducted extensive numerical experiments on real clinical data collected from local hospitals. Compared with state-of-the-art methods, the proposed M3S-Net not only achieved higher objective metrics and better generated tumors, but also performed better in preserving edges and suppressing noise and artifacts.Significance.The experimental results of quantitative metrics and qualitative displays demonstrate that the proposed M3S-Net can generate high-quality PET images from low-dose ones, which are competable to standard-dose PET images. This is valuable in reducing PET acquisition time and has potential applications in dynamic PET imaging.
Collapse
Affiliation(s)
- Dong Wang
- School of Mathematics/S.T.Yau Center of Southeast University, Southeast University, 210096, People's Republic of China
- Nanjing Center of Applied Mathematics, Nanjing, 211135, People's Republic of China
| | - Chong Jiang
- Department of Nuclear Medicine, West China Hospital of Sichuan University, Sichuan University, Chengdu, 610041, People's Republic of China
| | - Jian He
- Department of Nuclear Medicine, Nanjing Drum Tower Hospital, the Affiliated Hospital of Nanjing University Medical School, Nanjing, 210008, People's Republic of China
| | - Yue Teng
- Department of Nuclear Medicine, Nanjing Drum Tower Hospital, the Affiliated Hospital of Nanjing University Medical School, Nanjing, 210008, People's Republic of China
| | - Hourong Qin
- Department of Mathematics, Nanjing University, Nanjing, 210093, People's Republic of China
| | - Jijun Liu
- School of Mathematics/S.T.Yau Center of Southeast University, Southeast University, 210096, People's Republic of China
- Nanjing Center of Applied Mathematics, Nanjing, 211135, People's Republic of China
| | - Xiaoping Yang
- Department of Mathematics, Nanjing University, Nanjing, 210093, People's Republic of China
| |
Collapse
|
8
|
Chen K, Weng Y, Hosseini AA, Dening T, Zuo G, Zhang Y. A comparative study of GNN and MLP based machine learning for the diagnosis of Alzheimer's Disease involving data synthesis. Neural Netw 2024; 169:442-452. [PMID: 37939533 DOI: 10.1016/j.neunet.2023.10.040] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/12/2023] [Revised: 09/23/2023] [Accepted: 10/25/2023] [Indexed: 11/10/2023]
Abstract
Alzheimer's Disease (AD) is a neurodegenerative disease that commonly occurs in older people. It is characterized by both cognitive and functional impairment. However, as AD has an unclear pathological cause, it can be hard to diagnose with confidence. This is even more so in the early stage of Mild Cognitive Impairment (MCI). This paper proposes a U-Net based Generative Adversarial Network (GAN) to synthesize fluorodeoxyglucose - positron emission tomography (FDG-PET) from magnetic resonance imaging - T1 weighted imaging (MRI-T1WI) for further usage in AD diagnosis including its early-stage MCI. The experiments have displayed promising results with Structural Similarity Index Measure (SSIM) reaching 0.9714. Furthermore, three types of classifiers are developed, i.e., one Multi-Layer Perceptron (MLP) based classifier, two Graph Neural Network (GNN) based classifiers where one is for graph classification and the other is for node classification. 10-fold cross-validation has been conducted on all trials of experiments for classifier comparison. The performance of these three types of classifiers has been compared with the different input modalities setting and data fusion strategies. The results have shown that GNN based node classifier surpasses the other two types of classifiers, and has achieved the state-of-the-art (SOTA) performance with the best accuracy at 90.18% for 3-class classification, namely AD, MCI and normal control (NC) with the synthesized fluorodeoxyglucose - positron emission tomography (FDG-PET) features fused at the input level. Moreover, involving synthesized FDG-PET as part of the input with proper data fusion strategies has also proved to enhance all three types of classifiers' performance. This work provides support for the notion that machine learning-derived image analysis may be a useful approach to improving the diagnosis of AD.
Collapse
Affiliation(s)
- Ke Chen
- School of Computer Science, University of Nottingham Ningbo China, Ningbo, 315100, China; Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Sciences, Ningbo, 315201, China; School of Medicine, University of Nottingham, Nottingham, NG7 2RD, UK.
| | - Ying Weng
- School of Computer Science, University of Nottingham Ningbo China, Ningbo, 315100, China.
| | - Akram A Hosseini
- Neurology Department, Nottingham University Hospitals NHS Trust, Nottingham, NG7 2UH, UK.
| | - Tom Dening
- School of Medicine, University of Nottingham, Nottingham, NG7 2RD, UK.
| | - Guokun Zuo
- Ningbo Institute of Materials Technology and Engineering, Chinese Academy of Sciences, Ningbo, 315201, China.
| | - Yiming Zhang
- School of Computer Science, University of Nottingham Ningbo China, Ningbo, 315100, China; School of Medicine, University of Nottingham, Nottingham, NG7 2RD, UK.
| |
Collapse
|
9
|
Wang C, Piao S, Huang Z, Gao Q, Zhang J, Li Y, Shan H. Joint learning framework of cross-modal synthesis and diagnosis for Alzheimer's disease by mining underlying shared modality information. Med Image Anal 2024; 91:103032. [PMID: 37995628 DOI: 10.1016/j.media.2023.103032] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/15/2022] [Revised: 08/31/2023] [Accepted: 11/13/2023] [Indexed: 11/25/2023]
Abstract
Alzheimer's disease (AD) is one of the most common neurodegenerative disorders presenting irreversible progression of cognitive impairment. How to identify AD as early as possible is critical for intervention with potential preventive measures. Among various neuroimaging modalities used to diagnose AD, functional positron emission tomography (PET) has higher sensitivity than structural magnetic resonance imaging (MRI), but it is also costlier and often not available in many hospitals. How to leverage massive unpaired unlabeled PET to improve the diagnosis performance of AD from MRI becomes rather important. To address this challenge, this paper proposes a novel joint learning framework of unsupervised cross-modal synthesis and AD diagnosis by mining underlying shared modality information, improving the AD diagnosis from MRI while synthesizing more discriminative PET images. We mine underlying shared modality information in two aspects: diversifying modality information through the cross-modal synthesis network and locating critical diagnosis-related patterns through the AD diagnosis network. First, to diversify the modality information, we propose a novel unsupervised cross-modal synthesis network, which implements the inter-conversion between 3D PET and MRI in a single model modulated by the AdaIN module. Second, to locate shared critical diagnosis-related patterns, we propose an interpretable diagnosis network based on fully 2D convolutions, which takes either 3D synthesized PET or original MRI as input. Extensive experimental results on the ADNI dataset show that our framework can synthesize more realistic images, outperform the state-of-the-art AD diagnosis methods, and have better generalization on external AIBL and NACC datasets.
Collapse
Affiliation(s)
- Chenhui Wang
- Institute of Science and Technology for Brain-inspired Intelligence, Fudan University, Shanghai 200433, China
| | - Sirong Piao
- Department of Radiology, Huashan Hospital, Fudan University, Shanghai 200040, China
| | - Zhizhong Huang
- Shanghai Key Lab of Intelligent Information Processing, Fudan University, Shanghai 200433, China; School of Computer Science, Fudan University, Shanghai 200433, China
| | - Qi Gao
- Institute of Science and Technology for Brain-inspired Intelligence, Fudan University, Shanghai 200433, China
| | - Junping Zhang
- Shanghai Key Lab of Intelligent Information Processing, Fudan University, Shanghai 200433, China; School of Computer Science, Fudan University, Shanghai 200433, China
| | - Yuxin Li
- Department of Radiology, Huashan Hospital, Fudan University, Shanghai 200040, China.
| | - Hongming Shan
- Institute of Science and Technology for Brain-inspired Intelligence, Fudan University, Shanghai 200433, China; MOE Frontiers Center for Brain Science, Fudan University, Shanghai, 200433, China; MOE Key Laboratory of Computational Neuroscience and Brain-Inspired Intelligence, Fudan University, Shanghai, 200433, China; Shanghai Center for Brain Science and Brain-inspired Technology, Shanghai 201210, China.
| |
Collapse
|
10
|
Flaus A, Jung J, Ostrowky‐Coste K, Rheims S, Guénot M, Bouvard S, Janier M, Yaakub SN, Lartizien C, Costes N, Hammers A. Deep-learning predicted PET can be subtracted from the true clinical fluorodeoxyglucose PET co-registered to MRI to identify the epileptogenic zone in focal epilepsy. Epilepsia Open 2023; 8:1440-1451. [PMID: 37602538 PMCID: PMC10690662 DOI: 10.1002/epi4.12820] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/27/2023] [Accepted: 08/16/2023] [Indexed: 08/22/2023] Open
Abstract
OBJECTIVE Normal interictal [18 F]FDG-PET can be predicted from the corresponding T1w MRI with Generative Adversarial Networks (GANs). A technique we call SIPCOM (Subtraction Interictal PET Co-registered to MRI) can then be used to compare epilepsy patients' predicted and clinical PET. We assessed the ability of SIPCOM to identify the Resection Zone (RZ) in patients with drug-resistant epilepsy (DRE) with reference to visual and statistical parametric mapping (SPM) analysis. METHODS Patients with complete presurgical work-up and subsequent SEEG and cortectomy were included. RZ localisation, the reference region, was assigned to one of eighteen anatomical brain regions. SIPCOM was implemented using healthy controls to train a GAN. To compare, the clinical PET coregistered to MRI was visually assessed by two trained readers, and a standard SPM analysis was performed. RESULTS Twenty patients aged 17-50 (32 ± 7.8) years were included, 14 (70%) with temporal lobe epilepsy (TLE). Eight (40%) were MRI-negative. After surgery, 14 patients (70%) had a good outcome (Engel I-II). RZ localisation rate was 60% with SIPCOM vs 35% using SPM (P = 0.015) and vs 85% using visual analysis (P = 0.54). Results were similar for Engel I-II patients, the RZ localisation rate was 64% with SIPCOM vs 36% with SPM. With SIPCOM localisation was correct in 67% in MRI-positive vs 50% in MRI-negative patients, and 64% in TLE vs 43% in extra-TLE. The average number of false-positive clusters was 2.2 ± 1.3 using SIPCOM vs 2.3 ± 3.1 using SPM. All RZs localized with SPM were correctly localized with SIPCOM. In one case, PET and MRI were visually reported as negative, but both SIPCOM and SPM localized the RZ. SIGNIFICANCE SIPCOM performed better than the reference computer-assisted method (SPM) for RZ detection in a group of operated DRE patients. SIPCOM's impact on epilepsy management needs to be prospectively validated.
Collapse
Affiliation(s)
- Anthime Flaus
- Department of Nuclear MedicineHospices Civils de LyonLyonFrance
- Medical Faculty of Lyon EstUniversity Claude Bernard Lyon 1LyonFrance
- King's College London & Guy's and St Thomas' PET Centre, School of Biomedical Engineering and Imaging SciencesKing's College LondonLondonUK
- Lyon Neuroscience Research CenterINSERM U1028/CNRS UMR5292LyonFrance
| | - Julien Jung
- Lyon Neuroscience Research CenterINSERM U1028/CNRS UMR5292LyonFrance
- Department of Functional Neurology and Epileptology, Hospices Civils de Lyon, Member of the ERN EpiCARELyon 1 UniversityLyonFrance
| | - Karine Ostrowky‐Coste
- Lyon Neuroscience Research CenterINSERM U1028/CNRS UMR5292LyonFrance
- Department of Pediatric Clinical Epileptology, Sleep Disorders, and Functional NeurologyHospices Civils de Lyon, Member of the ERN EpiCARELyonFrance
| | - Sylvain Rheims
- Lyon Neuroscience Research CenterINSERM U1028/CNRS UMR5292LyonFrance
- Department of Functional Neurology and Epileptology, Hospices Civils de Lyon, Member of the ERN EpiCARELyon 1 UniversityLyonFrance
| | - Marc Guénot
- Lyon Neuroscience Research CenterINSERM U1028/CNRS UMR5292LyonFrance
- Department of Functional Neurosurgery, Hospices Civils de Lyon, Member of the ERN EpiCARELyon 1 UniversityLyonFrance
| | - Sandrine Bouvard
- Lyon Neuroscience Research CenterINSERM U1028/CNRS UMR5292LyonFrance
| | - Marc Janier
- Department of Nuclear MedicineHospices Civils de LyonLyonFrance
- Medical Faculty of Lyon EstUniversity Claude Bernard Lyon 1LyonFrance
| | - Siti N. Yaakub
- Brain Research & Imaging CentreUniversity of PlymouthPlymouthUK
| | - Carole Lartizien
- INSA‐Lyon, CNRS, Inserm, CREATIS UMR 5220, U1294University Claude Bernard Lyon 1LyonFrance
| | - Nicolas Costes
- Lyon Neuroscience Research CenterINSERM U1028/CNRS UMR5292LyonFrance
- CERMEP‐Life ImagingLyonFrance
| | - Alexander Hammers
- King's College London & Guy's and St Thomas' PET Centre, School of Biomedical Engineering and Imaging SciencesKing's College LondonLondonUK
| |
Collapse
|
11
|
Wang Z, Zhang L, Shu X, Wang Y, Feng Y. Consistent representation via contrastive learning for skin lesion diagnosis. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 242:107826. [PMID: 37837885 DOI: 10.1016/j.cmpb.2023.107826] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/28/2023] [Revised: 09/19/2023] [Accepted: 09/21/2023] [Indexed: 10/16/2023]
Abstract
BACKGROUND Skin lesions are a prevalent ailment, with melanoma emerging as a particularly perilous variant. Encouragingly, artificial intelligence displays promising potential in early detection, yet its integration within clinical contexts, particularly involving multi-modal data, presents challenges. While multi-modal approaches enhance diagnostic efficacy, the influence of modal bias is often disregarded. METHODS In this investigation, a multi-modal feature learning technique termed "Contrast-based Consistent Representation Disentanglement" for dermatological diagnosis is introduced. This approach employs adversarial domain adaptation to disentangle features from distinct modalities, fostering a shared representation. Furthermore, a contrastive learning strategy is devised to incentivize the model to preserve uniformity in common lesion attributes across modalities. Emphasizing the learning of a uniform representation among models, this approach circumvents reliance on supplementary data. RESULTS Assessment of the proposed technique on a 7-point criteria evaluation dataset yields an average accuracy of 76.1% for multi-classification tasks, surpassing researched state-of-the-art methods. The approach tackles modal bias, enabling the acquisition of a consistent representation of common lesion appearances across diverse modalities, which transcends modality boundaries. This study underscores the latent potential of multi-modal feature learning in dermatological diagnosis. CONCLUSION In summation, a multi-modal feature learning strategy is posited for dermatological diagnosis. This approach outperforms other state-of-the-art methods, underscoring its capacity to enhance diagnostic precision for skin lesions.
Collapse
Affiliation(s)
- Zizhou Wang
- College of Computer Science, Sichuan University, Chengdu 610065, China; Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR), Singapore 138632, Singapore.
| | - Lei Zhang
- College of Computer Science, Sichuan University, Chengdu 610065, China.
| | - Xin Shu
- College of Computer Science, Sichuan University, Chengdu 610065, China.
| | - Yan Wang
- Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR), Singapore 138632, Singapore.
| | - Yangqin Feng
- Institute of High Performance Computing, Agency for Science, Technology and Research (A*STAR), Singapore 138632, Singapore.
| |
Collapse
|
12
|
Zhang Y, Li X, Ji Y, Ding H, Suo X, He X, Xie Y, Liang M, Zhang S, Yu C, Qin W. MRAβ: A multimodal MRI-derived amyloid-β biomarker for Alzheimer's disease. Hum Brain Mapp 2023; 44:5139-5152. [PMID: 37578386 PMCID: PMC10502620 DOI: 10.1002/hbm.26452] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/11/2022] [Revised: 04/30/2023] [Accepted: 08/01/2023] [Indexed: 08/15/2023] Open
Abstract
Florbetapir 18 F (AV45), a highly sensitive and specific positron emission tomographic (PET) molecular biomarker binding to the amyloid-β of Alzheimer's disease (AD), is constrained by radiation and cost. We sought to combat it by combining multimodal magnetic resonance imaging (MRI) images and a collaborative generative adversarial networks model (CollaGAN) to develop a multimodal MRI-derived Amyloid-β (MRAβ) biomarker. We collected multimodal MRI and PET AV45 data of 380 qualified participants from the ADNI dataset and 64 subjects from OASIS3 dataset. A five-fold cross-validation CollaGAN were applied to generate MRAβ. In the ADNI dataset, we found MRAβ could characterize the subject-level AV45 spatial variations in both AD and mild cognitive impairment (MCI). Voxel-wise two-sample t-tests demonstrated amyloid-β depositions identified by MRAβ in AD and MCI were significantly higher than healthy controls (HCs) in widespread cortices (p < .05, corrected) and were much similar to those by AV45 (r > .92, p < .001). Moreover, a 3D ResNet classifier demonstrated that MRAβ was comparable to AV45 in discriminating AD from HC in both the ADNI and OASIS3 datasets, and in discriminate MCI from HC in ADNI. Finally, we found MRAβ could mimic cortical hyper-AV45 in HCs who later converted to MCI (r = .79, p < .001) and was comparable to AV45 in discriminating them from stable HC (p > .05). In summary, our work illustrates that MRAβ synthesized by multimodal MRI could mimic the cerebral amyloid-β depositions like AV45 and lends credence to the feasibility of advancing MRI toward molecular-explainable biomarkers.
Collapse
Affiliation(s)
- Yu Zhang
- Department of Radiology and Tianjin Key Lab of Functional ImagingTianjin Medical University General HospitalTianjinChina
| | - Xi Li
- Department of Radiology and Tianjin Key Lab of Functional ImagingTianjin Medical University General HospitalTianjinChina
- Department of RadiologyFirst Clinical Medical College and First Hospital of Shanxi Medical UniversityTaiyuanShanxi ProvinceChina
| | - Yi Ji
- Department of Radiology and Tianjin Key Lab of Functional ImagingTianjin Medical University General HospitalTianjinChina
| | - Hao Ding
- Department of Radiology and Tianjin Key Lab of Functional ImagingTianjin Medical University General HospitalTianjinChina
- School of Medical ImagingTianjin Medical UniversityTianjinChina
| | - Xinjun Suo
- Department of Radiology and Tianjin Key Lab of Functional ImagingTianjin Medical University General HospitalTianjinChina
| | - Xiaoxi He
- Department of Radiology and Tianjin Key Lab of Functional ImagingTianjin Medical University General HospitalTianjinChina
| | - Yingying Xie
- Department of Radiology and Tianjin Key Lab of Functional ImagingTianjin Medical University General HospitalTianjinChina
| | - Meng Liang
- School of Medical ImagingTianjin Medical UniversityTianjinChina
| | - Shijie Zhang
- Department of PharmacologyTianjin Medical UniversityTianjinChina
| | - Chunshui Yu
- Department of Radiology and Tianjin Key Lab of Functional ImagingTianjin Medical University General HospitalTianjinChina
- School of Medical ImagingTianjin Medical UniversityTianjinChina
| | - Wen Qin
- Department of Radiology and Tianjin Key Lab of Functional ImagingTianjin Medical University General HospitalTianjinChina
| |
Collapse
|
13
|
Tang C, Wei M, Sun J, Wang S, Zhang Y. CsAGP: Detecting Alzheimer's disease from multimodal images via dual-transformer with cross-attention and graph pooling. JOURNAL OF KING SAUD UNIVERSITY. COMPUTER AND INFORMATION SCIENCES 2023; 35:101618. [PMID: 38559705 PMCID: PMC7615783 DOI: 10.1016/j.jksuci.2023.101618] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Indexed: 04/04/2024]
Abstract
Alzheimer's disease (AD) is a terrible and degenerative disease commonly occurring in the elderly. Early detection can prevent patients from further damage, which is crucial in treating AD. Over the past few decades, it has been demonstrated that neuroimaging can be a critical diagnostic tool for AD, and the feature fusion of different neuroimaging modalities can enhance diagnostic performance. Most previous studies in multimodal feature fusion have only concatenated the high-level features extracted by neural networks from various neuroimaging images simply. However, a major problem of these studies is over-looking the low-level feature interactions between modalities in the feature extraction stage, resulting in suboptimal performance in AD diagnosis. In this paper, we develop a dual-branch vision transformer with cross-attention and graph pooling, namely CsAGP, which enables multi-level feature interactions between the inputs to learn a shared feature representation. Specifically, we first construct a brand-new cross-attention fusion module (CAFM), which processes MRI and PET images by two independent branches of differing computational complexity. These features are fused merely by the cross-attention mechanism to enhance each other. After that, a concise graph pooling algorithm-based Reshape-Pooling-Reshape (RPR) framework is developed for token selection to reduce token redundancy in the proposed model. Extensive experiments on the Alzheimer's Disease Neuroimaging Initiative (ADNI) database demonstrated that the suggested method obtains 99.04%, 97.43%, 98.57%, and 98.72% accuracy for the classification of AD vs. CN, AD vs. MCI, CN vs. MCI, and AD vs. CN vs. MCI, respectively.
Collapse
Affiliation(s)
- Chaosheng Tang
- School of Computer Science and Technology, Henan Polytechnic University, Jiaozuo, Henan 454000, PR China
| | - Mingyang Wei
- School of Computer Science and Technology, Henan Polytechnic University, Jiaozuo, Henan 454000, PR China
| | - Junding Sun
- School of Computer Science and Technology, Henan Polytechnic University, Jiaozuo, Henan 454000, PR China
| | - Shuihua Wang
- School of Computer Science and Technology, Henan Polytechnic University, Jiaozuo, Henan 454000, PR China
- School of Computing and Mathematical Sciences, University of Leicester, Leicester LE1 7RH, UK
- Department of Information Systems, Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah 21589, Saudi Arabia
| | - Yudong Zhang
- School of Computer Science and Technology, Henan Polytechnic University, Jiaozuo, Henan 454000, PR China
- School of Computing and Mathematical Sciences, University of Leicester, Leicester LE1 7RH, UK
- Department of Information Systems, Faculty of Computing and Information Technology, King Abdulaziz University, Jeddah 21589, Saudi Arabia
| | | |
Collapse
|
14
|
Zhang Y, Huang K, Li M, Yuan S, Chen Q. Learn Single-horizon Disease Evolution for Predictive Generation of Post-therapeutic Neovascular Age-related Macular Degeneration. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 230:107364. [PMID: 36716636 DOI: 10.1016/j.cmpb.2023.107364] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 10/06/2022] [Revised: 01/16/2023] [Accepted: 01/20/2023] [Indexed: 06/18/2023]
Abstract
BACKGROUND AND OBJECTIVE Most of the existing disease prediction methods in the field of medical image processing fall into two classes, namely image-to-category predictions and image-to-parameter predictions.Few works have focused on image-to-image predictions. Different from multi-horizon predictions in other fields, ophthalmologists prefer to show more confidence in single-horizon predictions due to the low tolerance of predictive risk. METHODS We propose a single-horizon disease evolution network (SHENet) to predictively generate post-therapeutic SD-OCT images by inputting pre-therapeutic SD-OCT images with neovascular age-related macular degeneration (nAMD). In SHENet, a feature encoder converts the input SD-OCT images to deep features, then a graph evolution module predicts the process of disease evolution in high-dimensional latent space and outputs the predicted deep features, and lastly, feature decoder recovers the predicted deep features to SD-OCT images. We further propose an evolution reinforcement module to ensure the effectiveness of disease evolution learning and obtain realistic SD-OCT images by adversarial training. RESULTS SHENet is validated on 383 SD-OCT cubes of 22 nAMD patients based on three well-designed schemes (P-0, P-1 and P-M) based on the quantitative and qualitative evaluations. Three metrics (PSNR, SSIM, 1-LPIPS) are used here for quantitative evaluations. Compared with other generative methods, the generative SD-OCT images of SHENet have the highest image quality (P-0: 23.659, P-1: 23.875, P-M: 24.198) by PSNR. Besides, SHENet achieves the best structure protection (P-0: 0.326, P-1: 0.337, P-M: 0.349) by SSIM and content prediction (P-0: 0.609, P-1: 0.626, P-M: 0.642) by 1-LPIPS. Qualitative evaluations also demonstrate that SHENet has a better visual effect than other methods. CONCLUSIONS SHENet can generate post-therapeutic SD-OCT images with both high prediction performance and good image quality, which has great potential to help ophthalmologists forecast the therapeutic effect of nAMD.
Collapse
Affiliation(s)
- Yuhan Zhang
- School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, 210094, China.
| | - Kun Huang
- School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, 210094, China.
| | - Mingchao Li
- School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, 210094, China.
| | - Songtao Yuan
- Department of Ophthalmology, The First Affiliated Hospital with Nanjing Medical University, Nanjing, 210094, China.
| | - Qiang Chen
- School of Computer Science and Engineering, Nanjing University of Science and Technology, Nanjing, 210094, China.
| |
Collapse
|
15
|
Li J, Qu Z, Yang Y, Zhang F, Li M, Hu S. TCGAN: a transformer-enhanced GAN for PET synthetic CT. BIOMEDICAL OPTICS EXPRESS 2022; 13:6003-6018. [PMID: 36733758 PMCID: PMC9872870 DOI: 10.1364/boe.467683] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/21/2022] [Revised: 08/06/2022] [Accepted: 10/05/2022] [Indexed: 06/18/2023]
Abstract
Multimodal medical images can be used in a multifaceted approach to resolve a wide range of medical diagnostic problems. However, these images are generally difficult to obtain due to various limitations, such as cost of capture and patient safety. Medical image synthesis is used in various tasks to obtain better results. Recently, various studies have attempted to use generative adversarial networks for missing modality image synthesis, making good progress. In this study, we propose a generator based on a combination of transformer network and a convolutional neural network (CNN). The proposed method can combine the advantages of transformers and CNNs to promote a better detail effect. The network is designed for positron emission tomography (PET) to computer tomography synthesis, which can be used for PET attenuation correction. We also experimented on two datasets for magnetic resonance T1- to T2-weighted image synthesis. Based on qualitative and quantitative analyses, our proposed method outperforms the existing methods.
Collapse
Affiliation(s)
- Jitao Li
- College of Information Science and Engineering, Linyi University, Linyi, 276000, China
- College of Chemistry and Chemical Engineering, Linyi University, Linyi, 276000, China
- These authors contributed equally
| | - Zongjin Qu
- College of Chemistry and Chemical Engineering, Linyi University, Linyi, 276000, China
- These authors contributed equally
| | - Yue Yang
- College of Information Science and Engineering, Linyi University, Linyi, 276000, China
| | - Fuchun Zhang
- College of Information Science and Engineering, Linyi University, Linyi, 276000, China
| | - Meng Li
- College of Information Science and Engineering, Linyi University, Linyi, 276000, China
| | - Shunbo Hu
- College of Information Science and Engineering, Linyi University, Linyi, 276000, China
| |
Collapse
|
16
|
Yan B, Li Y, Li L, Yang X, Li TQ, Yang G, Jiang M. Quantifying the impact of Pyramid Squeeze Attention mechanism and filtering approaches on Alzheimer's disease classification. Comput Biol Med 2022; 148:105944. [PMID: 35969934 DOI: 10.1016/j.compbiomed.2022.105944] [Citation(s) in RCA: 16] [Impact Index Per Article: 8.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2022] [Revised: 07/23/2022] [Accepted: 08/06/2022] [Indexed: 11/20/2022]
Abstract
Brain medical imaging and deep learning are important foundations for diagnosing and predicting Alzheimer's disease. In this study, we explored the impact of different image filtering approaches and Pyramid Squeeze Attention (PSA) mechanism on the image classification of Alzheimer's disease. First, during the image preprocessing, we register MRI images and remove skulls, then apply median filtering, Gaussian blur filtering, and anisotropic diffusion filtering to obtain different experimental images. After that, we add the Squeeze and Excitation (SE) mechanism and Pyramid Squeeze Attention (PSA) mechanism to the Fully Convolutional Network (FCN) model respectively, to obtain each MRI image's corresponding feature information of disease probability map. Besides, we also construct Multi-Layer Perceptron (MLP) model's framework, combining feature information of disease probability map with age, gender, and Mini-Mental State Examination (MMSE) of each sample, to get the final classification performance of model. Among them, the accuracy of the MLP-C model combining anisotropic diffusion filtering with the Pyramid Squeeze Attention mechanism can reach 98.85%. The corresponding quantitative experimental results show that different image filtering approaches and attention mechanisms provide effective assistance for the diagnosis and classification of Alzheimer's disease.
Collapse
Affiliation(s)
- Bin Yan
- School of Computer Science and Technology, Zhejiang Sci-Tech University, Hangzhou, 310018, China
| | - Yang Li
- School of Computer Science and Technology, Zhejiang Sci-Tech University, Hangzhou, 310018, China
| | - Lin Li
- School of Computer Science and Technology, Zhejiang Sci-Tech University, Hangzhou, 310018, China
| | - Xiaocheng Yang
- School of Computer Science and Technology, Zhejiang Sci-Tech University, Hangzhou, 310018, China
| | - Tie-Qiang Li
- Department of Clinical Science, Intervention and Technology, Karolinska Institutet, 171 77, Stockholm, Sweden.
| | - Guang Yang
- Cardiovascular Research Centre, Royal Brompton Hospital, London, SW3 6NP, UK; National Heart and Lung Institute, Imperial College London, London, SW7 2AZ, UK.
| | - Mingfeng Jiang
- School of Computer Science and Technology, Zhejiang Sci-Tech University, Hangzhou, 310018, China.
| |
Collapse
|
17
|
Wan MD, Liu H, Liu XX, Zhang WW, Xiao XW, Zhang SZ, Jiang YL, Zhou H, Liao XX, Zhou YF, Tang BS, Wang JL, Guo JF, Jiao B, Shen L. Associations of multiple visual rating scales based on structural magnetic resonance imaging with disease severity and cerebrospinal fluid biomarkers in patients with Alzheimer’s disease. Front Aging Neurosci 2022; 14:906519. [PMID: 35966797 PMCID: PMC9374170 DOI: 10.3389/fnagi.2022.906519] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2022] [Accepted: 07/13/2022] [Indexed: 12/11/2022] Open
Abstract
The relationships between multiple visual rating scales based on structural magnetic resonance imaging (sMRI) with disease severity and cerebrospinal fluid (CSF) biomarkers in patients with Alzheimer’s disease (AD) were ambiguous. In this study, a total of 438 patients with clinically diagnosed AD were recruited. All participants underwent brain sMRI scan, and medial temporal lobe atrophy (MTA), posterior atrophy (PA), global cerebral atrophy-frontal sub-scale (GCA-F), and Fazekas rating scores were visually evaluated. Meanwhile, disease severity was assessed by neuropsychological tests such as the Mini-Mental State Examination (MMSE), Montreal Cognitive Assessment (MoCA), and Clinical Dementia Rating (CDR). Among them, 95 patients were tested for CSF core biomarkers, including Aβ1–42, Aβ1–40, Aβ1–42/Aβ1–40, p-tau, and t-tau. As a result, the GCA-F and Fazekas scales showed positively significant correlations with onset age (r = 0.181, p < 0.001; r = 0.411, p < 0.001, respectively). Patients with late-onset AD (LOAD) showed higher GCA-F and Fazekas scores (p < 0.001, p < 0.001). With regard to the disease duration, the MTA and GCA-F were positively correlated (r = 0.137, p < 0.05; r = 0.106, p < 0.05, respectively). In terms of disease severity, a positively significant association emerged between disease severity and the MTA, PA GCA-F, and Fazekas scores (p < 0.001, p < 0.001, p < 0.001, p < 0.05, respectively). Moreover, after adjusting for age, gender, and APOE alleles, the MTA scale contributed to moderate to severe AD in statistical significance independently by multivariate logistic regression analysis (p < 0.05). The model combining visual rating scales, age, gender, and APOE alleles showed the best performance for the prediction of moderate to severe AD significantly (AUC = 0.712, sensitivity = 51.5%, specificity = 84.6%). In addition, we observed that the MTA and Fazekas scores were associated with a lower concentration of Aβ1–42 (p < 0.031, p < 0.022, respectively). In summary, we systematically analyzed the benefits of multiple visual rating scales in predicting the clinical status of AD. The visual rating scales combined with age, gender, and APOE alleles showed best performance in predicting the severity of AD. MRI biomarkers in combination with CSF biomarkers can be used in clinical practice.
Collapse
Affiliation(s)
- Mei-dan Wan
- Department of Neurology, Xiangya Hospital, Central South University, Changsha, China
| | - Hui Liu
- Department of Neurology, Xiangya Hospital, Central South University, Changsha, China
| | - Xi-xi Liu
- Department of Neurology, Xiangya Hospital, Central South University, Changsha, China
| | - Wei-wei Zhang
- Department of Radiology, Xiangya Hospital, Central South University, Changsha, China
| | - Xue-wen Xiao
- Department of Neurology, Xiangya Hospital, Central South University, Changsha, China
| | - Si-zhe Zhang
- Department of Neurology, Xiangya Hospital, Central South University, Changsha, China
| | - Ya-ling Jiang
- Department of Neurology, Xiangya Hospital, Central South University, Changsha, China
| | - Hui Zhou
- Department of Neurology, Xiangya Hospital, Central South University, Changsha, China
| | - Xin-xin Liao
- National Clinical Research Center for Geriatric Disorders, Central South University, Changsha, China
- Key Laboratory of Hunan Province in Neurodegenerative Disorders, Central South University, Changsha, China
- Department of Geriatrics, Xiangya Hospital, Central South University, Changsha, China
| | - Ya-fang Zhou
- National Clinical Research Center for Geriatric Disorders, Central South University, Changsha, China
- Key Laboratory of Hunan Province in Neurodegenerative Disorders, Central South University, Changsha, China
- Department of Geriatrics, Xiangya Hospital, Central South University, Changsha, China
| | - Bei-sha Tang
- Department of Neurology, Xiangya Hospital, Central South University, Changsha, China
- National Clinical Research Center for Geriatric Disorders, Central South University, Changsha, China
- Key Laboratory of Hunan Province in Neurodegenerative Disorders, Central South University, Changsha, China
- Engineering Research Center of Hunan Province in Cognitive Impairment Disorders, Central South University, Changsha, China
- Hunan International Scientific and Technological Cooperation Base of Neurodegenerative and Neurogenetic Diseases, Changsha, China
| | - Jun-Ling Wang
- Department of Neurology, Xiangya Hospital, Central South University, Changsha, China
- National Clinical Research Center for Geriatric Disorders, Central South University, Changsha, China
- Key Laboratory of Hunan Province in Neurodegenerative Disorders, Central South University, Changsha, China
- Engineering Research Center of Hunan Province in Cognitive Impairment Disorders, Central South University, Changsha, China
- Hunan International Scientific and Technological Cooperation Base of Neurodegenerative and Neurogenetic Diseases, Changsha, China
| | - Ji-feng Guo
- Department of Neurology, Xiangya Hospital, Central South University, Changsha, China
- National Clinical Research Center for Geriatric Disorders, Central South University, Changsha, China
- Key Laboratory of Hunan Province in Neurodegenerative Disorders, Central South University, Changsha, China
- Engineering Research Center of Hunan Province in Cognitive Impairment Disorders, Central South University, Changsha, China
- Hunan International Scientific and Technological Cooperation Base of Neurodegenerative and Neurogenetic Diseases, Changsha, China
| | - Bin Jiao
- Department of Neurology, Xiangya Hospital, Central South University, Changsha, China
- National Clinical Research Center for Geriatric Disorders, Central South University, Changsha, China
- Key Laboratory of Hunan Province in Neurodegenerative Disorders, Central South University, Changsha, China
- Engineering Research Center of Hunan Province in Cognitive Impairment Disorders, Central South University, Changsha, China
- Hunan International Scientific and Technological Cooperation Base of Neurodegenerative and Neurogenetic Diseases, Changsha, China
- Bin Jiao,
| | - Lu Shen
- Department of Neurology, Xiangya Hospital, Central South University, Changsha, China
- National Clinical Research Center for Geriatric Disorders, Central South University, Changsha, China
- Key Laboratory of Hunan Province in Neurodegenerative Disorders, Central South University, Changsha, China
- Engineering Research Center of Hunan Province in Cognitive Impairment Disorders, Central South University, Changsha, China
- Hunan International Scientific and Technological Cooperation Base of Neurodegenerative and Neurogenetic Diseases, Changsha, China
- Key Laboratory of Organ Injury, Aging and Regenerative Medicine of Hunan Province, Changsha, China
- *Correspondence: Lu Shen,
| |
Collapse
|