1
|
Zhang Z, Yin W, Wang S, Zheng X, Dong S. MBFusion: Multi-modal balanced fusion and multi-task learning for cancer diagnosis and prognosis. Comput Biol Med 2024; 181:109042. [PMID: 39180856 DOI: 10.1016/j.compbiomed.2024.109042] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/27/2024] [Revised: 07/11/2024] [Accepted: 08/17/2024] [Indexed: 08/27/2024]
Abstract
Pathological images and molecular omics are important information for predicting diagnosis and prognosis. The two kinds of heterogeneous modal data contain complementary information, and the effective fusion of the two modals can better reveal the complex mechanisms of cancer. However, due to the different representation learning methods, the expression strength of different modals in different tasks varies greatly, so that many multimodal fusions do not achieve the best results. In this paper, MBFusion is proposed, to achieve multiple tasks such as prediction of diagnosis and prognosis through multi-modal balanced fusion. The MBFusion framework uses two kinds of specially constructed graph convolutional network to extract the features of molecular omics data, and uses ResNet to extract the features of pathological image data and retain important deep features by using attention and clustering, which effectively improves both kinds of the features representation, making their expressive ability balanced and comparable. The features of these two modal data are then fused through cross-attention Transformer, and the fused features are used to learn both tasks of cancer subtype classification and survival analysis by using multi-task learning. In this paper, MBFusion and other state of the art methods are compared on two public cancer datasets, and MBFusion shows an improvement of up to 10.1% by three kinds of evaluation metrics. In the ablation experiment, MBFusion explores the contribution of each modal data and each framework module to the performance. Furthermore, the interpretability of MBFusion is explained in detail to show the value of application.
Collapse
Affiliation(s)
- Ziye Zhang
- Guangdong Provincial Key Laboratory of Multimodal Big Data Intelligent Analysis, School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510641, Guangdong, China
| | - Wendong Yin
- Guangdong Provincial Key Laboratory of Multimodal Big Data Intelligent Analysis, School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510641, Guangdong, China
| | - Shijin Wang
- Guangdong Provincial Key Laboratory of Multimodal Big Data Intelligent Analysis, School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510641, Guangdong, China
| | - Xiaorou Zheng
- Guangdong Provincial Key Laboratory of Multimodal Big Data Intelligent Analysis, School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510641, Guangdong, China
| | - Shoubin Dong
- Guangdong Provincial Key Laboratory of Multimodal Big Data Intelligent Analysis, School of Computer Science and Engineering, South China University of Technology, Guangzhou, 510641, Guangdong, China.
| |
Collapse
|
2
|
Ji Y, Silva RF, Adali T, Wen X, Zhu Q, Jiang R, Zhang D, Qi S, Calhoun VD. Joint multi-site domain adaptation and multi-modality feature selection for the diagnosis of psychiatric disorders. Neuroimage Clin 2024; 43:103663. [PMID: 39226701 DOI: 10.1016/j.nicl.2024.103663] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/03/2024] [Revised: 08/18/2024] [Accepted: 08/25/2024] [Indexed: 09/05/2024]
Abstract
Identifying biomarkers for computer-aided diagnosis (CAD) is crucial for early intervention of psychiatric disorders. Multi-site data have been utilized to increase the sample size and improve statistical power, while multi-modality classification offers significant advantages over traditional single-modality based approaches for diagnosing psychiatric disorders. However, inter-site heterogeneity and intra-modality heterogeneity present challenges to multi-site and multi-modality based classification. In this paper, brain functional and structural networks (BFNs/BSNs) from multiple sites were constructed to establish a joint multi-site multi-modality framework for psychiatric diagnosis. To do this we developed a hypergraph based multi-source domain adaptation (HMSDA) which allowed us to transform source domain subjects into a target domain. A local ordinal structure based multi-task feature selection (LOSMFS) approach was developed by integrating the transformed functional and structural connections (FCs/SCs). The effectiveness of our method was validated by evaluating diagnosis of both schizophrenia (SZ) and autism spectrum disorder (ASD). The proposed method obtained accuracies of 92.2 %±2.22 % and 84.8 %±2.68 % for the diagnosis of SZ and ASD, respectively. We also compared with 6 DA, 10 multi-modality feature selection, and 8 multi-site and multi-modality methods. Results showed the proposed HMSDA+LOSMFS effectively integrated multi-site and multi-modality data to enhance psychiatric diagnosis and identify disorder-specific diagnostic brain connections.
Collapse
Affiliation(s)
- Yixin Ji
- Department of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China; Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China
| | - Rogers F Silva
- Tri-institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS), Georgia State University, Georgia Institute of Technology, Emory University, Atlanta, GA, USA
| | - Tülay Adali
- Department of CSEE, University of Maryland, USA
| | - Xuyun Wen
- Department of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China; Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China
| | - Qi Zhu
- Department of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China; Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China
| | - Rongtao Jiang
- Department of Psychiatry and Neuroscience, Yale School of Medicine, New Haven, CT, USA
| | - Daoqiang Zhang
- Department of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China; Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China.
| | - Shile Qi
- Department of Artificial Intelligence, Nanjing University of Aeronautics and Astronautics, Nanjing, China; Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing, China.
| | - Vince D Calhoun
- Tri-institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS), Georgia State University, Georgia Institute of Technology, Emory University, Atlanta, GA, USA
| |
Collapse
|
3
|
Chen Z, Chen Y, Sun Y, Tang L, Zhang L, Hu Y, He M, Li Z, Cheng S, Yuan J, Wang Z, Wang Y, Zhao J, Gong J, Zhao L, Cao B, Li G, Zhang X, Dong B, Shen L. Predicting gastric cancer response to anti-HER2 therapy or anti-HER2 combined immunotherapy based on multi-modal data. Signal Transduct Target Ther 2024; 9:222. [PMID: 39183247 PMCID: PMC11345439 DOI: 10.1038/s41392-024-01932-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2024] [Revised: 07/04/2024] [Accepted: 07/17/2024] [Indexed: 08/27/2024] Open
Abstract
The sole use of single modality data often fails to capture the complex heterogeneity among patients, including the variability in resistance to anti-HER2 therapy and outcomes of combined treatment regimens, for the treatment of HER2-positive gastric cancer (GC). This modality deficit has not been fully considered in many studies. Furthermore, the application of artificial intelligence in predicting the treatment response, particularly in complex diseases such as GC, is still in its infancy. Therefore, this study aimed to use a comprehensive analytic approach to accurately predict treatment responses to anti-HER2 therapy or anti-HER2 combined immunotherapy in patients with HER2-positive GC. We collected multi-modal data, comprising radiology, pathology, and clinical information from a cohort of 429 patients: 310 treated with anti-HER2 therapy and 119 treated with a combination of anti-HER2 and anti-PD-1/PD-L1 inhibitors immunotherapy. We introduced a deep learning model, called the Multi-Modal model (MuMo), that integrates these data to make precise treatment response predictions. MuMo achieved an area under the curve score of 0.821 for anti-HER2 therapy and 0.914 for combined immunotherapy. Moreover, patients classified as low-risk by MuMo exhibited significantly prolonged progression-free survival and overall survival (log-rank test, P < 0.05). These findings not only highlight the significance of multi-modal data analysis in enhancing treatment evaluation and personalized medicine for HER2-positive gastric cancer, but also the potential and clinical value of our model.
Collapse
Affiliation(s)
- Zifan Chen
- Center for Data Science, Peking University, Beijing, China
| | - Yang Chen
- Department of Gastrointestinal Oncology, Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education), Peking University Cancer Hospital and Institute, Beijing, China
| | - Yu Sun
- Department of Pathology, Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education), Peking University Cancer Hospital and Institute, Beijing, China
| | - Lei Tang
- Department of Radiology, Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education), Peking University Cancer Hospital and Institute, Beijing, China
| | - Li Zhang
- Center for Data Science, Peking University, Beijing, China
- National Biomedical Imaging Center, Peking University, Beijing, China
| | - Yajie Hu
- Department of Pathology, Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education), Peking University Cancer Hospital and Institute, Beijing, China
| | - Meng He
- Department of Radiology, Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education), Peking University Cancer Hospital and Institute, Beijing, China
| | - Zhiwei Li
- Department of General Surgery, Nanfang Hospital, Southern Medical University, Guangzhou, China
- Guangdong Provincial Key Laboratory of Precision Medicine for Gastrointestinal Tumor, Guangzhou, China
| | - Siyuan Cheng
- Department of Medical Oncology and Radiation Sickness, Peking University Third Hospital, Beijing, China
| | - Jiajia Yuan
- Department of Gastrointestinal Oncology, Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education), Peking University Cancer Hospital and Institute, Beijing, China
| | - Zhenghang Wang
- Department of Gastrointestinal Oncology, Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education), Peking University Cancer Hospital and Institute, Beijing, China
| | - Yakun Wang
- Department of Gastrointestinal Oncology, Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education), Peking University Cancer Hospital and Institute, Beijing, China
| | - Jie Zhao
- National Engineering Laboratory for Big Data Analysis and Applications, Peking University, Beijing, China
| | - Jifang Gong
- Department of Gastrointestinal Oncology, Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education), Peking University Cancer Hospital and Institute, Beijing, China
| | - Liying Zhao
- Department of General Surgery, Nanfang Hospital, Southern Medical University, Guangzhou, China
- Guangdong Provincial Key Laboratory of Precision Medicine for Gastrointestinal Tumor, Guangzhou, China
| | - Baoshan Cao
- Department of Medical Oncology and Radiation Sickness, Peking University Third Hospital, Beijing, China
| | - Guoxin Li
- Department of General Surgery, Nanfang Hospital, Southern Medical University, Guangzhou, China
- Guangdong Provincial Key Laboratory of Precision Medicine for Gastrointestinal Tumor, Guangzhou, China
| | - Xiaotian Zhang
- Department of Gastrointestinal Oncology, Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education), Peking University Cancer Hospital and Institute, Beijing, China.
| | - Bin Dong
- National Biomedical Imaging Center, Peking University, Beijing, China.
- Beijing International Center for Mathematical Research (BICMR), Peking University, Beijing, China.
- Center for Machine Learning Research, Peking University, Beijing, China.
| | - Lin Shen
- Department of Gastrointestinal Oncology, Key Laboratory of Carcinogenesis and Translational Research (Ministry of Education), Peking University Cancer Hospital and Institute, Beijing, China.
| |
Collapse
|
4
|
Odusami M, Damaševičius R, Milieškaitė-Belousovienė E, Maskeliūnas R. Alzheimer's disease stage recognition from MRI and PET imaging data using Pareto-optimal quantum dynamic optimization. Heliyon 2024; 10:e34402. [PMID: 39145034 PMCID: PMC11320145 DOI: 10.1016/j.heliyon.2024.e34402] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/07/2023] [Revised: 07/06/2024] [Accepted: 07/09/2024] [Indexed: 08/16/2024] Open
Abstract
The threat posed by Alzheimer's disease (AD) to human health has grown significantly. However, the precise diagnosis and classification of AD stages remain a challenge. Neuroimaging methods such as structural magnetic resonance imaging (sMRI) and fluorodeoxyglucose positron emission tomography (FDG-PET) have been used to diagnose and categorize AD. However, feature selection approaches that are frequently used to extract additional data from multimodal imaging are prone to errors. This paper suggests using a static pulse-coupled neural network and a Laplacian pyramid to combine sMRI and FDG-PET data. After that, the fused images are used to train the Mobile Vision Transformer (MViT), optimized with Pareto-Optimal Quantum Dynamic Optimization for Neural Architecture Search, while the fused images are augmented to avoid overfitting and then classify unfused MRI and FDG-PET images obtained from the AD Neuroimaging Initiative (ADNI) and Open Access Series of Imaging Studies (OASIS) datasets into various stages of AD. The architectural hyperparameters of MViT are optimized using Quantum Dynamic Optimization, which ensures a Pareto-optimal solution. The Peak Signal-to-Noise Ratio (PSNR), the Mean Squared Error (MSE), and the Structured Similarity Indexing Method (SSIM) are used to measure the quality of the fused image. We found that the fused image was consistent in all metrics, having 0.64 SIMM, 35.60 PSNR, and 0.21 MSE for the FDG-PET image. In the classification of AD vs. cognitive normal (CN), AD vs. mild cognitive impairment (MCI), and CN vs. MCI, the precision of the proposed method is 94.73%, 92.98% and 89.36%, respectively. The sensitivity is 90. 70%, 90. 70%, and 90. 91% while the specificity is 100%, 100%, and 85. 71%, respectively, in the ADNI MRI test data.
Collapse
Affiliation(s)
- Modupe Odusami
- Faculty of Informatics, Kaunas University of Technology, Kaunas, Lithuania
| | | | | | - Rytis Maskeliūnas
- Faculty of Informatics, Kaunas University of Technology, Kaunas, Lithuania
| |
Collapse
|
5
|
Chen H, Zhang H, Yang Y, Zhang Q. F-Norm-Based Soft LDA Algorithm for Fault Detection in Chemical Production Processes. ACS OMEGA 2024; 9:34725-34734. [PMID: 39157156 PMCID: PMC11325430 DOI: 10.1021/acsomega.4c03747] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 04/18/2024] [Revised: 07/12/2024] [Accepted: 07/25/2024] [Indexed: 08/20/2024]
Abstract
In chemical production processes, outliers are inevitable. Many existing feature extraction algorithms are overly sensitive to outliers and excessively focus on secondary features while ignoring the key features in the data. To address this problem, the Frobenius norm based soft linear discriminant analysis algorithm (FBSLA) is proposed in this paper. Specifically, FBSLA uses the Frobenius norm instead of its square as a metric to enhance the robustness of the algorithm. Furthermore, a nonreduced dimensionality projection matrix is introduced to make the training data features more obvious. Additionally, soft constraint is adopted instead of the traditional hard constraint to reduce the sensitivity caused by outliers. To validate the effectiveness of FBSLA, in this paper, experiments are conducted on the Tennessee Eastman Process and the Penicillin Fermentation Process data sets. According to experimental results, FBSLA significantly outperforms other state-of-the-art algorithms in terms of fault detection accuracy.
Collapse
Affiliation(s)
- Hao Chen
- School of Information Engineering, Nantong Institute of Technology, Nantong 226002, China
| | - Haifei Zhang
- School of Information Engineering, Nantong Institute of Technology, Nantong 226002, China
| | - Yuwei Yang
- School of Information Engineering, Nantong Institute of Technology, Nantong 226002, China
| | - Qiong Zhang
- School of Information Engineering, Nantong Institute of Technology, Nantong 226002, China
| |
Collapse
|
6
|
Ganesan P, Ramesh GP, Falkowski-Gilski P, Falkowska-Gilska B. Detection of Alzheimer's disease using Otsu thresholding with tunicate swarm algorithm and deep belief network. Front Physiol 2024; 15:1380459. [PMID: 39045216 PMCID: PMC11263168 DOI: 10.3389/fphys.2024.1380459] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2024] [Accepted: 06/10/2024] [Indexed: 07/25/2024] Open
Abstract
Introduction: Alzheimer's Disease (AD) is a degenerative brain disorder characterized by cognitive and memory dysfunctions. The early detection of AD is necessary to reduce the mortality rate through slowing down its progression. The prevention and detection of AD is the emerging research topic for many researchers. The structural Magnetic Resonance Imaging (sMRI) is an extensively used imaging technique in detection of AD, because it efficiently reflects the brain variations. Methods: Machine learning and deep learning models are widely applied on sMRI images for AD detection to accelerate the diagnosis process and to assist clinicians for timely treatment. In this article, an effective automated framework is implemented for early detection of AD. At first, the Region of Interest (RoI) is segmented from the acquired sMRI images by employing Otsu thresholding method with Tunicate Swarm Algorithm (TSA). The TSA finds the optimal segmentation threshold value for Otsu thresholding method. Then, the vectors are extracted from the RoI by applying Local Binary Pattern (LBP) and Local Directional Pattern variance (LDPv) descriptors. At last, the extracted vectors are passed to Deep Belief Networks (DBN) for image classification. Results and Discussion: The proposed framework achieves supreme classification accuracy of 99.80% and 99.92% on the Alzheimer's Disease Neuroimaging Initiative (ADNI) and Australian Imaging, Biomarker and Lifestyle flagship work of ageing (AIBL) datasets, which is higher than the conventional detection models.
Collapse
Affiliation(s)
- Praveena Ganesan
- Department of Electronics and Communication Engineering, St. Peter’s Institute of Higher Education and Research, Chennai, India
| | - G. P. Ramesh
- Department of Electronics and Communication Engineering, St. Peter’s Institute of Higher Education and Research, Chennai, India
| | | | | |
Collapse
|
7
|
Odusami M, Maskeliūnas R, Damaševičius R, Misra S. Machine learning with multimodal neuroimaging data to classify stages of Alzheimer's disease: a systematic review and meta-analysis. Cogn Neurodyn 2024; 18:775-794. [PMID: 38826669 PMCID: PMC11143094 DOI: 10.1007/s11571-023-09993-5] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/16/2023] [Revised: 06/23/2023] [Accepted: 07/17/2023] [Indexed: 06/04/2024] Open
Abstract
In recent years, Alzheimer's disease (AD) has been a serious threat to human health. Researchers and clinicians alike encounter a significant obstacle when trying to accurately identify and classify AD stages. Several studies have shown that multimodal neuroimaging input can assist in providing valuable insights into the structural and functional changes in the brain related to AD. Machine learning (ML) algorithms can accurately categorize AD phases by identifying patterns and linkages in multimodal neuroimaging data using powerful computational methods. This study aims to assess the contribution of ML methods to the accurate classification of the stages of AD using multimodal neuroimaging data. A systematic search is carried out in IEEE Xplore, Science Direct/Elsevier, ACM DigitalLibrary, and PubMed databases with forward snowballing performed on Google Scholar. The quantitative analysis used 47 studies. The explainable analysis was performed on the classification algorithm and fusion methods used in the selected studies. The pooled sensitivity and specificity, including diagnostic efficiency, were evaluated by conducting a meta-analysis based on a bivariate model with the hierarchical summary receiver operating characteristics (ROC) curve of multimodal neuroimaging data and ML methods in the classification of AD stages. Wilcoxon signed-rank test is further used to statistically compare the accuracy scores of the existing models. With a 95% confidence interval of 78.87-87.71%, the combined sensitivity for separating participants with mild cognitive impairment (MCI) from healthy control (NC) participants was 83.77%; for separating participants with AD from NC, it was 94.60% (90.76%, 96.89%); for separating participants with progressive MCI (pMCI) from stable MCI (sMCI), it was 80.41% (74.73%, 85.06%). With a 95% confidence interval (78.87%, 87.71%), the Pooled sensitivity for distinguishing mild cognitive impairment (MCI) from healthy control (NC) participants was 83.77%, with a 95% confidence interval (90.76%, 96.89%), the Pooled sensitivity for distinguishing AD from NC was 94.60%, likewise (MCI) from healthy control (NC) participants was 83.77% progressive MCI (pMCI) from stable MCI (sMCI) was 80.41% (74.73%, 85.06%), and early MCI (EMCI) from NC was 86.63% (82.43%, 89.95%). Pooled specificity for differentiating MCI from NC was 79.16% (70.97%, 87.71%), AD from NC was 93.49% (91.60%, 94.90%), pMCI from sMCI was 81.44% (76.32%, 85.66%), and EMCI from NC was 85.68% (81.62%, 88.96%). The Wilcoxon signed rank test showed a low P-value across all the classification tasks. Multimodal neuroimaging data with ML is a promising future in classifying the stages of AD but more research is required to increase the validity of its application in clinical practice.
Collapse
Affiliation(s)
- Modupe Odusami
- Department of Multimedia Engineering, Kaunas University of Technology, Kaunas, Lithuania
| | - Rytis Maskeliūnas
- Department of Multimedia Engineering, Kaunas University of Technology, Kaunas, Lithuania
| | | | - Sanjay Misra
- Department of Applied Data Science, Institute for Energy Technology, Halden, Norway
| |
Collapse
|
8
|
Chen Z, Liu Y, Zhang Y, Zhu J, Li Q, Wu X. Shared Manifold Regularized Joint Feature Selection for Joint Classification and Regression in Alzheimer's Disease Diagnosis. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2024; 33:2730-2745. [PMID: 38578858 DOI: 10.1109/tip.2024.3382600] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 04/07/2024]
Abstract
In Alzheimer's disease (AD) diagnosis, joint feature selection for predicting disease labels (classification) and estimating cognitive scores (regression) with neuroimaging data has received increasing attention. In this paper, we propose a model named Shared Manifold regularized Joint Feature Selection (SMJFS) that performs classification and regression in a unified framework for AD diagnosis. For classification, unlike the existing works that build least squares regression models which are insufficient in the ability of extracting discriminative information for classification, we design an objective function that integrates linear discriminant analysis and subspace sparsity regularization for acquiring an informative feature subset. Furthermore, the local data relationships are learned according to the samples' transformed distances to exploit the local data structure adaptively. For regression, in contrast to previous works that overlook the correlations among cognitive scores, we learn a latent score space to capture the correlations and employ the latent space to design a regression model with l2,1 -norm regularization, facilitating the feature selection in regression task. Moreover, the missing cognitive scores can be recovered in the latent space for increasing the number of available training samples. Meanwhile, to capture the correlations between the two tasks and describe the local relationships between samples, we construct an adaptive shared graph to guide the subspace learning in classification and the latent cognitive score learning in regression simultaneously. An efficient iterative optimization algorithm is proposed to solve the optimization problem. Extensive experiments on three datasets validate the discriminability of the features selected by SMJFS.
Collapse
|
9
|
Xu X, Li J, Zhu Z, Zhao L, Wang H, Song C, Chen Y, Zhao Q, Yang J, Pei Y. A Comprehensive Review on Synergy of Multi-Modal Data and AI Technologies in Medical Diagnosis. Bioengineering (Basel) 2024; 11:219. [PMID: 38534493 DOI: 10.3390/bioengineering11030219] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/29/2023] [Revised: 02/15/2024] [Accepted: 02/21/2024] [Indexed: 03/28/2024] Open
Abstract
Disease diagnosis represents a critical and arduous endeavor within the medical field. Artificial intelligence (AI) techniques, spanning from machine learning and deep learning to large model paradigms, stand poised to significantly augment physicians in rendering more evidence-based decisions, thus presenting a pioneering solution for clinical practice. Traditionally, the amalgamation of diverse medical data modalities (e.g., image, text, speech, genetic data, physiological signals) is imperative to facilitate a comprehensive disease analysis, a topic of burgeoning interest among both researchers and clinicians in recent times. Hence, there exists a pressing need to synthesize the latest strides in multi-modal data and AI technologies in the realm of medical diagnosis. In this paper, we narrow our focus to five specific disorders (Alzheimer's disease, breast cancer, depression, heart disease, epilepsy), elucidating advanced endeavors in their diagnosis and treatment through the lens of artificial intelligence. Our survey not only delineates detailed diagnostic methodologies across varying modalities but also underscores commonly utilized public datasets, the intricacies of feature engineering, prevalent classification models, and envisaged challenges for future endeavors. In essence, our research endeavors to contribute to the advancement of diagnostic methodologies, furnishing invaluable insights for clinical decision making.
Collapse
Affiliation(s)
- Xi Xu
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Jianqiang Li
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Zhichao Zhu
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Linna Zhao
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Huina Wang
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Changwei Song
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Yining Chen
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Qing Zhao
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Jijiang Yang
- Tsinghua National Laboratory for Information Science and Technology, Tsinghua University, Beijing 100084, China
| | - Yan Pei
- School of Computer Science and Engineering, The University of Aizu, Aizuwakamatsu 965-8580, Japan
| |
Collapse
|
10
|
Xie J, Zhong W, Yang R, Wang L, Zhen X. Discriminative fusion of moments-aligned latent representation of multimodality medical data. Phys Med Biol 2023; 69:015015. [PMID: 38052076 DOI: 10.1088/1361-6560/ad1271] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/10/2023] [Accepted: 12/05/2023] [Indexed: 12/07/2023]
Abstract
Fusion of multimodal medical data provides multifaceted, disease-relevant information for diagnosis or prognosis prediction modeling. Traditional fusion strategies such as feature concatenation often fail to learn hidden complementary and discriminative manifestations from high-dimensional multimodal data. To this end, we proposed a methodology for the integration of multimodality medical data by matching their moments in a latent space, where the hidden, shared information of multimodal data is gradually learned by optimization with multiple feature collinearity and correlation constrains. We first obtained the multimodal hidden representations by learning mappings between the original domain and shared latent space. Within this shared space, we utilized several relational regularizations, including data attribute preservation, feature collinearity and feature-task correlation, to encourage learning of the underlying associations inherent in multimodal data. The fused multimodal latent features were finally fed to a logistic regression classifier for diagnostic prediction. Extensive evaluations on three independent clinical datasets have demonstrated the effectiveness of the proposed method in fusing multimodal data for medical prediction modeling.
Collapse
Affiliation(s)
- Jincheng Xie
- School of Biomedical Engineering, Southern Medical University, Guangzhou, Guangdong 510515, People's Republic of China
| | - Weixiong Zhong
- School of Biomedical Engineering, Southern Medical University, Guangzhou, Guangdong 510515, People's Republic of China
| | - Ruimeng Yang
- Department of Radiology, the Second Affiliated Hospital, School of Medicine, South China University of Technology, Guangzhou, Guangdong, 510180, People's Republic of China
| | - Linjing Wang
- Radiotherapy Center, Affiliated Cancer Hospital & Institute of Guangzhou Medical University, Guangzhou, Guangdong 510095, People's Republic of China
| | - Xin Zhen
- School of Biomedical Engineering, Southern Medical University, Guangzhou, Guangdong 510515, People's Republic of China
| |
Collapse
|
11
|
Dai Y, Zou B, Zhu C, Li Y, Chen Z, Ji Z, Kui X, Zhang W. DE-JANet: A unified network based on dual encoder and joint attention for Alzheimer's disease classification using multi-modal data. Comput Biol Med 2023; 165:107396. [PMID: 37703717 DOI: 10.1016/j.compbiomed.2023.107396] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2023] [Revised: 07/28/2023] [Accepted: 08/26/2023] [Indexed: 09/15/2023]
Abstract
Structural magnetic resonance imaging (sMRI), which can reflect cerebral atrophy, plays an important role in the early detection of Alzheimer's disease (AD). However, the information provided by analyzing only the morphological changes in sMRI is relatively limited, and the assessment of the atrophy degree is subjective. Therefore, it is meaningful to combine sMRI with other clinical information to acquire complementary diagnosis information and achieve a more accurate classification of AD. Nevertheless, how to fuse these multi-modal data effectively is still challenging. In this paper, we propose DE-JANet, a unified AD classification network that integrates image data sMRI with non-image clinical data, such as age and Mini-Mental State Examination (MMSE) score, for more effective multi-modal analysis. DE-JANet consists of three key components: (1) a dual encoder module for extracting low-level features from the image and non-image data according to specific encoding regularity, (2) a joint attention module for fusing multi-modal features, and (3) a token classification module for performing AD-related classification according to the fused multi-modal features. Our DE-JANet is evaluated on the ADNI dataset, with a mean accuracy of 0.9722 and 0.9538 for AD classification and mild cognition impairment (MCI) classification, respectively, which is superior to existing methods and indicates advanced performance on AD-related diagnosis tasks.
Collapse
Affiliation(s)
- Yulan Dai
- School of Computer Science and Engineering, Central South University, Changsha, China; Hunan Engineering Research Center of Machine Vision and Intelligent Medicine, Changsha, China
| | - Beiji Zou
- School of Computer Science and Engineering, Central South University, Changsha, China; Hunan Engineering Research Center of Machine Vision and Intelligent Medicine, Changsha, China
| | - Chengzhang Zhu
- School of Computer Science and Engineering, Central South University, Changsha, China; Hunan Engineering Research Center of Machine Vision and Intelligent Medicine, Changsha, China.
| | - Yang Li
- School of Computer Science and Engineering, Central South University, Changsha, China; Hunan Engineering Research Center of Machine Vision and Intelligent Medicine, Changsha, China
| | - Zhi Chen
- School of Computer Science and Engineering, Central South University, Changsha, China; Hunan Engineering Research Center of Machine Vision and Intelligent Medicine, Changsha, China
| | - Zexin Ji
- School of Computer Science and Engineering, Central South University, Changsha, China; Hunan Engineering Research Center of Machine Vision and Intelligent Medicine, Changsha, China
| | - Xiaoyan Kui
- School of Computer Science and Engineering, Central South University, Changsha, China
| | - Wensheng Zhang
- Institute of Automation, Chinese Academy of Sciences, Beijing, China
| |
Collapse
|
12
|
Zhang ZC, Zhao X, Dong G, Zhao XM. Improving Alzheimer's Disease Diagnosis With Multi-Modal PET Embedding Features by a 3D Multi-Task MLP-Mixer Neural Network. IEEE J Biomed Health Inform 2023; 27:4040-4051. [PMID: 37247318 DOI: 10.1109/jbhi.2023.3280823] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 05/31/2023]
Abstract
Positron emission tomography (PET) with fluorodeoxyglucose (FDG) or florbetapir (AV45) has been proved effective in the diagnosis of Alzheimer's disease. However, the expensive and radioactive nature of PET has limited its application. Here, employing multi-layer perceptron mixer architecture, we present a deep learning model, namely 3-dimensional multi-task multi-layer perceptron mixer, for simultaneously predicting the standardized uptake value ratios (SUVRs) for FDG-PET and AV45-PET from the cheap and widely used structural magnetic resonance imaging data, and the model can be further used for Alzheimer's disease diagnosis based on embedding features derived from SUVR prediction. Experiment results demonstrate the high prediction accuracy of the proposed method for FDG/AV45-PET SUVRs, where we achieved Pearson's correlation coefficients of 0.66 and 0.61 respectively between the estimated and actual SUVR and the estimated SUVRs also show high sensitivity and distinct longitudinal patterns for different disease status. By taking into account PET embedding features, the proposed method outperforms other competing methods on five independent datasets in the diagnosis of Alzheimer's disease and discriminating between stable and progressive mild cognitive impairments, achieving the area under receiver operating characteristic curves of 0.968 and 0.776 respectively on ADNI dataset, and generalizes better to other external datasets. Moreover, the top-weighted patches extracted from the trained model involve important brain regions related to Alzheimer's disease, suggesting good biological interpretability of our proposed method."
Collapse
|
13
|
Xu F, Qiao C, Zhou H, Calhoun VD, Stephen JM, Wilson TW, Wang Y. An explainable autoencoder with multi-paradigm fMRI fusion for identifying differences in dynamic functional connectivity during brain development. Neural Netw 2023; 159:185-197. [PMID: 36580711 DOI: 10.1016/j.neunet.2022.12.007] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/27/2022] [Revised: 10/19/2022] [Accepted: 12/12/2022] [Indexed: 12/24/2022]
Abstract
Multi-paradigm deep learning models show great potential for dynamic functional connectivity (dFC) analysis by integrating complementary information. However, many of them cannot use information from different paradigms effectively and have poor explainability, that is, the ability to identify significant features that contribute to decision making. In this paper, we propose a multi-paradigm fusion-based explainable deep sparse autoencoder (MF-EDSAE) to address these issues. Considering explainability, the MF-EDSAE is constructed based on a deep sparse autoencoder (DSAE). For integrating information effectively, the MF-EDASE contains the nonlinear fusion layer and multi-paradigm hypergraph regularization. We apply the model to the Philadelphia Neurodevelopmental Cohort and demonstrate it achieves better performance in detecting dynamic FC (dFC) that differ significantly during brain development than the single-paradigm DSAE. The experimental results show that children have more dispersive dFC patterns than adults. The function of the brain transits from undifferentiated systems to specialized networks during brain development. Meanwhile, adults have stronger connectivities between task-related functional networks for a given task than children. As the brain develops, the patterns of the global dFC change more quickly when stimulated by a task.
Collapse
Affiliation(s)
- Faming Xu
- School of Mathematics and Statistics, Xi'an Jiaotong University, Xi'an, 710049, PR China.
| | - Chen Qiao
- School of Mathematics and Statistics, Xi'an Jiaotong University, Xi'an, 710049, PR China.
| | - Huiyu Zhou
- School of Computing and Mathematical Sciences, University of Leicester, LE1 7RH, UK.
| | - Vince D Calhoun
- Tri-Institutional Center for Translational Research in Neuroimaging and Data Science (TReNDS), Georgia State University, Georgia Institute of Technology, Emory University, Atlanta, GA 30030, USA.
| | | | - Tony W Wilson
- Institute for Human Neuroscience, Boys Town National Research Hospital, Boys Town, NE 68010, USA.
| | - Yuping Wang
- Department of Biomedical Engineering, Tulane University, New Orleans, LA 70118, USA.
| |
Collapse
|
14
|
Chen Z, Liu Y, Zhang Y, Li Q. Orthogonal latent space learning with feature weighting and graph learning for multimodal Alzheimer's disease diagnosis. Med Image Anal 2023; 84:102698. [PMID: 36462372 DOI: 10.1016/j.media.2022.102698] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/10/2022] [Revised: 10/18/2022] [Accepted: 11/17/2022] [Indexed: 11/23/2022]
Abstract
Recent studies have shown that multimodal neuroimaging data provide complementary information of the brain and latent space-based methods have achieved promising results in fusing multimodal data for Alzheimer's disease (AD) diagnosis. However, most existing methods treat all features equally and adopt nonorthogonal projections to learn the latent space, which cannot retain enough discriminative information in the latent space. Besides, they usually preserve the relationships among subjects in the latent space based on the similarity graph constructed on original features for performance boosting. However, the noises and redundant features significantly corrupt the graph. To address these limitations, we propose an Orthogonal Latent space learning with Feature weighting and Graph learning (OLFG) model for multimodal AD diagnosis. Specifically, we map multiple modalities into a common latent space by orthogonal constrained projection to capture the discriminative information for AD diagnosis. Then, a feature weighting matrix is utilized to sort the importance of features in AD diagnosis adaptively. Besides, we devise a regularization term with learned graph to preserve the local structure of the data in the latent space and integrate the graph construction into the learning processing for accurately encoding the relationships among samples. Instead of constructing a similarity graph for each modality, we learn a joint graph for multiple modalities to capture the correlations among modalities. Finally, the representations in the latent space are projected into the target space to perform AD diagnosis. An alternating optimization algorithm with proved convergence is developed to solve the optimization objective. Extensive experimental results show the effectiveness of the proposed method.
Collapse
Affiliation(s)
- Zhi Chen
- Knowledge and Data Engineering Laboratory of Chinese Medicine, School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu 610054, China
| | - Yongguo Liu
- Knowledge and Data Engineering Laboratory of Chinese Medicine, School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu 610054, China.
| | - Yun Zhang
- Knowledge and Data Engineering Laboratory of Chinese Medicine, School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu 610054, China
| | - Qiaoqin Li
- Knowledge and Data Engineering Laboratory of Chinese Medicine, School of Information and Software Engineering, University of Electronic Science and Technology of China, Chengdu 610054, China
| |
Collapse
|
15
|
Huang W, Tan K, Zhang Z, Hu J, Dong S. A Review of Fusion Methods for Omics and Imaging Data. IEEE/ACM TRANSACTIONS ON COMPUTATIONAL BIOLOGY AND BIOINFORMATICS 2023; 20:74-93. [PMID: 35044920 DOI: 10.1109/tcbb.2022.3143900] [Citation(s) in RCA: 4] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/04/2023]
Abstract
The development of omics data and biomedical images has greatly advanced the progress of precision medicine in diagnosis, treatment, and prognosis. The fusion of omics and imaging data, i.e., omics-imaging fusion, offers a new strategy for understanding complex diseases. However, due to a variety of issues such as the limited number of samples, high dimensionality of features, and heterogeneity of different data types, efficiently learning complementary or associated discriminative fusion information from omics and imaging data remains a challenge. Recently, numerous machine learning methods have been proposed to alleviate these problems. In this review, from the perspective of fusion levels and fusion methods, we first provide an overview of preprocessing and feature extraction methods for omics and imaging data, and comprehensively analyze and summarize the basic forms and variations of commonly used and newly emerging fusion methods, along with their advantages, disadvantages and the applicable scope. We then describe public datasets and compare experimental results of various fusion methods on the ADNI and TCGA datasets. Finally, we discuss future prospects and highlight remaining challenges in the field.
Collapse
|
16
|
Alorf A, Khan MUG. Multi-label classification of Alzheimer's disease stages from resting-state fMRI-based correlation connectivity data and deep learning. Comput Biol Med 2022; 151:106240. [PMID: 36423532 DOI: 10.1016/j.compbiomed.2022.106240] [Citation(s) in RCA: 13] [Impact Index Per Article: 6.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2022] [Revised: 09/21/2022] [Accepted: 10/22/2022] [Indexed: 11/11/2022]
Abstract
Alzheimer's disease is a neurodegenerative condition that gradually impairs cognitive abilities. Recently, various neuroimaging modalities and machine learning methods have surfaced to diagnose Alzheimer's disease. Resting-state fMRI is a neuroimaging modality that has been widely utilized to study brain activity related to neurodegenerative diseases. In literature, the previous studies are limited to the binary classification of Alzheimer's disease and Mild Cognitive Impairment. The application of computer-aided diagnosis for the numerous advancing phases of Alzheimer's disease, on the other hand, remains understudied. This research analyzes and presents methods for multi-label classification of six Alzheimer's stages using rs-fMRI and deep learning. The proposed model solves the multi-class classification problem by extracting the brain's functional connectivity networks from rs-fMRI data and employing two deep learning approaches, Stacked Sparse Autoencoder and Brain Connectivity Graph Convolutional Network. The suggested models' results were assessed using the k-fold cross-validation approach, and an average accuracy of 77.13% and 84.03% was reached for multi-label classification using Stacked Sparse Autoencoders and Brain Connectivity Based Convolutional Network, respectively. An analysis of brain regions was also performed by using the network's learned weights, leading to the conclusion that the precentral gyrus, frontal gyrus, lingual gyrus, and supplementary motor area are the significant brain regions of interest.
Collapse
Affiliation(s)
- Abdulaziz Alorf
- Department of Electrical Engineering, College of Engineering, Qassim University, Buraydah 52571, Saudi Arabia.
| | - Muhammad Usman Ghani Khan
- Department of Computer Science, University of Engineering and Technology (UET), Lahore, Pakistan; National Center of Artificial Intelligence (NCAI), Al-Khwarizmi Institute of Computer Science (KICS), Lahore, Pakistan.
| |
Collapse
|
17
|
Zheng S, Zhu Z, Liu Z, Guo Z, Liu Y, Yang Y, Zhao Y. Multi-Modal Graph Learning for Disease Prediction. IEEE TRANSACTIONS ON MEDICAL IMAGING 2022; 41:2207-2216. [PMID: 35286257 DOI: 10.1109/tmi.2022.3159264] [Citation(s) in RCA: 15] [Impact Index Per Article: 7.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Benefiting from the powerful expressive capability of graphs, graph-based approaches have been popularly applied to handle multi-modal medical data and achieved impressive performance in various biomedical applications. For disease prediction tasks, most existing graph-based methods tend to define the graph manually based on specified modality (e.g., demographic information), and then integrated other modalities to obtain the patient representation by Graph Representation Learning (GRL). However, constructing an appropriate graph in advance is not a simple matter for these methods. Meanwhile, the complex correlation between modalities is ignored. These factors inevitably yield the inadequacy of providing sufficient information about the patient's condition for a reliable diagnosis. To this end, we propose an end-to-end Multi-modal Graph Learning framework (MMGL) for disease prediction with multi-modality. To effectively exploit the rich information across multi-modality associated with the disease, modality-aware representation learning is proposed to aggregate the features of each modality by leveraging the correlation and complementarity between the modalities. Furthermore, instead of defining the graph manually, the latent graph structure is captured through an effective way of adaptive graph learning. It could be jointly optimized with the prediction model, thus revealing the intrinsic connections among samples. Our model is also applicable to the scenario of inductive learning for those unseen data. An extensive group of experiments on two disease prediction tasks demonstrates that the proposed MMGL achieves more favorable performance. The code of MMGL is available at https://github.com/SsGood/MMGL.
Collapse
|
18
|
Zhu J, Tan Y, Lin R, Miao J, Fan X, Zhu Y, Liang P, Gong J, He H. Efficient self-attention mechanism and structural distilling model for Alzheimer’s disease diagnosis. Comput Biol Med 2022; 147:105737. [DOI: 10.1016/j.compbiomed.2022.105737] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/02/2022] [Revised: 05/23/2022] [Accepted: 06/11/2022] [Indexed: 11/27/2022]
|
19
|
Tan K, Huang W, Liu X, Hu J, Dong S. A multi-modal fusion framework based on multi-task correlation learning for cancer prognosis prediction. Artif Intell Med 2022; 126:102260. [DOI: 10.1016/j.artmed.2022.102260] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/11/2021] [Revised: 01/07/2022] [Accepted: 02/16/2022] [Indexed: 12/30/2022]
|
20
|
Abstract
Medical images of brain tumors are critical for characterizing the pathology of tumors and early diagnosis. There are multiple modalities for medical images of brain tumors. Fusing the unique features of each modality of the magnetic resonance imaging (MRI) scans can accurately determine the nature of brain tumors. The current genetic analysis approach is time-consuming and requires surgical extraction of brain tissue samples. Accurate classification of multi-modal brain tumor images can speed up the detection process and alleviate patient suffering. Medical image fusion refers to effectively merging the significant information of multiple source images of the same tissue into one image, which will carry abundant information for diagnosis. This paper proposes a novel attentive deep-learning-based classification model that integrates multi-modal feature aggregation, lite attention mechanism, separable embedding, and modal-wise shortcuts for performance improvement. We evaluate our model on the RSNA-MICCAI dataset, a scenario-specific medical image dataset, and demonstrate that the proposed method outperforms the state-of-the-art (SOTA) by around 3%.
Collapse
|
21
|
Ning Z, Du D, Tu C, Feng Q, Zhang Y. Relation-Aware Shared Representation Learning for Cancer Prognosis Analysis With Auxiliary Clinical Variables and Incomplete Multi-Modality Data. IEEE TRANSACTIONS ON MEDICAL IMAGING 2022; 41:186-198. [PMID: 34460368 DOI: 10.1109/tmi.2021.3108802] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
The integrative analysis of complementary phenotype information contained in multi-modality data (e.g., histopathological images and genomic data) has advanced the prognostic evaluation of cancers. However, multi-modality based prognosis analysis confronts two challenges: (1) how to explore underlying relations inherent in different modalities data for learning compact and discriminative multi-modality representations; (2) how to take full consideration of incomplete multi-modality data for constructing accurate and robust prognostic model, since a host of complete multi-modality data are not always available. Additionally, many existing multi-modality based prognostic methods commonly ignore relevant clinical variables (e.g., grade and stage), which, however, may provide supplemental information to promote the performance of model. In this paper, we propose a relation-aware shared representation learning method for prognosis analysis of cancers, which makes full use of clinical information and incomplete multi-modality data. The proposed method learns multi-modal shared space tailored for prognostic model via a dual mapping. Within the shared space, it equips with relational regularizers to explore the potential relations (i.e., feature-label and feature-feature relations) among multi-modality data for inducing discriminatory representations and simultaneously obtaining extra sparsity for alleviating overfitting. Moreover, it regresses and incorporates multiple auxiliary clinical attributes with dynamic coefficients to meliorate performance. Furthermore, in training stage, a partial mapping strategy is employed to extend and train a more reliable model with incomplete multi-modality data. We have evaluated our method on three public datasets derived from The Cancer Genome Atlas (TCGA) project, and the experimental results demonstrate the superior performance of the proposed method.
Collapse
|
22
|
Ning Z, Tu C, Di X, Feng Q, Zhang Y. Deep cross-view co-regularized representation learning for glioma subtype identification. Med Image Anal 2021; 73:102160. [PMID: 34303890 DOI: 10.1016/j.media.2021.102160] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/23/2021] [Revised: 05/04/2021] [Accepted: 06/29/2021] [Indexed: 10/20/2022]
Abstract
The new subtypes of diffuse gliomas are recognized by the World Health Organization (WHO) on the basis of genotypes, e.g., isocitrate dehydrogenase and chromosome arms 1p/19q, in addition to the histologic phenotype. Glioma subtype identification can provide valid guidances for both risk-benefit assessment and clinical decision. The feature representations of gliomas in magnetic resonance imaging (MRI) have been prevalent for revealing underlying subtype status. However, since gliomas are highly heterogeneous tumors with quite variable imaging phenotypes, learning discriminative feature representations in MRI for gliomas remains challenging. In this paper, we propose a deep cross-view co-regularized representation learning framework for glioma subtype identification, in which view representation learning and multiple constraints are integrated into a unified paradigm. Specifically, we first learn latent view-specific representations based on cross-view images generated from MRI via a bi-directional mapping connecting original imaging space and latent space, and view-correlated regularizer and output-consistent regularizer in the latent space are employed to explore view correlation and derive view consistency, respectively. We further learn view-sharable representations which can explore complementary information of multiple views by projecting the view-specific representations into a holistically shared space and enhancing via adversary learning strategy. Finally, the view-specific and view-sharable representations are incorporated for identifying glioma subtype. Experimental results on multi-site datasets demonstrate the proposed method outperforms several state-of-the-art methods in detection of glioma subtype status.
Collapse
Affiliation(s)
- Zhenyuan Ning
- School of Biomedical Engineering, Southern Medical University, Guangzhou 510515, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou 510515, China
| | - Chao Tu
- School of Biomedical Engineering, Southern Medical University, Guangzhou 510515, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou 510515, China
| | - Xiaohui Di
- School of Biomedical Engineering, Southern Medical University, Guangzhou 510515, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou 510515, China
| | - Qianjin Feng
- School of Biomedical Engineering, Southern Medical University, Guangzhou 510515, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou 510515, China
| | - Yu Zhang
- School of Biomedical Engineering, Southern Medical University, Guangzhou 510515, China; Guangdong Provincial Key Laboratory of Medical Image Processing, Southern Medical University, Guangzhou 510515, China; Guangdong Province Engineering Laboratory for Medical Imaging and Diagnostic Technology, Southern Medical University, Guangzhou 510515, China.
| |
Collapse
|