Disease-Image-Specific Learning for Diagnosis-Oriented Neuroimage Synthesis With Incomplete Multi-Modality Data

A review of deep learning-based information fusion techniques for multimodal medical image classification

Multimodal medical imaging plays a pivotal role in clinical diagnosis and research, as it combines information from various imaging modalities to provide a more comprehensive understanding of the underlying pathology. Recently, deep learning-based multimodal fusion techniques have emerged as powerful tools for improving medical image classification. This review offers a thorough analysis of the developments in deep learning-based multimodal fusion for medical classification tasks. We explore the complementary relationships among prevalent clinical modalities and outline three main fusion schemes for multimodal classification networks: input fusion, intermediate fusion (encompassing single-level fusion, hierarchical fusion, and attention-based fusion), and output fusion. By evaluating the performance of these fusion techniques, we provide insight into the suitability of different network architectures for various multimodal fusion scenarios and application domains. Furthermore, we delve into challenges related to network architecture selection, handling incomplete multimodal data management, and the potential limitations of multimodal fusion. Finally, we spotlight the promising future of Transformer-based multimodal fusion techniques and give recommendations for future research in this rapidly evolving field.

MGIML: Cancer Grading With Incomplete Radiology-Pathology Data via Memory Learning and Gradient Homogenization

Taking advantage of multi-modal radiology-pathology data with complementary clinical information for cancer grading is helpful for doctors to improve diagnosis efficiency and accuracy. However, radiology and pathology data have distinct acquisition difficulties and costs, which leads to incomplete-modality data being common in applications. In this work, we propose a Memory- and Gradient-guided Incomplete Modal-modal Learning (MGIML) framework for cancer grading with incomplete radiology-pathology data. Firstly, to remedy missing-modality information, we propose a Memory-driven Hetero-modality Complement (MH-Complete) scheme, which constructs modal-specific memory banks constrained by a coarse-grained memory boosting (CMB) loss to record generic radiology and pathology feature patterns, and develops a cross-modal memory reading strategy enhanced by a fine-grained memory consistency (FMC) loss to take missing-modality information from well-stored memories. Secondly, as gradient conflicts exist between missing-modality situations, we propose a Rotation-driven Gradient Homogenization (RG-Homogenize) scheme, which estimates instance-specific rotation matrices to smoothly change the feature-level gradient directions, and computes confidence-guided homogenization weights to dynamically balance gradient magnitudes. By simultaneously mitigating gradient direction and magnitude conflicts, this scheme well avoids the negative transfer and optimization imbalance problems. Extensive experiments on CPTAC-UCEC and CPTAC-PDA datasets show that the proposed MGIML framework performs favorably against state-of-the-art multi-modal methods on missing-modality situations.

Attention-Like Multimodality Fusion With Data Augmentation for Diagnosis of Mental Disorders Using MRI

The globally rising prevalence of mental disorders leads to shortfalls in timely diagnosis and therapy to reduce patients' suffering. Facing such an urgent public health problem, professional efforts based on symptom criteria are seriously overstretched. Recently, the successful applications of computer-aided diagnosis approaches have provided timely opportunities to relieve the tension in healthcare services. Particularly, multimodal representation learning gains increasing attention thanks to the high temporal and spatial resolution information extracted from neuroimaging fusion. In this work, we propose an efficient multimodality fusion framework to identify multiple mental disorders based on the combination of functional and structural magnetic resonance imaging. A multioutput conditional generative adversarial network (GAN) is developed to address the scarcity of multimodal data for augmentation. Based on the augmented training data, the multiheaded gating fusion model is proposed for classification by extracting the complementary features across different modalities. The experiments demonstrate that the proposed model can achieve robust accuracies of 75.1 ± 1.5 %, 72.9 ± 1.1 %, and 87.2 ± 1.5 % for autism spectrum disorder (ASD), attention deficit/hyperactivity disorder, and schizophrenia, respectively. In addition, the interpretability of our model is expected to enable the identification of remarkable neuropathology diagnostic biomarkers, leading to well-informed therapeutic decisions.

Disentangle Then Calibrate With Gradient Guidance: A Unified Framework for Common and Rare Disease Diagnosis

The computer-aided diagnosis (CAD) for rare diseases using medical imaging poses a significant challenge due to the requirement of large volumes of labeled training data, which is particularly difficult to collect for rare diseases. Although Few-shot learning (FSL) methods have been developed for this task, these methods focus solely on rare disease diagnosis, failing to preserve the performance in common disease diagnosis. To address this issue, we propose the Disentangle then Calibrate with Gradient Guidance (DCGG) framework under the setting of generalized few-shot learning, i.e., using one model to diagnose both common and rare diseases. The DCGG framework consists of a network backbone, a gradient-guided network disentanglement (GND) module, and a gradient-induced feature calibration (GFC) module. The GND module disentangles the network into a disease-shared component and a disease-specific component based on gradient guidance, and devises independent optimization strategies for both components, respectively, when learning from rare diseases. The GFC module transfers only the disease-shared channels of common-disease features to rare diseases, and incorporates the optimal transport theory to identify the best transport scheme based on the semantic relationship among different diseases. Based on the best transport scheme, the GFC module calibrates the distribution of rare-disease features at the disease-shared channels, deriving more informative rare-disease features for better diagnosis. The proposed DCGG framework has been evaluated on three public medical image classification datasets. Our results suggest that the DCGG framework achieves state-of-the-art performance in diagnosing both common and rare diseases.

MHD-Net: Memory-Aware Hetero-Modal Distillation Network for Thymic Epithelial Tumor Typing With Missing Pathology Modality

Fusing multi-modal radiology and pathology data with complementary information can improve the accuracy of tumor typing. However, collecting pathology data is difficult since it is high-cost and sometimes only obtainable after the surgery, which limits the application of multi-modal methods in diagnosis. To address this problem, we propose comprehensively learning multi-modal radiology-pathology data in training, and only using uni-modal radiology data in testing. Concretely, a Memory-aware Hetero-modal Distillation Network (MHD-Net) is proposed, which can distill well-learned multi-modal knowledge with the assistance of memory from the teacher to the student. In the teacher, to tackle the challenge in hetero-modal feature fusion, we propose a novel spatial-differentiated hetero-modal fusion module (SHFM) that models spatial-specific tumor information correlations across modalities. As only radiology data is accessible to the student, we store pathology features in the proposed contrast-boosted typing memory module (CTMM) that achieves type-wise memory updating and stage-wise contrastive memory boosting to ensure the effectiveness and generalization of memory items. In the student, to improve the cross-modal distillation, we propose a multi-stage memory-aware distillation (MMD) scheme that reads memory-aware pathology features from CTMM to remedy missing modal-specific information. Furthermore, we construct a Radiology-Pathology Thymic Epithelial Tumor (RPTET) dataset containing paired CT and WSI images with annotations. Experiments on the RPTET and CPTAC-LUAD datasets demonstrate that MHD-Net significantly improves tumor typing and outperforms existing multi-modal methods on missing modality situations.

A novel loss function to reproduce texture features for deep learning-based MRI-to-CT synthesis

BACKGROUND

Studies on computed tomography (CT) synthesis based on magnetic resonance imaging (MRI) have mainly focused on pixel-wise consistency, but the texture features of regions of interest (ROIs) have not received appropriate attention.

PURPOSE

This study aimed to propose a novel loss function to reproduce texture features of ROIs and pixel-wise consistency for deep learning-based MRI-to-CT synthesis. The method was expected to assist the multi-modality studies for radiomics.

METHODS

The study retrospectively enrolled 127 patients with nasopharyngeal carcinoma. CT and MRI images were collected for each patient, and then rigidly registered as pre-procession. We proposed a gray-level co-occurrence matrix (GLCM)-based loss function to improve the reproducibility of texture features. This novel loss function could be embedded into the present deep learning-based framework for image synthesis. In this study, a typical image synthesis model was selected as the baseline, which contained a Unet trained mean square error (MSE) loss function. We embedded the proposed loss function and designed experiments to supervise different ROIs to prove its effectiveness. The concordance correlation coefficient (CCC) of the GLCM feature was employed to evaluate the reproducibility of GLCM features, which are typical texture features. Besides, we used a publicly available dataset of brain tumors to verify our loss function.

RESULTS

Compared with the baseline, the proposed method improved the pixel-wise image quality metrics (MAE: 107.5 to 106.8 HU; SSIM: 0.9728 to 0.9730). CCC values of the GLCM features in GTVnx were significantly improved from 0.78 ± 0.12 to 0.82 ± 0.11 (p < 0.05 for paired t-test). Generally, > 90% (22/24) of the GLCM-based features were improved compared with the baseline, where the Informational Measure of Correlation feature was improved the most (CCC: 0.74 to 0.83). For the public dataset, the loss function also shows its effectiveness. With our proposed loss function added, the ability to reproduce texture features was improved in the ROIs.

CONCLUSIONS

The proposed method reproduced texture features for MRI-to-CT synthesis, which would benefit radiomics studies based on image multi-modality synthesis.

Multimodal classification of Alzheimer's disease and mild cognitive impairment using custom MKSCDDL kernel over CNN with transparent decision-making for explainable diagnosis

The study presents an innovative diagnostic framework that synergises Convolutional Neural Networks (CNNs) with a Multi-feature Kernel Supervised within-class-similar Discriminative Dictionary Learning (MKSCDDL). This integrative methodology is designed to facilitate the precise classification of individuals into categories of Alzheimer's Disease, Mild Cognitive Impairment (MCI), and Cognitively Normal (CN) statuses while also discerning the nuanced phases within the MCI spectrum. Our approach is distinguished by its robustness and interpretability, offering clinicians an exceptionally transparent tool for diagnosis and therapeutic strategy formulation. We use scandent decision trees to deal with the unpredictability and complexity of neuroimaging data. Considering that different people's brain scans are different, this enables the model to make more detailed individualised assessments and explains how the algorithm illuminates the specific neuroanatomical regions that are indicative of cognitive impairment. This explanation is beneficial for clinicians because it gives them concrete ideas for early intervention and targeted care. The empirical review of our model shows that it makes diagnoses with a level of accuracy that is unmatched, with a classification efficacy of 98.27%. This shows that the model is good at finding important parts of the brain that may be damaged by cognitive diseases.

Addiction-related brain networks identification via Graph Diffusion Reconstruction Network

Functional magnetic resonance imaging (fMRI) provides insights into complex patterns of brain functional changes, making it a valuable tool for exploring addiction-related brain connectivity. However, effectively extracting addiction-related brain connectivity from fMRI data remains challenging due to the intricate and non-linear nature of brain connections. Therefore, this paper proposed the Graph Diffusion Reconstruction Network (GDRN), a novel framework designed to capture addiction-related brain connectivity from fMRI data acquired from addicted rats. The proposed GDRN incorporates a diffusion reconstruction module that effectively maintains the unity of data distribution by reconstructing the training samples, thereby enhancing the model's ability to reconstruct nicotine addiction-related brain networks. Experimental evaluations conducted on a nicotine addiction rat dataset demonstrate that the proposed GDRN effectively explores nicotine addiction-related brain connectivity. The findings suggest that the GDRN holds promise for uncovering and understanding the complex neural mechanisms underlying addiction using fMRI data.

Joint learning framework of cross-modal synthesis and diagnosis for Alzheimer's disease by mining underlying shared modality information

Alzheimer's disease (AD) is one of the most common neurodegenerative disorders presenting irreversible progression of cognitive impairment. How to identify AD as early as possible is critical for intervention with potential preventive measures. Among various neuroimaging modalities used to diagnose AD, functional positron emission tomography (PET) has higher sensitivity than structural magnetic resonance imaging (MRI), but it is also costlier and often not available in many hospitals. How to leverage massive unpaired unlabeled PET to improve the diagnosis performance of AD from MRI becomes rather important. To address this challenge, this paper proposes a novel joint learning framework of unsupervised cross-modal synthesis and AD diagnosis by mining underlying shared modality information, improving the AD diagnosis from MRI while synthesizing more discriminative PET images. We mine underlying shared modality information in two aspects: diversifying modality information through the cross-modal synthesis network and locating critical diagnosis-related patterns through the AD diagnosis network. First, to diversify the modality information, we propose a novel unsupervised cross-modal synthesis network, which implements the inter-conversion between 3D PET and MRI in a single model modulated by the AdaIN module. Second, to locate shared critical diagnosis-related patterns, we propose an interpretable diagnosis network based on fully 2D convolutions, which takes either 3D synthesized PET or original MRI as input. Extensive experimental results on the ADNI dataset show that our framework can synthesize more realistic images, outperform the state-of-the-art AD diagnosis methods, and have better generalization on external AIBL and NACC datasets.

Disentangle First, Then Distill: A Unified Framework for Missing Modality Imputation and Alzheimer's Disease Diagnosis

Multi-modality medical data provide complementary information, and hence have been widely explored for computer-aided AD diagnosis. However, the research is hindered by the unavoidable missing-data problem, i.e., one data modality was not acquired on some subjects due to various reasons. Although the missing data can be imputed using generative models, the imputation process may introduce unrealistic information to the classification process, leading to poor performance. In this paper, we propose the Disentangle First, Then Distill (DFTD) framework for AD diagnosis using incomplete multi-modality medical images. First, we design a region-aware disentanglement module to disentangle each image into inter-modality relevant representation and intra-modality specific representation with emphasis on disease-related regions. To progressively integrate multi-modality knowledge, we then construct an imputation-induced distillation module, in which a lateral inter-modality transition unit is created to impute representation of the missing modality. The proposed DFTD framework has been evaluated against six existing methods on an ADNI dataset with 1248 subjects. The results show that our method has superior performance in both AD-CN classification and MCI-to-AD prediction tasks, substantially over-performing all competing methods.

A model for predicting lymph node metastasis of thyroid carcinoma: a multimodality convolutional neural network study

Background

Early preoperative evaluation of cervical lymph node metastasis (LNM) in papillary thyroid carcinoma (PTC) is critical for further surgical treatment. However, insufficient accuracy in predicting LNM status for PTC based on ultrasound images is a problem that needs to be urgently resolved. This study aimed to clarify the role of convolutional neural networks (CNNs) in predicting LNM for PTC based on multimodality ultrasound.

Methods

In this study, the data of 308 patients who were clinically diagnosed with PTC and had confirmed LNM status via postoperative pathology at Beijing Tiantan Hospital, Capital Medical University, from August 2018 to April 2022 were incorporated into CNN algorithm development and evaluation. Of these patients, 80% were randomly included into the training set and 20% into the test set. The ultrasound examination of cervical LNM was performed to assess possible metastasis. Residual network 50 (Resnet50) was employed for feature extraction from the B-mode and contrast-enhanced ultrasound (CEUS) images. For each case, all of features were extracted from B-mode ultrasound images and CEUS images separately, and the ultrasound examination data of cervical LNM information were concatenated together to produce a final multimodality LNM prediction. Sensitivity, specificity, accuracy, and area under the receiver operating characteristic curve (AUC) were used to evaluate the performance of the predictive model. Heatmaps were further developed for visualizing the attention region of the images of the best-working model.

Results

Of the 308 patients with PTC included in the analysis, 158 (51.3%) were diagnosed as LNM and 150 (48.7%) as non-LNM. In the test set, when a triple-modality method (i.e., B-mode image, CEUS image, and ultrasound examination of cervical LNM) was used, accuracy was maximized at 80.65% (AUC =0.831; sensitivity =80.65%; specificity =82.26%), which showed an expected increased performance over B-mode alone (accuracy =69.00%; AUC =0.720; sensitivity =70.00%; specificity =73.00%) and a dual-modality method (B-mode image plus CEUS image: accuracy =75.81%; AUC =0.742; sensitivity =74.19%; specificity =77.42%). The heatmaps of our triple-modality model demonstrated a possible focus area and revealed the model's flaws.

Conclusions

The PTC lymph node prediction model based on the triple-modality features significantly outperformed all the other feature configurations. This deep learning model mimics the workflow of a human expert and leverages multimodal data from patients with PTC, thus further supporting clinical decision-making.

Deep learning enables automatic adult age estimation based on CT reconstruction images of the costal cartilage

OBJECTIVE

Adult age estimation (AAE) is a challenging task. Deep learning (DL) could be a supportive tool. This study aimed to develop DL models for AAE based on CT images and compare their performance to the manual visual scoring method.

METHODS

Chest CT were reconstructed using volume rendering (VR) and maximum intensity projection (MIP) separately. Retrospective data of 2500 patients aged 20.00-69.99 years were obtained. The cohort was split into training (80%) and validation (20%) sets. Additional independent data from 200 patients were used as the test set and external validation set. Different modality DL models were developed accordingly. Comparisons were hierarchically performed by VR versus MIP, single-modality versus multi-modality, and DL versus manual method. Mean absolute error (MAE) was the primary parameter of comparison.

RESULTS

A total of 2700 patients (mean age = 45.24 years ± 14.03 [SD]) were evaluated. Of single-modality models, MAEs yielded by VR were lower than MIP. Multi-modality models generally yielded lower MAEs than the optimal single-modality model. The best-performing multi-modality model obtained the lowest MAEs of 3.78 in males and 3.40 in females. On the test set, DL achieved MAEs of 3.78 in males and 3.92 in females, which were far better than the MAEs of 8.90 and 6.42 respectively, for the manual method. For the external validation, MAEs were 6.05 in males and 6.68 in females for DL, and 6.93 and 8.28 for the manual method.

CONCLUSIONS

DL demonstrated better performance than the manual method in AAE based on CT reconstruction of the costal cartilage.

CLINICAL RELEVANCE STATEMENT

Aging leads to diseases, functional performance deterioration, and both physical and physiological damage over time. Accurate AAE may aid in diagnosing the personalization of aging processes.

KEY POINTS

• VR-based DL models outperformed MIP-based models with lower MAEs and higher R2 values. • All multi-modality DL models showed better performance than single-modality models in adult age estimation. • DL models achieved a better performance than expert assessments.

A neuroimaging biomarker for Individual Brain-Related Abnormalities In Neurodegeneration (IBRAIN): a cross-sectional study

Background

Alzheimer's disease (AD) is a prevalent neurodegenerative disorder that poses a worldwide public health challenge. A neuroimaging biomarker would significantly improve early diagnosis and intervention, ultimately enhancing the quality of life for affected individuals and reducing the burden on healthcare systems.

Methods

Cross-sectional and longitudinal data (10,099 participants with 13,380 scans) from 12 independent datasets were used in the present study (this study was performed between September 1, 2021 and February 15, 2023). The Individual Brain-Related Abnormalities In Neurodegeneration (IBRAIN) score was developed via integrated regional- and network-based measures under an ensemble machine learning model based on structural MRI data. We systematically assessed whether IBRAIN could be a neuroimaging biomarker for AD.

Findings

IBRAIN accurately differentiated individuals with AD from NCs (AUC = 0.92) and other neurodegenerative diseases, including Frontotemporal dementia (FTD), Parkinson's disease (PD), Vascular dementia (VaD) and Amyotrophic Lateral Sclerosis (ALS) (AUC = 0.92). IBRAIN was significantly correlated to clinical measures and gene expression, enriched in immune process and protein metabolism. The IBRAIN score exhibited a significant ability to reveal the distinct progression of prodromal AD (i.e., Mild cognitive impairment, MCI) (Hazard Ratio (HR) = 6.52 [95% CI: 4.42∼9.62], p < 1 × 10-16), which offers similar powerful performance with Cerebrospinal Fluid (CSF) Aβ (HR = 3.78 [95% CI: 2.63∼5.43], p = 2.13 × 10-14) and CSF Tau (HR = 3.77 [95% CI: 2.64∼5.39], p = 9.53 × 10-15) based on the COX and Log-rank test. Notably, the IBRAIN shows comparable sensitivity (beta = -0.70, p < 1 × 10-16) in capturing longitudinal changes in individuals with conversion to AD than CSF Aβ (beta = -0.26, p = 4.40 × 10-9) and CSF Tau (beta = 0.12, p = 1.02 × 10-5).

Interpretation

Our findings suggested that IBRAIN is a biologically relevant, specific, and sensitive neuroimaging biomarker that can serve as a clinical measure to uncover prodromal AD progression. It has strong potential for application in future clinical practice and treatment trials.

Funding

Science and Technology Innovation 2030 Major Projects, the National Natural Science Foundation of China, Beijing Natural Science Funds, the Fundamental Research Funds for the CentralUniversity, and the Startup Funds for Talents at Beijing Normal University.

Hybrid Multimodality Fusion with Cross-Domain Knowledge Transfer to Forecast Progression Trajectories in Cognitive Decline

Magnetic resonance imaging (MRI) and positron emission tomography (PET) are increasingly used to forecast progression trajectories of cognitive decline caused by preclinical and prodromal Alzheimer's disease (AD). Many existing studies have explored the potential of these two distinct modalities with diverse machine and deep learning approaches. But successfully fusing MRI and PET can be complex due to their unique characteristics and missing modalities. To this end, we develop a hybrid multimodality fusion (HMF) framework with cross-domain knowledge transfer for joint MRI and PET representation learning, feature fusion, and cognitive decline progression forecasting. Our HMF consists of three modules: 1) a module to impute missing PET images, 2) a module to extract multimodality features from MRI and PET images, and 3) a module to fuse the extracted multimodality features. To address the issue of small sample sizes, we employ a cross-domain knowledge transfer strategy from the ADNI dataset, which includes 795 subjects, to independent small-scale AD-related cohorts, in order to leverage the rich knowledge present within the ADNI. The proposed HMF is extensively evaluated in three AD-related studies with 272 subjects across multiple disease stages, such as subjective cognitive decline and mild cognitive impairment. Experimental results demonstrate the superiority of our method over several state-of-the-art approaches in forecasting progression trajectories of AD-related cognitive decline.

Improving Alzheimer's Disease Diagnosis With Multi-Modal PET Embedding Features by a 3D Multi-Task MLP-Mixer Neural Network

Positron emission tomography (PET) with fluorodeoxyglucose (FDG) or florbetapir (AV45) has been proved effective in the diagnosis of Alzheimer's disease. However, the expensive and radioactive nature of PET has limited its application. Here, employing multi-layer perceptron mixer architecture, we present a deep learning model, namely 3-dimensional multi-task multi-layer perceptron mixer, for simultaneously predicting the standardized uptake value ratios (SUVRs) for FDG-PET and AV45-PET from the cheap and widely used structural magnetic resonance imaging data, and the model can be further used for Alzheimer's disease diagnosis based on embedding features derived from SUVR prediction. Experiment results demonstrate the high prediction accuracy of the proposed method for FDG/AV45-PET SUVRs, where we achieved Pearson's correlation coefficients of 0.66 and 0.61 respectively between the estimated and actual SUVR and the estimated SUVRs also show high sensitivity and distinct longitudinal patterns for different disease status. By taking into account PET embedding features, the proposed method outperforms other competing methods on five independent datasets in the diagnosis of Alzheimer's disease and discriminating between stable and progressive mild cognitive impairments, achieving the area under receiver operating characteristic curves of 0.968 and 0.776 respectively on ADNI dataset, and generalizes better to other external datasets. Moreover, the top-weighted patches extracted from the trained model involve important brain regions related to Alzheimer's disease, suggesting good biological interpretability of our proposed method."

Quantitative Cerebral Blood Volume Image Synthesis from Standard MRI Using Image-to-Image Translation for Brain Tumors

Background Cerebral blood volume (CBV) maps derived from dynamic susceptibility contrast-enhanced (DSC) MRI are useful but not commonly available in clinical scenarios. Purpose To test image-to-image translation techniques for generating CBV maps from standard MRI sequences of brain tumors using the bookend technique DSC MRI as ground-truth references. Materials and Methods A total of 756 MRI examinations, including quantitative CBV maps produced from bookend DSC MRI, were included in this retrospective study. Two algorithms, the feature-consistency generative adversarial network (GAN) and three-dimensional encoder-decoder network with only mean absolute error loss, were trained to synthesize CBV maps. The performance of the two algorithms was evaluated quantitatively using the structural similarity index (SSIM) and qualitatively by two neuroradiologists using a four-point Likert scale. The clinical value of combining synthetic CBV maps and standard MRI scans of brain tumors was assessed in several clinical scenarios (tumor grading, prognosis prediction, differential diagnosis) using multicenter data sets (four external and one internal). Differences in diagnostic and predictive accuracy were tested using the z test. Results The three-dimensional encoder-decoder network with T1-weighted images, contrast-enhanced T1-weighted images, and apparent diffusion coefficient maps as the input achieved the highest synthetic performance (SSIM, 86.29% ± 4.30). The mean qualitative score of the synthesized CBV maps by neuroradiologists was 2.63. Combining synthetic CBV with standard MRI improved the accuracy of grading gliomas (standard MRI scans area under the receiver operating characteristic curve [AUC], 0.707; standard MRI scans with CBV maps AUC, 0.857; z = 15.17; P < .001), prediction of prognosis in gliomas (standard MRI scans AUC, 0.654; standard MRI scans with CBV maps AUC, 0.793; z = 9.62; P < .001), and differential diagnosis between tumor recurrence and treatment response in gliomas (standard MRI scans AUC, 0.778; standard MRI scans with CBV maps AUC, 0.853; z = 4.86; P < .001) and brain metastases (standard MRI scans AUC, 0.749; standard MRI scans with CBV maps AUC, 0.857; z = 6.13; P < .001). Conclusion GAN image-to-image translation techniques produced accurate synthetic CBV maps from standard MRI scans, which could be used for improving the clinical evaluation of brain tumors. Published under a CC BY 4.0 license. Supplemental material is available for this article. See also the editorial by Branstetter in this issue.

Generative AI for brain image computing and brain network computing: a review

Recent years have witnessed a significant advancement in brain imaging techniques that offer a non-invasive approach to mapping the structure and function of the brain. Concurrently, generative artificial intelligence (AI) has experienced substantial growth, involving using existing data to create new content with a similar underlying pattern to real-world data. The integration of these two domains, generative AI in neuroimaging, presents a promising avenue for exploring various fields of brain imaging and brain network computing, particularly in the areas of extracting spatiotemporal brain features and reconstructing the topological connectivity of brain networks. Therefore, this study reviewed the advanced models, tasks, challenges, and prospects of brain imaging and brain network computing techniques and intends to provide a comprehensive picture of current generative AI techniques in brain imaging. This review is focused on novel methodological approaches and applications of related new methods. It discussed fundamental theories and algorithms of four classic generative models and provided a systematic survey and categorization of tasks, including co-registration, super-resolution, enhancement, classification, segmentation, cross-modality, brain network analysis, and brain decoding. This paper also highlighted the challenges and future directions of the latest work with the expectation that future research can be beneficial.

Deep multimodality-disentangled association analysis network for imaging genetics in neurodegenerative diseases

Imaging genetics is a crucial tool that is applied to explore potentially disease-related biomarkers, particularly for neurodegenerative diseases (NDs). With the development of imaging technology, the association analysis between multimodal imaging data and genetic data is gradually being concerned by a wide range of imaging genetics studies. However, multimodal data are fused first and then correlated with genetic data in traditional methods, which leads to an incomplete exploration of their common and complementary information. In addition, the inaccurate formulation in the complex relationships between imaging and genetic data and information loss caused by missing multimodal data are still open problems in imaging genetics studies. Therefore, in this study, a deep multimodality-disentangled association analysis network (DMAAN) is proposed to solve the aforementioned issues and detect the disease-related biomarkers of NDs simultaneously. First, the imaging data are nonlinearly projected into a latent space and imaging representations can be achieved. The imaging representations are further disentangled into common and specific parts by using a multimodal-disentangled module. Second, the genetic data are encoded to achieve genetic representations, and then, the achieved genetic representations are nonlinearly mapped to the common and specific imaging representations to build nonlinear associations between imaging and genetic data through an association analysis module. Moreover, modality mask vectors are synchronously synthesized to integrate the genetic and imaging data, which helps the following disease diagnosis. Finally, the proposed method achieves reasonable diagnosis performance via a disease diagnosis module and utilizes the label information to detect the disease-related modality-shared and modality-specific biomarkers. Furthermore, the genetic representation can be used to impute the missing multimodal data with our learning strategy. Two publicly available datasets with different NDs are used to demonstrate the effectiveness of the proposed DMAAN. The experimental results show that the proposed DMAAN can identify the disease-related biomarkers, which suggests the proposed DMAAN may provide new insights into the pathological mechanism and early diagnosis of NDs. The codes are publicly available at https://github.com/Meiyan88/DMAAN.

COM: Contrastive Masked-attention model for incomplete multimodal learning

Most multimodal learning methods assume that all modalities are always available in data. However, in real-world applications, the assumption is often violated due to privacy protection, sensor failure etc. Previous works for incomplete multimodal learning often suffer from one of the following drawbacks: introducing noise, lacking flexibility to missing patterns and failing to capture interactions between modalities. To overcome these challenges, we propose a COntrastive Masked-attention model (COM). The framework performs cross-modal contrastive learning with GAN-based augmentation to reduce modality gap, and employs a masked-attention model to capture interactions between modalities. The augmentation adapts cross-modal contrastive learning to suit incomplete case by a two-player game, improving the effectiveness of multimodal representations. Interactions between modalities are modeled by stacking self-attention blocks, and attention masks limit them on the observed modalities to avoid extra noise. All kinds of modality combinations share a unified architecture, so the model is flexible to different missing patterns. Extensive experiments on six datasets demonstrate the effectiveness and robustness of the proposed method for incomplete multimodal learning.

A Hybrid Deep Learning Method for Early and Late Mild Cognitive Impairment Diagnosis With Incomplete Multimodal Data

Multimodality neuroimages have been widely applied to diagnose mild cognitive impairment (MCI). However, the missing data problem is unavoidable. Most previously developed methods first train a generative adversarial network (GAN) to synthesize missing data and then train a classification network with the completed data. These methods independently train two networks with no information communication. Thus, the resulting GAN cannot focus on the crucial regions that are helpful for classification. To overcome this issue, we propose a hybrid deep learning method. First, a classification network is pretrained with paired MRI and PET images. Afterward, we use the pretrained classification network to guide a GAN by focusing on the features that are helpful for classification. Finally, we synthesize the missing PET images and use them with real MR images to fine-tune the classification model to make it better adapt to the synthesized images. We evaluate our proposed method on the ADNI dataset, and the results show that our method improves the accuracies obtained on the validation and testing sets by 3.84 and 5.82%, respectively. Moreover, our method increases the accuracies for the validation and testing sets by 7.7 and 9.09%, respectively, when we synthesize the missing PET images via our method. An ablation experiment shows that the last two stages are essential for our method. We also compare our method with other state-of-the-art methods, and our method achieves better classification performance.

BMNet: A New Region-Based Metric Learning Method for Early Alzheimer's Disease Identification With FDG-PET Images

18F-fluorodeoxyglucose (FDG)-positron emission tomography (PET) reveals altered brain metabolism in individuals with mild cognitive impairment (MCI) and Alzheimer's disease (AD). Some biomarkers derived from FDG-PET by computer-aided-diagnosis (CAD) technologies have been proved that they can accurately diagnosis normal control (NC), MCI, and AD. However, existing FDG-PET-based researches are still insufficient for the identification of early MCI (EMCI) and late MCI (LMCI). Compared with methods based other modalities, current methods with FDG-PET are also inadequate in using the inter-region-based features for the diagnosis of early AD. Moreover, considering the variability in different individuals, some hard samples which are very similar with both two classes limit the classification performance. To tackle these problems, in this paper, we propose a novel bilinear pooling and metric learning network (BMNet), which can extract the inter-region representation features and distinguish hard samples by constructing the embedding space. To validate the proposed method, we collect 898 FDG-PET images from Alzheimer's disease neuroimaging initiative (ADNI) including 263 normal control (NC) patients, 290 EMCI patients, 147 LMCI patients, and 198 AD patients. Following the common preprocessing steps, 90 features are extracted from each FDG-PET image according to the automatic anatomical landmark (AAL) template and then sent into the proposed network. Extensive fivefold cross-validation experiments are performed for multiple two-class classifications. Experiments show that most metrics are improved after adding the bilinear pooling module and metric losses to the Baseline model respectively. Specifically, in the classification task between EMCI and LMCI, the specificity improves 6.38% after adding the triple metric loss, and the negative predictive value (NPV) improves 3.45% after using the bilinear pooling module. In addition, the accuracy of classification between EMCI and LMCI achieves 79.64% using imbalanced FDG-PET images, which illustrates that the proposed method yields a state-of-the-art result of the classification accuracy between EMCI and LMCI based on PET images.

Assessing clinical progression from subjective cognitive decline to mild cognitive impairment with incomplete multi-modal neuroimages

Accurately assessing clinical progression from subjective cognitive decline (SCD) to mild cognitive impairment (MCI) is crucial for early intervention of pathological cognitive decline. Multi-modal neuroimaging data such as T1-weighted magnetic resonance imaging (MRI) and positron emission tomography (PET), help provide objective and supplementary disease biomarkers for computer-aided diagnosis of MCI. However, there are few studies dedicated to SCD progression prediction since subjects usually lack one or more imaging modalities. Besides, one usually has a limited number (e.g., tens) of SCD subjects, negatively affecting model robustness. To this end, we propose a Joint neuroimage Synthesis and Representation Learning (JSRL) framework for SCD conversion prediction using incomplete multi-modal neuroimages. The JSRL contains two components: 1) a generative adversarial network to synthesize missing images and generate multi-modal features, and 2) a classification network to fuse multi-modal features for SCD conversion prediction. The two components are incorporated into a joint learning framework by sharing the same features, encouraging effective fusion of multi-modal features for accurate prediction. A transfer learning strategy is employed in the proposed framework by leveraging model trained on the Alzheimer's Disease Neuroimaging Initiative (ADNI) with MRI and fluorodeoxyglucose PET from 863 subjects to both the Chinese Longitudinal Aging Study (CLAS) with only MRI from 76 SCD subjects and the Australian Imaging, Biomarkers and Lifestyle (AIBL) with MRI from 235 subjects. Experimental results suggest that the proposed JSRL yields superior performance in SCD and MCI conversion prediction and cross-database neuroimage synthesis, compared with several state-of-the-art methods.