Latent Representation Learning for Alzheimer's Disease Diagnosis With Incomplete Multi-Modality Neuroimaging and Genetic Data

A novel diagnosis method utilizing MDBO-SVM and imaging genetics for Alzheimer's disease

Alzheimer's disease (AD) is the most common neurodegenerative disorder, yet its underlying mechanisms remain elusive. Early and accurate diagnosis is crucial for timely intervention and disease management. In this paper, a multi-strategy improved dung beetle optimizer (MDBO) was proposed to establish a new framework for AD diagnosis. The unique aspect of this algorithm lies in its integration of the Osprey Optimization Algorithm, Lévy flight, and adaptive t-distribution. This combination endows MDBO with superior global search capabilities and the ability to avoid local optima. Then, we presented a novel fitness function for integrating imaging genetics data. In experiments, MDBO demonstrated outstanding performance on the CEC2017 benchmark functions, proving its effectiveness in optimization problems. Furthermore, it was used to classify individuals with AD, mild cognitive impairment (MCI), and control normal (CN) using limited features. In the multi-classification of CN, MCI, and AD, the algorithm achieved excellent results, with an average accuracy of 81.7 % and a best accuracy of 92 %. Overall, the proposed MDBO algorithm provides a more comprehensive and efficient diagnostic tool, offering new possibilities for early intervention and disease progression control.

Dual-way magnetic resonance image translation with transformer-based adversarial network

BACKGROUND

The magnetic resonance (MR) image translation model is designed to generate MR images of required sequence from the images of existing sequence. However, the generalization performance of MR image generation models on external datasets tends to be unsatisfactory due to the inconsistency in the data distribution of MR images across different centers or scanners.

PURPOSE

The aim of this study is to propose a cross-sequence MR image synthesis model that could generate high-quality MR synthetic images with high transferability for small-sized external datasets.

METHODS

We proposed a dual-way magnetic resonance image translation model using transformer-based adversarial network (DMTrans) for MR image synthesis across sequences. It integrates a transformer-based generative architecture with an innovative discriminator design. The shifted window-based multi-head self-attention mechanism in DMTrans enables efficient capture of global and local features from MR images. The sequential dual-scale discriminator is designed to distinguish features of the generated images at multi-scale.

RESULTS

We pre-trained DMTrans model for bi-directional image synthesis on a T1/T2-weighted MR image dataset comprising 4229 slices. It demonstrates superior performance to baseline methods on both qualitative and quantitative measurements. The SSIM, PSNR, and MAE metrics for synthetic T1 images generation based on T2 images are 0.91 ± 0.04, 25.30 ± 2.40, and 24.65 ± 10.46, while the metric values are 0.90 ± 0.04, 24.72 ± 1.62, and 23.28 ± 7.40 for the opposite direction. Fine-tuning is then utilized to adapt the model to another public dataset with T1/T2/proton-weighted (PD) images, so that only 6 patients of 500 slices are required for model adaptation to achieve high-quality T1/T2, T1/PD, and T2/PD image translation results.

CONCLUSIONS

The proposed DMTrans achieves the state-of-the-art performance for cross-sequence MR image conversion, which could provide more information assisting clinical diagnosis and treatment. It also offered a versatile and efficient solution to the needs of high-quality MR image synthesis in data-scarce conditions at different centers.

CSEPC: a deep learning framework for classifying small-sample multimodal medical image data in Alzheimer's disease

BACKGROUND

Alzheimer's disease (AD) is a neurodegenerative disorder that significantly impacts health care worldwide, particularly among the elderly population. The accurate classification of AD stages is essential for slowing disease progression and guiding effective interventions. However, limited sample sizes continue to present a significant challenge in classifying the stages of AD progression. Addressing this obstacle is crucial for improving diagnostic accuracy and optimizing treatment strategies for those affected by AD.

METHODS

In this study, we proposed cross-scale equilibrium pyramid coupling (CSEPC), which is a novel diagnostic algorithm designed for small-sample multimodal medical imaging data. CSEPC leverages scale equilibrium theory and modal coupling properties to integrate semantic features from different imaging modalities and across multiple scales within each modality. The architecture first extracts balanced multiscale features from structural MRI (sMRI) data and functional MRI (fMRI) data using a cross-scale pyramid module. These features are then combined through a contrastive learning-based cosine similarity coupling mechanism to capture intermodality associations effectively. This approach enhances the representation of both inter- and intramodal features while significantly reducing the number of learning parameters, making it highly suitable for small sample environments. We validated the effectiveness of the CSEPC model through experiments on the Alzheimer's Disease Neuroimaging Initiative (ADNI) dataset and demonstrated its superior performance in diagnosing and staging AD.

RESULTS

Our experimental results demonstrate that the proposed model matches or exceeds the performance of models used in previous studies in AD classification. Specifically, the model achieved an accuracy of 85.67% and an area under the curve (AUC) of 0.98 in classifying the progression from mild cognitive impairment (MCI) to AD. To further validate its effectiveness, we used our method to diagnose different stages of AD. In both classification tasks, our approach delivered superior performance.

CONCLUSIONS

In conclusion, the performance of our model in various tasks has demonstrated its significant potential in the field of small-sample multimodal medical imaging classification, particularly AD classification. This advancement could significantly assist clinicians in effectively managing and intervening in the disease progression of patients with early-stage AD.

AN INTERPRETABLE GENERATIVE MULTIMODAL NEUROIMAGING-GENOMICS FRAMEWORK FOR DECODING ALZHEIMER'S DISEASE

Objective

Alzheimer's disease (AD) is the most prevalent form of dementia worldwide, encompassing a prodromal stage known as Mild Cognitive Impairment (MCI), where patients may either progress to AD or remain stable. The objective of the work was to capture structural and functional modulations of brain structure and function relying on multimodal MRI data and Single Nucleotide Polymorphisms, also in case of missing views, with the twofold goal of classifying AD patients versus healthy controls and detecting MCI converters.

Approach

We propose a multimodal DL-based classification framework where a generative module employing Cycle Generative Adversarial Networks was introduced in the latent space for imputing missing data (a common issue of multimodal approaches). Explainable AI method was then used to extract input features' relevance allowing for post-hoc validation and enhancing the interpretability of the learned representations.

Main results

Experimental results on two tasks, AD detection and MCI conversion, showed that our framework reached competitive performance in the state-of-the-art with an accuracy of 0.926 ± 0.02 and 0.711 ± 0.01 in the two tasks, respectively. The interpretability analysis revealed gray matter modulations in cortical and subcortical brain areas typically associated with AD. Moreover, impairments in sensory-motor and visual resting state networks along the disease continuum, as well as genetic mutations defining biological processes linked to endocytosis, amyloid-beta, and cholesterol, were identified.

Significance

Our integrative and interpretable DL approach shows promising performance for AD detection and MCI prediction while shedding light on important biological insights.

Enhanced Multimodal Low-Rank Embedding-Based Feature Selection Model for Multimodal Alzheimer's Disease Diagnosis

Identification of Alzheimer's disease (AD) with multimodal neuroimaging data has been receiving increasing attention. However, the presence of numerous redundant features and corrupted neuroimages within multimodal datasets poses significant challenges for existing methods. In this paper, we propose a feature selection method named Enhanced Multimodal Low-rank Embedding (EMLE) for multimodal AD diagnosis. Unlike previous methods utilizing convex relaxations of the -norm, EMLE exploits an -norm regularized projection matrix to obtain an embedding representation and select informative features jointly for each modality. The -norm, employing an upper-bounded nonconvex Minimax Concave Penalty (MCP) function to characterize sparsity, offers a superior approximation for the -norm compared to other convex relaxations. Next, a similarity graph is learned based on the self-expressiveness property to increase the robustness to corrupted data. As the approximation coefficient vectors of samples from the same class should be highly correlated, an MCP function introduced norm, i.e., matrix -norm, is applied to constrain the rank of the graph. Furthermore, recognizing that diverse modalities should share an underlying structure related to AD, we establish a consensus graph for all modalities to unveil intrinsic structures across multiple modalities. Finally, we fuse the embedding representations of all modalities into the label space to incorporate supervisory information. The results of extensive experiments on the Alzheimer's Disease Neuroimaging Initiative datasets verify the discriminability of the features selected by EMLE.

BIGFormer: A Graph Transformer With Local Structure Awareness for Diagnosis and Pathogenesis Identification of Alzheimer's Disease Using Imaging Genetic Data

Alzheimer's disease (AD) is a highly inheritable neurological disorder, and brain imaging genetics (BIG) has become a rapidly advancing field for comprehensive understanding its pathogenesis. However, most of the existing approaches underestimate the complexity of the interactions among factors that cause AD. To take full appreciate of these complexity interactions, we propose BIGFormer, a graph Transformer with local structural awareness, for AD diagnosis and identification of pathogenic mechanisms. Specifically, the factors interaction graph is constructed with lesion brain regions and risk genes as nodes, where the connection between nodes intuitively represents the interaction between nodes. After that, a perception with local structure awareness is built to extract local structure around nodes, which is then injected into node representation. Then, the global reliance inference component assembles the local structure into higher-order structure, and multi-level interaction structures are jointly aggregated into a classification projection head for disease state prediction. Experimental results show that BIGFormer demonstrated superiority in four classification tasks on the AD neuroimaging initiative dataset and proved to identify biomarkers closely intimately related to AD.

A Modality-Flexible Framework for Alzheimer's Disease Diagnosis Following Clinical Routine

Dementia has high incidence among the elderly, and Alzheimer's disease (AD) is the most common dementia. The procedure of AD diagnosis in clinics usually follows a standard routine consisting of different phases, from acquiring non-imaging tabular data in the screening phase to MR imaging and ultimately to PET imaging. Most of the existing AD diagnosis studies are dedicated to a specific phase using either single or multi-modal data. In this paper, we introduce a modality-flexible classification framework, which is applicable for different AD diagnosis phases following the clinical routine. Specifically, our framework consists of three branches corresponding to three diagnosis phases: 1) a tabular branch using only tabular data for screening phase, 2) an MRI branch using both MRI and tabular data for uncertain cases in screening phase, and 3) ultimately a PET branch for the challenging cases using all the modalities including PET, MRI, and tabular data. To achieve effective fusion of imaging and non-imaging modalities, we introduce an image-tabular transformer block to adaptively scale and shift the image and tabular features according to modality importance determined by the network. The proposed framework is extensively validated on four cohorts containing 6495 subjects. Experiments demonstrate that our framework achieves superior diagnostic performance than the other representative methods across various AD diagnosis tasks, and shows promising performance for all the diagnosis phases, which exhibits great potential for clinical application.

Longitudinal Alzheimer's Disease Progression Prediction With Modality Uncertainty and Optimization of Information Flow

Alzheimer's disease (AD) is a global neurodegenerative disorder that affects millions of individuals worldwide. Actual AD imaging datasets challenge the construction of reliable longitudinal models owing to imaging modality uncertainty. In addition, they are still unable to retain or obtain important information during disease progression from previous to followup time points. For example, the output values of current gates in recurrent models should be close to a specific value that indicates the model is uncertain about retaining or forgetting information. In this study, we propose a model which can extract and constrain each modality into a common representation space to capture intermodality interactions among different modalities associated with modality uncertainty to predict AD progression. In addition, we provide an auxiliary function to enhance the ability of recurrent gate robustly and effectively in controlling the flow of information over time using longitudinal data. We conducted comparative analysis on data from the Alzheimer's Disease Neuroimaging Initiative database. Our model outperformed other methods across all evaluation metrics. Therefore, the proposed model provides a promising solution for addressing modality uncertainty challenges in multimodal longitudinal AD progression prediction.

Domain adaptation in small-scale and heterogeneous biological datasets

Machine-learning models are key to modern biology, yet models trained on one dataset are often not generalizable to other datasets from different cohorts or laboratories due to both technical and biological differences. Domain adaptation, a type of transfer learning, alleviates this problem by aligning different datasets so that models can be applied across them. However, most state-of-the-art domain adaptation methods were designed for large-scale data such as images, whereas biological datasets are smaller and have more features, and these are also complex and heterogeneous. This Review discusses domain adaptation methods in the context of such biological data to inform biologists and guide future domain adaptation research. We describe the benefits and challenges of domain adaptation in biological research and critically explore some of its objectives, strengths, and weaknesses. We argue for the incorporation of domain adaptation techniques to the computational biologist's toolkit, with further development of customized approaches.

CE-GAN: Community Evolutionary Generative Adversarial Network for Alzheimer's Disease Risk Prediction

In the studies of neurodegenerative diseases such as Alzheimer's Disease (AD), researchers often focus on the associations among multi-omics pathogeny based on imaging genetics data. However, current studies overlook the communities in brain networks, leading to inaccurate models of disease development. This paper explores the developmental patterns of AD from the perspective of community evolution. We first establish a mathematical model to describe functional degeneration in the brain as the community evolution driven by entropy information propagation. Next, we propose an interpretable Community Evolutionary Generative Adversarial Network (CE-GAN) to predict disease risk. In the generator of CE-GAN, community evolutionary convolutions are designed to capture the evolutionary patterns of AD. The experiments are conducted using functional magnetic resonance imaging (fMRI) data and single nucleotide polymorphism (SNP) data. CE-GAN achieves 91.67% accuracy and 91.83% area under curve (AUC) in AD risk prediction tasks, surpassing advanced methods on the same dataset. In addition, we validated the effectiveness of CE-GAN for pathogeny extraction. The source code of this work is available at https://github.com/fmri123456/CE-GAN.

A Cross-Modal Mutual Knowledge Distillation Framework for Alzheimer's Disease Diagnosis: Addressing Incomplete Modalities

Early detection of Alzheimer's Disease (AD) is crucial for timely interventions and optimizing treatment outcomes. Despite the promise of integrating multimodal neuroimages such as MRI and PET, handling datasets with incomplete modalities remains under-researched. This phenomenon, however, is common in real-world scenarios as not every patient has all modalities due to practical constraints such as cost, access, and safety concerns. We propose a deep learning framework employing cross-modal Mutual Knowledge Distillation (MKD) to model different sub-cohorts of patients based on their available modalities. In MKD, the multimodal model (e.g., MRI and PET) serves as a teacher, while the single-modality model (e.g., MRI only) is the student. Our MKD framework features three components: a Modality-Disentangling Teacher (MDT) model designed through information disentanglement, a student model that learns from classification errors and MDT's knowledge, and the teacher model enhanced via distilling the student's single-modal feature extraction capabilities. Moreover, we show the effectiveness of the proposed method through theoretical analysis and validate its performance with simulation studies. In addition, our method is demonstrated through a case study with Alzheimer's Disease Neuroimaging Initiative (ADNI) datasets, underscoring the potential of artificial intelligence in addressing incomplete multimodal neuroimaging datasets and advancing early AD detection.

Note to Practitioners—

This paper was motivated by the challenge of early AD diagnosis, particularly in scenarios when clinicians encounter varied availability of patient imaging data, such as MRI and PET scans, often constrained by cost or accessibility issues. We propose an incomplete multimodal learning framework that produces tailored models for patients with only MRI and patients with both MRI and PET. This approach improves the accuracy and effectiveness of early AD diagnosis, especially when imaging resources are limited, via bi-directional knowledge transfer. We introduced a teacher model that prioritizes extracting common information between different modalities, significantly enhancing the student model's learning process. This paper includes theoretical analysis, simulation study, and real-world case study to illustrate the method's promising potential in early AD detection. However, practitioners should be mindful of the complexities involved in model tuning. Future work will focus on improving model interpretability and expanding its application. This includes developing methods to discover the key brain regions for predictions, enhancing clinical trust, and extending the framework to incorporate a broader range of imaging modalities, demographic information, and clinical data. These advancements aim to provide a more comprehensive view of patient health and improve diagnostic accuracy across various neurodegenerative diseases.

Cost-Sensitive Weighted Contrastive Learning Based on Graph Convolutional Networks for Imbalanced Alzheimer's Disease Staging

Identifying the progression stages of Alzheimer's disease (AD) can be considered as an imbalanced multi-class classification problem in machine learning. It is challenging due to the class imbalance issue and the heterogeneity of the disease. Recently, graph convolutional networks (GCNs) have been successfully applied in AD classification. However, these works did not handle the class imbalance issue in classification. Besides, they ignore the heterogeneity of the disease. To this end, we propose a novel cost-sensitive weighted contrastive learning method based on graph convolutional networks (CSWCL-GCNs) for imbalanced AD staging using resting-state functional magnetic resonance imaging (rs-fMRI). The proposed method is developed on a multi-view graph constructed by the functional connectivity (FC) and high-order functional connectivity (HOFC) features of the subjects. A novel cost-sensitive weighted contrastive learning procedure is proposed to capture discriminative information from the minority classes, encouraging the samples in the minority class to provide adequate supervision. Considering the heterogeneity of the disease, the weights of the negative pairs are introduced into contrastive learning and they are computed based on the distance to class prototypes, which are automatically learned from the training data. Meanwhile, the cost-sensitive mechanism is further introduced into contrastive learning to handle the class imbalance issue. The proposed CSWCL-GCN is evaluated on 720 subjects (including 184 NCs, 40 SMC patients, 208 EMCI patients, 172 LMCI patients and 116 AD patients) from the ADNI (Alzheimer's Disease Neuroimaging Initiative). Experimental results show that the proposed CSWCL-GCN outperforms state-of-the-art methods on the ADNI database.

[An MRI multi-sequence feature imputation and fusion mutual-aid model based on sequence deletion for differentiation of high-grade from low-grade glioma]

OBJECTIVE

To evaluate the performance of magnetic resonance imaging (MRI) multi-sequence feature imputation and fusion mutual model based on sequence deletion in differentiating high-grade glioma (HGG) from low-grade glioma (LGG).

METHODS

We retrospectively collected multi-sequence MR images from 305 glioma patients, including 189 HGG patients and 116 LGG patients. The region of interest (ROI) of T1-weighted images (T1WI), T2-weighted images (T2WI), T2 fluid attenuated inversion recovery (T2_FLAIR) and post-contrast enhancement T1WI (CE_T1WI) were delineated to extract the radiomics features. A mutual-aid model of MRI multi-sequence feature imputation and fusion based on sequence deletion was used for imputation and fusion of the feature matrix with missing data. The discriminative ability of the model was evaluated using 5-fold cross-validation method and by assessing the accuracy, balanced accuracy, area under the ROC curve (AUC), specificity, and sensitivity. The proposed model was quantitatively compared with other non-holonomic multimodal classification models for discriminating HGG and LGG. Class separability experiments were performed on the latent features learned by the proposed feature imputation and fusion methods to observe the classification effect of the samples in twodimensional plane. Convergence experiments were used to verify the feasibility of the model.

RESULTS

For differentiation of HGG from LGG with a missing rate of 10%, the proposed model achieved accuracy, balanced accuracy, AUC, specificity, and sensitivity of 0.777, 0.768, 0.826, 0.754 and 0.780, respectively. The fused latent features showed excellent performance in the class separability experiment, and the algorithm could be iterated to convergence with superior classification performance over other methods at the missing rates of 30% and 50%.

CONCLUSION

The proposed model has excellent performance in classification task of HGG and LGG and outperforms other non-holonomic multimodal classification models, demonstrating its potential for efficient processing of non-holonomic multimodal data.

Latent relation shared learning for endometrial cancer diagnosis with incomplete multi-modality medical images

Magnetic resonance imaging (MRI), ultrasound (US), and contrast-enhanced ultrasound (CEUS) can provide different image data about uterus, which have been used in the preoperative assessment of endometrial cancer. In practice, not all the patients have complete multi-modality medical images due to the high cost or long examination period. Most of the existing methods need to perform data cleansing or discard samples with missing modalities, which will influence the performance of the model. In this work, we propose an incomplete multi-modality images data fusion method based on latent relation shared to overcome this limitation. The shared space contains the common latent feature representation and modality-specific latent feature representation from the complete and incomplete multi-modality data, which jointly exploits both consistent and complementary information among multiple images. The experimental results show that our method outperforms the current representative approaches in terms of classification accuracy, sensitivity, specificity, and area under curve (AUC). Furthermore, our method performs well under varying imaging missing rates.

A feature-aware multimodal framework with auto-fusion for Alzheimer's disease diagnosis

Alzheimer's disease (AD), one of the most common dementias, has about 4.6 million new cases yearly worldwide. Due to the significant amount of suspected AD patients, early screening for the disease has become particularly important. There are diversified types of AD diagnosis data, such as cognitive tests, images, and risk factors, many prior investigations have primarily concentrated on integrating only high-dimensional features and simple fusion concatenation, resulting in less-than-optimal outcomes for AD diagnosis. Therefore, We propose an enhanced multimodal AD diagnostic framework comprising a feature-aware module and an automatic model fusion strategy (AMFS). To preserve the correlation and significance features within a low-dimensional space, the feature-aware module employs a low-dimensional SHapley Additive exPlanation (SHAP) boosting feature selection as the initial step, following this analysis, diverse tiers of low-dimensional features are extracted from patients' biological data. Besides, in the high-dimensional stage, the feature-aware module integrates cross-modal attention mechanisms to capture subtle relationships among different cognitive domains, neuroimaging modalities, and risk factors. Subsequently, we integrate the aforementioned feature-aware module with graph convolutional networks (GCN) to address heterogeneous data in multimodal AD, while also possessing the capability to perceive relationships between different modalities. Lastly, our proposed AMFS autonomously learns optimal parameters for aligning two sub-models. The validation tests using two ADNI datasets show the high accuracies of 95.9% and 91.9% respectively, in AD diagnosis. The methods efficiently select features from multimodal AD data, optimizing model fusion for potential clinical assistance in diagnostics.

Multi-Modal Modality-Masked Diffusion Network for Brain MRI Synthesis With Random Modality Missing

Synthesis of unavailable imaging modalities from available ones can generate modality-specific complementary information and enable multi-modality based medical images diagnosis or treatment. Existing generative methods for medical image synthesis are usually based on cross-modal translation between acquired and missing modalities. These methods are usually dedicated to specific missing modality and perform synthesis in one shot, which cannot deal with varying number of missing modalities flexibly and construct the mapping across modalities effectively. To address the above issues, in this paper, we propose a unified Multi-modal Modality-masked Diffusion Network (M2DN), tackling multi-modal synthesis from the perspective of "progressive whole-modality inpainting", instead of "cross-modal translation". Specifically, our M2DN considers the missing modalities as random noise and takes all the modalities as a unity in each reverse diffusion step. The proposed joint synthesis scheme performs synthesis for the missing modalities and self-reconstruction for the available ones, which not only enables synthesis for arbitrary missing scenarios, but also facilitates the construction of common latent space and enhances the model representation ability. Besides, we introduce a modality-mask scheme to encode availability status of each incoming modality explicitly in a binary mask, which is adopted as condition for the diffusion model to further enhance the synthesis performance of our M2DN for arbitrary missing scenarios. We carry out experiments on two public brain MRI datasets for synthesis and downstream segmentation tasks. Experimental results demonstrate that our M2DN outperforms the state-of-the-art models significantly and shows great generalizability for arbitrary missing modalities.

MGIML: Cancer Grading With Incomplete Radiology-Pathology Data via Memory Learning and Gradient Homogenization

Taking advantage of multi-modal radiology-pathology data with complementary clinical information for cancer grading is helpful for doctors to improve diagnosis efficiency and accuracy. However, radiology and pathology data have distinct acquisition difficulties and costs, which leads to incomplete-modality data being common in applications. In this work, we propose a Memory- and Gradient-guided Incomplete Modal-modal Learning (MGIML) framework for cancer grading with incomplete radiology-pathology data. Firstly, to remedy missing-modality information, we propose a Memory-driven Hetero-modality Complement (MH-Complete) scheme, which constructs modal-specific memory banks constrained by a coarse-grained memory boosting (CMB) loss to record generic radiology and pathology feature patterns, and develops a cross-modal memory reading strategy enhanced by a fine-grained memory consistency (FMC) loss to take missing-modality information from well-stored memories. Secondly, as gradient conflicts exist between missing-modality situations, we propose a Rotation-driven Gradient Homogenization (RG-Homogenize) scheme, which estimates instance-specific rotation matrices to smoothly change the feature-level gradient directions, and computes confidence-guided homogenization weights to dynamically balance gradient magnitudes. By simultaneously mitigating gradient direction and magnitude conflicts, this scheme well avoids the negative transfer and optimization imbalance problems. Extensive experiments on CPTAC-UCEC and CPTAC-PDA datasets show that the proposed MGIML framework performs favorably against state-of-the-art multi-modal methods on missing-modality situations.

Comparing multi-class classifier performance by multi-class ROC analysis: A nonparametric approach

The area under the Receiver Operating Characteristic (ROC) curve (AUC) is a standard metric for quantifying and comparing binary classifiers. Real world applications often require classification into multiple (more than two) classes. For multi-class classifiers that produce class membership scores, a popular multi-class AUC (MAUC) variant is to average the pairwise AUC values [1]. Due to the complicated correlation patterns, the variance of MAUC is often estimated numerically using resampling techniques. This work is a generalization of DeLong's nonparameteric approach for binary AUC analysis [2] to MAUC. We first derive the closed-form expression of the covariance matrix of the pairwise AUCs within a single MAUC. Then by dropping higher order terms, we obtain an approximate covariance matrix with a compact, matrix factorization form, which then serves as the basis for variance estimation of a single MAUC. We further extend this approach to estimate the covariance of correlated MAUCs that arise from multiple competing classifiers. For the special case of binary correlated AUCs, our results coincide with that of DeLong. Our numerical studies confirm the accuracy of the variance and covariance estimates. We provide the source code of the proposed covariance estimation of correlated MAUCs on GitHub (https://tinyurl.com/euj6wvsz) for its easy adoption by machine learning and statistical analysis packages to quantify and compare multi-class classifiers.

Inferring brain causal and temporal-lag networks for recognizing abnormal patterns of dementia

Brain functional network analysis has become a popular method to explore the laws of brain organization and identify biomarkers of neurological diseases. However, it is still a challenging task to construct an ideal brain network due to the limited understanding of the human brain. Existing methods often ignore the impact of temporal-lag on the results of brain network modeling, which may lead to some unreliable conclusions. To overcome this issue, we propose a novel brain functional network estimation method, which can simultaneously infer the causal mechanisms and temporal-lag values among brain regions. Specifically, our method converts the lag learning into an instantaneous effect estimation problem, and further embeds the search objectives into a deep neural network model as parameters to be learned. To verify the effectiveness of the proposed estimation method, we perform experiments on the Alzheimer's Disease Neuroimaging Initiative (ADNI) database by comparing the proposed model with several existing methods, including correlation-based and causality-based methods. The experimental results show that our brain networks constructed by the proposed estimation method can not only achieve promising classification performance, but also exhibit some characteristics of physiological mechanisms. Our approach provides a new perspective for understanding the pathogenesis of brain diseases. The source code is released at https://github.com/NJUSTxiazw/CTLN.

A Comprehensive Review on Synergy of Multi-Modal Data and AI Technologies in Medical Diagnosis

Disease diagnosis represents a critical and arduous endeavor within the medical field. Artificial intelligence (AI) techniques, spanning from machine learning and deep learning to large model paradigms, stand poised to significantly augment physicians in rendering more evidence-based decisions, thus presenting a pioneering solution for clinical practice. Traditionally, the amalgamation of diverse medical data modalities (e.g., image, text, speech, genetic data, physiological signals) is imperative to facilitate a comprehensive disease analysis, a topic of burgeoning interest among both researchers and clinicians in recent times. Hence, there exists a pressing need to synthesize the latest strides in multi-modal data and AI technologies in the realm of medical diagnosis. In this paper, we narrow our focus to five specific disorders (Alzheimer's disease, breast cancer, depression, heart disease, epilepsy), elucidating advanced endeavors in their diagnosis and treatment through the lens of artificial intelligence. Our survey not only delineates detailed diagnostic methodologies across varying modalities but also underscores commonly utilized public datasets, the intricacies of feature engineering, prevalent classification models, and envisaged challenges for future endeavors. In essence, our research endeavors to contribute to the advancement of diagnostic methodologies, furnishing invaluable insights for clinical decision making.

Credit and Loan Approval Classification Using a Bio-Inspired Neural Network

Numerous people are applying for bank loans as a result of the banking industry's expansion, but because banks only have a certain amount of assets to lend to, they can only do so to a certain number of applicants. Therefore, the banking industry is very interested in finding ways to reduce the risk factor involved in choosing the safe applicant in order to save lots of bank resources. These days, machine learning greatly reduces the amount of work needed to choose the safe applicant. Taking this into account, a novel weights and structure determination (WASD) neural network has been built to meet the aforementioned two challenges of credit approval and loan approval, as well as to handle the unique characteristics of each. Motivated by the observation that WASD neural networks outperform conventional back-propagation neural networks in terms of sluggish training speed and being stuck in local minima, we created a bio-inspired WASD algorithm for binary classification problems (BWASD) for best adapting to the credit or loan approval model by utilizing the metaheuristic beetle antennae search (BAS) algorithm to improve the learning procedure of the WASD algorithm. Theoretical and experimental study demonstrate superior performance and problem adaptability. Furthermore, we provide a complete MATLAB package to support our experiments together with full implementation and extensive installation instructions.

[A multi-modal feature fusion classification model based on distance matching and discriminative representation learning for differentiation of high-grade glioma from solitary brain metastasis]

OBJECTIVE

To explore the performance of a new multimodal feature fusion classification model based on distance matching and discriminative representation learning for differentiating high-grade glioma (HGG) from solitary brain metastasis (SBM).

METHODS

We collected multi-parametric magnetic resonance imaging (MRI) data from 61 patients with HGG and 60 with SBM, and delineated regions of interest (ROI) on T1WI, T2WI, T2-weighted fluid attenuated inversion recovery (T2_FLAIR) and post-contrast enhancement T1WI (CE_T1WI) images. The radiomics features were extracted from each sequence using Pyradiomics and fused using a multimodal feature fusion classification model based on distance matching and discriminative representation learning to obtain a classification model. The discriminative performance of the classification model for differentiating HGG from SBM was evaluated using five-fold cross-validation with metrics of specificity, sensitivity, accuracy, and the area under the ROC curve (AUC) and quantitatively compared with other feature fusion models. Visual experiments were conducted to examine the fused features obtained by the proposed model to validate its feasibility and effectiveness.

RESULTS

The five-fold cross-validation results showed that the proposed multimodal feature fusion classification model had a specificity of 0.871, a sensitivity of 0.817, an accuracy of 0.843, and an AUC of 0.930 for distinguishing HGG from SBM. This feature fusion method exhibited excellent discriminative performance in the visual experiments.

CONCLUSION

The proposed multimodal feature fusion classification model has an excellent ability for differentiating HGG from SBM with significant advantages over other feature fusion classification models in discrimination and classification tasks between HGG and SBM.

Dysfunctions of multiscale dynamic brain functional networks in subjective cognitive decline

Subjective cognitive decline is potentially the earliest symptom of Alzheimer's disease, whose objective neurological basis remains elusive. To explore the potential biomarkers for subjective cognitive decline, we developed a novel deep learning method based on multiscale dynamical brain functional networks to identify subjective cognitive declines. We retrospectively constructed an internal data set (with 112 subjective cognitive decline and 64 healthy control subjects) to develop and internally validate the deep learning model. Conventional deep learning methods based on static and dynamic brain functional networks are compared. After the model is established, we prospectively collect an external data set (26 subjective cognitive decline and 12 healthy control subjects) for testing. Meanwhile, our method provides monitoring of the transitions between normal and abnormal (subjective cognitive decline-related) dynamical functional network states. The features of abnormal dynamical functional network states are quantified by network and variability metrics and associated with individual cognitions. Our method achieves an area under the receiver operating characteristic curve of 0.807 ± 0.046 in the internal validation data set and of 0.707 (P = 0.007) in the external testing data set, which shows improvements compared to conventional methods. The method further suggests that, at the local level, the abnormal dynamical functional network states are characterized by decreased connectivity strength and increased connectivity variability at different spatial scales. At the network level, the abnormal states are featured by scale-specifically altered modularity and all-scale decreased efficiency. Low tendencies to stay in abnormal states and high state transition variabilities are significantly associated with high general, language and executive functions. Overall, our work supports the deficits in multiscale brain dynamical functional networks detected by the deep learning method as reliable and meaningful neural alternation underpinning subjective cognitive decline.

Multi-omics fusion with soft labeling for enhanced prediction of distant metastasis in nasopharyngeal carcinoma patients after radiotherapy

Omics fusion has emerged as a crucial preprocessing approach in medical image processing, significantly assisting several studies. One of the challenges encountered in integrating omics data is the unpredictability arising from disparities in data sources and medical imaging equipment. Due to these differences, the distribution of omics futures exhibits spatial heterogeneity, diminishing their capacity to enhance subsequent tasks. To overcome this challenge and facilitate the integration of their joint application to specific medical objectives, this study aims to develop a fusion methodology for nasopharyngeal carcinoma (NPC) distant metastasis prediction to mitigate the disparities inherent in omics data. The multi-kernel late-fusion method can reduce the impact of these differences by mapping the features using the most suiTable single-kernel function and then combining them in a high-dimensional space that can effectively represent the data. The proposed approach in this study employs a distinctive framework incorporating a label-softening technique alongside a multi-kernel-based Radial basis function (RBF) neural network to address these limitations. An efficient representation of the data may be achieved by utilizing the multi-kernel to map the inherent features and then merging them in a space with many dimensions. However, the inflexibility of label fitting poses a constraint on using multi-kernel late-fusion methods in complex NPC datasets, hence affecting the efficacy of general classifiers in dealing with high-dimensional characteristics. The label softening increases the disparity between the two cohorts, providing a more flexible structure for allocating labels. The proposed model is evaluated on multi-omics datasets, and the results demonstrate its strength and effectiveness in predicting distant metastasis of NPC patients.

Identifying subgroups of eating behavior traits unrelated to obesity using functional connectivity and feature representation learning

Eating behavior is highly heterogeneous across individuals and cannot be fully explained using only the degree of obesity. We utilized unsupervised machine learning and functional connectivity measures to explore the heterogeneity of eating behaviors measured by a self-assessment instrument using 424 healthy adults (mean ± standard deviation [SD] age = 47.07 ± 18.89 years; 67% female). We generated low-dimensional representations of functional connectivity using resting-state functional magnetic resonance imaging and estimated latent features using the feature representation capabilities of an autoencoder by nonlinearly compressing the functional connectivity information. The clustering approaches applied to latent features identified three distinct subgroups. The subgroups exhibited different levels of hunger traits, while their body mass indices were comparable. The results were replicated in an independent dataset consisting of 212 participants (mean ± SD age = 38.97 ± 19.80 years; 35% female). The model interpretation technique of integrated gradients revealed that the between-group differences in the integrated gradient maps were associated with functional reorganization in heteromodal association and limbic cortices and reward-related subcortical structures such as the accumbens, amygdala, and caudate. The cognitive decoding analysis revealed that these systems are associated with reward- and emotion-related systems. Our findings provide insights into the macroscopic brain organization of eating behavior-related subgroups independent of obesity.

Verifying and refining early statuses in Alzheimer's disease progression: a possibility from deep feature comparison

Defining the early status of Alzheimer's disease is challenging. Theoretically, the statuses in the Alzheimer's disease continuum are expected to share common features. Here, we explore to verify and refine candidature early statuses of Alzheimer's disease with features learned from deep learning. We train models on brain functional networks to accurately classify between amnestic and non-amnestic mild cognitive impairments and between healthy controls and mild cognitive impairments. The trained models are applied to Alzheimer's disease and subjective cognitive decline groups to suggest feature similarities among the statuses and identify informative subpopulations. The amnestic mild cognitive impairment vs non-amnestic mild cognitive impairments classifier believes that 71.8% of Alzheimer's disease are amnestic mild cognitive impairment. And 73.5% of subjective cognitive declines are labeled as mild cognitive impairments, 88.8% of which are further suggested as "amnestic mild cognitive impairment." Further multimodal analyses suggest that the amnestic mild cognitive impairment-like Alzheimer's disease, mild cognitive impairment-like subjective cognitive decline, and amnestic mild cognitive impairment-like subjective cognitive decline exhibit more Alzheimer's disease -related pathological changes (elaborated β-amyloid depositions, reduced glucose metabolism, and gray matter atrophy) than non-amnestic mild cognitive impairments -like Alzheimer's disease, healthy control-like subjective cognitive decline, and non-amnestic mild cognitive impairments -like subjective cognitive decline. The test-retest reliability of the subpopulation identification is fair to good in general. The study indicates overall similarity among subjective cognitive decline, amnestic mild cognitive impairment, and Alzheimer's disease and implies their progression relationships. The results support "deep feature comparison" as a potential beneficial framework to verify and refine early Alzheimer's disease status.

Multimodal transformer network for incomplete image generation and diagnosis of Alzheimer's disease

Multimodal images such as magnetic resonance imaging (MRI) and positron emission tomography (PET) could provide complementary information about the brain and have been widely investigated for the diagnosis of neurodegenerative disorders such as Alzheimer's disease (AD). However, multimodal brain images are often incomplete in clinical practice. It is still challenging to make use of multimodality for disease diagnosis with missing data. In this paper, we propose a deep learning framework with the multi-level guided generative adversarial network (MLG-GAN) and multimodal transformer (Mul-T) for incomplete image generation and disease classification, respectively. First, MLG-GAN is proposed to generate the missing data, guided by multi-level information from voxels, features, and tasks. In addition to voxel-level supervision and task-level constraint, a feature-level auto-regression branch is proposed to embed the features of target images for an accurate generation. With the complete multimodal images, we propose a Mul-T network for disease diagnosis, which can not only combine the global and local features but also model the latent interactions and correlations from one modality to another with the cross-modal attention mechanism. Comprehensive experiments on three independent datasets (i.e., ADNI-1, ADNI-2, and OASIS-3) show that the proposed method achieves superior performance in the tasks of image generation and disease diagnosis compared to state-of-the-art methods.

Disentangle First, Then Distill: A Unified Framework for Missing Modality Imputation and Alzheimer's Disease Diagnosis

Multi-modality medical data provide complementary information, and hence have been widely explored for computer-aided AD diagnosis. However, the research is hindered by the unavoidable missing-data problem, i.e., one data modality was not acquired on some subjects due to various reasons. Although the missing data can be imputed using generative models, the imputation process may introduce unrealistic information to the classification process, leading to poor performance. In this paper, we propose the Disentangle First, Then Distill (DFTD) framework for AD diagnosis using incomplete multi-modality medical images. First, we design a region-aware disentanglement module to disentangle each image into inter-modality relevant representation and intra-modality specific representation with emphasis on disease-related regions. To progressively integrate multi-modality knowledge, we then construct an imputation-induced distillation module, in which a lateral inter-modality transition unit is created to impute representation of the missing modality. The proposed DFTD framework has been evaluated against six existing methods on an ADNI dataset with 1248 subjects. The results show that our method has superior performance in both AD-CN classification and MCI-to-AD prediction tasks, substantially over-performing all competing methods.

Federated Domain Adaptation via Transformer for Multi-Site Alzheimer's Disease Diagnosis

In multi-site studies of Alzheimer's disease (AD), the difference of data in multi-site datasets leads to the degraded performance of models in the target sites. The traditional domain adaptation method requires sharing data from both source and target domains, which will lead to data privacy issue. To solve it, federated learning is adopted as it can allow models to be trained with multi-site data in a privacy-protected manner. In this paper, we propose a multi-site federated domain adaptation framework via Transformer (FedDAvT), which not only protects data privacy, but also eliminates data heterogeneity. The Transformer network is used as the backbone network to extract the correlation between the multi-template region of interest features, which can capture the brain abundant information. The self-attention maps in the source and target domains are aligned by applying mean squared error for subdomain adaptation. Finally, we evaluate our method on the multi-site databases based on three AD datasets. The experimental results show that the proposed FedDAvT is quite effective, achieving accuracy rates of 88.75%, 69.51%, and 69.88% on the AD vs. NC, MCI vs. NC, and AD vs. MCI two-way classification tasks, respectively.

Multimodal learning of clinically accessible tests to aid diagnosis of neurodegenerative disorders: a scoping review

With ageing populations around the world, there is a rapid rise in the number of people with Alzheimer's disease (AD) and Parkinson's disease (PD), the two most common types of neurodegenerative disorders. There is an urgent need to find new ways of aiding early diagnosis of these conditions. Multimodal learning of clinically accessible data is a relatively new approach that holds great potential to support early precise diagnosis. This scoping review follows the PRSIMA guidelines and we analysed 46 papers, comprising 11,750 participants, 3569 with AD, 978 with PD, and 2482 healthy controls; the recency of this topic was highlighted by nearly all papers being published in the last 5 years. It highlights the effectiveness of combining different types of data, such as brain scans, cognitive scores, speech and language, gait, hand and eye movements, and genetic assessments for the early detection of AD and PD. The review also outlines the AI methods and the model used in each study, which includes feature extraction, feature selection, feature fusion, and using multi-source discriminative features for classification. The review identifies knowledge gaps around the need to validate findings and address limitations such as small sample sizes. Applying multimodal learning of clinically accessible tests holds strong potential to aid the development of low-cost, reliable, and non-invasive methods for early detection of AD and PD.

Integration of incomplete multi-omics data using Knowledge Distillation and Supervised Variational Autoencoders for disease progression prediction

OBJECTIVE

The rapid advancement of high-throughput technologies in the biomedical field has resulted in the accumulation of diverse omics data types, such as mRNA expression, DNA methylation, and microRNA expression, for studying various diseases. Integrating these multi-omics datasets enables a comprehensive understanding of the molecular basis of cancer and facilitates accurate prediction of disease progression.

METHODS

However, conventional approaches face challenges due to the dimensionality curse problem. This paper introduces a novel framework called Knowledge Distillation and Supervised Variational AutoEncoders utilizing View Correlation Discovery Network (KD-SVAE-VCDN) to address the integration of high-dimensional multi-omics data with limited common samples. Through our experimental evaluation, we demonstrate that the proposed KD-SVAE-VCDN architecture accurately predicts the progression of breast and kidney carcinoma by effectively classifying patients as long- or short-term survivors. Furthermore, our approach outperforms other state-of-the-art multi-omics integration models.

RESULTS

Our findings highlight the efficacy of the KD-SVAE-VCDN architecture in predicting the disease progression of breast and kidney carcinoma. By enabling the classification of patients based on survival outcomes, our model contributes to personalized and targeted treatments. The favorable performance of our approach in comparison to several existing models suggests its potential to contribute to the advancement of cancer understanding and management.

CONCLUSION

The development of a robust predictive model capable of accurately forecasting disease progression at the time of diagnosis holds immense promise for advancing personalized medicine. By leveraging multi-omics data integration, our proposed KD-SVAE-VCDN framework offers an effective solution to this challenge, paving the way for more precise and tailored treatment strategies for patients with different types of cancer.

Multi-Scale Transformer Network With Edge-Aware Pre-Training for Cross-Modality MR Image Synthesis

Cross-modality magnetic resonance (MR) image synthesis can be used to generate missing modalities from given ones. Existing (supervised learning) methods often require a large number of paired multi-modal data to train an effective synthesis model. However, it is often challenging to obtain sufficient paired data for supervised training. In reality, we often have a small number of paired data while a large number of unpaired data. To take advantage of both paired and unpaired data, in this paper, we propose a Multi-scale Transformer Network (MT-Net) with edge-aware pre-training for cross-modality MR image synthesis. Specifically, an Edge-preserving Masked AutoEncoder (Edge-MAE) is first pre-trained in a self-supervised manner to simultaneously perform 1) image imputation for randomly masked patches in each image and 2) whole edge map estimation, which effectively learns both contextual and structural information. Besides, a novel patch-wise loss is proposed to enhance the performance of Edge-MAE by treating different masked patches differently according to the difficulties of their respective imputations. Based on this proposed pre-training, in the subsequent fine-tuning stage, a Dual-scale Selective Fusion (DSF) module is designed (in our MT-Net) to synthesize missing-modality images by integrating multi-scale features extracted from the encoder of the pre-trained Edge-MAE. Furthermore, this pre-trained encoder is also employed to extract high-level features from the synthesized image and corresponding ground-truth image, which are required to be similar (consistent) in the training. Experimental results show that our MT-Net achieves comparable performance to the competing methods even using 70% of all available paired data. Our code will be released at https://github.com/lyhkevin/MT-Net.

Identifying Alzheimer's disease and mild cognitive impairment with atlas-based multi-modal metrics

Introduction

Multi-modal neuroimaging metrics in combination with advanced machine learning techniques have attracted more and more attention for an effective multi-class identification of Alzheimer's disease (AD), mild cognitive impairment (MCI) and health controls (HC) recently.

Methods

In this paper, a total of 180 subjects consisting of 44 AD, 66 MCI and 58 HC subjects were enrolled, and the multi-modalities of the resting-state functional magnetic resonance imaging (rs-fMRI) and the structural MRI (sMRI) for all participants were obtained. Then, four kinds of metrics including the Hurst exponent (HE) metric and bilateral hippocampus seed independently based connectivity metrics generated from fMRI data, and the gray matter volume (GMV) metric obtained from sMRI data, were calculated and extracted in each region of interest (ROI) based on a newly proposed automated anatomical Labeling (AAL3) atlas after data pre-processing. Next, these metrics were selected with a minimal redundancy maximal relevance (MRMR) method and a sequential feature collection (SFC) algorithm, and only a subset of optimal features were retained after this step. Finally, the support vector machine (SVM) based classification methods and artificial neural network (ANN) algorithm were utilized to identify the multi-class of AD, MCI and HC subjects in single modal and multi-modal metrics respectively, and a nested ten-fold cross-validation was utilized to estimate the final classification performance.

Results

The results of the SVM and ANN based methods indicated the best accuracies of 80.36 and 74.40%, respectively, by utilizing all the multi-modal metrics, and the optimal accuracies for AD, MCI and HC were 79.55, 78.79 and 82.76%, respectively, in the SVM based method. In contrast, when using single modal metric, the SVM based method obtained a best accuracy of 72.62% with the HE metric, and the accuracies for AD, MCI and HC subjects were just 56.82, 80.30 and 75.86%, respectively. Moreover, the overlapping abnormal brain regions detected by multi-modal metrics were mainly located at posterior cingulate gyrus, superior frontal gyrus and cuneus.

Conclusion

Taken together, the SVM based method with multi-modal metrics could provide effective diagnostic information for identifying AD, MCI and HC subjects.

Early Prediction of Progression to Alzheimer's Disease using Multi-Modality Neuroimages by a Novel Ordinal Learning Model ADPacer

Machine learning has shown great promise for integrating multi-modality neuroimaging datasets to predict the risk of progression/conversion to Alzheimer's Disease (AD) for individuals with Mild Cognitive Impairment (MCI). Most existing work aims to classify MCI patients into converters versus non-converters using a pre-defined timeframe. The limitation is a lack of granularity in differentiating MCI patients who convert at different paces. Progression pace prediction has important clinical values, which allow from more personalized interventional strategies, better preparation of patients and their caregivers, and facilitation of patient selection in clinical trials. We proposed a novel ADPacer model which formulated the pace prediction into an ordinal learning problem with a unique capability of leveraging training samples with label ambiguity to augment the training set. This capability differentiates ADPacer from existing ordinal learning algorithms. We applied ADPacer to MCI patient cohorts from the Alzheimer's Disease Neuroimaging Initiative (ADNI) and the Australian Imaging, Biomarker & Lifestyle Flagship Study of Ageing (AIBL), and demonstrated the superior performance of ADPacer compared to existing ordinal learning algorithms. We also integrated the SHapley Additive exPlanations (SHAP) method with ADPacer to assess the contributions from different modalities to the model prediction. The findings are consistent with the AD literature.

Multi-modality cardiac image computing: A survey

Multi-modality cardiac imaging plays a key role in the management of patients with cardiovascular diseases. It allows a combination of complementary anatomical, morphological and functional information, increases diagnosis accuracy, and improves the efficacy of cardiovascular interventions and clinical outcomes. Fully-automated processing and quantitative analysis of multi-modality cardiac images could have a direct impact on clinical research and evidence-based patient management. However, these require overcoming significant challenges including inter-modality misalignment and finding optimal methods to integrate information from different modalities. This paper aims to provide a comprehensive review of multi-modality imaging in cardiology, the computing methods, the validation strategies, the related clinical workflows and future perspectives. For the computing methodologies, we have a favored focus on the three tasks, i.e., registration, fusion and segmentation, which generally involve multi-modality imaging data, either combining information from different modalities or transferring information across modalities. The review highlights that multi-modality cardiac imaging data has the potential of wide applicability in the clinic, such as trans-aortic valve implantation guidance, myocardial viability assessment, and catheter ablation therapy and its patient selection. Nevertheless, many challenges remain unsolved, such as missing modality, modality selection, combination of imaging and non-imaging data, and uniform analysis and representation of different modalities. There is also work to do in defining how the well-developed techniques fit in clinical workflows and how much additional and relevant information they introduce. These problems are likely to continue to be an active field of research and the questions to be answered in the future.

Multimodal learning on graphs for disease relation extraction

OBJECTIVE

Disease knowledge graphs have emerged as a powerful tool for AI, enabling the connection, organization, and access to diverse information about diseases. However, the relations between disease concepts are often distributed across multiple data formats, including plain language and incomplete disease knowledge graphs. As a result, extracting disease relations from multimodal data sources is crucial for constructing accurate and comprehensive disease knowledge graphs.

METHODS

We introduce REMAP, a multimodal approach for disease relation extraction. The REMAP machine learning approach jointly embeds a partial, incomplete knowledge graph and a medical language dataset into a compact latent vector space, aligning the multimodal embeddings for optimal disease relation extraction. Additionally, REMAP utilizes a decoupled model structure to enable inference in single-modal data, which can be applied under missing modality scenarios.

RESULTS

We apply the REMAP approach to a disease knowledge graph with 96,913 relations and a text dataset of 1.24 million sentences. On a dataset annotated by human experts, REMAP improves language-based disease relation extraction by 10.0% (accuracy) and 17.2% (F1-score) by fusing disease knowledge graphs with language information. Furthermore, REMAP leverages text information to recommend new relationships in the knowledge graph, outperforming graph-based methods by 8.4% (accuracy) and 10.4% (F1-score).

CONCLUSION

In summary, REMAP is a flexible multimodal approach for extracting disease relations by fusing structured knowledge and language information. This approach provides a powerful model to easily find, access, and evaluate relations between disease concepts.

Multimodal data fusion for cancer biomarker discovery with deep learning

Technological advances now make it possible to study a patient from multiple angles with high-dimensional, high-throughput multi-scale biomedical data. In oncology, massive amounts of data are being generated ranging from molecular, histopathology, radiology to clinical records. The introduction of deep learning has significantly advanced the analysis of biomedical data. However, most approaches focus on single data modalities leading to slow progress in methods to integrate complementary data types. Development of effective multimodal fusion approaches is becoming increasingly important as a single modality might not be consistent and sufficient to capture the heterogeneity of complex diseases to tailor medical care and improve personalised medicine. Many initiatives now focus on integrating these disparate modalities to unravel the biological processes involved in multifactorial diseases such as cancer. However, many obstacles remain, including lack of usable data as well as methods for clinical validation and interpretation. Here, we cover these current challenges and reflect on opportunities through deep learning to tackle data sparsity and scarcity, multimodal interpretability, and standardisation of datasets.

Supervised Phenotype Discovery From Multimodal Brain Imaging

Data-driven discovery of image-derived phenotypes (IDPs) from large-scale multimodal brain imaging data has enormous potential for neuroscientific and clinical research by linking IDPs to subjects' demographic, behavioural, clinical and cognitive measures (i.e., non-imaging derived phenotypes or nIDPs). However, current approaches are primarily based on unsupervised approaches, without the use of information in nIDPs. In this paper, we proposed a semi-supervised, multimodal, and multi-task fusion approach, termed SuperBigFLICA, for IDP discovery, which simultaneously integrates information from multiple imaging modalities as well as multiple nIDPs. SuperBigFLICA is computationally efficient and largely avoids the need for parameter tuning. Using the UK Biobank brain imaging dataset with around 40,000 subjects and 47 modalities, along with more than 17,000 nIDPs, we showed that SuperBigFLICA enhances the prediction power of nIDPs, benchmarked against IDPs derived by conventional expert-knowledge and unsupervised-learning approaches (with average nIDP prediction accuracy improvements of up to 46%). It also enables the learning of generic imaging features that can predict new nIDPs. Further empirical analysis of the SuperBigFLICA algorithm demonstrates its robustness in different prediction tasks and the ability to derive biologically meaningful IDPs in predicting health outcomes and cognitive nIDPs, such as fluid intelligence and hypertension.

Orthogonal latent space learning with feature weighting and graph learning for multimodal Alzheimer's disease diagnosis

Recent studies have shown that multimodal neuroimaging data provide complementary information of the brain and latent space-based methods have achieved promising results in fusing multimodal data for Alzheimer's disease (AD) diagnosis. However, most existing methods treat all features equally and adopt nonorthogonal projections to learn the latent space, which cannot retain enough discriminative information in the latent space. Besides, they usually preserve the relationships among subjects in the latent space based on the similarity graph constructed on original features for performance boosting. However, the noises and redundant features significantly corrupt the graph. To address these limitations, we propose an Orthogonal Latent space learning with Feature weighting and Graph learning (OLFG) model for multimodal AD diagnosis. Specifically, we map multiple modalities into a common latent space by orthogonal constrained projection to capture the discriminative information for AD diagnosis. Then, a feature weighting matrix is utilized to sort the importance of features in AD diagnosis adaptively. Besides, we devise a regularization term with learned graph to preserve the local structure of the data in the latent space and integrate the graph construction into the learning processing for accurately encoding the relationships among samples. Instead of constructing a similarity graph for each modality, we learn a joint graph for multiple modalities to capture the correlations among modalities. Finally, the representations in the latent space are projected into the target space to perform AD diagnosis. An alternating optimization algorithm with proved convergence is developed to solve the optimization objective. Extensive experimental results show the effectiveness of the proposed method.

A Review of Fusion Methods for Omics and Imaging Data

The development of omics data and biomedical images has greatly advanced the progress of precision medicine in diagnosis, treatment, and prognosis. The fusion of omics and imaging data, i.e., omics-imaging fusion, offers a new strategy for understanding complex diseases. However, due to a variety of issues such as the limited number of samples, high dimensionality of features, and heterogeneity of different data types, efficiently learning complementary or associated discriminative fusion information from omics and imaging data remains a challenge. Recently, numerous machine learning methods have been proposed to alleviate these problems. In this review, from the perspective of fusion levels and fusion methods, we first provide an overview of preprocessing and feature extraction methods for omics and imaging data, and comprehensively analyze and summarize the basic forms and variations of commonly used and newly emerging fusion methods, along with their advantages, disadvantages and the applicable scope. We then describe public datasets and compare experimental results of various fusion methods on the ADNI and TCGA datasets. Finally, we discuss future prospects and highlight remaining challenges in the field.

Latent feature representation learning for Alzheimer's disease classification

Early detection and treatment of Alzheimer's Disease (AD) are significant. Recently, multi-modality imaging data have promoted the development of the automatic diagnosis of AD. This paper proposes a method based on latent feature fusion to make full use of multi-modality image data information. Specifically, we learn a specific projection matrix for each modality by introducing a binary label matrix and local geometry constraints and then project the original features of each modality into a low-dimensional target space. In this space, we fuse latent feature representations of different modalities for AD classification. The experimental results on Alzheimer's Disease Neuroimaging Initiative database demonstrate the proposed methods effectiveness in classifying AD.

Self-supervised learning of neighborhood embedding for longitudinal MRI

In recent years, several deep learning models recommend first to represent Magnetic Resonance Imaging (MRI) as latent features before performing a downstream task of interest (such as classification or regression). The performance of the downstream task generally improves when these latent representations are explicitly associated with factors of interest. For example, we derived such a representation for capturing brain aging by applying self-supervised learning to longitudinal MRIs and then used the resulting encoding to automatically identify diseases accelerating the aging of the brain. We now propose a refinement of this representation by replacing the linear modeling of brain aging with one that is consistent in local neighborhoods in the latent space. Called Longitudinal Neighborhood Embedding (LNE), we derive an encoding so that neighborhoods are age-consistent (i.e., brain MRIs of different subjects with similar brain ages are in close proximity of each other) and progression-consistent, i.e., the latent space is defined by a smooth trajectory field where each trajectory captures changes in brain ages between a pair of MRIs extracted from a longitudinal sequence. To make the problem computationally tractable, we further propose a strategy for mini-batch sampling so that the resulting local neighborhoods accurately approximate the ones that would be defined based on the whole cohort. We evaluate LNE on three different downstream tasks: (1) to predict chronological age from T1-w MRI of 274 healthy subjects participating in a study at SRI International; (2) to distinguish Normal Control (NC) from Alzheimer's Disease (AD) and stable Mild Cognitive Impairment (sMCI) from progressive Mild Cognitive Impairment (pMCI) based on T1-w MRI of 632 participants of the Alzheimer's Disease Neuroimaging Initiative (ADNI); and (3) to distinguish no-to-low from moderate-to-heavy alcohol drinkers based on fractional anisotropy derived from diffusion tensor MRIs of 764 adolescents recruited by the National Consortium on Alcohol and NeuroDevelopment in Adolescence (NCANDA). Across the three data sets, the visualization of the smooth trajectory vector fields and superior accuracy on downstream tasks demonstrate the strength of the proposed method over existing self-supervised methods in extracting information related to brain aging, which could help study the impact of substance use and neurodegenerative disorders. The code is available at https://github.com/ouyangjiahong/longitudinal-neighbourhood-embedding.

Disentangling Normal Aging From Severity of Disease via Weak Supervision on Longitudinal MRI

The continuous progression of neurological diseases are often categorized into conditions according to their severity. To relate the severity to changes in brain morphometry, there is a growing interest in replacing these categories with a continuous severity scale that longitudinal MRIs are mapped onto via deep learning algorithms. However, existing methods based on supervised learning require large numbers of samples and those that do not, such as self-supervised models, fail to clearly separate the disease effect from normal aging. Here, we propose to explicitly disentangle those two factors via weak-supervision. In other words, training is based on longitudinal MRIs being labelled either normal or diseased so that the training data can be augmented with samples from disease categories that are not of primary interest to the analysis. We do so by encouraging trajectories of controls to be fully encoded by the direction associated with brain aging. Furthermore, an orthogonal direction linked to disease severity captures the residual component from normal aging in the diseased cohort. Hence, the proposed method quantifies disease severity and its progression speed in individuals without knowing their condition. We apply the proposed method on data from the Alzheimer's Disease Neuroimaging Initiative (ADNI, N =632 ). We then show that the model properly disentangled normal aging from the severity of cognitive impairment by plotting the resulting disentangled factors of each subject and generating simulated MRIs for a given chronological age and condition. Moreover, our representation obtains higher balanced accuracy when used for two downstream classification tasks compared to other pre-training approaches. The code for our weak-supervised approach is available at https://github.com/ouyangjiahong/longitudinal-direction-disentangle.

A Deep Generative-Discriminative Learning for Multimodal Representation in Imaging Genetics

Imaging genetics, one of the foremost emerging topics in the medical imaging field, analyzes the inherent relations between neuroimaging and genetic data. As deep learning has gained widespread acceptance in many applications, pioneering studies employed deep learning frameworks for imaging genetics. However, existing approaches suffer from some limitations. First, they often adopt a simple strategy for joint learning of phenotypic and genotypic features. Second, their findings have not been extended to biomedical applications, e.g., degenerative brain disease diagnosis and cognitive score prediction. Finally, existing studies perform insufficient and inappropriate analyses from the perspective of data science and neuroscience. In this work, we propose a novel deep learning framework to simultaneously tackle the aforementioned issues. Our proposed framework learns to effectively represent the neuroimaging and the genetic data jointly, and achieves state-of-the-art performance when used for Alzheimer's disease and mild cognitive impairment identification. Furthermore, unlike the existing methods, the framework enables learning the relation between imaging phenotypes and genotypes in a nonlinear way without any prior neuroscientific knowledge. To demonstrate the validity of our proposed framework, we conducted experiments on a publicly available dataset and analyzed the results from diverse perspectives. Based on our experimental results, we believe that the proposed framework has immense potential to provide new insights and perspectives in deep learning-based imaging genetics studies.

Doubly Supervised Transfer Classifier for Computer-Aided Diagnosis With Imbalanced Modalities

Transfer learning (TL) can effectively improve diagnosis accuracy of single-modal-imaging-based computer-aided diagnosis (CAD) by transferring knowledge from other related imaging modalities, which offers a way to alleviate the small-sample-size problem. However, medical imaging data generally have the following characteristics for the TL-based CAD: 1) The source domain generally has limited data, which increases the difficulty to explore transferable information for the target domain; 2) Samples in both domains often have been labeled for training the CAD model, but the existing TL methods cannot make full use of label information to improve knowledge transfer. In this work, we propose a novel doubly supervised transfer classifier (DSTC) algorithm. In particular, DSTC integrates the support vector machine plus (SVM+) classifier and the low-rank representation (LRR) into a unified framework. The former makes full use of the shared labels to guide the knowledge transfer between the paired data, while the latter adopts the block-diagonal low-rank (BLR) to perform supervised TL between the unpaired data. Furthermore, we introduce the Schatten-p norm for BLR to obtain a tighter approximation to the rank function. The proposed DSTC algorithm is evaluated on the Alzheimer's disease neuroimaging initiative (ADNI) dataset and the bimodal breast ultrasound image (BBUI) dataset. The experimental results verify the effectiveness of the proposed DSTC algorithm.

Improving EEG Decoding via Clustering-Based Multitask Feature Learning

Accurate electroencephalogram (EEG) pattern decoding for specific mental tasks is one of the key steps for the development of brain-computer interface (BCI), which is quite challenging due to the considerably low signal-to-noise ratio of EEG collected at the brain scalp. Machine learning provides a promising technique to optimize EEG patterns toward better decoding accuracy. However, existing algorithms do not effectively explore the underlying data structure capturing the true EEG sample distribution and, hence, can only yield a suboptimal decoding accuracy. To uncover the intrinsic distribution structure of EEG data, we propose a clustering-based multitask feature learning algorithm for improved EEG pattern decoding. Specifically, we perform affinity propagation-based clustering to explore the subclasses (i.e., clusters) in each of the original classes and then assign each subclass a unique label based on a one-versus-all encoding strategy. With the encoded label matrix, we devise a novel multitask learning algorithm by exploiting the subclass relationship to jointly optimize the EEG pattern features from the uncovered subclasses. We then train a linear support vector machine with the optimized features for EEG pattern decoding. Extensive experimental studies are conducted on three EEG data sets to validate the effectiveness of our algorithm in comparison with other state-of-the-art approaches. The improved experimental results demonstrate the outstanding superiority of our algorithm, suggesting its prominent performance for EEG pattern decoding in BCI applications.