Representation Disentanglement for Multi-modal Brain MRI Analysis

Disentangling Neurodegeneration From Aging in Multiple Sclerosis Using Deep Learning: The Brain-Predicted Disease Duration Gap

BACKGROUND AND OBJECTIVES

Disentangling brain aging from disease-related neurodegeneration in patients with multiple sclerosis (PwMS) is increasingly topical. The brain-age paradigm offers a window into this problem but may miss disease-specific effects. In this study, we investigated whether a disease-specific model might complement the brain-age gap (BAG) by capturing aspects unique to MS.

METHODS

In this retrospective study, we collected 3D T1-weighted brain MRI scans of PwMS to build (1) a cross-sectional multicentric cohort for age and disease duration (DD) modeling and (2) a longitudinal single-center cohort of patients with early MS as a clinical use case. We trained and evaluated a 3D DenseNet architecture to predict DD from minimally preprocessed images while age predictions were obtained with the DeepBrainNet model. The brain-predicted DD gap (the difference between predicted and actual duration) was proposed as a DD-adjusted global measure of MS-specific brain damage. Model predictions were scrutinized to assess the influence of lesions and brain volumes while the DD gap was biologically and clinically validated within a linear model framework assessing its relationship with BAG and physical disability measured with the Expanded Disability Status Scale (EDSS).

RESULTS

We gathered MRI scans of 4,392 PwMS (69.7% female, age: 42.8 ± 10.6 years, DD: 11.4 ± 9.3 years) from 15 centers while the early MS cohort included 749 sessions from 252 patients (64.7% female, age: 34.5 ± 8.3 years, DD: 0.7 ± 1.2 years). Our model predicted DD better than chance (mean absolute error = 5.63 years, R2 = 0.34) and was nearly orthogonal to the brain-age model (correlation between DD and BAGs: r = 0.06 [0.00-0.13], p = 0.07). Predictions were influenced by distributed variations in brain volume and, unlike brain-predicted age, were sensitive to MS lesions (difference between unfilled and filled scans: 0.55 years [0.51-0.59], p < 0.001). DD gap significantly explained EDSS changes (B = 0.060 [0.038-0.082], p < 0.001), adding to BAG (ΔR2 = 0.012, p < 0.001). Longitudinally, increasing DD gap was associated with greater annualized EDSS change (r = 0.50 [0.39-0.60], p < 0.001), with an incremental contribution in explaining disability worsening compared with changes in BAG alone (ΔR2 = 0.064, p < 0.001).

DISCUSSION

The brain-predicted DD gap is sensitive to MS-related lesions and brain atrophy, adds to the brain-age paradigm in explaining physical disability both cross-sectionally and longitudinally, and may be used as an MS-specific biomarker of disease severity and progression.

MGIML: Cancer Grading With Incomplete Radiology-Pathology Data via Memory Learning and Gradient Homogenization

Taking advantage of multi-modal radiology-pathology data with complementary clinical information for cancer grading is helpful for doctors to improve diagnosis efficiency and accuracy. However, radiology and pathology data have distinct acquisition difficulties and costs, which leads to incomplete-modality data being common in applications. In this work, we propose a Memory- and Gradient-guided Incomplete Modal-modal Learning (MGIML) framework for cancer grading with incomplete radiology-pathology data. Firstly, to remedy missing-modality information, we propose a Memory-driven Hetero-modality Complement (MH-Complete) scheme, which constructs modal-specific memory banks constrained by a coarse-grained memory boosting (CMB) loss to record generic radiology and pathology feature patterns, and develops a cross-modal memory reading strategy enhanced by a fine-grained memory consistency (FMC) loss to take missing-modality information from well-stored memories. Secondly, as gradient conflicts exist between missing-modality situations, we propose a Rotation-driven Gradient Homogenization (RG-Homogenize) scheme, which estimates instance-specific rotation matrices to smoothly change the feature-level gradient directions, and computes confidence-guided homogenization weights to dynamically balance gradient magnitudes. By simultaneously mitigating gradient direction and magnitude conflicts, this scheme well avoids the negative transfer and optimization imbalance problems. Extensive experiments on CPTAC-UCEC and CPTAC-PDA datasets show that the proposed MGIML framework performs favorably against state-of-the-art multi-modal methods on missing-modality situations.

MHD-Net: Memory-Aware Hetero-Modal Distillation Network for Thymic Epithelial Tumor Typing With Missing Pathology Modality

Fusing multi-modal radiology and pathology data with complementary information can improve the accuracy of tumor typing. However, collecting pathology data is difficult since it is high-cost and sometimes only obtainable after the surgery, which limits the application of multi-modal methods in diagnosis. To address this problem, we propose comprehensively learning multi-modal radiology-pathology data in training, and only using uni-modal radiology data in testing. Concretely, a Memory-aware Hetero-modal Distillation Network (MHD-Net) is proposed, which can distill well-learned multi-modal knowledge with the assistance of memory from the teacher to the student. In the teacher, to tackle the challenge in hetero-modal feature fusion, we propose a novel spatial-differentiated hetero-modal fusion module (SHFM) that models spatial-specific tumor information correlations across modalities. As only radiology data is accessible to the student, we store pathology features in the proposed contrast-boosted typing memory module (CTMM) that achieves type-wise memory updating and stage-wise contrastive memory boosting to ensure the effectiveness and generalization of memory items. In the student, to improve the cross-modal distillation, we propose a multi-stage memory-aware distillation (MMD) scheme that reads memory-aware pathology features from CTMM to remedy missing modal-specific information. Furthermore, we construct a Radiology-Pathology Thymic Epithelial Tumor (RPTET) dataset containing paired CT and WSI images with annotations. Experiments on the RPTET and CPTAC-LUAD datasets demonstrate that MHD-Net significantly improves tumor typing and outperforms existing multi-modal methods on missing modality situations.

Generating Realistic Brain MRIs via a Conditional Diffusion Probabilistic Model

As acquiring MRIs is expensive, neuroscience studies struggle to attain a sufficient number of them for properly training deep learning models. This challenge could be reduced by MRI synthesis, for which Generative Adversarial Networks (GANs) are popular. GANs, however, are commonly unstable and struggle with creating diverse and high-quality data. A more stable alternative is Diffusion Probabilistic Models (DPMs) with a fine-grained training strategy. To overcome their need for extensive computational resources, we propose a conditional DPM (cDPM) with a memory-efficient process that generates realistic-looking brain MRIs. To this end, we train a 2D cDPM to generate an MRI subvolume conditioned on another subset of slices from the same MRI. By generating slices using arbitrary combinations between condition and target slices, the model only requires limited computational resources to learn interdependencies between slices even if they are spatially far apart. After having learned these dependencies via an attention network, a new anatomy-consistent 3D brain MRI is generated by repeatedly applying the cDPM. Our experiments demonstrate that our method can generate high-quality 3D MRIs that share a similar distribution to real MRIs while still diversifying the training set. The code is available at https://github.com/xiaoiker/mask3DMRI_diffusion and also will be released as part of MONAI, at https://github.com/Project-MONAI/GenerativeModels.

Latent shape image learning via disentangled representation for cross-sequence image registration and segmentation

PURPOSE

Cross-sequence magnetic resonance image (MRI) registration and segmentation are two essential steps in a variety of medical image analysis tasks. And have attracted considerable research interest. However, they remain challenging due to domain shifts between different sequences. This study is aiming at proposing a novel method via disentangled representations, latent shape image learning (LSIL), for cross-sequence image registration and segmentation.

METHODS

Images from different sequences were firstly decomposed into a shared domain-invariant shape space and a domain-specific appearance space via an unsupervised image-to-image translation approach. A latent shape image learning model is then built on the disentangled shape representations to generate latent shape images. A series of experiments including cross-sequence image registration and segmentation were performed to qualitatively and quantitatively verify the validity of our method. Dice similarity coefficient (DSC) and 95th percentile Hausdorff distance (HD95) were adopted as our evaluation metrics.

RESULTS

The performance of our method was evaluated based on 2 datasets total of 50 MRIs. The experimental results showed the superiority of the proposed framework over the state-of-the-art cross-sequence registration and segmentation approaches. The proposed method shows the mean DSCs of 0.711 and 0.867, respectively, in cross-sequence registration and segmentation.

CONCLUSION

We proposed a novel method based on representation disentangling to solve the cross-sequence registration and segmentation problem. Experimental results prove the feasibility and generalization of the generated latent shape images. The proposed method demonstrates significant potential for use in clinical environments of missing sequences. The source code is available at https://github.com/wujiong-hub/LSIL .

D2-Net: Dual Disentanglement Network for Brain Tumor Segmentation With Missing Modalities

Multi-modal Magnetic Resonance Imaging (MRI) can provide complementary information for automatic brain tumor segmentation, which is crucial for diagnosis and prognosis. While missing modality data is common in clinical practice and it can result in the collapse of most previous methods relying on complete modality data. Current state-of-the-art approaches cope with the situations of missing modalities by fusing multi-modal images and features to learn shared representations of tumor regions, which often ignore explicitly capturing the correlations among modalities and tumor regions. Inspired by the fact that modality information plays distinct roles to segment different tumor regions, we aim to explicitly exploit the correlations among various modality-specific information and tumor-specific knowledge for segmentation. To this end, we propose a Dual Disentanglement Network (D²-Net) for brain tumor segmentation with missing modalities, which consists of a modality disentanglement stage (MD-Stage) and a tumor-region disentanglement stage (TD-Stage). In the MD-Stage, a spatial-frequency joint modality contrastive learning scheme is designed to directly decouple the modality-specific information from MRI data. To decompose tumor-specific representations and extract discriminative holistic features, we propose an affinity-guided dense tumor-region knowledge distillation mechanism in the TD-Stage through aligning the features of a disentangled binary teacher network with a holistic student network. By explicitly discovering relations among modalities and tumor regions, our model can learn sufficient information for segmentation even if some modalities are missing. Extensive experiments on the public BraTS-2018 database demonstrate the superiority of our framework over state-of-the-art methods in missing modalities situations. Codes are available at https://github.com/CityU-AIM-Group/D2Net.

Denoising Single Images by Feature Ensemble Revisited

Image denoising is still a challenging issue in many computer vision subdomains. Recent studies have shown that significant improvements are possible in a supervised setting. However, a few challenges, such as spatial fidelity and cartoon-like smoothing, remain unresolved or decisively overlooked. Our study proposes a simple yet efficient architecture for the denoising problem that addresses the aforementioned issues. The proposed architecture revisits the concept of modular concatenation instead of long and deeper cascaded connections, to recover a cleaner approximation of the given image. We find that different modules can capture versatile representations, and a concatenated representation creates a richer subspace for low-level image restoration. The proposed architecture's number of parameters remains smaller than in most of the previous networks and still achieves significant improvements over the current state-of-the-art networks.