Deep Generative Medical Image Harmonization for Improving Cross-Site Generalization in Deep Learning Predictors

Exploring the potential of representation and transfer learning for anatomical neuroimaging: Application to psychiatry

The perspective of personalized medicine for brain disorders requires efficient learning models for anatomical neuroimaging-based prediction of clinical conditions. There is now a consensus on the benefit of deep learning (DL) in addressing many medical imaging tasks, such as image segmentation. However, for single-subject prediction problems, recent studies yielded contradictory results when comparing DL with Standard Machine Learning (SML) on top of classical feature extraction. Most existing comparative studies were limited in predicting phenotypes of little clinical interest, such as sex and age, and using a single dataset. Moreover, they conducted a limited analysis of the employed image pre-processing and feature selection strategies. This paper extensively compares DL and SML prediction capacity on five multi-site problems, including three increasingly complex clinical applications in psychiatry namely schizophrenia, bipolar disorder, and Autism Spectrum Disorder (ASD) diagnosis. To compensate for the relative scarcity of neuroimaging data on these clinical datasets, we also evaluate three pre-training strategies for transfer learning from brain imaging of the general healthy population: self-supervised learning, generative modeling and supervised learning with age. Overall, we find similar performance between randomly initialized DL and SML for the three clinical tasks and a similar scaling trend for sex prediction. This was replicated on an external dataset. We also show highly correlated discriminative brain regions between DL and linear ML models in all problems. Nonetheless, we demonstrate that self-supervised pre-training on large-scale healthy population imaging datasets (N≈10k), along with Deep Ensemble, allows DL to learn robust and transferable representations to smaller-scale clinical datasets (N≤1k). It largely outperforms SML on 2 out of 3 clinical tasks both in internal and external test sets. These findings suggest that the improvement of DL over SML in anatomical neuroimaging mainly comes from its capacity to learn meaningful and useful abstract representations of the brain anatomy, and it sheds light on the potential of transfer learning for personalized medicine in psychiatry.

The effect of harmonization on the variability of PET radiomic features extracted using various segmentation methods

PURPOSE

This study aimed to examine the robustness of positron emission tomography (PET) radiomic features extracted via different segmentation methods before and after ComBat harmonization in patients with non-small cell lung cancer (NSCLC).

METHODS

We included 120 patients (positive recurrence = 46 and negative recurrence = 74) referred for PET scanning as a routine part of their care. All patients had a biopsy-proven NSCLC. Nine segmentation methods were applied to each image, including manual delineation, K-means (KM), watershed, fuzzy-C-mean, region-growing, local active contour (LAC), and iterative thresholding (IT) with 40, 45, and 50% thresholds. Diverse image discretizations, both without a filter and with different wavelet decompositions, were applied to PET images. Overall, 6741 radiomic features were extracted from each image (749 radiomic features from each segmented area). Non-parametric empirical Bayes (NPEB) ComBat harmonization was used to harmonize the features. Linear Support Vector Classifier (LinearSVC) with L1 regularization For feature selection and Support Vector Machine classifier (SVM) with fivefold nested cross-validation was performed using StratifiedKFold with 'n_splits' set to 5 to predict recurrence in NSCLC patients and assess the impact of ComBat harmonization on the outcome.

RESULTS

From 749 extracted radiomic features, 206 (27%) and 389 (51%) features showed excellent reliability (ICC ≥ 0.90) against segmentation method variation before and after NPEB ComBat harmonization, respectively. Among all, 39 features demonstrated poor reliability, which declined to 10 after ComBat harmonization. The 64 fixed bin widths (without any filter) and wavelets (LLL)-based radiomic features set achieved the best performance in terms of robustness against diverse segmentation techniques before and after ComBat harmonization. The first-order and GLRLM and also first-order and NGTDM feature families showed the largest number of robust features before and after ComBat harmonization, respectively. In terms of predicting recurrence in NSCLC, our findings indicate that using ComBat harmonization can significantly enhance machine learning outcomes, particularly improving the accuracy of watershed segmentation, which initially had fewer reliable features than manual contouring. Following the application of ComBat harmonization, the majority of cases saw substantial increase in sensitivity and specificity.

CONCLUSION

Radiomic features are vulnerable to different segmentation methods. ComBat harmonization might be considered a solution to overcome the poor reliability of radiomic features.

Hybrid representation learning for cognitive diagnosis in late-life depression over 5 years with structural MRI

Late-life depression (LLD) is a highly prevalent mood disorder occurring in older adults and is frequently accompanied by cognitive impairment (CI). Studies have shown that LLD may increase the risk of Alzheimer's disease (AD). However, the heterogeneity of presentation of geriatric depression suggests that multiple biological mechanisms may underlie it. Current biological research on LLD progression incorporates machine learning that combines neuroimaging data with clinical observations. There are few studies on incident cognitive diagnostic outcomes in LLD based on structural MRI (sMRI). In this paper, we describe the development of a hybrid representation learning (HRL) framework for predicting cognitive diagnosis over 5 years based on T1-weighted sMRI data. Specifically, we first extract prediction-oriented MRI features via a deep neural network, and then integrate them with handcrafted MRI features via a Transformer encoder for cognitive diagnosis prediction. Two tasks are investigated in this work, including (1) identifying cognitively normal subjects with LLD and never-depressed older healthy subjects, and (2) identifying LLD subjects who developed CI (or even AD) and those who stayed cognitively normal over five years. We validate the proposed HRL on 294 subjects with T1-weighted MRIs from two clinically harmonized studies. Experimental results suggest that the HRL outperforms several classical machine learning and state-of-the-art deep learning methods in LLD identification and prediction tasks.

A novel loss function to reproduce texture features for deep learning-based MRI-to-CT synthesis

BACKGROUND

Studies on computed tomography (CT) synthesis based on magnetic resonance imaging (MRI) have mainly focused on pixel-wise consistency, but the texture features of regions of interest (ROIs) have not received appropriate attention.

PURPOSE

This study aimed to propose a novel loss function to reproduce texture features of ROIs and pixel-wise consistency for deep learning-based MRI-to-CT synthesis. The method was expected to assist the multi-modality studies for radiomics.

METHODS

The study retrospectively enrolled 127 patients with nasopharyngeal carcinoma. CT and MRI images were collected for each patient, and then rigidly registered as pre-procession. We proposed a gray-level co-occurrence matrix (GLCM)-based loss function to improve the reproducibility of texture features. This novel loss function could be embedded into the present deep learning-based framework for image synthesis. In this study, a typical image synthesis model was selected as the baseline, which contained a Unet trained mean square error (MSE) loss function. We embedded the proposed loss function and designed experiments to supervise different ROIs to prove its effectiveness. The concordance correlation coefficient (CCC) of the GLCM feature was employed to evaluate the reproducibility of GLCM features, which are typical texture features. Besides, we used a publicly available dataset of brain tumors to verify our loss function.

RESULTS

Compared with the baseline, the proposed method improved the pixel-wise image quality metrics (MAE: 107.5 to 106.8 HU; SSIM: 0.9728 to 0.9730). CCC values of the GLCM features in GTVnx were significantly improved from 0.78 ± 0.12 to 0.82 ± 0.11 (p < 0.05 for paired t-test). Generally, > 90% (22/24) of the GLCM-based features were improved compared with the baseline, where the Informational Measure of Correlation feature was improved the most (CCC: 0.74 to 0.83). For the public dataset, the loss function also shows its effectiveness. With our proposed loss function added, the ability to reproduce texture features was improved in the ROIs.

CONCLUSIONS

The proposed method reproduced texture features for MRI-to-CT synthesis, which would benefit radiomics studies based on image multi-modality synthesis.

Identifying Biases in a Multicenter MRI Database for Parkinson's Disease Classification: Is the Disease Classifier a Secret Site Classifier?

Sharing multicenter imaging datasets can be advantageous to increase data diversity and size but may lead to spurious correlations between site-related biological and non-biological image features and target labels, which machine learning (ML) models may exploit as shortcuts. To date, studies analyzing how and if deep learning models may use such effects as a shortcut are scarce. Thus, the aim of this work was to investigate if site-related effects are encoded in the feature space of an established deep learning model designed for Parkinson's disease (PD) classification based on T1-weighted MRI datasets. Therefore, all layers of the PD classifier were frozen, except for the last layer of the network, which was replaced by a linear layer that was exclusively re-trained to predict three potential bias types (biological sex, scanner type, and originating site). Our findings based on a large database consisting of 1880 MRI scans collected across 41 centers show that the feature space of the established PD model (74% accuracy) can be used to classify sex (75% accuracy), scanner type (79% accuracy), and site location (71% accuracy) with high accuracies despite this information never being explicitly provided to the PD model during original training. Overall, the results of this study suggest that trained image-based classifiers may use unwanted shortcuts that are not meaningful for the actual clinical task at hand. This finding may explain why many image-based deep learning models do not perform well when applied to data from centers not contributing to the training set.

Maximum Classifier Discrepancy Generative Adversarial Network for Jointly Harmonizing Scanner Effects and Improving Reproducibility of Downstream Tasks

OBJECTIVE

Multi-site collaboration is essential for overcoming small-sample problems when exploring reproducible biomarkers in MRI studies. However, various scanner-specific factors dramatically reduce the cross-scanner replicability. Moreover, existing harmony methods mostly could not guarantee the improved performance of downstream tasks.

METHODS

we proposed a new multi-scanner harmony framework, called 'maximum classifier discrepancy generative adversarial network', or MCD-GAN, for removing scanner effects in the original feature space while preserving substantial biological information for downstream tasks. Specifically, the adversarial generative network was utilized for persisting the structural layout of each sample, and the maximum classifier discrepancy module was introduced for regulating GAN generators by incorporating the downstream tasks.

RESULTS

We compared the MCD-GAN with other state-of-the-art data harmony approaches (e.g., ComBat, CycleGAN) on simulated data and the Adolescent Brain Cognitive Development (ABCD) dataset. Results demonstrate that MCD-GAN outperformed other approaches in improving cross-scanner classification performance while preserving the anatomical layout of the original images.

SIGNIFICANCE

To the best of our knowledge, the proposed MCD-GAN is the first generative model which incorporates downstream tasks while harmonizing, and is a promising solution for facilitating cross-site reproducibility in various tasks such as classification and regression.

Harmonizing T1-Weighted Images to Improve Consistency of Brain Morphology Among Different Scanner Manufacturers in Alzheimer's disease

BACKGROUND

Brain MRI scanner variability can introduce bias in measurements. Harmonizing scanner variability is crucial.

PURPOSE

To develop a harmonization method aimed at removing scanner variability, and to evaluate the consistency of results in multicenter studies.

STUDY TYPE

Retrospective.

POPULATION

Multicenter data from 170 healthy participants (males/females = 98/72; age = 73.8 ± 7.3) and 170 Alzheimer's disease patients (males/females = 98/72; age = 76.2 ± 8.5) were compared with reference data from another 340 participants.

FIELD STRENGTH/SEQUENCE

3-T, magnetization prepared rapid gradient echo and turbo field echo; 1.5-T, inversion recovery prepared fast spoiled gradient echo T1-weighted sequences.

ASSESSMENT

Gray matter (GM) brain images, obtained through segmentation of T1-weighted images, were utilized to evaluate the performance of the harmonization method using common orthogonal basis extraction (HCOBE) and four other methods (removal of artificial voxel effect by linear regression, RAVEL; Z_score; general linear model, GLM; ComBat). Linear discriminant analysis (LDA) was used to access the effectiveness of different methods in reducing scanner variability. The performance of harmonization methods in preserving GM volumes heterogeneity was evaluated by the similarity of the relationship between GM proportion and age in the reference and multicenter data. Furthermore, the consistency of the harmonized multicenter data with the reference data were evaluated based on classification results (train/test = 7/3) and brain atrophy.

STATISTICAL TESTS

Two-sample t-tests, area under the curve (AUC), and Dice coefficients were used to analyze the consistency of results from the reference and harmonized multicenter data. A P-value <0.01 was considered statistically significant.

RESULTS

HCOBE reduced the scanner variability from 0.09 before harmonization to 0.003 (ideal: 0, RAVEL/Z_score/GLM/ComBat = 0.087/0.003/0.006/0.013). GM volumes showed no significant difference (P = 0.52) between the reference and HCOBE-harmonized multicenter data. Consistency evaluation showed that AUC values of 0.95 for both reference and HCOBE-harmonized multicenter data (RAVEL/Z_score/GLM/ComBat = 0.86/0.86/0.84/0.89), and the Dice coefficient increased from 0.73 before harmonization to 0.82 (ideal: 1, RAVEL/Z_score/GLM/ComBat = 0.39/0.64/0.59/0.74).

DATA CONCLUSION

HCOBE may help to remove scanner variability and could improve the consistency of results in multicenter studies.

LEVEL OF EVIDENCE

2 TECHNICAL EFFICACY STAGE: 1.

Technical Note: Generalizable and Promptable Artificial Intelligence Model to Augment Clinical Delineation in Radiation Oncology

BACKGROUND

Efficient and accurate delineation of organs at risk (OARs) is a critical procedure for treatment planning and dose evaluation. Deep learning-based auto-segmentation of OARs has shown promising results and is increasingly being used in radiation therapy. However, existing deep learning-based auto-segmentation approaches face two challenges in clinical practice: generalizability and human-AI interaction. A generalizable and promptable auto-segmentation model, which segments OARs of multiple disease sites simultaneously and supports on-the-fly human-AI interaction, can significantly enhance the efficiency of radiation therapy treatment planning.

PURPOSE

Meta's segment anything model (SAM) was proposed as a generalizable and promptable model for next-generation natural image segmentation. We further evaluated the performance of SAM in radiotherapy segmentation.

METHODS

Computed tomography (CT) images of clinical cases from four disease sites at our institute were collected: prostate, lung, gastrointestinal, and head & neck. For each case, we selected the OARs important in radiotherapy treatment planning. We then compared both the Dice coefficients and Jaccard indices derived from three distinct methods: manual delineation (ground truth), automatic segmentation using SAM's 'segment anything' mode, and automatic segmentation using SAM's 'box prompt' mode that implements manual interaction via live prompts during segmentation.

RESULTS

Our results indicate that SAM's segment anything mode can achieve clinically acceptable segmentation results in most OARs with Dice scores higher than 0.7. SAM's box prompt mode further improves Dice scores by 0.1∼0.5. Similar results were observed for Jaccard indices. The results show that SAM performs better for prostate and lung, but worse for gastrointestinal and head & neck. When considering the size of organs and the distinctiveness of their boundaries, SAM shows better performance for large organs with distinct boundaries, such as lung and liver, and worse for smaller organs with less distinct boundaries, like parotid and cochlea.

CONCLUSIONS

Our results demonstrate SAM's robust generalizability with consistent accuracy in automatic segmentation for radiotherapy. Furthermore, the advanced box-prompt method enables the users to augment auto-segmentation interactively and dynamically, leading to patient-specific auto-segmentation in radiation therapy. SAM's generalizability across different disease sites and different modalities makes it feasible to develop a generic auto-segmentation model in radiotherapy.

Image harmonization improves consistency of intra-rater delineations of MS lesions in heterogeneous MRI

Clinical magnetic resonance images (MRIs) lack a standard intensity scale due to differences in scanner hardware and the pulse sequences used to acquire the images. When MRIs are used for quantification, as in the evaluation of white matter lesions (WMLs) in multiple sclerosis, this lack of intensity standardization becomes a critical problem affecting both the staging and tracking of the disease and its treatment. This paper presents a study of harmonization on WML segmentation consistency, which is evaluated using an object detection classification scheme that incorporates manual delineations from both the original and harmonized MRIs. A cohort of ten people scanned on two different imaging platforms was studied. An expert rater, blinded to the image source, manually delineated WMLs on images from both scanners before and after harmonization. It was found that there is closer agreement in both global and per-lesion WML volume and spatial distribution after harmonization, demonstrating the importance of image harmonization prior to the creation of manual delineations. These results could lead to better truth models in both the development and evaluation of automated lesion segmentation algorithms.

DeepResBat: deep residual batch harmonization accounting for covariate distribution differences

Pooling MRI data from multiple datasets requires harmonization to reduce undesired inter-site variabilities, while preserving effects of biological variables (or covariates). The popular harmonization approach ComBat uses a mixed effect regression framework that explicitly accounts for covariate distribution differences across datasets. There is also significant interest in developing harmonization approaches based on deep neural networks (DNNs), such as conditional variational autoencoder (cVAE). However, current DNN approaches do not explicitly account for covariate distribution differences across datasets. Here, we provide mathematical results, suggesting that not accounting for covariates can lead to suboptimal harmonization outcomes. We propose two DNN-based harmonization approaches that explicitly account for covariate distribution differences across datasets: covariate VAE (coVAE) and DeepResBat. The coVAE approach is a natural extension of cVAE by concatenating covariates and site information with site- and covariate-invariant latent representations. DeepResBat adopts a residual framework inspired by ComBat. DeepResBat first removes the effects of covariates with nonlinear regression trees, followed by eliminating site differences with cVAE. Finally, covariate effects are added back to the harmonized residuals. Using three datasets from three different continents with a total of 2787 participants and 10085 anatomical T1 scans, we find that DeepResBat and coVAE outperformed ComBat, CovBat and cVAE in terms of removing dataset differences, while enhancing biological effects of interest. However, coVAE hallucinates spurious associations between anatomical MRI and covariates even when no association exists. Therefore, future studies proposing DNN-based harmonization approaches should be aware of this false positive pitfall. Overall, our results suggest that DeepResBat is an effective deep learning alternative to ComBat.

Learning multi-site harmonization of magnetic resonance images without traveling human phantoms

Harmonization improves Magn. Reson. Imaging (MRI) data consistency and is central to effective integration of diverse imaging data acquired across multiple sites. Recent deep learning techniques for harmonization are predominantly supervised in nature and hence require imaging data of the same human subjects to be acquired at multiple sites. Data collection as such requires the human subjects to travel across sites and is hence challenging, costly, and impractical, more so when sufficient sample size is needed for reliable network training. Here we show how harmonization can be achieved with a deep neural network that does not rely on traveling human phantom data. Our method disentangles site-specific appearance information and site-invariant anatomical information from images acquired at multiple sites and then employs the disentangled information to generate the image of each subject for any target site. We demonstrate with more than 6,000 multi-site T1- and T2-weighted images that our method is remarkably effective in generating images with realistic site-specific appearances without altering anatomical details. Our method allows retrospective harmonization of data in a wide range of existing modern large-scale imaging studies, conducted via different scanners and protocols, without additional data collection.

Strategies for Implementing Machine Learning Algorithms in the Clinical Practice of Radiology

Despite recent advancements in machine learning (ML) applications in health care, there have been few benefits and improvements to clinical medicine in the hospital setting. To facilitate clinical adaptation of methods in ML, this review proposes a standardized framework for the step-by-step implementation of artificial intelligence into the clinical practice of radiology that focuses on three key components: problem identification, stakeholder alignment, and pipeline integration. A review of the recent literature and empirical evidence in radiologic imaging applications justifies this approach and offers a discussion on structuring implementation efforts to help other hospital practices leverage ML to improve patient care. Clinical trial registration no. 04242667 © RSNA, 2024 Supplemental material is available for this article.

CROSS-AGE AND CROSS-SITE DOMAIN SHIFT IMPACTS ON DEEP LEARNING-BASED WHITE MATTER FIBER ESTIMATION IN NEWBORN AND BABY BRAINS

Deep learning models have shown great promise in estimating tissue microstructure from limited diffusion magnetic resonance imaging data. However, these models face domain shift challenges when test and train data are from different scanners and protocols, or when the models are applied to data with inherent variations such as the developing brains of infants and children scanned at various ages. Several techniques have been proposed to address some of these challenges, such as data harmonization or domain adaptation in the adult brain. However, those techniques remain unexplored for the estimation of fiber orientation distribution functions in the rapidly developing brains of infants. In this work, we extensively investigate the age effect and domain shift within and across two different cohorts of 201 newborns and 165 babies using the Method of Moments and fine-tuning strategies. Our results show that reduced variations in the microstructural development of babies in comparison to newborns directly impact the deep learning models' cross-age performance. We also demonstrate that a small number of target domain samples can significantly mitigate domain shift problems.

Enhancing Interoperability and Harmonisation of Nuclear Medicine Image Data and Associated Clinical Data

Nuclear imaging techniques such as positron emission tomography (PET) and single photon emission computed tomography (SPECT) in combination with computed tomography (CT) are established imaging modalities in clinical practice, particularly for oncological problems. Due to a multitude of manufacturers, different measurement protocols, local demographic or clinical workflow variations as well as various available reconstruction and analysis software, very heterogeneous datasets are generated. This review article examines the current state of interoperability and harmonisation of image data and related clinical data in the field of nuclear medicine. Various approaches and standards to improve data compatibility and integration are discussed. These include, for example, structured clinical history, standardisation of image acquisition and reconstruction as well as standardised preparation of image data for evaluation. Approaches to improve data acquisition, storage and analysis will be presented. Furthermore, approaches are presented to prepare the datasets in such a way that they become usable for projects applying artificial intelligence (AI) (machine learning, deep learning, etc.). This review article concludes with an outlook on future developments and trends related to AI in nuclear medicine, including a brief research of commercial solutions.

A three-dimensional deep learning model for inter-site harmonization of structural MR images of the brain: Extensive validation with a multicenter dataset

In multicenter MRI studies, pooling the imaging data can introduce site-related variabilities and can therefore bias the subsequent analyses. To harmonize the intensity distributions of brain MR images in a multicenter dataset, unsupervised deep learning methods can be employed. Here, we developed a model based on cycle-consistent adversarial networks for the harmonization of T1-weighted brain MR images. In contrast to previous works, it was designed to process three-dimensional whole-brain images in a stable manner while optimizing computation resources. Using six different MRI datasets for healthy adults (n=1525 in total) with different acquisition parameters, we tested the model in (i) three pairwise harmonizations with site effects of various sizes, (ii) an overall harmonization of the six datasets with different age distributions, and (iii) a traveling-subject dataset. Our results for intensity distributions, brain volumes, image quality metrics and radiomic features indicated that the MRI characteristics at the various sites had been effectively homogenized. Next, brain age prediction experiments and the observed correlation between the gray-matter volume and age showed that thanks to an appropriate training strategy and despite biological differences between the dataset populations, the model reinforced biological patterns. Furthermore, radiologic analyses of the harmonized images attested to the conservation of the radiologic information in the original images. The robustness of the harmonization model (as judged with various datasets and metrics) demonstrates its potential for application in retrospective multicenter studies.

Image-encoded biological and non-biological variables may be used as shortcuts in deep learning models trained on multisite neuroimaging data

OBJECTIVE

This work investigates if deep learning (DL) models can classify originating site locations directly from magnetic resonance imaging (MRI) scans with and without correction for intensity differences.

MATERIAL AND METHODS

A large database of 1880 T1-weighted MRI scans collected across 41 sites originally for Parkinson's disease (PD) classification was used to classify sites in this study. Forty-six percent of the datasets are from PD patients, while 54% are from healthy participants. After preprocessing the T1-weighted scans, 2 additional data types were generated: intensity-harmonized T1-weighted scans and log-Jacobian deformation maps resulting from nonlinear atlas registration. Corresponding DL models were trained to classify sites for each data type. Additionally, logistic regression models were used to investigate the contribution of biological (age, sex, disease status) and non-biological (scanner type) variables to the models' decision.

RESULTS

A comparison of the 3 different types of data revealed that DL models trained using T1-weighted and intensity-harmonized T1-weighted scans can classify sites with an accuracy of 85%, while the model using log-Jacobian deformation maps achieved a site classification accuracy of 54%. Disease status and scanner type were found to be significant confounders.

DISCUSSION

Our results demonstrate that MRI scans encode relevant site-specific information that models could use as shortcuts that cannot be removed using simple intensity harmonization methods.

CONCLUSION

The ability of DL models to exploit site-specific biases as shortcuts raises concerns about their reliability, generalization, and deployability in clinical settings.

Style transfer generative adversarial networks to harmonize multisite MRI to a single reference image to avoid overcorrection

Recent work within neuroimaging consortia have aimed to identify reproducible, and often subtle, brain signatures of psychiatric or neurological conditions. To allow for high-powered brain imaging analyses, it is often necessary to pool MR images that were acquired with different protocols across multiple scanners. Current retrospective harmonization techniques have shown promise in removing site-related image variation. However, most statistical approaches may over-correct for technical, scanning-related, variation as they cannot distinguish between confounded image-acquisition based variability and site-related population variability. Such statistical methods often require that datasets contain subjects or patient groups with similar clinical or demographic information to isolate the acquisition-based variability. To overcome this limitation, we consider site-related magnetic resonance (MR) imaging harmonization as a style transfer problem rather than a domain transfer problem. Using a fully unsupervised deep-learning framework based on a generative adversarial network (GAN), we show that MR images can be harmonized by inserting the style information encoded from a single reference image, without knowing their site/scanner labels a priori. We trained our model using data from five large-scale multisite datasets with varied demographics. Results demonstrated that our style-encoding model can harmonize MR images, and match intensity profiles, without relying on traveling subjects. This model also avoids the need to control for clinical, diagnostic, or demographic information. We highlight the effectiveness of our method for clinical research by comparing extracted cortical and subcortical features, brain-age estimates, and case-control effect sizes before and after the harmonization. We showed that our harmonization removed the site-related variances, while preserving the anatomical information and clinical meaningful patterns. We further demonstrated that with a diverse training set, our method successfully harmonized MR images collected from unseen scanners and protocols, suggesting a promising tool for ongoing collaborative studies. Source code is released in USC-IGC/style_transfer_harmonization (github.com).

Image harmonization: A review of statistical and deep learning methods for removing batch effects and evaluation metrics for effective harmonization

Magnetic resonance imaging and computed tomography from multiple batches (e.g. sites, scanners, datasets, etc.) are increasingly used alongside complex downstream analyses to obtain new insights into the human brain. However, significant confounding due to batch-related technical variation, called batch effects, is present in this data; direct application of downstream analyses to the data may lead to biased results. Image harmonization methods seek to remove these batch effects and enable increased generalizability and reproducibility of downstream results. In this review, we describe and categorize current approaches in statistical and deep learning harmonization methods. We also describe current evaluation metrics used to assess harmonization methods and provide a standardized framework to evaluate newly-proposed methods for effective harmonization and preservation of biological information. Finally, we provide recommendations to end-users to advocate for more effective use of current methods and to methodologists to direct future efforts and accelerate development of the field.

Artificial Intelligence in CT and MR Imaging for Oncological Applications

Cancer care increasingly relies on imaging for patient management. The two most common cross-sectional imaging modalities in oncology are computed tomography (CT) and magnetic resonance imaging (MRI), which provide high-resolution anatomic and physiological imaging. Herewith is a summary of recent applications of rapidly advancing artificial intelligence (AI) in CT and MRI oncological imaging that addresses the benefits and challenges of the resultant opportunities with examples. Major challenges remain, such as how best to integrate AI developments into clinical radiology practice, the vigorous assessment of quantitative CT and MR imaging data accuracy, and reliability for clinical utility and research integrity in oncology. Such challenges necessitate an evaluation of the robustness of imaging biomarkers to be included in AI developments, a culture of data sharing, and the cooperation of knowledgeable academics with vendor scientists and companies operating in radiology and oncology fields. Herein, we will illustrate a few challenges and solutions of these efforts using novel methods for synthesizing different contrast modality images, auto-segmentation, and image reconstruction with examples from lung CT as well as abdome, pelvis, and head and neck MRI. The imaging community must embrace the need for quantitative CT and MRI metrics beyond lesion size measurement. AI methods for the extraction and longitudinal tracking of imaging metrics from registered lesions and understanding the tumor environment will be invaluable for interpreting disease status and treatment efficacy. This is an exciting time to work together to move the imaging field forward with narrow AI-specific tasks. New AI developments using CT and MRI datasets will be used to improve the personalized management of cancer patients.

Reducing segmentation failures in cardiac MRI via late feature fusion and GAN-based augmentation

Cardiac magnetic resonance (CMR) image segmentation is an integral step in the analysis of cardiac function and diagnosis of heart related diseases. While recent deep learning-based approaches in automatic segmentation have shown great promise to alleviate the need for manual segmentation, most of these are not applicable to realistic clinical scenarios. This is largely due to training on mainly homogeneous datasets, without variation in acquisition, which typically occurs in multi-vendor and multi-site settings, as well as pathological data. Such approaches frequently exhibit a degradation in prediction performance, particularly on outlier cases commonly associated with difficult pathologies, artifacts and extensive changes in tissue shape and appearance. In this work, we present a model aimed at segmenting all three cardiac structures in a multi-center, multi-disease and multi-view scenario. We propose a pipeline, addressing different challenges with segmentation of such heterogeneous data, consisting of heart region detection, augmentation through image synthesis and a late-fusion segmentation approach. Extensive experiments and analysis demonstrate the ability of the proposed approach to tackle the presence of outlier cases during both training and testing, allowing for better adaptation to unseen and difficult examples. Overall, we show that the effective reduction of segmentation failures on outlier cases has a positive impact on not only the average segmentation performance, but also on the estimation of clinical parameters, leading to a better consistency in derived metrics.

DeepComBat: A Statistically Motivated, Hyperparameter-Robust, Deep Learning Approach to Harmonization of Neuroimaging Data

Neuroimaging data from multiple batches (i.e. acquisition sites, scanner manufacturer, datasets, etc.) are increasingly necessary to gain new insights into the human brain. However, multi-batch data, as well as extracted radiomic features, exhibit pronounced technical artifacts across batches. These batch effects introduce confounding into the data and can obscure biological effects of interest, decreasing the generalizability and reproducibility of findings. This is especially true when multi-batch data is used alongside complex downstream analysis models, such as machine learning methods. Image harmonization methods seeking to remove these batch effects are important for mitigating these issues; however, significant multivariate batch effects remain in the data following harmonization by current state-of-the-art statistical and deep learning methods. We present DeepCombat, a deep learning harmonization method based on a conditional variational autoencoder architecture and the ComBat harmonization model. DeepCombat learns and removes subject-level batch effects by accounting for the multivariate relationships between features. Additionally, DeepComBat relaxes a number of strong assumptions commonly made by previous deep learning harmonization methods and is empirically robust across a wide range of hyperparameter choices. We apply this method to neuroimaging data from a large cognitive-aging cohort and find that DeepCombat outperforms existing methods, as assessed by a battery of machine learning methods, in removing scanner effects from cortical thickness measurements while preserving biological heterogeneity. Additionally, DeepComBat provides a new perspective for statistically-motivated deep learning harmonization methods.

Applications of generative adversarial networks in neuroimaging and clinical neuroscience

Generative adversarial networks (GANs) are one powerful type of deep learning models that have been successfully utilized in numerous fields. They belong to the broader family of generative methods, which learn to generate realistic data with a probabilistic model by learning distributions from real samples. In the clinical context, GANs have shown enhanced capabilities in capturing spatially complex, nonlinear, and potentially subtle disease effects compared to traditional generative methods. This review critically appraises the existing literature on the applications of GANs in imaging studies of various neurological conditions, including Alzheimer's disease, brain tumors, brain aging, and multiple sclerosis. We provide an intuitive explanation of various GAN methods for each application and further discuss the main challenges, open questions, and promising future directions of leveraging GANs in neuroimaging. We aim to bridge the gap between advanced deep learning methods and neurology research by highlighting how GANs can be leveraged to support clinical decision making and contribute to a better understanding of the structural and functional patterns of brain diseases.

AI-based dimensional neuroimaging system for characterizing heterogeneity in brain structure and function in major depressive disorder: COORDINATE-MDD consortium design and rationale

BACKGROUND

Efforts to develop neuroimaging-based biomarkers in major depressive disorder (MDD), at the individual level, have been limited to date. As diagnostic criteria are currently symptom-based, MDD is conceptualized as a disorder rather than a disease with a known etiology; further, neural measures are often confounded by medication status and heterogeneous symptom states.

METHODS

We describe a consortium to quantify neuroanatomical and neurofunctional heterogeneity via the dimensions of novel multivariate coordinate system (COORDINATE-MDD). Utilizing imaging harmonization and machine learning methods in a large cohort of medication-free, deeply phenotyped MDD participants, patterns of brain alteration are defined in replicable and neurobiologically-based dimensions and offer the potential to predict treatment response at the individual level. International datasets are being shared from multi-ethnic community populations, first episode and recurrent MDD, which are medication-free, in a current depressive episode with prospective longitudinal treatment outcomes and in remission. Neuroimaging data consist of de-identified, individual, structural MRI and resting-state functional MRI with additional positron emission tomography (PET) data at specific sites. State-of-the-art analytic methods include automated image processing for extraction of anatomical and functional imaging variables, statistical harmonization of imaging variables to account for site and scanner variations, and semi-supervised machine learning methods that identify dominant patterns associated with MDD from neural structure and function in healthy participants.

RESULTS

We are applying an iterative process by defining the neural dimensions that characterise deeply phenotyped samples and then testing the dimensions in novel samples to assess specificity and reliability. Crucially, we aim to use machine learning methods to identify novel predictors of treatment response based on prospective longitudinal treatment outcome data, and we can externally validate the dimensions in fully independent sites.

CONCLUSION

We describe the consortium, imaging protocols and analytics using preliminary results. Our findings thus far demonstrate how datasets across many sites can be harmonized and constructively pooled to enable execution of this large-scale project.

Ten quick tips for computational analysis of medical images

Medical imaging is a great asset for modern medicine, since it allows physicians to spatially interrogate a disease site, resulting in precise intervention for diagnosis and treatment, and to observe particular aspect of patients' conditions that otherwise would not be noticeable. Computational analysis of medical images, moreover, can allow the discovery of disease patterns and correlations among cohorts of patients with the same disease, thus suggesting common causes or providing useful information for better therapies and cures. Machine learning and deep learning applied to medical images, in particular, have produced new, unprecedented results that can pave the way to advanced frontiers of medical discoveries. While computational analysis of medical images has become easier, however, the possibility to make mistakes or generate inflated or misleading results has become easier, too, hindering reproducibility and deployment. In this article, we provide ten quick tips to perform computational analysis of medical images avoiding common mistakes and pitfalls that we noticed in multiple studies in the past. We believe our ten guidelines, if taken into practice, can help the computational-medical imaging community to perform better scientific research that eventually can have a positive impact on the lives of patients worldwide.

Radiomics in clinical trials: perspectives on standardization

The term biomarker is used to describe a biological measure of the disease behavior. The existing imaging biomarkers are associated with the known tissue biological characteristics and follow a well-established roadmap to be implemented in routine clinical practice. Recently, a new quantitative imaging analysis approach named radiomics has emerged. It refers to the extraction of a large number of advanced imaging features with high-throughput computing. Extensive research has demonstrated its value in predicting disease behavior, progression, and response to therapeutic options. However, there are numerous challenges to establishing it as a clinically viable solution, including lack of reproducibility and transparency. The data-driven nature also does not offer insights into the underpinning biology of the observed relationships. As such, additional effort is needed to establish it as a qualified biomarker to inform clinical decisions. Here we review the technical difficulties encountered in the clinical applications of radiomics and current effort in addressing some of these challenges in clinical trial designs. By addressing these challenges, the true potential of radiomics can be unleashed.

Contrastive semi-supervised harmonization of single-shell to multi-shell diffusion MRI

Diffusion weighted MRI (DW-MRI) harmonization is necessary for multi-site or multi-acquisition studies. Current statistical methods address the need to harmonize from one site to another, but do not simultaneously consider the use of multiple datasets which are comprised of multiple sites, acquisitions protocols, and age demographics. This work explores deep learning methods which can generalize across these variations through semi-supervised and unsupervised learning while also learning to estimate multi-shell data from single-shell data using the Multi-shell Diffusion MRI Harmonization Challenge (MUSHAC) and Baltimore Longitudinal Study on Aging (BLSA) datasets. We compare disentanglement harmonization models, which seek to encode anatomy and acquisition in separate latent spaces, and a CycleGAN harmonization model, which uses generative adversarial networks (GAN) to perform style transfer between sites, to the baseline preprocessing and to SHORE interpolation. We find that the disentanglement models achieve superior performance in harmonizing all data while at the same transforming the input data to a single target space across several diffusion metrics (fractional anisotropy, mean diffusivity, mean kurtosis, primary eigenvector).

Identifying acute ischemic stroke patients within the thrombolytic treatment window using deep learning

BACKGROUND AND PURPOSE

Treatment of acute ischemic stroke is heavily contingent upon time, as there is a strong relationship between time clock and tissue progression. Work has established imaging biomarker assessments as surrogates for time since stroke (TSS), namely, by comparing signal mismatch between diffusion-weighted imaging (DWI) and fluid-attenuated inversion recovery (FLAIR) imaging. Our goal was to develop an automatic technique for determining TSS from imaging that does not require subspecialist radiology expertise.

METHODS

Using 772 patients (66 ± 9 years, 319 women), we developed and externally evaluated a deep learning network for classifying TSS from MR images and compared algorithm predictions to neuroradiologist assessments of DWI-FLAIR mismatch. Models were trained to classify TSS within 4.5 hours and performance metrics with confidence intervals were reported on both internal and external evaluation sets.

RESULTS

Three board-certified neuroradiologists' DWI-FLAIR mismatch assessments, based on majority vote, yielded a sensitivity of .62, a specificity of .86, and a Fleiss' kappa of .46 when used to classify TSS. The deep learning method performed similarly to radiologists and outperformed previously reported methods, with the best model achieving an average evaluation accuracy, sensitivity, and specificity of .726, .712, and .741, respectively, on an internal cohort and .724, .757, and .679, respectively, on an external cohort.

CONCLUSION

Our model achieved higher generalization performance on external evaluation datasets than the current state-of-the-art for TSS classification. These results demonstrate the potential of automatic assessment of onset time from imaging without the need for expertly trained radiologists.

Improving reproducibility and performance of radiomics in low-dose CT using cycle GANs

Background

As a means to extract biomarkers from medical imaging, radiomics has attracted increased attention from researchers. However, reproducibility and performance of radiomics in low‐dose CT scans are still poor, mostly due to noise. Deep learning generative models can be used to denoise these images and in turn improve radiomics’ reproducibility and performance. However, most generative models are trained on paired data, which can be difficult or impossible to collect.

Purpose

In this article, we investigate the possibility of denoising low‐dose CTs using cycle generative adversarial networks (GANs) to improve radiomics reproducibility and performance based on unpaired datasets.

Methods and materials

Two cycle GANs were trained: (1) from paired data, by simulating low‐dose CTs (i.e., introducing noise) from high‐dose CTs and (2) from unpaired real low dose CTs. To accelerate convergence, during GAN training, a slice‐paired training strategy was introduced. The trained GANs were applied to three scenarios: (1) improving radiomics reproducibility in simulated low‐dose CT images and (2) same‐day repeat low dose CTs (RIDER dataset), and (3) improving radiomics performance in survival prediction. Cycle GAN results were compared with a conditional GAN (CGAN) and an encoder–decoder network (EDN) trained on simulated paired data.

Results

The cycle GAN trained on simulated data improved concordance correlation coefficients (CCC) of radiomic features from 0.87 (95%CI, [0.833,0.901]) to 0.93 (95%CI, [0.916,0.949]) on simulated noise CT and from 0.89 (95%CI, [0.881,0.914]) to 0.92 (95%CI, [0.908,0.937]) on the RIDER dataset, as well improving the area under the receiver operating characteristic curve (AUC) of survival prediction from 0.52 (95%CI, [0.511,0.538]) to 0.59 (95%CI, [0.578,0.602]). The cycle GAN trained on real data increased the CCCs of features in RIDER to 0.95 (95%CI, [0.933,0.961]) and the AUC of survival prediction to 0.58 (95%CI, [0.576,0.596]).

Conclusion

The results show that cycle GANs trained on both simulated and real data can improve radiomics’ reproducibility and performance in low‐dose CT and achieve similar results compared to CGANs and EDNs.

Commentary to "Translational machine learning for child and adolescent psychiatry"

In this commentary on 'Translational Machine Learning for Child and Adolescent Psychiatry,' by Dwyer and Koutsouleris, we summarize some of the main points made by the authors, which highlight the importance of emerging applications of machine learning for psychiatric disorders in youth but also emphasize principles of good practice. We also offer complementary insights regarding large-scale training, harmonization, and the ability of these artificial intelligence models to adapt to new datasets, which is critical for their stability across imaging centers, and hence for their widespread clinical adoption.

Applications of Radiomics and Radiogenomics in High-Grade Gliomas in the Era of Precision Medicine

Simple Summary

Radiomics and radiogenomics offer new insight into high-grade glioma biology, as well as into glioma behavior in response to standard therapies. In this article, we provide neuro-oncology, neuropathology, and computational perspectives on the role of radiomics in providing more accurate diagnoses, prognostication, and surveillance of patients with high-grade glioma, and on the potential application of radiomics in clinical practice, with the overarching goal of advancing precision medicine for optimal patient care.

Abstract

Machine learning (ML) integrated with medical imaging has introduced new perspectives in precision diagnostics of high-grade gliomas, through radiomics and radiogenomics. This has raised hopes for characterizing noninvasive and in vivo biomarkers for prediction of patient survival, tumor recurrence, and genomics and therefore encouraging treatments tailored to individualized needs. Characterization of tumor infiltration based on pre-operative multi-parametric magnetic resonance imaging (MP-MRI) scans may allow prediction of the loci of future tumor recurrence and thereby aid in planning the course of treatment for the patients, such as optimizing the extent of resection and the dose and target area of radiation. Imaging signatures of tumor genomics can help in identifying the patients who benefit from certain targeted therapies. Specifying molecular properties of gliomas and prediction of their changes over time and with treatment would allow optimization of treatment. In this article, we provide neuro-oncology, neuropathology, and computational perspectives on the promise of radiomics and radiogenomics for allowing personalized treatments of patients with gliomas and discuss the challenges and limitations of these methods in multi-institutional clinical trials and suggestions to mitigate the issues and the future directions.