The reliability of a deep learning model in clinical out-of-distribution MRI data: A multicohort study

A study of generalization and compatibility performance of 3D U-Net segmentation on multiple heterogeneous liver CT datasets

BACKGROUND

Most existing algorithms have been focused on the segmentation from several public Liver CT datasets scanned regularly (no pneumoperitoneum and horizontal supine position). This study primarily segmented datasets with unconventional liver shapes and intensities deduced by contrast phases, irregular scanning conditions, different scanning objects of pigs and patients with large pathological tumors, which formed the multiple heterogeneity of datasets used in this study.

METHODS

The multiple heterogeneous datasets used in this paper includes: (1) One public contrast-enhanced CT dataset and one public non-contrast CT dataset; (2) A contrast-enhanced dataset that has abnormal liver shape with very long left liver lobes and large-sized liver tumors with abnormal presets deduced by microvascular invasion; (3) One artificial pneumoperitoneum dataset under the pneumoperitoneum and three scanning profiles (horizontal/left/right recumbent position); (4) Two porcine datasets of Bama type and domestic type that contains pneumoperitoneum cases but with large anatomy discrepancy with humans. The study aimed to investigate the segmentation performances of 3D U-Net in: (1) generalization ability between multiple heterogeneous datasets by cross-testing experiments; (2) the compatibility when hybrid training all datasets in different sampling and encoder layer sharing schema. We further investigated the compatibility of encoder level by setting separate level for each dataset (i.e., dataset-wise convolutions) while sharing the decoder.

RESULTS

Model trained on different datasets has different segmentation performance. The prediction accuracy between LiTS dataset and Zhujiang dataset was about 0.955 and 0.958 which shows their good generalization ability due to that they were all contrast-enhanced clinical patient datasets scanned regularly. For the datasets scanned under pneumoperitoneum, their corresponding datasets scanned without pneumoperitoneum showed good generalization ability. Dataset-wise convolution module in high-level can improve the dataset unbalance problem. The experimental results will facilitate researchers making solutions when segmenting those special datasets.

CONCLUSIONS

(1) Regularly scanned datasets is well generalized to irregularly ones. (2) The hybrid training is beneficial but the dataset imbalance problem always exits due to the multi-domain homogeneity. The higher levels encoded more domain specific information than lower levels and thus were less compatible in terms of our datasets.

Multi-Institutional Validation of a Mammography-Based Breast Cancer Risk Model

PURPOSE

Accurate risk assessment is essential for the success of population screening programs in breast cancer. Models with high sensitivity and specificity would enable programs to target more elaborate screening efforts to high-risk populations, while minimizing overtreatment for the rest. Artificial intelligence (AI)-based risk models have demonstrated a significant advance over risk models used today in clinical practice. However, the responsible deployment of novel AI requires careful validation across diverse populations. To this end, we validate our AI-based model, Mirai, across globally diverse screening populations.

METHODS

We collected screening mammograms and pathology-confirmed breast cancer outcomes from Massachusetts General Hospital, USA; Novant, USA; Emory, USA; Maccabi-Assuta, Israel; Karolinska, Sweden; Chang Gung Memorial Hospital, Taiwan; and Barretos, Brazil. We evaluated Uno's concordance-index for Mirai in predicting risk of breast cancer at one to five years from the mammogram.

RESULTS

A total of 128,793 mammograms from 62,185 patients were collected across the seven sites, of which 3,815 were followed by a cancer diagnosis within 5 years. Mirai obtained concordance indices of 0.75 (95% CI, 0.72 to 0.78), 0.75 (95% CI, 0.70 to 0.80), 0.77 (95% CI, 0.75 to 0.79), 0.77 (95% CI, 0.73 to 0.81), 0.81 (95% CI, 0.79 to 0.82), 0.79 (95% CI, 0.76 to 0.83), and 0.84 (95% CI, 0.81 to 0.88) at Massachusetts General Hospital, Novant, Emory, Maccabi-Assuta, Karolinska, Chang Gung Memorial Hospital, and Barretos, respectively.

CONCLUSION

Mirai, a mammography-based risk model, maintained its accuracy across globally diverse test sets from seven hospitals across five countries. This is the broadest validation to date of an AI-based breast cancer model and suggests that the technology can offer broad and equitable improvements in care.

Machine Learning for Health: Algorithm Auditing & Quality Control

Developers proposing new machine learning for health (ML4H) tools often pledge to match or even surpass the performance of existing tools, yet the reality is usually more complicated. Reliable deployment of ML4H to the real world is challenging as examples from diabetic retinopathy or Covid-19 screening show. We envision an integrated framework of algorithm auditing and quality control that provides a path towards the effective and reliable application of ML systems in healthcare. In this editorial, we give a summary of ongoing work towards that vision and announce a call for participation to the special issue  Machine Learning for Health: Algorithm Auditing & Quality Control in this journal to advance the practice of ML4H auditing.

Evaluating the progress of deep learning for visual relational concepts

Convolutional neural networks have become the state-of-the-art method for image classification in the last 10 years. Despite the fact that they achieve superhuman classification accuracy on many popular datasets, they often perform much worse on more abstract image classification tasks. We will show that these difficult tasks are linked to relational concepts from cognitive psychology and that despite progress over the last few years, such relational reasoning tasks still remain difficult for current neural network architectures. We will review deep learning research that is linked to relational concept learning, even if it was not originally presented from this angle. Reviewing the current literature, we will argue that some form of attention will be an important component of future systems to solve relational tasks. In addition, we will point out the shortcomings of currently used datasets, and we will recommend steps to make future datasets more relevant for testing systems on relational reasoning.

Radiomics in PET Imaging:: A Practical Guide for Newcomers

Radiomics has undergone considerable development in recent years. In PET imaging, very promising results concerning the ability of handcrafted features to predict the biological characteristics of lesions and to assess patient prognosis or response to treatment have been reported in the literature. This article presents a checklist for designing a reliable radiomic study, gives an overview of the steps of the pipeline, and outlines approaches for data harmonization. Tips are provided for critical reading of the content of articles. The advantages and limitations of handcrafted radiomics compared with deep-learning approaches for the characterization of PET images are also discussed.

A Contrast Augmentation Approach to Improve Multi-Scanner Generalization in MRI

Most data-driven methods are very susceptible to data variability. This problem is particularly apparent when applying Deep Learning (DL) to brain Magnetic Resonance Imaging (MRI), where intensities and contrasts vary due to acquisition protocol, scanner- and center-specific factors. Most publicly available brain MRI datasets originate from the same center and are homogeneous in terms of scanner and used protocol. As such, devising robust methods that generalize to multi-scanner and multi-center data is crucial for transferring these techniques into clinical practice. We propose a novel data augmentation approach based on Gaussian Mixture Models (GMM-DA) with the goal of increasing the variability of a given dataset in terms of intensities and contrasts. The approach allows to augment the training dataset such that the variability in the training set compares to what is seen in real world clinical data, while preserving anatomical information. We compare the performance of a state-of-the-art U-Net model trained for segmenting brain structures with and without the addition of GMM-DA. The models are trained and evaluated on single- and multi-scanner datasets. Additionally, we verify the consistency of test-retest results on same-patient images (same and different scanners). Finally, we investigate how the presence of bias field influences the performance of a model trained with GMM-DA. We found that the addition of the GMM-DA improves the generalization capability of the DL model to other scanners not present in the training data, even when the train set is already multi-scanner. Besides, the consistency between same-patient segmentation predictions is improved, both for same-scanner and different-scanner repetitions. We conclude that GMM-DA could increase the transferability of DL models into clinical scenarios.

An [18F]FDG-PET/CT deep learning method for fully automated detection of pathological mediastinal lymph nodes in lung cancer patients

Purpose

The identification of pathological mediastinal lymph nodes is an important step in the staging of lung cancer, with the presence of metastases significantly affecting survival rates. Nodes are currently identified by a physician, but this process is time-consuming and prone to errors. In this paper, we investigate the use of artificial intelligence–based methods to increase the accuracy and consistency of this process.

Methods

Whole-body ¹⁸F-labelled fluoro-2-deoxyglucose ([18F]FDG) positron emission tomography/computed tomography ([18F]FDG-PET/CT) scans (Philips Gemini TF) from 134 patients were retrospectively analysed. The thorax was automatically located, and then slices were fed into a U-Net to identify candidate regions. These regions were split into overlapping 3D cubes, which were individually predicted as positive or negative using a 3D CNN. From these predictions, pathological mediastinal nodes could be identified. A second cohort of 71 patients was then acquired from a different, newer scanner (GE Discovery MI), and the performance of the model on this dataset was tested with and without transfer learning.

Results

On the test set from the first scanner, our model achieved a sensitivity of 0.87 (95% confidence intervals [0.74, 0.94]) with 0.41 [0.22, 0.71] false positives/patient. This was comparable to the performance of an expert. Without transfer learning, on the test set from the second scanner, the corresponding results were 0.53 [0.35, 0.70] and 0.24 [0.10, 0.49], respectively. With transfer learning, these metrics were 0.88 [0.73, 0.97] and 0.69 [0.43, 1.04], respectively.

Conclusion

Model performance was comparable to that of an expert on data from the same scanner. With transfer learning, the model can be applied to data from a different scanner. To our knowledge it is the first study of its kind to go directly from whole-body [18F]FDG-PET/CT scans to pathological mediastinal lymph node localisation.

Supplementary Information

The online version contains supplementary material available at 10.1007/s00259-021-05513-x.

Normalization of breast MRIs using cycle-consistent generative adversarial networks

OBJECTIVES

Dynamic Contrast Enhanced-Magnetic Resonance Imaging (DCE-MRI) is widely used to complement ultrasound examinations and x-ray mammography for early detection and diagnosis of breast cancer. However, images generated by various MRI scanners (e.g., GE Healthcare, and Siemens) differ both in intensity and noise distribution, preventing algorithms trained on MRIs from one scanner to generalize to data from other scanners. In this work, we propose a method to solve this problem by normalizing images between various scanners.

METHODS

MRI normalization is challenging because it requires normalizing intensity values and mapping noise distributions between scanners. We utilize a cycle-consistent generative adversarial network to learn a bidirectional mapping and perform normalization between MRIs produced by GE Healthcare and Siemens scanners in an unpaired setting. Initial experiments demonstrate that the traditional CycleGAN architecture struggles to preserve the anatomical structures of the breast during normalization. Thus, we propose two technical innovations in order to preserve both the shape of the breast as well as the tissue structures within the breast. First, we incorporate mutual information loss during training in order to ensure anatomical consistency. Second, we propose a modified discriminator architecture that utilizes a smaller field-of-view to ensure the preservation of finer details in the breast tissue.

RESULTS

Quantitative and qualitative evaluations show that the second innovation consistently preserves the breast shape and tissue structures while also performing the proper intensity normalization and noise distribution mapping.

CONCLUSION

Our results demonstrate that the proposed model can successfully learn a bidirectional mapping and perform normalization between MRIs produced by different vendors, potentially enabling improved diagnosis and detection of breast cancer. All the data used in this study are publicly available at https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=70226903.

Differences in Brain Structural Covariance Network Characteristics in Children and Adults With Autism Spectrum Disorder

Systematically describing the structural topological configuration of human brain during development is an essential task. Autism spectrum disorder (ASD) represents a powerful challenge for psychiatry and neuroscience researchers. In this study, we investigated variations in the structural covariance network properties of 441 patients with ASD ranging in age from 7 to 45 years and in 426 age-matched healthy controls (HCs) using structural magnetic resonance neuroimaging from the ABIDE database. We applied a sliding window approach to study topological variation during development using comprehensive graph theoretical analysis. The main findings are as follows: (1) Cross-sectional trajectories of the network characteristics exhibited inverted U-shapes in both HCs and participants with ASD, with the latter exhibiting a 7-year delay in reaching the maximum value, (2) network resilience to targeted attacks peaked at 18' and 19' in the HCs and at 25' in the participants with ASD, and the weakest resilience occurred at age 7', (3) the HCs and participants with ASD exhibited normalized mean degree differences in the right amygdala, and (4) significant differences in the network characteristics were observed in the 18' age group at most of the densities analyzed. We used cross-sectional analysis to infer distinct neurodevelopmental trajectories in ASD in the brain structural connectome. Our findings are consistent with the notion that adolescence is a sensitive period of brain development with strong potential for brain plasticity, offering opportunities for environmental adaptation and social integration and for increasing vulnerability. ASD may be a product of susceptibility. LAY SUMMARY: We used cross-sectional analysis to preliminarily infer distinct neurodevelopmental trajectories in ASD in the brain structural connectome. The main findings are as follows: (1) Cross-sectional trajectories of the network characteristics exhibited inverted U-shapes in both HCs and participants with ASD, with the latter exhibiting a 7-year delay in reaching the maximum value, (2) Network resilience to targeted attacks peaked at 18' and 19' in the HCs and at 25' in the participants with ASD, and the weakest resilience occurred at age 7', (3) The HCs and participants with ASD exhibited normalized mean degree differences in the right amygdala, and (4) significant differences in the network characteristics were observed in the 18' age group at most of the densities analyzed.

Comparison of subtyping methods for neuroimaging studies in Alzheimer's disease: a call for harmonization

Biological subtypes in Alzheimer's disease, originally identified on neuropathological data, have been translated to in vivo biomarkers such as structural magnetic resonance imaging and positron emission tomography, to disentangle the heterogeneity within Alzheimer's disease. Although there is methodological variability across studies, comparable characteristics of subtypes are reported at the group level. In this study, we investigated whether group-level similarities translate to individual-level agreement across subtyping methods, in a head-to-head context. We compared five previously published subtyping methods. Firstly, we validated the subtyping methods in 89 amyloid-beta positive Alzheimer's disease dementia patients (reference group: 70 amyloid-beta negative healthy individuals) using structural magnetic resonance imaging. Secondly, we extended and applied the subtyping methods to 53 amyloid-beta positive prodromal Alzheimer's disease and 30 amyloid-beta positive Alzheimer's disease dementia patients (reference group: 200 amyloid-beta negative healthy individuals) using structural magnetic resonance imaging and tau positron emission tomography. Subtyping methods were implemented as outlined in each original study. Group-level and individual-level comparisons across methods were performed. Each individual subtyping method was replicated, and the proof-of-concept was established. At the group level, all methods captured subtypes with similar patterns of demographic and clinical characteristics, and with similar cortical thinning and tau positron emission tomography uptake patterns. However, at the individual level, large disagreements were found in subtype assignments. Although characteristics of subtypes are comparable at the group level, there is a large disagreement at the individual level across subtyping methods. Therefore, there is an urgent need for consensus and harmonization across subtyping methods. We call for the establishment of an open benchmarking framework to overcome this problem.

Medial temporal atrophy in preclinical dementia: Visual and automated assessment during six year follow-up

•

Visual MTA ratings can detect longitudinal changes in preclinical dementia patients.

•

Medial temporal atrophy rate is greater in individuals with an AD biomarker profile.

•

Visual MTA ratings provide a robust alternative to automated measures.

Medial temporal lobe (MTL) atrophy is an important morphological marker of many dementias and is closely related to cognitive decline. In this study we aimed to characterize longitudinal progression of MTL atrophy in 93 individuals with subjective cognitive decline and mild cognitive impairment followed up over six years, and to assess if clinical rating scales are able to detect these changes. All MRI images were visually rated according to Scheltens’ scale of medial temporal atrophy (MTA) by two neuroradiologists and AVRA, a software for automated MTA ratings. The images were also segmented using FreeSurfer’s longitudinal pipeline in order to compare the MTA ratings to volumes of the hippocampi and inferior lateral ventricles. We found that MTL atrophy rates increased with CSF biomarker abnormality, used to define preclinical stages of Alzheimer’s Disease. Both AVRA’s and the radiologists’ MTA ratings showed similar longitudinal trends as the subcortical volumes, suggesting that visual rating scales provide a valid alternative to automatic segmentations. Our results further showed that it took more than 8 years on average for individuals with mild cognitive impairment, and an Alzheimer’s disease biomarker profile, to increase the MTA score by one. This suggests that discrete MTA ratings are too coarse for tracking individual MTL atrophy in short time spans. While the MTA scores from each radiologist showed strong correlations to subcortical volumes, the inter-rater agreement was low. We conclude that the main limitation of quantifying MTL atrophy with visual ratings in clinics is the subjectiveness of the assessment.