Domain-specific data augmentation for segmenting MR images of fatty infiltrated human thighs with neural networks

Synthesis of Hybrid Data Consisting of Chest Radiographs and Tabular Clinical Records Using Dual Generative Models for COVID-19 Positive Cases

To generate synthetic medical data incorporating image-tabular hybrid data by merging an image encoding/decoding model with a table-compatible generative model and assess their utility. We used 1342 cases from the Stony Brook University Covid-19-positive cases, comprising chest X-ray radiographs (CXRs) and tabular clinical data as a private dataset (pDS). We generated a synthetic dataset (sDS) through the following steps: (I) dimensionally reducing CXRs in the pDS using a pretrained encoder of the auto-encoding generative adversarial networks (αGAN) and integrating them with the correspondent tabular clinical data; (II) training the conditional tabular GAN (CTGAN) on this combined data to generate synthetic records, encompassing encoded image features and clinical data; and (III) reconstructing synthetic images from these encoded image features in the sDS using a pretrained decoder of the αGAN. The utility of sDS was assessed by the performance of the prediction models for patient outcomes (deceased or discharged). For the pDS test set, the area under the receiver operating characteristic (AUC) curve was calculated to compare the performance of prediction models trained separately with pDS, sDS, or a combination of both. We created an sDS comprising CXRs with a resolution of 256 × 256 pixels and tabular data containing 13 variables. The AUC for the outcome was 0.83 when the model was trained with the pDS, 0.74 with the sDS, and 0.87 when combining pDS and sDS for training. Our method is effective for generating synthetic records consisting of both images and tabular clinical data.

The Impact of Fatty Infiltration on MRI Segmentation of Lower Limb Muscles in Neuromuscular Diseases: A Comparative Study of Deep Learning Approaches

BACKGROUND

Deep learning methods have been shown to be useful for segmentation of lower limb muscle MRIs of healthy subjects but, have not been sufficiently evaluated on neuromuscular disease (NDM) patients.

PURPOSE

Evaluate the influence of fat infiltration on convolutional neural network (CNN) segmentation of MRIs from NMD patients.

STUDY TYPE

Retrospective study.

SUBJECTS

Data were collected from a hospital database of 67 patients with NMDs and 14 controls (age: 53 ± 17 years, sex: 48 M, 33 F). Ten individual muscles were segmented from the thigh and six from the calf (20 slices, 200 cm section).

FIELD STRENGTH/SEQUENCE

A 1.5 T. Sequences: 2D T1 -weighted fast spin echo. Fat fraction (FF): three-point Dixon 3D GRE, magnetization transfer ratio (MTR): 3D MT-prepared GRE, T2: 2D multispin-echo sequence.

ASSESSMENT

U-Net 2D, U-Net 3D, TransUNet, and HRNet were trained to segment thigh and leg muscles (101/11 and 95/11 training/validation images, 10-fold cross-validation). Automatic and manual segmentations were compared based on geometric criteria (Dice coefficient [DSC], outlier rate, absence rate) and reliability of measured MRI quantities (FF, MTR, T2, volume).

STATISTICAL TESTS

Bland-Altman plots were chosen to describe agreement between manual vs. automatic estimated FF, MTR, T2 and volume. Comparisons were made between muscle populations with an FF greater than 20% (G20+) and lower than 20% (G20-).

RESULTS

The CNNs achieved equivalent results, yet only HRNet recognized every muscle in the database, with a DSC of 0.91 ± 0.08, and measurement biases reaching -0.32% ± 0.92% for FF, 0.19 ± 0.77 for MTR, -0.55 ± 1.95 msec for T2, and - 0.38 ± 3.67 cm3 for volume. The performances of HRNet, between G20- and G20+ decreased significantly.

DATA CONCLUSION

HRNet was the most appropriate network, as it did not omit any muscle. The accuracy obtained shows that CNNs could provide fully automated methods for studying NMDs. However, the accuracy of the methods may be degraded on the most infiltrated muscles (>20%).

EVIDENCE LEVEL

4.

TECHNICAL EFFICACY

Stage 1.

Artificial Intelligence Applications in the Diagnosis of Neuromuscular Diseases: A Narrative Review

The accurate diagnosis of neuromuscular diseases (NMD) is in many cases difficult; the starting point is the clinical approach based on the course of the disease and a careful physical examination of the patient. Electrodiagnostic tests, imaging, muscle biopsy, and genetics are fundamental complementary studies for the diagnosis of NMD. The large volume of data obtained from such studies makes it necessary to look for efficient solutions, such as artificial intelligence (AI) applications, which can help classify, synthesize, and organize the information of patients with NMD to facilitate their accurate and timely diagnosis. The objective of this study was to describe the usefulness of AI applications in the diagnosis of patients with neuromuscular diseases. A narrative review was done, including publications on artificial intelligence applied to the diagnostic methods of NMD currently existing. Twelve studies were included. Two of the studies focused on muscle ultrasound, five of the studies on muscle MRI, two studies on electromyography, two studies on amyotrophic lateral sclerosis (ALS) biomarkers, and one study on genes related to myopathies. The accuracy of classification using different classification algorithms used in each of the studies included in this narrative review was already 90% in most studies. In conclusion, the future design of more accurate algorithms applied to NMD with greater precision will have an impact on the earlier diagnosis of this group of diseases.

Data augmentation for medical imaging: A systematic literature review

Recent advances in Deep Learning have largely benefited from larger and more diverse training sets. However, collecting large datasets for medical imaging is still a challenge due to privacy concerns and labeling costs. Data augmentation makes it possible to greatly expand the amount and variety of data available for training without actually collecting new samples. Data augmentation techniques range from simple yet surprisingly effective transformations such as cropping, padding, and flipping, to complex generative models. Depending on the nature of the input and the visual task, different data augmentation strategies are likely to perform differently. For this reason, it is conceivable that medical imaging requires specific augmentation strategies that generate plausible data samples and enable effective regularization of deep neural networks. Data augmentation can also be used to augment specific classes that are underrepresented in the training set, e.g., to generate artificial lesions. The goal of this systematic literature review is to investigate which data augmentation strategies are used in the medical domain and how they affect the performance of clinical tasks such as classification, segmentation, and lesion detection. To this end, a comprehensive analysis of more than 300 articles published in recent years (2018-2022) was conducted. The results highlight the effectiveness of data augmentation across organs, modalities, tasks, and dataset sizes, and suggest potential avenues for future research.

Multi-Contrast MRI Segmentation Trained on Synthetic Images

In our comprehensive experiments and evaluations, we show that it is possible to generate multiple contrast (even all synthetically) and use synthetically generated images to train an image segmentation engine. We showed promising segmentation results tested on real multi-contrast MRI scans when delineating muscle, fat, bone and bone marrow, all trained on synthetic images. Based on synthetic image training, our segmentation results were as high as 93.91%, 94.11%, 91.63%, 95.33%, for muscle, fat, bone, and bone marrow delineation, respectively. Results were not significantly different from the ones obtained when real images were used for segmentation training: 94.68%, 94.67%, 95.91%, and 96.82%, respectively. Clinical relevance- Synthetically generated images could potentially be used in large-scale training of deep networks for segmentation purpose. Small data set problem of many clinical imaging problems can potentially be addressed with the proposed algorithm.

Generative adversarial networks in medical image segmentation: A review

PURPOSE

Since Generative Adversarial Network (GAN) was introduced into the field of deep learning in 2014, it has received extensive attention from academia and industry, and a lot of high-quality papers have been published. GAN effectively improves the accuracy of medical image segmentation because of its good generating ability and capability to capture data distribution. This paper introduces the origin, working principle, and extended variant of GAN, and it reviews the latest development of GAN-based medical image segmentation methods.

METHOD

To find the papers, we searched on Google Scholar and PubMed with the keywords like "segmentation", "medical image", and "GAN (or generative adversarial network)". Also, additional searches were performed on Semantic Scholar, Springer, arXiv, and the top conferences in computer science with the above keywords related to GAN.

RESULTS

We reviewed more than 120 GAN-based architectures for medical image segmentation that were published before September 2021. We categorized and summarized these papers according to the segmentation regions, imaging modality, and classification methods. Besides, we discussed the advantages, challenges, and future research directions of GAN in medical image segmentation.

CONCLUSIONS

We discussed in detail the recent papers on medical image segmentation using GAN. The application of GAN and its extended variants has effectively improved the accuracy of medical image segmentation. Obtaining the recognition of clinicians and patients and overcoming the instability, low repeatability, and uninterpretability of GAN will be an important research direction in the future.

Automated major psoas muscle volumetry in computed tomography using machine learning algorithms

Purpose

The psoas major muscle (PMM) volume serves as an opportunistic imaging marker in cross-sectional imaging datasets for various clinical applications. Since manual segmentation is time consuming, two different automated segmentation methods, a generative adversarial network architecture (GAN) and a multi-atlas segmentation (MAS), as well as a combined approach of both, were investigated in terms of accuracy of automated volumetrics in given CT datasets.

Materials and methods

The bilateral PMM was manually segmented by a radiologist in 34 abdominal CT scans, resulting in 68 single 3D muscle segmentations as training data. Three different methods were tested for their ability to generate automated image segmentations: a GAN- and MAS-based approach and a combined approach of both methods (COM). Bilateral PMM volume (PMMV) was calculated in cm³ by each algorithm for every CT. Results were compared to the corresponding ground truth using the Dice similarity coefficient (DSC), Spearman’s correlation coefficient and Wilcoxon signed-rank test.

Results

Mean PMMV was 239 ± 7.0 cm³ and 308 ± 9.6 cm³, 306 ± 9.5 cm³ and 243 ± 7.3 cm³ for the CNN, MAS and COM, respectively. Compared to the ground truth the CNN and MAS overestimated the PMMV significantly (+ 28.9% and + 28.0%, p < 0.001), while results of the COM were quite accurate (+ 0.7%, p = 0.33). Spearman’s correlation coefficients were 0.38, 0.62 and 0.73, and the DSCs were 0.75 [95%CI: 0.56–0.88], 0.73 [95%CI: 0.54–0.85] and 0.82 [95%CI: 0.65–0.90] for the CNN, MAS and COM, respectively.

Conclusion

The combined approach was able to efficiently exploit the advantages of both methods (GAN and MAS), resulting in a significantly higher accuracy in PMMV predictions compared to the isolated implementations of both methods. Even with the relatively small set of training data, the segmentation accuracy of this hybrid approach was relatively close to that of the radiologist.

Supplementary Information

The online version contains supplementary material available at 10.1007/s11548-021-02539-2.

Deep Generative Adversarial Networks: Applications in Musculoskeletal Imaging

In recent years, deep learning techniques have been applied in musculoskeletal radiology to increase the diagnostic potential of acquired images. Generative adversarial networks (GANs), which are deep neural networks that can generate or transform images, have the potential to aid in faster imaging by generating images with a high level of realism across multiple contrast and modalities from existing imaging protocols. This review introduces the key architectures of GANs as well as their technical background and challenges. Key research trends are highlighted, including: (a) reconstruction of high-resolution MRI; (b) image synthesis with different modalities and contrasts; (c) image enhancement that efficiently preserves high-frequency information suitable for human interpretation; (d) pixel-level segmentation with annotation sharing between domains; and (e) applications to different musculoskeletal anatomies. In addition, an overview is provided of the key issues wherein clinical applicability is challenging to capture with conventional performance metrics and expert evaluation. When clinically validated, GANs have the potential to improve musculoskeletal imaging. Keywords: Adults and Pediatrics, Computer Aided Diagnosis (CAD), Computer Applications-General (Informatics), Informatics, Skeletal-Appendicular, Skeletal-Axial, Soft Tissues/Skin © RSNA, 2021.

Overview of MR Image Segmentation Strategies in Neuromuscular Disorders

Neuromuscular disorders are rare diseases for which few therapeutic strategies currently exist. Assessment of therapeutic strategies efficiency is limited by the lack of biomarkers sensitive to the slow progression of neuromuscular diseases (NMD). Magnetic resonance imaging (MRI) has emerged as a tool of choice for the development of qualitative scores for the study of NMD. The recent emergence of quantitative MRI has enabled to provide quantitative biomarkers more sensitive to the evaluation of pathological changes in muscle tissue. However, in order to extract these biomarkers from specific regions of interest, muscle segmentation is mandatory. The time-consuming aspect of manual segmentation has limited the evaluation of these biomarkers on large cohorts. In recent years, several methods have been proposed to make the segmentation step automatic or semi-automatic. The purpose of this study was to review these methods and discuss their reliability, reproducibility, and limitations in the context of NMD. A particular attention has been paid to recent deep learning methods, as they have emerged as an effective method of image segmentation in many other clinical contexts.

Realistic High-Resolution Body Computed Tomography Image Synthesis by Using Progressive Growing Generative Adversarial Network: Visual Turing Test

BACKGROUND

Generative adversarial network (GAN)-based synthetic images can be viable solutions to current supervised deep learning challenges. However, generating highly realistic images is a prerequisite for these approaches.

OBJECTIVE

The aim of this study was to investigate and validate the unsupervised synthesis of highly realistic body computed tomography (CT) images by using a progressive growing GAN (PGGAN) trained to learn the probability distribution of normal data.

METHODS

We trained the PGGAN by using 11,755 body CT scans. Ten radiologists (4 radiologists with <5 years of experience [Group I], 4 radiologists with 5-10 years of experience [Group II], and 2 radiologists with >10 years of experience [Group III]) evaluated the results in a binary approach by using an independent validation set of 300 images (150 real and 150 synthetic) to judge the authenticity of each image.

RESULTS

The mean accuracy of the 10 readers in the entire image set was higher than random guessing (1781/3000, 59.4% vs 1500/3000, 50.0%, respectively; P<.001). However, in terms of identifying synthetic images as fake, there was no significant difference in the specificity between the visual Turing test and random guessing (779/1500, 51.9% vs 750/1500, 50.0%, respectively; P=.29). The accuracy between the 3 reader groups with different experience levels was not significantly different (Group I, 696/1200, 58.0%; Group II, 726/1200, 60.5%; and Group III, 359/600, 59.8%; P=.36). Interreader agreements were poor (κ=0.11) for the entire image set. In subgroup analysis, the discrepancies between real and synthetic CT images occurred mainly in the thoracoabdominal junction and in the anatomical details.

CONCLUSIONS

The GAN can synthesize highly realistic high-resolution body CT images that are indistinguishable from real images; however, it has limitations in generating body images of the thoracoabdominal junction and lacks accuracy in the anatomical details.

A deep learning model for diagnosing dystrophinopathies on thigh muscle MRI images

Background

Dystrophinopathies are the most common type of inherited muscular diseases. Muscle biopsy and genetic tests are effective to diagnose the disease but cost much more than primary hospitals can reach. The more available muscle MRI is promising but its diagnostic results highly depends on doctors’ experiences. This study intends to explore a way of deploying a deep learning model for muscle MRI images to diagnose dystrophinopathies.

Methods

This study collected 2536 T1WI images from 432 cases who had been diagnosed by genetic analysis and/or muscle biopsy, including 148 cases with dystrophinopathies and 284 cases with other diseases. The data was randomly divided into three sets: the data from 233 cases were used to train the CNN model, the data from 97 cases for the validation experiments, and the data from 102 cases for the test experiments. We also validated our models expertise at diagnosing by comparing the model’s results on the 102 cases with those of three skilled radiologists.

Results

The proposed model achieved 91% (95% CI: 0.88, 0.93) accuracy on the test set, higher than the best accuracy of 84% in radiologists. It also performed better than the skilled radiologists in sensitivity : sensitivities of the models and the doctors were 0.89 (95% CI: 0.85 0.93) versus 0.79 (95% CI:0.73, 0.84; p = 0.190).

Conclusions

The deep model achieved excellent accuracy and sensitivity in identifying cases with dystrophinopathies. The comparable performance of the model and skilled radiologists demonstrates the potential application of the model in diagnosing dystrophinopathies through MRI images.

Supervised segmentation framework for evaluation of diffusion tensor imaging indices in skeletal muscle

Diffusion tensor imaging (DTI) is becoming a relevant diagnostic tool to understand muscle disease and map muscle recovery processes following physical activity or after injury. Segmenting all the individual leg muscles, necessary for quantification, is still a time-consuming manual process. The purpose of this study was to evaluate the impact of a supervised semi-automatic segmentation pipeline on the quantification of DTI indices in individual upper leg muscles. Longitudinally acquired MRI datasets (baseline, post-marathon and follow-up) of the upper legs of 11 subjects were used in this study. MR datasets consisted of a DTI and Dixon acquisition. Semi-automatic segmentations for the upper leg muscles were performed using a transversal propagation approach developed by Ogier et al on the out-of-phase Dixon images at baseline. These segmentations were longitudinally propagated for the post-marathon and follow-up time points. Manual segmentations were performed on the water image of the Dixon for each of the time points. Dice similarity coefficients (DSCs) were calculated to compare the manual and semi-automatic segmentations. Bland-Altman and regression analyses were performed, to evaluate the impact of the two segmentation methods on mean diffusivity (MD), fractional anisotropy (FA) and the third eigenvalue (λ₃ ). The average DSC for all analyzed muscles over all time points was 0.92 ± 0.01, ranging between 0.48 and 0.99. Bland-Altman analysis showed that the 95% limits of agreement for MD, FA and λ₃ ranged between 0.5% and 3.0% for the transversal propagation and between 0.7% and 3.0% for the longitudinal propagations. Similarly, regression analysis showed good correlation for MD, FA and λ₃ (r = 0.99, p < 60; 0.0001). In conclusion, the supervised semi-automatic segmentation framework successfully quantified DTI indices in the upper-leg muscles compared with manual segmentation while only requiring manual input of 30% of the slices, resulting in a threefold reduction in segmentation time.

Segmentation of the fascia lata and reproducible quantification of intermuscular adipose tissue (IMAT) of the thigh

Objective

To develop a precise semi-automated segmentation of the fascia lata (FL) of the thigh to quantify IMAT volume in T₁w MR images and fat fraction (FF) in Dixon MR images.

Materials and methods

A multi-step segmentation approach was developed to identify fibrous structures of the FL and combining them into a closed 3D surface. 23 healthy young men with low and 50 elderly sarcopenic men with moderate levels of IMAT were measured by T₁w and 6pt Dixon MRI at 3T. 20 datasets were used to determine reanalysis precision errors. IMAT volume was compared using the new FL segmentation versus an easier to segment but less accurate, tightly fitting envelope of the thigh muscle ensemble.

Results

The segmentation was successfully applied to all 73 datasets and took about 7 min per 28 slices. In particular, in elderly subjects, it includes a large amount of adipose tissue below the FL typically not accounted for in other segmentation approaches. Inter- and intra-operator RMS-CVs were 0.33% and 0.14%, respectively, for IMAT volume and 0.04% and 0.02%, respectively, for FF_MT.

Discussion

The FL segmentation is an important step to quantify IMAT with high precision and may be useful to investigate effects of aging and treatment on changes of IMAT and FF. ClinicalTrials.gov identifier NCT2857660, August 5, 2016.

Trial registration

ClinicalTrials.gov identifier NCT2857660, August 5, 2016.

Creating Artificial Images for Radiology Applications Using Generative Adversarial Networks (GANs) - A Systematic Review

RATIONALE AND OBJECTIVES

Generative adversarial networks (GANs) are deep learning models aimed at generating fake realistic looking images. These novel models made a great impact on the computer vision field. Our study aims to review the literature on GANs applications in radiology.

MATERIALS AND METHODS

This systematic review followed the PRISMA guidelines. Electronic datasets were searched for studies describing applications of GANs in radiology. We included studies published up-to September 2019.

RESULTS

Data were extracted from 33 studies published between 2017 and 2019. Eighteen studies focused on CT images generation, ten on MRI, three on PET/MRI and PET/CT, one on ultrasound and one on X-ray. Applications in radiology included image reconstruction and denoising for dose and scan time reduction (fourteen studies), data augmentation (six studies), transfer between modalities (eight studies) and image segmentation (five studies). All studies reported that generated images improved the performance of the developed algorithms.

CONCLUSION

GANs are increasingly studied for various radiology applications. They enable the creation of new data, which can be used to improve clinical care, education and research.