Generation of synthetic PET/MR fusion images from MR images using a combination of generative adversarial networks and conditional denoising diffusion probabilistic models based on simultaneous 18F-FDG PET/MR image data of pyogenic spondylodiscitis

BACKGROUND CONTEXT

Cross-modality image generation from magnetic resonance (MR) to positron emission tomography (PET) using the generative model can be expected to have complementary effects by addressing the limitations and maximizing the advantages inherent in each modality.

PURPOSE

This study aims to generate synthetic PET/MR fusion images from MR images using a combination of generative adversarial networks (GANs) and conditional denoising diffusion probabilistic models (cDDPMs) based on simultaneous 18F-fluorodeoxyglucose (18F-FDG) PET/MR image data.

STUDY DESIGN

Retrospective study with prospectively collected clinical and radiological data.

PATIENT SAMPLE

This study included 94 patients (60 men and 34 women) with thoraco-lumbar pyogenic spondylodiscitis (PSD) from February 2017 to January 2020 in a single tertiary institution.

OUTCOME MEASURES

Quantitative and qualitative image similarity were analyzed between the real and synthetic PET/ T2-weighted fat saturation MR (T2FS) fusion images on the test data set.

METHODS

We used paired spinal sagittal T2FS and PET/T2FS fusion images of simultaneous 18F-FDG PET/MR imaging examination in patients with PSD, which were employed to generate synthetic PET/T2FS fusion images from T2FS images using a combination of Pix2Pix (U-Net generator + Least Squares GANs discriminator) and cDDPMs algorithms. In the analyses of image similarity between the real and synthetic PET/T2FS fusion images, we adopted the values of mean peak signal to noise ratio (PSNR), mean structural similarity measurement (SSIM), mean absolute error (MAE), and mean squared error (MSE) for quantitative analysis, while the discrimination accuracy by three spine surgeons was applied for qualitative analysis.

RESULTS

Total 2082 pairs of T2FS and PET/T2FS fusion images were obtained from 172 examinations on 94 patients, which were randomly assigned to training, validation, and test data sets in 8:1:1 ratio (1664, 209, and 209 pairs). The quantitative analysis revealed PSNR of 30.634 ± 3.437, SSIM of 0.910 ± 0.067, MAE of 0.017 ± 0.008, and MSE of 0.001 ± 0.001, respectively. The values of PSNR, MAE, and MSE significantly decreased as FDG uptake increase in real PET/T2FS fusion image, with no significant correlation on SSIM. In the qualitative analysis, the overall discrimination accuracy between real and synthetic PET/T2FS fusion images was 47.4%.

CONCLUSIONS

The combination of Pix2Pix and cDDPMs demonstrated the potential for cross-modal image generation from MR to PET images, with reliable quantitative and qualitative image similarities.

Generative artificial intelligence to produce high-fidelity blastocyst-stage embryo images

STUDY QUESTION

Can generative artificial intelligence (AI) models produce high-fidelity images of human blastocysts?

SUMMARY ANSWER

Generative AI models exhibit the capability to generate high-fidelity human blastocyst images, thereby providing substantial training datasets crucial for the development of robust AI models.

WHAT IS KNOWN ALREADY

The integration of AI into IVF procedures holds the potential to enhance objectivity and automate embryo selection for transfer. However, the effectiveness of AI is limited by data scarcity and ethical concerns related to patient data privacy. Generative adversarial networks (GAN) have emerged as a promising approach to alleviate data limitations by generating synthetic data that closely approximate real images.

STUDY DESIGN, SIZE, DURATION

Blastocyst images were included as training data from a public dataset of time-lapse microscopy (TLM) videos (n = 136). A style-based GAN was fine-tuned as the generative model.

PARTICIPANTS/MATERIALS, SETTING, METHODS

We curated a total of 972 blastocyst images as training data, where frames were captured within the time window of 110-120 h post-insemination at 1-h intervals from TLM videos. We configured the style-based GAN model with data augmentation (AUG) and pretrained weights (Pretrained-T: with translation equivariance; Pretrained-R: with translation and rotation equivariance) to compare their optimization on image synthesis. We then applied quantitative metrics including Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) to assess the quality and fidelity of the generated images. Subsequently, we evaluated qualitative performance by measuring the intelligence behavior of the model through the visual Turing test. To this end, 60 individuals with diverse backgrounds and expertise in clinical embryology and IVF evaluated the quality of synthetic embryo images.

MAIN RESULTS AND THE ROLE OF CHANCE

During the training process, we observed consistent improvement of image quality that was measured by FID and KID scores. Pretrained and AUG + Pretrained initiated with remarkably lower FID and KID values compared to both Baseline and AUG + Baseline models. Following 5000 training iterations, the AUG + Pretrained-R model showed the highest performance of the evaluated five configurations with FID and KID scores of 15.2 and 0.004, respectively. Subsequently, we carried out the visual Turing test, such that IVF embryologists, IVF laboratory technicians, and non-experts evaluated the synthetic blastocyst-stage embryo images and obtained similar performance in specificity with marginal differences in accuracy and sensitivity.

LIMITATIONS, REASONS FOR CAUTION

In this study, we primarily focused the training data on blastocyst images as IVF embryos are primarily assessed in blastocyst stage. However, generation of an array of images in different preimplantation stages offers further insights into the development of preimplantation embryos and IVF success. In addition, we resized training images to a resolution of 256 × 256 pixels to moderate the computational costs of training the style-based GAN models. Further research is needed to involve a more extensive and diverse dataset from the formation of the zygote to the blastocyst stage, e.g. video generation, and the use of improved image resolution to facilitate the development of comprehensive AI algorithms and to produce higher-quality images.

WIDER IMPLICATIONS OF THE FINDINGS

Generative AI models hold promising potential in generating high-fidelity human blastocyst images, which allows the development of robust AI models as it can provide sufficient training datasets while safeguarding patient data privacy. Additionally, this may help to produce sufficient embryo imaging training data with different (rare) abnormal features, such as embryonic arrest, tripolar cell division to avoid class imbalances and reach to even datasets. Thus, generative models may offer a compelling opportunity to transform embryo selection procedures and substantially enhance IVF outcomes.

STUDY FUNDING/COMPETING INTEREST(S)

This study was supported by a Horizon 2020 innovation grant (ERIN, grant no. EU952516) and a Horizon Europe grant (NESTOR, grant no. 101120075) of the European Commission to A.S. and M.Z.E., the Estonian Research Council (grant no. PRG1076) to A.S., and the EVA (Erfelijkheid Voortplanting & Aanleg) specialty program (grant no. KP111513) of Maastricht University Medical Centre (MUMC+) to M.Z.E.

TRIAL REGISTRATION NUMBER

Not applicable.

Interpreting generative adversarial networks to infer natural selection from genetic data

Understanding natural selection and other forms of non-neutrality is a major focus for the use of machine learning in population genetics. Existing methods rely on computationally intensive simulated training data. Unlike efficient neutral coalescent simulations for demographic inference, realistic simulations of selection typically require slow forward simulations. Because there are many possible modes of selection, a high dimensional parameter space must be explored, with no guarantee that the simulated models are close to the real processes. Finally, it is difficult to interpret trained neural networks, leading to a lack of understanding about what features contribute to classification. Here we develop a new approach to detect selection and other local evolutionary processes that requires relatively few selection simulations during training. We build upon a generative adversarial network trained to simulate realistic neutral data. This consists of a generator (fitted demographic model), and a discriminator (convolutional neural network) that predicts whether a genomic region is real or fake. As the generator can only generate data under neutral demographic processes, regions of real data that the discriminator recognizes as having a high probability of being "real" do not fit the neutral demographic model and are therefore candidates for targets of selection. To incentivize identification of a specific mode of selection, we fine-tune the discriminator with a small number of custom non-neutral simulations. We show that this approach has high power to detect various forms of selection in simulations, and that it finds regions under positive selection identified by state-of-the-art population genetic methods in three human populations. Finally, we show how to interpret the trained networks by clustering hidden units of the discriminator based on their correlation patterns with known summary statistics.

Virtual MOLLI Target: Generative Adversarial Networks Toward Improved Motion Correction in MRI Myocardial T1 Mapping

BACKGROUND

The modified Look-Locker inversion recovery (MOLLI) sequence is commonly used for myocardial T1 mapping. However, it acquires images with different inversion times, which causes difficulty in motion correction for respiratory-induced misregistration to a given target image.

HYPOTHESIS

Using a generative adversarial network (GAN) to produce virtual MOLLI images with consistent heart positions can reduce respiratory-induced misregistration of MOLLI datasets.

STUDY TYPE

Retrospective.

POPULATION

1071 MOLLI datasets from 392 human participants.

FIELD STRENGTH/SEQUENCE

Modified Look-Locker inversion recovery sequence at 3 T.

ASSESSMENT

A GAN model with a single inversion time image as input was trained to generate virtual MOLLI target (VMT) images at different inversion times which were subsequently used in an image registration algorithm. Four VMT models were investigated and the best performing model compared with the standard vendor-provided motion correction (MOCO) technique.

STATISTICAL TESTS

The effectiveness of the motion correction technique was assessed using the fitting quality index (FQI), mutual information (MI), and Dice coefficients of motion-corrected images, plus subjective quality evaluation of T1 maps by three independent readers using Likert score. Wilcoxon signed-rank test with Bonferroni correction for multiple comparison. Significance levels were defined as P < 0.01 for highly significant differences and P < 0.05 for significant differences.

RESULTS

The best performing VMT model with iterative registration demonstrated significantly better performance (FQI 0.88 ± 0.03, MI 1.78 ± 0.20, Dice 0.84 ± 0.23, quality score 2.26 ± 0.95) compared to other approaches, including the vendor-provided MOCO method (FQI 0.86 ± 0.04, MI 1.69 ± 0.25, Dice 0.80 ± 0.27, quality score 2.16 ± 1.01).

DATA CONCLUSION

Our GAN model generating VMT images improved motion correction, which may assist reliable T1 mapping in the presence of respiratory motion. Its robust performance, even with considerable respiratory-induced heart displacements, may be beneficial for patients with difficulties in breath-holding.

LEVEL OF EVIDENCE

3 TECHNICAL EFFICACY: Stage 1.

A Sustainable Approach to Asthma Diagnosis: Classification with Data Augmentation, Feature Selection, and Boosting Algorithm

Asthma is a diverse disease that affects over 300 million individuals globally. The prevalence of asthma has increased by 50% every decade since the 1960s, making it a serious global health issue. In addition to its associated high mortality, asthma generates large economic losses due to the degradation of patients' quality of life and the impairment of their physical fitness. Asthma research has evolved in recent years to fully analyze why certain diseases develop based on a variety of data and observations of patients' performance. The advent of new techniques offers good opportunities and application prospects for the development of asthma diagnosis methods. Over the last few decades, techniques like data mining and machine learning have been utilized to diagnose asthma. Nevertheless, these traditional methods are unable to address all of the difficulties associated with improving a small dataset to increase its quantity, quality, and feature space complexity at the same time. In this study, we propose a sustainable approach to asthma diagnosis using advanced machine learning techniques. To be more specific, we use feature selection to find the most important features, data augmentation to improve the dataset's resilience, and the extreme gradient boosting algorithm for classification. Data augmentation in the proposed method involves generating synthetic samples to increase the size of the training dataset, which is then utilized to enhance the training data initially. This could lessen the phenomenon of imbalanced data related to asthma. Then, to improve diagnosis accuracy and prioritize significant features, the extreme gradient boosting technique is used. The outcomes indicate that the proposed approach performs better in terms of diagnostic accuracy than current techniques. Furthermore, five essential features are extracted to help physicians diagnose asthma.

Impact of bias field correction on 0.35 T pelvic MR images: evaluation on generative adversarial network-based OARs' auto-segmentation and visual grading assessment

Purpose

Magnetic resonance imaging (MRI)-guided radiotherapy enables adaptive treatment plans based on daily anatomical changes and accurate organ visualization. However, the bias field artifact can compromise image quality, affecting diagnostic accuracy and quantitative analyses. This study aims to assess the impact of bias field correction on 0.35 T pelvis MRIs by evaluating clinical anatomy visualization and generative adversarial network (GAN) auto-segmentation performance.

Materials and methods

3D simulation MRIs from 60 prostate cancer patients treated on MR-Linac (0.35 T) were collected and preprocessed with the N4ITK algorithm for bias field correction. A 3D GAN architecture was trained, validated, and tested on 40, 10, and 10 patients, respectively, to auto-segment the organs at risk (OARs) rectum and bladder. The GAN was trained and evaluated either with the original or the bias-corrected MRIs. The Dice similarity coefficient (DSC) and 95th percentile Hausdorff distance (HD95th) were computed for the segmented volumes of each patient. The Wilcoxon signed-rank test assessed the statistical difference of the metrics within OARs, both with and without bias field correction. Five radiation oncologists blindly scored 22 randomly chosen patients in terms of overall image quality and visibility of boundaries (prostate, rectum, bladder, seminal vesicles) of the original and bias-corrected MRIs. Bennett's S score and Fleiss' kappa were used to assess the pairwise interrater agreement and the interrater agreement among all the observers, respectively.

Results

In the test set, the GAN trained and evaluated on original and bias-corrected MRIs showed DSC/HD95th of 0.92/5.63 mm and 0.92/5.91 mm for the bladder and 0.84/10.61 mm and 0.83/9.71 mm for the rectum. No statistical differences in the distribution of the evaluation metrics were found neither for the bladder (DSC: p = 0.07; HD95th: p = 0.35) nor for the rectum (DSC: p = 0.32; HD95th: p = 0.63). From the clinical visual grading assessment, the bias-corrected MRI resulted mostly in either no change or an improvement of the image quality and visualization of the organs' boundaries compared with the original MRI.

Conclusion

The bias field correction did not improve the anatomy visualization from a clinical point of view and the OARs' auto-segmentation outputs generated by the GAN.

Lung CT harmonization of paired reconstruction kernel images using generative adversarial networks

BACKGROUND

The kernel used in CT image reconstruction is an important factor that determines the texture of the CT image. Consistency of reconstruction kernel choice is important for quantitative CT-based assessment as kernel differences can lead to substantial shifts in measurements unrelated to underlying anatomical structures.

PURPOSE

In this study, we investigate kernel harmonization in a multi-vendor low-dose CT lung cancer screening cohort and evaluate our approach's validity in quantitative CT-based assessments.

METHODS

Using the National Lung Screening Trial, we identified CT scan pairs of the same sessions with one reconstructed from a soft tissue kernel and one from a hard kernel. In total, 1000 pairs of five different paired kernel types (200 each) were identified. We adopt the pix2pix architecture to train models for kernel conversion. Each model was trained on 100 pairs and evaluated on 100 withheld pairs. A total of 10 models were implemented. We evaluated the efficacy of kernel conversion based on image similarity metrics including root mean squared error (RMSE), peak signal-to-noise ratio (PSNR), and structural similarity index measure (SSIM) as well as the capability of the models to reduce measurement shifts in quantitative emphysema and body composition measurements. Additionally, we study the reproducibility of standard radiomic features for all kernel pairs before and after harmonization.

RESULTS

Our approach effectively converts CT images from one kernel to another in all paired kernel types, as indicated by the reduction in RMSE (p < 0.05) and an increase in the PSNR (p < 0.05) and SSIM (p < 0.05) for both directions of conversion for all pair types. In addition, there is an increase in the agreement for percent emphysema, skeletal muscle area, and subcutaneous adipose tissue (SAT) area for both directions of conversion. Furthermore, radiomic features were reproducible when compared with the ground truth features.

CONCLUSIONS

Kernel conversion using deep learning reduces measurement variation in percent emphysema, muscle area, and SAT area.

OA-GAN: organ-aware generative adversarial network for synthesizing contrast-enhanced medical images

Contrast-enhanced computed tomography (CE-CT) images are vital for clinical diagnosis of focal liver lesions (FLLs). However, the use of CE-CT images imposes a significant burden on patients due to the injection of contrast agents and extended shooting. Deep learning-based image synthesis models offer a promising solution that synthesizes CE-CT images from non-contrasted CT (NC-CT) images. Unlike natural images, medical image synthesis requires a specific focus on certain organs or localized regions to ensure accurate diagnosis. Determining how to effectively emphasize target organs poses a challenging issue in medical image synthesis. To solve this challenge, we present a novel CE-CT image synthesis model called, Organ-Aware Generative Adversarial Network (OA-GAN). The OA-GAN comprises an organ-aware (OA) network and a dual decoder-based generator. First, the OA network learns the most discriminative spatial features about the target organ (i.e. liver) by utilizing the ground truth organ mask as localization cues. Subsequently, NC-CT image and captured feature are fed into the dual decoder-based generator, which employs a local and global decoder network to simultaneously synthesize the organ and entire CECT image. Moreover, the semantic information extracted from the local decoder is transferred to the global decoder to facilitate better reconstruction of the organ in entire CE-CT image. The qualitative and quantitative evaluation on a CE-CT dataset demonstrates that the OA-GAN outperforms state-of-the-art approaches for synthesizing two types of CE-CT images such as arterial phase and portal venous phase. Additionally, subjective evaluations by expert radiologists and a deep learning-based FLLs classification also affirm that CE-CT images synthesized from the OA-GAN exhibit a remarkable resemblance to real CE-CT images.

Synthesis of gadolinium-enhanced glioma images on multisequence magnetic resonance images using contrastive learning

BACKGROUND

Gadolinium-based contrast agents are commonly used in brain magnetic resonance imaging (MRI), however, they cannot be used by patients with allergic reactions or poor renal function. For long-term follow-up patients, gadolinium deposition in the body can cause nephrogenic systemic fibrosis and other potential risks.

PURPOSE

Developing a new method of enhanced image synthesis based on the advantages of multisequence MRI has important clinical value for these patients. In this paper, an end-to-end synthesis model structure similarity index measure (SSIM)-based Dual Constrastive Learning with Attention (SDACL) based on contrastive learning is proposed to synthesize contrast-enhanced T1 (T1ce) using three unenhanced MRI images of T1, T2, and Flair in patients with glioma.

METHODS

The model uses the attention-dilation generator to enlarge the receptive field by expanding the residual blocks and to strengthen the feature representation and context learning of multisequence MRI. To enhance the detail and texture performance of the imaged tumor area, a comprehensive loss function combining patch-level contrast loss and structural similarity loss is created, which can effectively suppress noise and ensure the consistency of synthesized images and real images.

RESULTS

The normalized root-mean-square error (NRMSE), peak signal-to-noise ratio (PSNR), and SSIM of the model on the independent test set are 0.307  ± $\pm$  0.12, 23.337  ± $\pm$  3.21, and 0.881  ± $\pm$  0.05, respectively.

CONCLUSIONS

Results show this method can be used for the multisequence synthesis of T1ce images, which can provide valuable information for clinical diagnosis.

Semantic and Geometric-Aware Day-to-Night Image Translation Network

Autonomous driving systems heavily depend on perception tasks for optimal performance. However, the prevailing datasets are primarily focused on scenarios with clear visibility (i.e., sunny and daytime). This concentration poses challenges in training deep-learning-based perception models for environments with adverse conditions (e.g., rainy and nighttime). In this paper, we propose an unsupervised network designed for the translation of images from day-to-night to solve the ill-posed problem of learning the mapping between domains with unpaired data. The proposed method involves extracting both semantic and geometric information from input images in the form of attention maps. We assume that the multi-task network can extract semantic and geometric information during the estimation of semantic segmentation and depth maps, respectively. The image-to-image translation network integrates the two distinct types of extracted information, employing them as spatial attention maps. We compare our method with related works both qualitatively and quantitatively. The proposed method shows both qualitative and qualitative improvements in visual presentation over related work.

Predicting materials properties with generative models: applying generative adversarial networks for heat flux generation

In the realm of materials science, the integration of machine learning techniques has ushered in a transformative era. This study delves into the innovative application of generative adversarial networks (GANs) for generating heat flux data, a pivotal step in predicting lattice thermal conductivity within metallic materials. Leveraging GANs, this research explores the generation of meaningful heat flux data, which has a high degree of similarity with that calculated by molecular dynamics simulations. This study demonstrates the potential of artificial intelligence (AI) in understanding the complex physical meaning of data in materials science. By harnessing the power of such AI to generate data that is previously attainable only through experiments or simulations, new opportunities arise for exploring and predicting properties of materials.

Fast Model Selection and Hyperparameter Tuning for Generative Models

Generative models have gained significant attention in recent years. They are increasingly used to estimate the underlying structure of high-dimensional data and artificially generate various kinds of data similar to those from the real world. The performance of generative models depends critically on a good set of hyperparameters. Yet, finding the right hyperparameter configuration can be an extremely time-consuming task. In this paper, we focus on speeding up the hyperparameter search through adaptive resource allocation, early stopping underperforming candidates quickly and allocating more computational resources to promising ones by comparing their intermediate performance. The hyperparameter search is formulated as a non-stochastic best-arm identification problem where resources like iterations or training time constrained by some predetermined budget are allocated to different hyperparameter configurations. A procedure which uses hypothesis testing coupled with Successive Halving is proposed to make the resource allocation and early stopping decisions and compares the intermediate performance of generative models by their exponentially weighted Maximum Means Discrepancy (MMD). The experimental results show that the proposed method selects hyperparameter configurations that lead to a significant improvement in the model performance compared to Successive Halving for a wide range of budgets across several real-world applications.

An unsupervised two-step training framework for low-dose computed tomography denoising

BACKGROUND

Although low-dose computed tomography (CT) imaging has been more widely adopted in clinical practice to reduce radiation exposure to patients, the reconstructed CT images tend to have more noise, which impedes accurate diagnosis. Recently, deep neural networks using convolutional neural networks to reduce noise in the reconstructed low-dose CT images have shown considerable improvement. However, they need a large number of paired normal- and low-dose CT images to fully train the network via supervised learning methods.

PURPOSE

To propose an unsupervised two-step training framework for image denoising that uses low-dose CT images of one dataset and unpaired high-dose CT images from another dataset.

METHODS

Our proposed framework trains the denoising network in two steps. In the first training step, we train the network using 3D volumes of CT images and predict the center CT slice from them. This pre-trained network is used in the second training step to train the denoising network and is combined with the memory-efficient denoising generative adversarial network (DenoisingGAN), which further enhances both objective and perceptual quality.

RESULTS

The experimental results on phantom and clinical datasets show superior performance over the existing traditional machine learning and self-supervised deep learning methods, and the results are comparable to the fully supervised learning methods.

CONCLUSIONS

We proposed a new unsupervised learning framework for low-dose CT denoising, convincingly improving noisy CT images from both objective and perceptual quality perspectives. Because our denoising framework does not require physics-based noise models or system-dependent assumptions, our proposed method can be easily reproduced; consequently, it can also be generally applicable to various CT scanners or dose levels.

TSE-GAN: strain elastography using generative adversarial network for thyroid disease diagnosis

Over the past 35 years, studies conducted worldwide have revealed a threefold increase in the incidence of thyroid cancer. Strain elastography is a new imaging technique to identify benign and malignant thyroid nodules due to its sensitivity to tissue stiffness. However, there are certain limitations of this technique, particularly in terms of standardization of the compression process, evaluation of results and several assumptions used in commercial strain elastography modes for the purpose of simplifying imaging analysis. In this work, we propose a novel conditional generative adversarial network (TSE-GAN) for automatically generating thyroid strain elastograms, which adopts a global-to-local architecture to improve the ability of extracting multi-scale features and develops an adaptive deformable U-net structure in the sub-generator to apply effective deformation. Furthermore, we introduce a Lab-based loss function to induce the networks to generate realistic thyroid elastograms that conform to the probability distribution of the target domain. Qualitative and quantitative assessments are conducted on a clinical dataset provided by Shanghai Sixth People's Hospital. Experimental results demonstrate that thyroid elastograms generated by the proposed TSE-GAN outperform state-of-the-art image translation methods in meeting the needs of clinical diagnostic applications and providing practical value.

SAR Image Generation Method Using DH-GAN for Automatic Target Recognition

In recent years, target recognition technology for synthetic aperture radar (SAR) images has witnessed significant advancements, particularly with the development of convolutional neural networks (CNNs). However, acquiring SAR images requires significant resources, both in terms of time and cost. Moreover, due to the inherent properties of radar sensors, SAR images are often marred by speckle noise, a form of high-frequency noise. To address this issue, we introduce a Generative Adversarial Network (GAN) with a dual discriminator and high-frequency pass filter, named DH-GAN, specifically designed for generating simulated images. DH-GAN produces images that emulate the high-frequency characteristics of real SAR images. Through power spectral density (PSD) analysis and experiments, we demonstrate the validity of the DH-GAN approach. The experimental results show that not only do the SAR image generated using DH-GAN closely resemble the high-frequency component of real SAR images, but the proficiency of CNNs in target recognition, when trained with these simulated images, is also notably enhanced.

Generative Adversarial Network-Based Data Augmentation for Enhancing Wireless Physical Layer Authentication

Wireless physical layer authentication has emerged as a promising approach to wireless security. The topic of wireless node classification and recognition has experienced significant advancements due to the rapid development of deep learning techniques. The potential of using deep learning to address wireless security issues should not be overlooked due to its considerable capabilities. Nevertheless, the utilization of this approach in the classification of wireless nodes is impeded by the lack of available datasets. In this study, we provide two models based on a data-driven approach. First, we used generative adversarial networks to design an automated model for data augmentation. Second, we applied a convolutional neural network to classify wireless nodes for a wireless physical layer authentication model. To verify the effectiveness of the proposed model, we assessed our results using an original dataset as a baseline and a generated synthetic dataset. The findings indicate an improvement of approximately 19% in classification accuracy rate.

Enriching Data Science and Health Care Education: Application and Impact of Synthetic Data Sets Through the Health Gym Project

Large-scale medical data sets are vital for hands-on education in health data science but are often inaccessible due to privacy concerns. Addressing this gap, we developed the Health Gym project, a free and open-source platform designed to generate synthetic health data sets applicable to various areas of data science education, including machine learning, data visualization, and traditional statistical models. Initially, we generated 3 synthetic data sets for sepsis, acute hypotension, and antiretroviral therapy for HIV infection. This paper discusses the educational applications of Health Gym's synthetic data sets. We illustrate this through their use in postgraduate health data science courses delivered by the University of New South Wales, Australia, and a Datathon event, involving academics, students, clinicians, and local health district professionals. We also include adaptable worked examples using our synthetic data sets, designed to enrich hands-on tutorial and workshop experiences. Although we highlight the potential of these data sets in advancing data science education and health care artificial intelligence, we also emphasize the need for continued research into the inherent limitations of synthetic data.

How do personality traits modulate real-world gaze behavior? Generated gaze data shows situation-dependent modulations

It has both scientific and practical benefits to substantiate the theoretical prediction that personality (Big Five) traits systematically modulate gaze behavior in various real-world (working) situations. Nevertheless, previous methods that required controlled situations and large numbers of participants failed to incorporate real-world personality modulation analysis. One cause of this research gap is the mixed effects of individual attributes (e.g., the accumulated attributes of age, gender, and degree of measurement noise) and personality traits in gaze data. Previous studies may have used larger sample sizes to average out the possible concentration of specific individual attributes in some personality traits, and may have imposed control situations to prevent unexpected interactions between these possibly biased individual attributes and complex, realistic situations. Therefore, we generated and analyzed real-world gaze behavior where the effects of personality traits are separated out from individual attributes. In Experiment 1, we successfully provided a methodology for generating such sensor data on head and eye movements for a small sample of participants who performed realistic nonsocial (data-entry) and social (conversation) work tasks (i.e., the first contribution). In Experiment 2, we evaluated the effectiveness of generated gaze behavior for real-world personality modulation analysis. We successfully showed how openness systematically modulates the autocorrelation coefficients of sensor data, reflecting the period of head and eye movements in data-entry and conversation tasks (i.e., the second contribution). We found different openness modulations in the autocorrelation coefficients from the generated sensor data of the two tasks. These modulations could not be detected using real sensor data because of the contamination of individual attributes. In conclusion, our method is a potentially powerful tool for understanding theoretically expected, systematic situation-specific personality modulation of real-world gaze behavior.

TIE-GANs: single-shot quantitative phase imaging using transport of intensity equation with integration of GANs

Significance

Artificial intelligence (AI) has become a prominent technology in computational imaging over the past decade. The expeditious and label-free characteristics of quantitative phase imaging (QPI) render it a promising contender for AI investigation. Though interferometric methodologies exhibit potential efficacy, their implementation involves complex experimental platforms and computationally intensive reconstruction procedures. Hence, non-interferometric methods, such as transport of intensity equation (TIE), are preferred over interferometric methods.

Aim

TIE method, despite its effectiveness, is tedious as it requires the acquisition of many images at varying defocus planes. The proposed methodology holds the ability to generate a phase image utilizing a single intensity image using generative adversarial networks (GANs). We present a method called TIE-GANs to overcome the multi-shot scheme of conventional TIE.

Approach

The present investigation employs the TIE as a QPI methodology, which necessitates reduced experimental and computational efforts. TIE is being used for the dataset preparation as well. The proposed method captures images from different defocus planes for training. Our approach uses an image-to-image translation technique to produce phase maps and is based on GANs. The main contribution of this work is the introduction of GANs with TIE (TIE:GANs) that can give better phase reconstruction results with shorter computation times. This is the first time the GANs is proposed for TIE phase retrieval.

Results

The characterization of the system was carried out with microbeads of 4    μ m size and structural similarity index (SSIM) for microbeads was found to be 0.98. We demonstrated the application of the proposed method with oral cells, which yielded a maximum SSIM value of 0.95. The key characteristics include mean squared error and peak-signal-to-noise ratio values of 140 and 26.42 dB for oral cells and 100 and 28.10 dB for microbeads.

Conclusions

The proposed methodology holds the ability to generate a phase image utilizing a single intensity image. Our method is feasible for digital cytology because of its reported high value of SSIM. Our approach can handle defocused images in such a way that it can take intensity image from any defocus plane within the provided range and able to generate phase map.

Generative Adversarial Networks in Digital Histopathology: Current Applications, Limitations, Ethical Considerations, and Future Directions

Generative adversarial networks (GANs) have gained significant attention in the field of image synthesis, particularly in computer vision. GANs consist of a generative model and a discriminative model trained in an adversarial setting to generate realistic and novel data. In the context of image synthesis, the generator produces synthetic images, whereas the discriminator determines their authenticity by comparing them with real examples. Through iterative training, the generator allows the creation of images that are indistinguishable from real ones, leading to high-quality image generation. Considering their success in computer vision, GANs hold great potential for medical diagnostic applications. In the medical field, GANs can generate images of rare diseases, aid in learning, and be used as visualization tools. GANs can leverage unlabeled medical images, which are large in size, numerous in quantity, and challenging to annotate manually. GANs have demonstrated remarkable capabilities in image synthesis and have the potential to significantly impact digital histopathology. This review article focuses on the emerging use of GANs in digital histopathology, examining their applications and potential challenges. Histopathology plays a crucial role in disease diagnosis, and GANs can contribute by generating realistic microscopic images. However, ethical considerations arise because of the reliance on synthetic or pseudogenerated images. Therefore, the manuscript also explores the current limitations and highlights the ethical considerations associated with the use of this technology. In conclusion, digital histopathology has seen an emerging use of GANs for image enhancement, such as color (stain) normalization, virtual staining, and ink/marker removal. GANs offer significant potential in transforming digital pathology when applied to specific and narrow tasks (preprocessing enhancements). Evaluating data quality, addressing biases, protecting privacy, ensuring accountability and transparency, and developing regulation are imperative to ensure the ethical application of GANs.

Novel Computational Methods for Cancer Drug Design

Cancer is a complex and debilitating disease that is one of the leading causes of death in the modern world. Computational methods have contributed to the successful design and development of several drugs. The recent advances in computational methodology, coupled with the avalanche of data being acquired through high throughput genomics, proteomics, and metabolomics, are likely to increase the contribution of computational methods toward the development of more effective treatments for cancer. Recent advances in the application of neural networks for the prediction of the native conformation of proteins have provided structural information regarding the complete human proteome. In addition, advances in machine learning and network pharmacology have provided novel methods for target identification and for the utilization of biological, pharmacological, and clinical databases for the design and development of drugs. This is a review of the key advances in computational methods that have the potential for application in the design and development of drugs for cancer.

Clinical Utility of Breast Ultrasound Images Synthesized by a Generative Adversarial Network

BACKGROUND AND OBJECTIVES

This study compares the clinical properties of original breast ultrasound images and those synthesized by a generative adversarial network (GAN) to assess the clinical usefulness of GAN-synthesized images.

MATERIALS AND METHODS

We retrospectively collected approximately 200 breast ultrasound images for each of five representative histological tissue types (cyst, fibroadenoma, scirrhous, solid, and tubule-forming invasive ductal carcinomas) as training images. A deep convolutional GAN (DCGAN) image-generation model synthesized images of the five histological types. Two diagnostic radiologists (reader 1 with 13 years of experience and reader 2 with 7 years of experience) were given a reading test consisting of 50 synthesized and 50 original images (≥1-month interval between sets) to assign the perceived histological tissue type. The percentages of correct diagnoses were calculated, and the reader agreement was assessed using the kappa coefficient.

RESULTS

The synthetic and original images were indistinguishable. The correct diagnostic rates from the synthetic images for readers 1 and 2 were 86.0% and 78.0% and from the original images were 88.0% and 78.0%, respectively. The kappa values were 0.625 and 0.650 for the synthetic and original images, respectively. The diagnoses made from the DCGAN synthetic images and original images were similar.

CONCLUSION

The DCGAN-synthesized images closely resemble the original ultrasound images in clinical characteristics, suggesting their potential utility in clinical education and training, particularly for enhancing diagnostic skills in breast ultrasound imaging.

Cotton Fusarium wilt diagnosis based on generative adversarial networks in small samples

This study aimed to explore the feasibility of applying Generative Adversarial Networks (GANs) for the diagnosis of Verticillium wilt disease in cotton and compared it with traditional data augmentation methods and transfer learning. By designing a model based on small-sample learning, we proposed an innovative cotton Verticillium wilt disease diagnosis system. The system uses Convolutional Neural Networks (CNNs) as feature extractors and applies trained GAN models for sample augmentation to improve classification accuracy. This study collected and processed a dataset of cotton Verticillium wilt disease images, including samples from normal and infected plants. Data augmentation techniques were used to expand the dataset and train the CNNs. Transfer learning using InceptionV3 was applied to train the CNNs on the dataset. The dataset was augmented using GAN algorithms and used to train CNNs. The performances of the data augmentation, transfer learning, and GANs were compared and analyzed. The results have demonstrated that augmenting the cotton Verticillium wilt disease image dataset using GAN algorithms enhanced the diagnostic accuracy and recall rate of the CNNs. Compared to traditional data augmentation methods, GANs exhibit better performance and generated more representative and diverse samples. Unlike transfer learning, GANs ensured an adequate sample size. By visualizing the images generated, GANs were found to generate realistic cotton images of Verticillium wilt disease, highlighting their potential applications in agricultural disease diagnosis. This study has demonstrated the potential of GANs in the diagnosis of cotton Verticillium wilt disease diagnosis, offering an effective approach for agricultural disease detection and providing insights into disease detection in other crops.

Unveiling the future of breast cancer assessment: a critical review on generative adversarial networks in elastography ultrasound

Elastography Ultrasound provides elasticity information of the tissues, which is crucial for understanding the density and texture, allowing for the diagnosis of different medical conditions such as fibrosis and cancer. In the current medical imaging scenario, elastograms for B-mode Ultrasound are restricted to well-equipped hospitals, making the modality unavailable for pocket ultrasound. To highlight the recent progress in elastogram synthesis, this article performs a critical review of generative adversarial network (GAN) methodology for elastogram generation from B-mode Ultrasound images. Along with a brief overview of cutting-edge medical image synthesis, the article highlights the contribution of the GAN framework in light of its impact and thoroughly analyzes the results to validate whether the existing challenges have been effectively addressed. Specifically, This article highlights that GANs can successfully generate accurate elastograms for deep-seated breast tumors (without having artifacts) and improve diagnostic effectiveness for pocket US. Furthermore, the results of the GAN framework are thoroughly analyzed by considering the quantitative metrics, visual evaluations, and cancer diagnostic accuracy. Finally, essential unaddressed challenges that lie at the intersection of elastography and GANs are presented, and a few future directions are shared for the elastogram synthesis research.

MMGan: a multimodal MR brain tumor image segmentation method

Computer-aided diagnosis has emerged as a rapidly evolving field, garnering increased attention in recent years. At the forefront of this field is the segmentation of lesions in medical images, which is a critical preliminary stage in subsequent treatment procedures. Among the most challenging tasks in medical image analysis is the accurate and automated segmentation of brain tumors in various modalities of brain tumor MRI. In this article, we present a novel end-to-end network architecture called MMGan, which combines the advantages of residual learning and generative adversarial neural networks inspired by classical generative adversarial networks. The segmenter in the MMGan network, which has a U-Net architecture, is constructed using a deep residual network instead of the conventional convolutional neural network. The dataset used for this study is the BRATS dataset from the Brain Tumor Segmentation Challenge at the Medical Image Computing and Computer Assisted Intervention Society. Our proposed method has been extensively tested, and the results indicate that this MMGan framework is more efficient and stable for segmentation tasks. On BRATS 2019, the segmentation algorithm improved accuracy and sensitivity in whole tumor, tumor core, and enhanced tumor segmentation. Particularly noteworthy is the higher dice score of 0.86 achieved by our proposed method in tumor core segmentation, surpassing those of stateof-the-art models. This study improves the accuracy and sensitivity of the tumor segmentation task, which we believe is significant for medical image analysis. And it should be further improved by replacing different loss functions such as cross-entropy loss function and other methods.

Bridging the Camera Domain Gap With Image-to-Image Translation Improves Glaucoma Diagnosis

Purpose

The purpose of this study was to improve the automated diagnosis of glaucomatous optic neuropathy (GON), we propose a generative adversarial network (GAN) model that translates Optain images to Topcon images.

Methods

We trained the GAN model on 725 paired images from Topcon and Optain cameras and externally validated it using an additional 843 paired images collected from the Aravind Eye Hospital in India. An optic disc segmentation model was used to assess the disparities in disc parameters across cameras. The performance of the translated images was evaluated using root mean square error (RMSE), peak signal-to-noise ratio (PSNR), structural similarity index (SSIM), 95% limits of agreement (LOA), Pearson's correlations, and Cohen's Kappa coefficient. The evaluation compared the performance of the GON model on Topcon photographs as a reference to that of Optain photographs and GAN-translated photographs.

Results

The GAN model significantly reduced Optain false positive results for GON diagnosis, with RMSE, PSNR, and SSIM of GAN images being 0.067, 14.31, and 0.64, respectively, the mean difference of VCDR and cup-to-disc area ratio between Topcon and GAN images being 0.03, 95% LOA ranging from -0.09 to 0.15 and -0.05 to 0.10. Pearson correlation coefficients increased from 0.61 to 0.85 in VCDR and 0.70 to 0.89 in cup-to-disc area ratio, whereas Cohen's Kappa improved from 0.32 to 0.60 after GAN translation.

Conclusions

Image-to-image translation across cameras can be achieved by using GAN to solve the problem of disc overexposure in Optain cameras.

Translational Relevance

Our approach enhances the generalizability of deep learning diagnostic models, ensuring their performance on cameras that are outside of the original training data set.

Ideal Observer Computation by Use of Markov-Chain Monte Carlo With Generative Adversarial Networks

Medical imaging systems are often evaluated and optimized via objective, or task-specific, measures of image quality (IQ) that quantify the performance of an observer on a specific clinically-relevant task. The performance of the Bayesian Ideal Observer (IO) sets an upper limit among all observers, numerical or human, and has been advocated for use as a figure-of-merit (FOM) for evaluating and optimizing medical imaging systems. However, the IO test statistic corresponds to the likelihood ratio that is intractable to compute in the majority of cases. A sampling-based method that employs Markov-chain Monte Carlo (MCMC) techniques was previously proposed to estimate the IO performance. However, current applications of MCMC methods for IO approximation have been limited to a small number of situations where the considered distribution of to-be-imaged objects can be described by a relatively simple stochastic object model (SOM). As such, there remains an important need to extend the domain of applicability of MCMC methods to address a large variety of scenarios where IO-based assessments are needed but the associated SOMs have not been available. In this study, a novel MCMC method that employs a generative adversarial network (GAN)-based SOM, referred to as MCMC-GAN, is described and evaluated. The MCMC-GAN method was quantitatively validated by use of test-cases for which reference solutions were available. The results demonstrate that the MCMC-GAN method can extend the domain of applicability of MCMC methods for conducting IO analyses of medical imaging systems.

Synthetic Tabular Data Based on Generative Adversarial Networks in Health Care: Generation and Validation Using the Divide-and-Conquer Strategy

BACKGROUND

Synthetic data generation (SDG) based on generative adversarial networks (GANs) is used in health care, but research on preserving data with logical relationships with synthetic tabular data (STD) remains challenging. Filtering methods for SDG can lead to the loss of important information.

OBJECTIVE

This study proposed a divide-and-conquer (DC) method to generate STD based on the GAN algorithm, while preserving data with logical relationships.

METHODS

The proposed method was evaluated on data from the Korea Association for Lung Cancer Registry (KALC-R) and 2 benchmark data sets (breast cancer and diabetes). The DC-based SDG strategy comprises 3 steps: (1) We used 2 different partitioning methods (the class-specific criterion distinguished between survival and death groups, while the Cramer V criterion identified the highest correlation between columns in the original data); (2) the entire data set was divided into a number of subsets, which were then used as input for the conditional tabular generative adversarial network and the copula generative adversarial network to generate synthetic data; and (3) the generated synthetic data were consolidated into a single entity. For validation, we compared DC-based SDG and conditional sampling (CS)-based SDG through the performances of machine learning models. In addition, we generated imbalanced and balanced synthetic data for each of the 3 data sets and compared their performance using 4 classifiers: decision tree (DT), random forest (RF), Extreme Gradient Boosting (XGBoost), and light gradient-boosting machine (LGBM) models.

RESULTS

The synthetic data of the 3 diseases (non-small cell lung cancer [NSCLC], breast cancer, and diabetes) generated by our proposed model outperformed the 4 classifiers (DT, RF, XGBoost, and LGBM). The CS- versus DC-based model performances were compared using the mean area under the curve (SD) values: 74.87 (SD 0.77) versus 63.87 (SD 2.02) for NSCLC, 73.31 (SD 1.11) versus 67.96 (SD 2.15) for breast cancer, and 61.57 (SD 0.09) versus 60.08 (SD 0.17) for diabetes (DT); 85.61 (SD 0.29) versus 79.01 (SD 1.20) for NSCLC, 78.05 (SD 1.59) versus 73.48 (SD 4.73) for breast cancer, and 59.98 (SD 0.24) versus 58.55 (SD 0.17) for diabetes (RF); 85.20 (SD 0.82) versus 76.42 (SD 0.93) for NSCLC, 77.86 (SD 2.27) versus 68.32 (SD 2.37) for breast cancer, and 60.18 (SD 0.20) versus 58.98 (SD 0.29) for diabetes (XGBoost); and 85.14 (SD 0.77) versus 77.62 (SD 1.85) for NSCLC, 78.16 (SD 1.52) versus 70.02 (SD 2.17) for breast cancer, and 61.75 (SD 0.13) versus 61.12 (SD 0.23) for diabetes (LGBM). In addition, we found that balanced synthetic data performed better.

CONCLUSIONS

This study is the first attempt to generate and validate STD based on a DC approach and shows improved performance using STD. The necessity for balanced SDG was also demonstrated.

Synthetic 3D Spinal Vertebrae Reconstruction from Biplanar X-rays Utilizing Generative Adversarial Networks

Computed tomography (CT) offers detailed insights into the internal anatomy of patients, particularly for spinal vertebrae examination. However, CT scans are associated with higher radiation exposure and cost compared to conventional X-ray imaging. In this study, we applied a Generative Adversarial Network (GAN) framework to reconstruct 3D spinal vertebrae structures from synthetic biplanar X-ray images, specifically focusing on anterior and lateral views. The synthetic X-ray images were generated using the DRRGenerator module in 3D Slicer by incorporating segmentations of spinal vertebrae in CT scans for the region of interest. This approach leverages a novel feature fusion technique based on X2CT-GAN to combine information from both views and employs a combination of mean squared error (MSE) loss and adversarial loss to train the generator, resulting in high-quality synthetic 3D spinal vertebrae CTs. A total of n = 440 CT data were processed. We evaluated the performance of our model using multiple metrics, including mean absolute error (MAE) (for each slice of the 3D volume (MAE0) and for the entire 3D volume (MAE)), cosine similarity, peak signal-to-noise ratio (PSNR), 3D peak signal-to-noise ratio (PSNR-3D), and structural similarity index (SSIM). The average PSNR was 28.394 dB, PSNR-3D was 27.432, SSIM was 0.468, cosine similarity was 0.484, MAE0 was 0.034, and MAE was 85.359. The results demonstrated the effectiveness of this approach in reconstructing 3D spinal vertebrae structures from biplanar X-rays, although some limitations in accurately capturing the fine bone structures and maintaining the precise morphology of the vertebrae were present. This technique has the potential to enhance the diagnostic capabilities of low-cost X-ray machines while reducing radiation exposure and cost associated with CT scans, paving the way for future applications in spinal imaging and diagnosis.

An improved multi-scale gradient generative adversarial network for enhancing classification of colorectal cancer histological images

Introduction

Deep learning-based solutions for histological image classification have gained attention in recent years due to their potential for objective evaluation of histological images. However, these methods often require a large number of expert annotations, which are both time-consuming and labor-intensive to obtain. Several scholars have proposed generative models to augment labeled data, but these often result in label uncertainty due to incomplete learning of the data distribution.

Methods

To alleviate these issues, a method called InceptionV3-SMSG-GAN has been proposed to enhance classification performance by generating high-quality images. Specifically, images synthesized by Multi-Scale Gradients Generative Adversarial Network (MSG-GAN) are selectively added to the training set through a selection mechanism utilizing a trained model to choose generated images with higher class probabilities. The selection mechanism filters the synthetic images that contain ambiguous category information, thus alleviating label uncertainty.

Results

Experimental results show that compared with the baseline method which uses InceptionV3, the proposed method can significantly improve the performance of pathological image classification from 86.87% to 89.54% for overall accuracy. Additionally, the quality of generated images is evaluated quantitatively using various commonly used evaluation metrics.

Discussion

The proposed InceptionV3-SMSG-GAN method exhibited good classification ability, where histological image could be divided into nine categories. Future work could focus on further refining the image generation and selection processes to optimize classification performance.

Synthetic Megavoltage Cone Beam Computed Tomography Image Generation for Improved Contouring Accuracy of Cardiac Pacemakers

In this study, we aimed to enhance the contouring accuracy of cardiac pacemakers by improving their visualization using deep learning models to predict MV CBCT images based on kV CT or CBCT images. Ten pacemakers and four thorax phantoms were included, creating a total of 35 combinations. Each combination was imaged on a Varian Halcyon (kV/MV CBCT images) and Siemens SOMATOM CT scanner (kV CT images). Two generative adversarial network (GAN)-based models, cycleGAN and conditional GAN (cGAN), were trained to generate synthetic MV (sMV) CBCT images from kV CT/CBCT images using twenty-eight datasets (80%). The pacemakers in the sMV CBCT images and original MV CBCT images were manually delineated and reviewed by three users. The Dice similarity coefficient (DSC), 95% Hausdorff distance (HD95), and mean surface distance (MSD) were used to compare contour accuracy. Visual inspection showed the improved visualization of pacemakers on sMV CBCT images compared to original kV CT/CBCT images. Moreover, cGAN demonstrated superior performance in enhancing pacemaker visualization compared to cycleGAN. The mean DSC, HD95, and MSD for contours on sMV CBCT images generated from kV CT/CBCT images were 0.91 ± 0.02/0.92 ± 0.01, 1.38 ± 0.31 mm/1.18 ± 0.20 mm, and 0.42 ± 0.07 mm/0.36 ± 0.06 mm using the cGAN model. Deep learning-based methods, specifically cycleGAN and cGAN, can effectively enhance the visualization of pacemakers in thorax kV CT/CBCT images, therefore improving the contouring precision of these devices.

Gaming self-consistent field theory: Generative block polymer phase discovery

Block polymers are an attractive platform for uncovering the factors that give rise to self-assembly in soft matter owing to their relatively simple thermodynamic description, as captured in self-consistent field theory (SCFT). SCFT historically has found great success explaining experimental data, allowing one to construct phase diagrams from a set of candidate phases, and there is now strong interest in deploying SCFT as a screening tool to guide experimental design. However, using SCFT for phase discovery leads to a conundrum: How does one discover a new morphology if the set of candidate phases needs to be specified in advance? This long-standing challenge was surmounted by training a deep convolutional generative adversarial network (GAN) with trajectories from converged SCFT solutions, and then deploying the GAN to generate input fields for subsequent SCFT calculations. The power of this approach is demonstrated for network phase formation in neat diblock copolymer melts via SCFT. A training set of only five networks produced 349 candidate phases spanning known and previously unexplored morphologies, including a chiral network. This computational pipeline, constructed here entirely from open-source codes, should find widespread application in block polymer phase discovery and other forms of soft matter.

medigan: a Python library of pretrained generative models for medical image synthesis

Purpose

Deep learning has shown great promise as the backbone of clinical decision support systems. Synthetic data generated by generative models can enhance the performance and capabilities of data-hungry deep learning models. However, there is (1) limited availability of (synthetic) datasets and (2) generative models are complex to train, which hinders their adoption in research and clinical applications. To reduce this entry barrier, we explore generative model sharing to allow more researchers to access, generate, and benefit from synthetic data.

Approach

We propose medigan, a one-stop shop for pretrained generative models implemented as an open-source framework-agnostic Python library. After gathering end-user requirements, design decisions based on usability, technical feasibility, and scalability are formulated. Subsequently, we implement medigan based on modular components for generative model (i) execution, (ii) visualization, (iii) search & ranking, and (iv) contribution. We integrate pretrained models with applications across modalities such as mammography, endoscopy, x-ray, and MRI.

Results

The scalability and design of the library are demonstrated by its growing number of integrated and readily-usable pretrained generative models, which include 21 models utilizing nine different generative adversarial network architectures trained on 11 different datasets. We further analyze three medigan applications, which include (a) enabling community-wide sharing of restricted data, (b) investigating generative model evaluation metrics, and (c) improving clinical downstream tasks. In (b), we extract Fréchet inception distances (FID) demonstrating FID variability based on image normalization and radiology-specific feature extractors.

Conclusion

medigan allows researchers and developers to create, increase, and domain-adapt their training data in just a few lines of code. Capable of enriching and accelerating the development of clinical machine learning models, we show medigan's viability as platform for generative model sharing. Our multimodel synthetic data experiments uncover standards for assessing and reporting metrics, such as FID, in image synthesis studies.

StainGAN: Learning a structural preserving translation for white blood cell images

Analysis of white blood cells in blood smear images plays a vital role in computer-aided diagnosis for the analysis and treatment of many diseases. However, different techniques for blood smear preparation result in images with large appearance variations, which limits the performance of large-scale machine learning algorithms. In this paper, we propose StainGAN, an image translation framework to transform the conventional Wright-stained white blood cell images into their rapidly-stained counterpart. Moreover, we designed a cluster-based learning strategy that does not require manual annotations and a multi-scale discriminator that incorporates a richer hierarchy of the spatial context to generate sharper images with better semantic consistency. Experimental results on multiple real-world datasets prove the effectiveness of our proposed framework. Moreover, we show that the transformed images from StainGAN can be used to boost the downstream segmentation performance under the label-limiting scenario.

Impact of artificial intelligence-based color constancy on dermoscopical assessment of skin lesions: A comparative study

BACKGROUND

The quality of dermoscopic images is affected by lighting conditions, operator experience, and device calibration. Color constancy algorithms reduce this variability by making images appear as if they were acquired under the same conditions, allowing artificial intelligence (AI)-based methods to achieve better results. The impact of color constancy algorithms has not yet been evaluated from a clinical dermatologist's workflow point of view. Here we propose an in-depth investigation of the impact of an AI-based color constancy algorithm, called DermoCC-GAN, on the skin lesion diagnostic routine.

METHODS

Three dermatologists, with different experience levels, carried out two assignments. The clinical experts evaluated key parameters such as perceived image quality, lesion diagnosis, and diagnosis confidence.

RESULTS

When the DermoCC-GAN color constancy algorithm was applied, the dermoscopic images were perceived to be of better quality overall. An increase in classification performance was observed, reaching a maximum accuracy of 74.67% for a six-class classification task. Finally, the use of normalized images results in an increase in the level of self-confidence in the qualitative diagnostic routine.

CONCLUSIONS

From the conducted analysis, it is evident that the impact of AI-based color constancy algorithms, such as DermoCC-GAN, is positive and brings qualitative benefits to the clinical practitioner.

Segmentation of Brain Tumor Using a 3D Generative Adversarial Network

Images of brain tumors may only show up in a small subset of scans, so important details may be missed. Further, because labeling is typically a labor-intensive and time-consuming task, there are typically only a small number of medical imaging datasets available for analysis. The focus of this research is on the MRI images of the human brain, and an attempt has been made to propose a method for the accurate segmentation of these images to identify the correct location of tumors. In this study, GAN is utilized as a classification network to detect and segment of 3D MRI images. The 3D GAN network model provides dense connectivity, followed by rapid network convergence and improved information extraction. Mutual training in a generative adversarial network can bring the segmentation results closer to the labeled data to improve image segmentation. The BraTS 2021 dataset of 3D images was used to compare two experimental models.

ROP-GAN: an image synthesis method for retinopathy of prematurity based on generative adversarial network

Objective. Training data with annotations are scarce in the intelligent diagnosis of retinopathy of prematurity (ROP), and existing typical data augmentation methods cannot generate data with a high degree of diversity. In order to increase the sample size and the generalization ability of the classification model, we propose a method called ROP-GAN for image synthesis of ROP based on a generative adversarial network.Approach. To generate a binary vascular network from color fundus images, we first design an image segmentation model based on U2-Net that can extract multi-scale features without reducing the resolution of the feature map. The vascular network is then fed into an adversarial autoencoder for reconstruction, which increases the diversity of the vascular network diagram. Then, we design an ROP image synthesis algorithm based on a generative adversarial network, in which paired color fundus images and binarized vascular networks are input into the image generation model to train the generator and discriminator, and attention mechanism modules are added to the generator to improve its detail synthesis ability.Main results. Qualitative and quantitative evaluation indicators are applied to evaluate the proposed method, and experiments demonstrate that the proposed method is superior to the existing ROP image synthesis methods, as it can synthesize realistic ROP fundus images.Significance. Our method effectively alleviates the problem of data imbalance in ROP intelligent diagnosis, contributes to the implementation of ROP staging tasks, and lays the foundation for further research. In addition to classification tasks, our synthesized images can facilitate tasks that require large amounts of medical data, such as detecting lesions and segmenting medical images.

Uncertainty Quantification in Estimating Blood Alcohol Concentration From Transdermal Alcohol Level With Physics-Informed Neural Networks

We develop an approach to estimate a blood alcohol signal from a transdermal alcohol signal using physics-informed neural networks (PINNs). Specifically, we use a generative adversarial network (GAN) with a residual-augmented loss function to estimate the distribution of unknown parameters in a diffusion equation model for transdermal transport of alcohol in the human body. We design another PINN for the deconvolution of the blood alcohol signal from the transdermal alcohol signal. Based on the distribution of the unknown parameters, this network is able to estimate the blood alcohol signal and quantify the uncertainty in the form of conservative error bands. Finally, we show how a posterior latent variable can be used to sharpen these conservative error bands. We apply the techniques to an extensive dataset of drinking episodes and demonstrate the advantages and shortcomings of this approach.

Creating High Fidelity Synthetic Pelvis Radiographs Using Generative Adversarial Networks: Unlocking the Potential of Deep Learning Models Without Patient Privacy Concerns

BACKGROUND

In this work, we applied and validated an artificial intelligence technique known as generative adversarial networks (GANs) to create large volumes of high-fidelity synthetic anteroposterior (AP) pelvis radiographs that can enable deep learning (DL)-based image analyses, while ensuring patient privacy.

METHODS

AP pelvis radiographs with native hips were gathered from an institutional registry between 1998 and 2018. The data was used to train a model to create 512 × 512 pixel synthetic AP pelvis images. The network was trained on 25 million images produced through augmentation. A set of 100 random images (50/50 real/synthetic) was evaluated by 3 orthopaedic surgeons and 2 radiologists to discern real versus synthetic images. Two models (joint localization and segmentation) were trained using synthetic images and tested on real images.

RESULTS

The final model was trained on 37,640 real radiographs (16,782 patients). In a computer assessment of image fidelity, the final model achieved an "excellent" rating. In a blinded review of paired images (1 real, 1 synthetic), orthopaedic surgeon reviewers were unable to correctly identify which image was synthetic (accuracy = 55%, Kappa = 0.11), highlighting synthetic image fidelity. The synthetic and real images showed equivalent performance when they were assessed by established DL models.

CONCLUSION

This work shows the ability to use a DL technique to generate a large volume of high-fidelity synthetic pelvis images not discernible from real imaging by computers or experts. These images can be used for cross-institutional sharing and model pretraining, further advancing the performance of DL models without risk to patient data safety.

LEVEL OF EVIDENCE

Level III.

DeepSWI: Using Deep Learning to Enhance Susceptibility Contrast on T2*-Weighted MRI

BACKGROUND

Although susceptibility-weighted imaging (SWI) is the gold standard for visualizing cerebral microbleeds (CMBs) in the brain, the required phase data are not always available clinically. Having a postprocessing tool for generating SWI contrast from T2*-weighted magnitude images is therefore advantageous.

PURPOSE

To create synthetic SWI images from clinical T2*-weighted magnitude images using deep learning and evaluate the resulting images in terms of similarity to conventional SWI images and ability to detect radiation-associated CMBs.

STUDY TYPE

Retrospective.

POPULATION

A total of 145 adults (87 males/58 females; 43.9 years old) with radiation-associated CMBs were used to train (16,093 patches/121 patients), validate (484 patches/4 patients), and test (2420 patches/20 patients) our networks.

FIELD STRENGTH/SEQUENCE

3D T2*-weighted, gradient-echo acquired at 3 T.

ASSESSMENT

Structural similarity index (SSIM), peak signal-to-noise-ratio (PSNR), normalized mean-squared-error (nMSE), CMB counts, and line profiles were compared among magnitude, original SWI, and synthetic SWI images. Three blinded raters (J.E.V.M., M.A.M., B.B. with 8-, 6-, and 4-years of experience, respectively) independently rated and classified test-set images.

STATISTICAL TESTS

Kruskall-Wallis and Wilcoxon signed-rank tests were used to compare SSIM, PSNR, nMSE, and CMB counts among magnitude, original SWI, and predicted synthetic SWI images. Intraclass correlation assessed interrater variability. P values <0.005 were considered statistically significant.

RESULTS

SSIM values of the predicted vs. original SWI (0.972, 0.995, 0.9864) were statistically significantly higher than that of the magnitude vs. original SWI (0.970, 0.994, 0.9861) for whole brain, vascular structures, and brain tissue regions, respectively; 67% (19/28) CMBs detected on original SWI images were also detected on the predicted SWI, whereas only 10 (36%) were detected on magnitude images. Overall image quality was similar between the synthetic and original SWI images, with less artifacts on the former.

CONCLUSIONS

This study demonstrated that deep learning can increase the susceptibility contrast present in neurovasculature and CMBs on T2*-weighted magnitude images, without residual susceptibility-induced artifacts. This may be useful for more accurately estimating CMB burden from magnitude images alone.

EVIDENCE LEVEL

3.

TECHNICAL EFFICACY

Stage 2.

GammaGAN: Gamma-Scaled Class Embeddings for Conditional Video Generation

In this paper, we propose a new model for conditional video generation (GammaGAN). Generally, it is challenging to generate a plausible video from a single image with a class label as a condition. Traditional methods based on conditional generative adversarial networks (cGANs) often encounter difficulties in effectively utilizing a class label, typically by concatenating a class label to the input or hidden layer. In contrast, the proposed GammaGAN adopts the projection method to effectively utilize a class label and proposes scaling class embeddings and normalizing outputs. Concretely, our proposed architecture consists of two streams: a class embedding stream and a data stream. In the class embedding stream, class embeddings are scaled to effectively emphasize class-specific differences. Meanwhile, the outputs in the data stream are normalized. Our normalization technique balances the outputs of both streams, ensuring a balance between the importance of feature vectors and class embeddings during training. This results in enhanced video quality. We evaluated the proposed method using the MUG facial expression dataset, which consists of six facial expressions. Compared with the prior conditional video generation model, ImaGINator, our model yielded relative improvements of 1.61%, 1.66%, and 0.36% in terms of PSNR, SSIM, and LPIPS, respectively. These results suggest potential for further advancements in conditional video generation.

Attention-based generative adversarial networks improve prognostic outcome prediction of cancer from multimodal data

The prediction of prognostic outcome is critical for the development of efficient cancer therapeutics and potential personalized medicine. However, due to the heterogeneity and diversity of multimodal data of cancer, data integration and feature selection remain a challenge for prognostic outcome prediction. We proposed a deep learning method with generative adversarial network based on sequential channel-spatial attention modules (CSAM-GAN), a multimodal data integration and feature selection approach, for accomplishing prognostic stratification tasks in cancer. Sequential channel-spatial attention modules equipped with an encoder-decoder are applied for the input features of multimodal data to accurately refine selected features. A discriminator network was proposed to make the generator and discriminator learning in an adversarial way to accurately describe the complex heterogeneous information of multiple modal data. We conducted extensive experiments with various feature selection and classification methods and confirmed that the CSAM-GAN via the multilayer deep neural network (DNN) classifier outperformed these baseline methods on two different multimodal data sets with miRNA expression, mRNA expression and histopathological image data: lower-grade glioma and kidney renal clear cell carcinoma. The CSAM-GAN via the multilayer DNN classifier bridges the gap between heterogenous multimodal data and prognostic outcome prediction.

Hybrid Learning Models for IMU-Based HAR with Feature Analysis and Data Correction

This paper proposes a novel approach to tackle the human activity recognition (HAR) problem. Four classes of body movement datasets, namely stand-up, sit-down, run, and walk, are applied to perform HAR. Instead of using vision-based solutions, we address the HAR challenge by implementing a real-time HAR system architecture with a wearable inertial measurement unit (IMU) sensor, which aims to achieve networked sensing and data sampling of human activity, data pre-processing and feature analysis, data generation and correction, and activity classification using hybrid learning models. Referring to the experimental results, the proposed system selects the pre-trained eXtreme Gradient Boosting (XGBoost) model and the Convolutional Variational Autoencoder (CVAE) model as the classifier and generator, respectively, with 96.03% classification accuracy.

Quality control-driven deep ensemble for accountable automated segmentation of cardiac magnetic resonance LGE and VNE images

Background

Late gadolinium enhancement (LGE) cardiovascular magnetic resonance (CMR) imaging is the gold standard for non-invasive myocardial tissue characterisation. However, accurate segmentation of the left ventricular (LV) myocardium remains a challenge due to limited training data and lack of quality control. This study addresses these issues by leveraging generative adversarial networks (GAN)-generated virtual native enhancement (VNE) images to expand the training set and incorporating an automated quality control-driven (QCD) framework to improve segmentation reliability.

Methods

A dataset comprising 4,716 LGE images (from 1,363 patients with hypertrophic cardiomyopathy and myocardial infarction) was used for development. To generate additional clinically validated data, LGE data were augmented with a GAN-based generator to produce VNE images. LV was contoured on these images manually by clinical observers. To create diverse candidate segmentations, the QCD framework involved multiple U-Nets, which were combined using statistical rank filters. The framework predicted the Dice Similarity Coefficient (DSC) for each candidate segmentation, with the highest predicted DSC indicating the most accurate and reliable result. The performance of the QCD ensemble framework was evaluated on both LGE and VNE test datasets (309 LGE/VNE images from 103 patients), assessing segmentation accuracy (DSC) and quality prediction (mean absolute error (MAE) and binary classification accuracy).

Results

The QCD framework effectively and rapidly segmented the LV myocardium (<1 s per image) on both LGE and VNE images, demonstrating robust performance on both test datasets with similar mean DSC (LGE: 0.845 ± 0.075 ; VNE: 0.845 ± 0.071 ; p = n s ). Incorporating GAN-generated VNE data into the training process consistently led to enhanced performance for both individual models and the overall framework. The quality control mechanism yielded a high performance (MAE = 0.043 , accuracy = 0.951 ) emphasising the accuracy of the quality control-driven strategy in predicting segmentation quality in clinical settings. Overall, no statistical difference (p = n s ) was found when comparing the LGE and VNE test sets across all experiments.

Conclusions

The QCD ensemble framework, leveraging GAN-generated VNE data and an automated quality control mechanism, significantly improved the accuracy and reliability of LGE segmentation, paving the way for enhanced and accountable diagnostic imaging in routine clinical use.

SGAN-IDS: Self-Attention-Based Generative Adversarial Network against Intrusion Detection Systems

In cybersecurity, a network intrusion detection system (NIDS) is a critical component in networks. It monitors network traffic and flags suspicious activities. To effectively detect malicious traffic, several detection techniques, including machine learning-based NIDSs (ML-NIDSs), have been proposed and implemented. However, in much of the existing ML-NIDS research, the experimental settings do not accurately reflect real-world scenarios where new attacks are constantly emerging. Thus, the robustness of intrusion detection systems against zero-day and adversarial attacks is a crucial area that requires further investigation. In this paper, we introduce and develop a framework named SGAN-IDS. This framework constructs adversarial attack flows designed to evade detection by five BlackBox ML-based IDSs. SGAN-IDS employs generative adversarial networks and self-attention mechanisms to generate synthetic adversarial attack flows that are resilient to detection. Our evaluation results demonstrate that SGAN-IDS has successfully constructed adversarial flows for various attack types, reducing the detection rate of all five IDSs by an average of 15.93%. These findings underscore the robustness and broad applicability of the proposed model.

3D Generative Model Latent Disentanglement via Local Eigenprojection

Designing realistic digital humans is extremely complex. Most data-driven generative models used to simplify the creation of their underlying geometric shape do not offer control over the generation of local shape attributes. In this paper, we overcome this limitation by introducing a novel loss function grounded in spectral geometry and applicable to different neural-network-based generative models of 3D head and body meshes. Encouraging the latent variables of mesh variational autoencoders (VAEs) or generative adversarial networks (GANs) to follow the local eigenprojections of identity attributes, we improve latent disentanglement and properly decouple the attribute creation. Experimental results show that our local eigenprojection disentangled (LED) models not only offer improved disentanglement with respect to the state-of-the-art, but also maintain good generation capabilities with training times comparable to the vanilla implementations of the models. Our code and pre-trained models are available at github.com/simofoti/LocalEigenprojDisentangled.

Beautification of images by generative adversarial networks

Finding the properties underlying beauty has always been a prominent yet difficult problem. However, new technological developments have often aided scientific progress by expanding the scientists' toolkit. Currently in the spotlight of cognitive neuroscience and vision science are deep neural networks. In this study, we have used a generative adversarial network (GAN) to generate images of increasing aesthetic value. We validated that this network indeed was able to increase the aesthetic value of an image by letting participants decide which of two presented images they considered more beautiful. As our validation was successful, we were justified to use the generated images to extract low- and mid-level features contributing to their aesthetic value. We compared the brightness, contrast, sharpness, saturation, symmetry, colorfulness, and visual complexity levels of "low-aesthetic" images to those of "high-aesthetic" images. We found that all of these features increased for the beautiful images, implying that they may play an important role underlying the aesthetic value of an image. With this study, we have provided further evidence for the potential value GANs may have for research concerning beauty.

Cyclic Generative Attention-Adversarial Network for Low-Light Image Enhancement

Images captured under complex conditions frequently have low quality, and image performance obtained under low-light conditions is poor and does not satisfy subsequent engineering processing. The goal of low-light image enhancement is to restore low-light images to normal illumination levels. Although many methods have emerged in this field, they are inadequate for dealing with noise, color deviation, and exposure issues. To address these issues, we present CGAAN, a new unsupervised generative adversarial network that combines a new attention module and a new normalization function based on cycle generative adversarial networks and employs a global-local discriminator trained with unpaired low-light and normal-light images and stylized region loss. Our attention generates feature maps via global and average pooling, and the weights of different feature maps are calculated by multiplying learnable parameters and feature maps in the appropriate order. These weights indicate the significance of corresponding features. Specifically, our attention is a feature map attention mechanism that improves the network's feature-extraction ability by distinguishing the normal light domain from the low-light domain to obtain an attention map to solve the color bias and exposure problems. The style region loss guides the network to more effectively eliminate the effects of noise. The new normalization function we present preserves more semantic information while normalizing the image, which can guide the model to recover more details and improve image quality even further. The experimental results demonstrate that the proposed method can produce good results that are useful for practical applications.

Progressively Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization for Image-to-Image Translation

Unsupervised image-to-image translation has received considerable attention due to the recent remarkable advancements in generative adversarial networks (GANs). In image-to-image translation, state-of-the-art methods use unpaired image data to learn mappings between the source and target domains. However, despite their promising results, existing approaches often fail in challenging conditions, particularly when images have various target instances and a translation task involves significant transitions in shape and visual artifacts when translating low-level information rather than high-level semantics. To tackle the problem, we propose a novel framework called Progressive Unsupervised Generative Attentional Networks with Adaptive Layer-Instance Normalization (PRO-U-GAT-IT) for the unsupervised image-to-image translation task. In contrast to existing attention-based models that fail to handle geometric transitions between the source and target domains, our model can translate images requiring extensive and holistic changes in shape. Experimental results show the superiority of the proposed approach compared to the existing state-of-the-art models on different datasets.

Inside out: transforming images of lab-grown plants for machine learning applications in agriculture

Introduction

Machine learning tasks often require a significant amount of training data for the resultant network to perform suitably for a given problem in any domain. In agriculture, dataset sizes are further limited by phenotypical differences between two plants of the same genotype, often as a result of different growing conditions. Synthetically-augmented datasets have shown promise in improving existing models when real data is not available.

Methods

In this paper, we employ a contrastive unpaired translation (CUT) generative adversarial network (GAN) and simple image processing techniques to translate indoor plant images to appear as field images. While we train our network to translate an image containing only a single plant, we show that our method is easily extendable to produce multiple-plant field images.

Results

Furthermore, we use our synthetic multi-plant images to train several YoloV5 nano object detection models to perform the task of plant detection and measure the accuracy of the model on real field data images.

Discussion

The inclusion of training data generated by the CUT-GAN leads to better plant detection performance compared to a network trained solely on real data.