CycleGAN for style transfer in X-ray angiography

Artificial vascular image generation using blood vessel texture maps

BACKGROUND

Current methods for identifying blood vessels in digital images typically involve training neural networks on pixel-wise annotated data. However, manually outlining whole vessel trees in images tends to be very costly. One approach for reducing the amount of manual annotation is to pre-train networks on artificially generated vessel images. Recent pre-training approaches focus on generating proper artificial geometries for the vessels, while the appearance of the vessels is defined using general statistics of the real samples or generative networks requiring an additional training procedure to be defined. In contrast, we propose a methodology for generating blood vessels with realistic textures extracted directly from manually annotated vessel segments from real samples. The method allows the generation of artificial images having blood vessels with similar geometry and texture to the real samples using only a handful of manually annotated vessels.

METHODS

The first step of the method is the manual annotation of the borders of a small vessel segment, which takes only a few seconds. The annotation is then used for creating a reference image containing the texture of the vessel, called a texture map. A procedure is then defined to allow texture maps to be placed on top of any smooth curve using a piecewise linear transformation. Artificial images are then created by generating a set of vessel geometries using Bézier curves and assigning vessel texture maps to the curves.

RESULTS

The method is validated on a fluorescence microscopy (CORTEX) and a fundus photography (DRIVE) dataset. We show that manually annotating only 0.03% of the vessels in the CORTEX dataset allows pre-training a network to reach, on average, a Dice score of 0.87 ± 0.02, which is close to the baseline score of 0.92 obtained when all vessels of the training split of the dataset are annotated. For the DRIVE dataset, on average, a Dice score of 0.74 ± 0.02 is obtained by annotating only 0.29% of the vessels, which is also close to the baseline Dice score of 0.81 obtained when all vessels are annotated.

CONCLUSION

The proposed method can be used for disentangling the geometry and texture of blood vessels, which allows a significant improvement of network pre-training performance when compared to other pre-training methods commonly used in the literature.

Generating Synthetic Data for Medical Imaging

Artificial intelligence (AI) models for medical imaging tasks, such as classification or segmentation, require large and diverse datasets of images. However, due to privacy and ethical issues, as well as data sharing infrastructure barriers, these datasets are scarce and difficult to assemble. Synthetic medical imaging data generated by AI from existing data could address this challenge by augmenting and anonymizing real imaging data. In addition, synthetic data enable new applications, including modality translation, contrast synthesis, and professional training for radiologists. However, the use of synthetic data also poses technical and ethical challenges. These challenges include ensuring the realism and diversity of the synthesized images while keeping data unidentifiable, evaluating the performance and generalizability of models trained on synthetic data, and high computational costs. Since existing regulations are not sufficient to guarantee the safe and ethical use of synthetic images, it becomes evident that updated laws and more rigorous oversight are needed. Regulatory bodies, physicians, and AI developers should collaborate to develop, maintain, and continually refine best practices for synthetic data. This review aims to provide an overview of the current knowledge of synthetic data in medical imaging and highlights current key challenges in the field to guide future research and development.

Semi-supervised segmentation of metal-artifact contaminated industrial CT images using improved CycleGAN

Accurate segmentation of industrial CT images is of great significance in industrial fields such as quality inspection and defect analysis. However, reconstruction of industrial CT images often suffers from typical metal artifacts caused by factors like beam hardening, scattering, statistical noise, and partial volume effects. Traditional segmentation methods are difficult to achieve precise segmentation of CT images mainly due to the presence of these metal artifacts. Furthermore, acquiring paired CT image data required by fully supervised networks proves to be extremely challenging. To address these issues, this paper introduces an improved CycleGAN approach for achieving semi-supervised segmentation of industrial CT images. This method not only eliminates the need for removing metal artifacts and noise, but also enables the direct conversion of metal artifact-contaminated images into segmented images without the requirement of paired data. The average values of quantitative assessment of image segmentation performance can reach 0.96645 for Dice Similarity Coefficient(Dice) and 0.93718 for Intersection over Union(IoU). In comparison to traditional segmentation methods, it presents significant improvements in both quantitative metrics and visual quality, provides valuable insights for further research.

Unsupervised synthesis of realistic coronary artery X-ray angiogram

PURPOSE

Medical image analysis suffers from a sparsity of annotated data necessary in learning-based models. Cardiorespiratory simulators have been developed to counter the lack of data. However, the resulting data often lack realism. Hence, the proposed method aims to synthesize realistic and fully customizable angiograms of coronary arteries for the training of learning-based biomedical tasks, for cardiologists performing interventions, and for cardiologist trainees.

METHODS

3D models of coronary arteries are generated with a fully customizable realistic cardiorespiratory simulator. The transfer of X-ray angiography style to simulator-generated images is performed using a new vessel-specific adaptation of the CycleGAN model. The CycleGAN model is paired with a vesselness-based loss function that is designed as a vessel-specific structural integrity constraint.

RESULTS

Validation is performed both on the style and on the preservation of the shape of the arteries of the images. The results show a PSNR of 14.125, an SSIM of 0.898, and an overlapping of 89.5% using the Dice coefficient.

CONCLUSION

We proposed a novel fluoroscopy-based style transfer method for the enhancement of the realism of simulated coronary artery angiograms. The results show that the proposed model is capable of accurately transferring the style of X-ray angiograms to the simulations while keeping the integrity of the structures of interest (i.e., the topology of the coronary arteries).

Style Transfer-assisted Deep Learning Method for Kidney Segmentation at Multiphase MRI

Purpose

To develop and validate a semisupervised style transfer-assisted deep learning method for automated segmentation of the kidneys using multiphase contrast-enhanced (MCE) MRI acquisitions.

Materials and Methods

This retrospective, Health Insurance Portability and Accountability Act-compliant, institutional review board-approved study included 125 patients (mean age, 57.3 years; 67 male, 58 female) with renal masses. Cohort 1 consisted of 102 coronal T2-weighted MRI acquisitions and 27 MCE MRI acquisitions during the corticomedullary phase. Cohort 2 comprised 92 MCE MRI acquisitions (23 acquisitions during four phases each, including precontrast, corticomedullary, early nephrographic, and nephrographic phases). The kidneys were manually segmented on T2-weighted images. A cycle-consistent generative adversarial network (CycleGAN) was trained to generate anatomically coregistered synthetic corticomedullary style images using T2-weighted images as input. Synthetic images for precontrast, early nephrographic, and nephrographic phases were then generated using the synthetic corticomedullary images as input. Mask region-based convolutional neural networks were trained on the four synthetic phase series for kidney segmentation using T2-weighted masks. Segmentation performance was evaluated in a different cohort of 20 originally acquired MCE MRI examinations by using Dice and Jaccard scores.

Results

The CycleGAN network successfully generated anatomically coregistered synthetic MCE MRI-like datasets from T2-weighted acquisitions. The proposed deep learning approach for kidney segmentation achieved high mean Dice scores in all four phases of the original MCE MRI acquisitions (0.91 for precontrast, 0.92 for corticomedullary, 0.91 for early nephrographic, and 0.93 for nephrographic).

Conclusion

The proposed deep learning approach achieved high performance in kidney segmentation on different MCE MRI acquisitions.Keywords: Kidney Segmentation, Generative Adversarial Network, CycleGAN, Convolutional Neural Network, Transfer Learning Supplemental material is available for this article. Published under a CC BY 4.0 license.

A deep CT to MRI unpaired translation that preserve ischemic stroke lesions

Stroke is the second-leading cause of death world around. The immediate attention is key to patient prognosis. Ischemic stroke diagnosis typically involves neuroimaging studies (MRI and CT scans) and clinical protocols to characterize lesions and support decisions about treatment to be administered to the patient. Nowadays, multiparametric MRI images are the standard tool to visualize core and penumbra of ischemic stroke, supporting diagnosis and lesion prognosis. Specially, DWI modality (Diffusion Weighted Imaging) allows to quantify the cellular density of the tissue, and therefore allowing to quantify the lesion aggressiveness, and the recognition of micro-circulation properties. Nevertheless, MRI availability at hospitals is not widespread, and acquisition require special conditions requiring considerable time. Contrary, CT scans commonly have major availability but brain structures are poorly delineated, and even worse, ischemic lesions are only visible at advanced stages of the disease. This work introduces a deep generative strategy that allows ischemic stroke lesion translation over synthetic DWI-MRI images. This encoder-decoder architecture, include U-net modules, hierarchically organized, with inter-level connections that preserve brain structures, while codifying an embedding representation. Then a cyclic loss was here implemented to receive CT inputs and decode DWI-MRI images. To avoid mode collapse, this learning is inversely propagated, i.e., from synthetic DWI-MRI images to original CT-scans. Finally, an embedding projection is recovered to show a proper lesion-slice discrimination, regarding control studies. Clinical relevance- To recover synthetic DWI-MRI that preserved ischemic lesion using CT scans as an input and following an unpaired image translation setup.

Unsupervised-learning-based method for chest MRI-CT transformation using structure constrained unsupervised generative attention networks

The integrated positron emission tomography/magnetic resonance imaging (PET/MRI) scanner simultaneously acquires metabolic information via PET and morphological information using MRI. However, attenuation correction, which is necessary for quantitative PET evaluation, is difficult as it requires the generation of attenuation-correction maps from MRI, which has no direct relationship with the gamma-ray attenuation information. MRI-based bone tissue segmentation is potentially available for attenuation correction in relatively rigid and fixed organs such as the head and pelvis regions. However, this is challenging for the chest region because of respiratory and cardiac motions in the chest, its anatomically complicated structure, and the thin bone cortex. We propose a new method using unsupervised generative attentional networks with adaptive layer-instance normalisation for image-to-image translation (U-GAT-IT), which specialised in unpaired image transformation based on attention maps for image transformation. We added the modality-independent neighbourhood descriptor (MIND) to the loss of U-GAT-IT to guarantee anatomical consistency in the image transformation between different domains. Our proposed method obtained a synthesised computed tomography of the chest. Experimental results showed that our method outperforms current approaches. The study findings suggest the possibility of synthesising clinically acceptable computed tomography images from chest MRI with minimal changes in anatomical structures without human annotation.

Real-Time Cerebral Vessel Segmentation in Laser Speckle Contrast Image Based on Unsupervised Domain Adaptation

Laser speckle contrast imaging (LSCI) is a full-field, high spatiotemporal resolution and low-cost optical technique for measuring blood flow, which has been successfully used for neurovascular imaging. However, due to the low signal-noise ratio and the relatively small sizes, segmenting the cerebral vessels in LSCI has always been a technical challenge. Recently, deep learning has shown its advantages in vascular segmentation. Nonetheless, ground truth by manual labeling is usually required for training the network, which makes it difficult to implement in practice. In this manuscript, we proposed a deep learning-based method for real-time cerebral vessel segmentation of LSCI without ground truth labels, which could be further integrated into intraoperative blood vessel imaging system. Synthetic LSCI images were obtained with a synthesis network from LSCI images and public labeled dataset of Digital Retinal Images for Vessel Extraction, which were then used to train the segmentation network. Using matching strategies to reduce the size discrepancy between retinal images and laser speckle contrast images, we could further significantly improve image synthesis and segmentation performance. In the testing LSCI images of rodent cerebral vessels, the proposed method resulted in a dice similarity coefficient of over 75%.

Fetal ECG Extraction from Maternal ECG using Attention-based CycleGAN

A non-invasive fetal electrocardiogram (FECG) is used to monitor the electrical pulse of the fetal heart. Decomposing the FECG signal from the maternal ECG (MECG) is a blind source separation problem, which is hard due to the low amplitude of the FECG, the overlap of R waves, and the potential exposure to noise from different sources. Traditional decomposition techniques, such as adaptive filters, require tuning, alignment, or pre-configuration, such as modeling the noise or desired signal to map the MECG to the FECG. The high correlation between maternal and fetal ECG fragments decreases the performance of convolution layers. Therefore, the masking region of interest based on the attention mechanism was performed to improve the signal generators' precision. The sine activation function was also used to retain more details when converting two signal domains. Three available datasets from the Physionet, including the A&D FECG, NI-FECG, and NI-FECG challenge, and one synthetic dataset using FECGSYN toolbox, were used to evaluate the performance. The proposed method could map an abdominal MECG to a scalp FECG with an average of 98% R-Square [CI 95%: 97%, 99%] as the goodness of fit on the A&D FECG dataset. Moreover, it achieved 99.7 % F1-score [CI 95%: 97.8-99.9], 99.6% F1-score [CI 95%: 98.2%, 99.9%] and 99.3% F1-score [CI 95%: 95.3%, 99.9%] for fetal QRS detection on the A&D FECG, NI-FECG and NI-FECG challenge datasets, respectively. Also, the distortion was in the very good and good ranges. These results are comparable to the state-of-the-art results; thus, the proposed algorithm has the potential to be used for high-performance signal-to-signal conversion.

Generation of annotated multimodal ground truth datasets for abdominal medical image registration

PURPOSE

Sparsity of annotated data is a major limitation in medical image processing tasks such as registration. Registered multimodal image data are essential for the diagnosis of medical conditions and the success of interventional medical procedures. To overcome the shortage of data, we present a method that allows the generation of annotated multimodal 4D datasets.

METHODS

We use a CycleGAN network architecture to generate multimodal synthetic data from the 4D extended cardiac-torso (XCAT) phantom and real patient data. Organ masks are provided by the XCAT phantom; therefore, the generated dataset can serve as ground truth for image segmentation and registration. Realistic simulation of respiration and heartbeat is possible within the XCAT framework. To underline the usability as a registration ground truth, a proof of principle registration is performed.

RESULTS

Compared to real patient data, the synthetic data showed good agreement regarding the image voxel intensity distribution and the noise characteristics. The generated T1-weighted magnetic resonance imaging, computed tomography (CT), and cone beam CT images are inherently co-registered. Thus, the synthetic dataset allowed us to optimize registration parameters of a multimodal non-rigid registration, utilizing liver organ masks for evaluation.

CONCLUSION

Our proposed framework provides not only annotated but also multimodal synthetic data which can serve as a ground truth for various tasks in medical imaging processing. We demonstrated the applicability of synthetic data for the development of multimodal medical image registration algorithms.

Synthesize and Segment: Towards Improved Catheter Segmentation via Adversarial Augmentation

Automatic catheter and guidewire segmentation plays an important role in robot-assisted interventions that are guided by fluoroscopy. Existing learning based methods addressing the task of segmentation or tracking are often limited by the scarcity of annotated samples and difficulty in data collection. In the case of deep learning based methods, the demand for large amounts of labeled data further impedes successful application. We propose a synthesize and segment approach with plug in possibilities for segmentation to address this. We show that an adversarially learned image-to-image translation network can synthesize catheters in X-ray fluoroscopy enabling data augmentation in order to alleviate a low data regime. To make realistic synthesized images, we train the translation network via a perceptual loss coupled with similarity constraints. Then existing segmentation networks are used to learn accurate localization of catheters in a semi-supervised setting with the generated images. The empirical results on collected medical datasets show the value of our approach with significant improvements over existing translation baseline methods.

An accurate and universal approach for short-exposure-time microscopy image enhancement

Fluorescence microscopy imaging has become an essential technique in the biology and biomedical science which can provide comprehensive visualization of many biological processes, and the exposure time is one of the most critical parameters for fluorescence microscopy imaging. Short-exposure-time (SET) imaging overcomes the limitations of photo-bleaching and photo-toxicity, allowing comprehensive visualization of the biological processes. Unfortunately, SET images deteriorate the signal to noise ratio and the image quality when compared with the long-exposure-time (LET) images. Therefore, we introduce a data-driven microscopy image enhancement network (MIEN) to improve the quality of SET microscopy images without requiring any manual intervention, facilitating the production of high-resolution and high contrast images. The universal property and accuracy of the proposed network are validated on 38,500 real fluorescence microscopy images, which contain different object contents and are collected from various exposure time ratios and fluorescence microscopes platforms. Experimental results demonstrate that the proposed MIEN model is effective to enhance the quality of SET fluorescence microscopy images, and can be used to observe delicate changes in cells, tissues and organs with low photo-bleaching and photo-toxicity.

Bridging the Gap Between AI and Healthcare Sides: Towards Developing Clinically Relevant AI-Powered Diagnosis Systems

Despite the success of Convolutional Neural Network-based Computer-Aided Diagnosis research, its clinical applications remain challenging. Accordingly, developing medical Artificial Intelligence (AI) fitting into a clinical environment requires identifying/bridging the gap between AI and Healthcare sides. Since the biggest problem in Medical Imaging lies in data paucity, confirming the clinical relevance for diagnosis of research-proven image augmentation techniques is essential. Therefore, we hold a clinically valuable AI-envisioning workshop among Japanese Medical Imaging experts, physicians, and generalists in Healthcare/Informatics. Then, a questionnaire survey for physicians evaluates our pathology-aware Generative Adversarial Network (GAN)-based image augmentation projects in terms of Data Augmentation and physician training. The workshop reveals the intrinsic gap between AI/Healthcare sides and solutions on Why (i.e., clinical significance/interpretation) and How (i.e., data acquisition, commercial deployment, and safety/feeling safe). This analysis confirms our pathology-aware GANs’ clinical relevance as a clinical decision support system and non-expert physician training tool. Our findings would play a key role in connecting inter-disciplinary research and clinical applications, not limited to the Japanese medical context and pathology-aware GANs.

pix2xray: converting RGB images into X-rays using generative adversarial networks

PURPOSE

We propose a novel methodology for generating synthetic X-rays from 2D RGB images. This method creates accurate simulations for use in non-diagnostic visualization problems where the only input comes from a generic camera. Traditional methods are restricted to using simulation algorithms on 3D computer models. To solve this problem, we propose a method of synthetic X-ray generation using conditional generative adversarial networks (CGANs).

METHODS

We create a custom synthetic X-ray dataset generator to generate image triplets for X-ray images, pose images, and RGB images of natural hand poses sampled from the NYU hand pose dataset. This dataset is used to train two general-purpose CGAN networks, pix2pix and CycleGAN, as well as our novel architecture called pix2xray which expands upon the pix2pix architecture to include the hand pose into the network.

RESULTS

Our results demonstrate that our pix2xray architecture outperforms both pix2pix and CycleGAN in producing higher-quality X-ray images. We measure higher similarity metrics in our approach, with pix2pix coming in second, and CycleGAN producing the worst results. Our network performs better in the difficult cases which involve high occlusion due to occluded poses or large rotations.

CONCLUSION

Overall our work establishes a baseline that synthetic X-rays can be simulated using 2D RGB input. We establish the need for additional data such as the hand pose to produce clearer results and show that future research must focus on more specialized architectures to improve overall image clarity and structure.