Effective Immunohistochemistry Pathology Microscopy Image Generation Using CycleGAN

Diagnostic performance of deep-learning-based virtual chromoendoscopy in gastric neoplasms

BACKGROUNDS

Cycle-consistent generative adversarial network (CycleGAN) is a deep neural network model that performs image-to-image translations. We generated virtual indigo carmine (IC) chromoendoscopy images of gastric neoplasms using CycleGAN and compared their diagnostic performance with that of white light endoscopy (WLE).

METHODS

WLE and IC images of 176 patients with gastric neoplasms who underwent endoscopic resection were obtained. We used 1,633 images (911 WLE and 722 IC) of 146 cases in the training dataset to develop virtual IC images using CycleGAN. The remaining 30 WLE images were translated into 30 virtual IC images using the trained CycleGAN and used for validation. The lesion borders were evaluated by 118 endoscopists from 22 institutions using the 60 paired virtual IC and WLE images. The lesion area concordance rate and successful whole-lesion diagnosis were compared.

RESULTS

The lesion area concordance rate based on the pathological diagnosis in virtual IC was lower than in WLE (44.1% vs. 48.5%, p < 0.01). The successful whole-lesion diagnosis was higher in the virtual IC than in WLE images; however, the difference was insignificant (28.2% vs. 26.4%, p = 0.11). Conversely, subgroup analyses revealed a significantly higher diagnosis in virtual IC than in WLE for depressed morphology (41.9% vs. 36.9%, p = 0.02), differentiated histology (27.6% vs. 24.8%, p = 0.02), smaller lesion size (42.3% vs. 38.3%, p = 0.01), and assessed by expert endoscopists (27.3% vs. 23.6%, p = 0.03).

CONCLUSIONS

The diagnostic ability of virtual IC was higher for some lesions, but not completely superior to that of WLE. Adjustments are required to improve the imaging system's performance.

DSFF-GAN: A novel stain transfer network for generating immunohistochemical image of endometrial cancer

Immunohistochemistry (IHC) is a commonly used histological examination technique. Compared to Hematoxylin and Eosin (H&E) staining, it enables the examination of protein expression and localization in tissues, which is valuable for cancer treatment and prognosis assessment, such as the detection and diagnosis of endometrial cancer. However, IHC involves multiple staining steps, is time-consuming and expensive. One potential solution is to utilize deep learning networks to generate corresponding virtual IHC images from H&E images. However, the similarity of the IHC image generated by the existing methods needs to be further improved. In this work, we propose a novel dual-scale feature fusion (DSFF) generative adversarial network named DSFF-GAN, which comprises a cycle structure-color similarity loss, and DSFF block to constrain the model's training process and enhance its stain transfer capability. In addition, our method incorporates labeling information of positive cell regions as prior knowledge into the network to further improve the evaluation metrics. We train and test our model using endometrial cancer and publicly available breast cancer IHC datasets, and compare it with state-of-the-art methods. Compared to previous methods, our model demonstrates significant improvements in most evaluation metrics on both datasets. The research results show that our method further improves the quality of image generation and has potential value for the future clinical application of virtual IHC images.

Deep UV-excited fluorescence microscopy installed with CycleGAN-assisted image translation enhances precise detection of lymph node metastasis towards rapid intraoperative diagnosis

Rapid and precise intraoperative diagnosing systems are required for improving surgical outcomes and patient prognosis. Because of the poor quality and time-intensive process of the prevalent frozen section procedure, various intraoperative diagnostic imaging systems have been explored. Microscopy with ultraviolet surface excitation (MUSE) is an inexpensive, maintenance-free, and rapid imaging technique that yields images like thin-sectioned samples without sectioning. However, pathologists find it nearly impossible to assign diagnostic labels to MUSE images of unfixed specimens; thus, AI for intraoperative diagnosis cannot be trained in a supervised learning manner. In this study, we propose a deep-learning pipeline model for lymph node metastasis detection, in which CycleGAN translate MUSE images of unfixed lymph nodes to formalin-fixed paraffin-embedded (FFPE) sample, and diagnostic prediction is performed using deep convolutional neural network trained on FFPE sample images. Our pipeline yielded an average accuracy of 84.6% when using each of the three deep convolutional neural networks, which is a 18.3% increase over the classification-only model without CycleGAN. The modality translation to FFPE sample images using CycleGAN can be applied to various intraoperative diagnostic imaging systems and eliminate the difficulty for pathologists in labeling new modality images in clinical sites. We anticipate our pipeline to be a starting point for accurate rapid intraoperative diagnostic systems for new imaging modalities, leading to healthcare quality improvement.

Deep learning based registration of serial whole-slide histopathology images in different stains

For routine pathology diagnosis and imaging-based biomedical research, Whole-slide image (WSI) analyses have been largely limited to a 2D tissue image space. For a more definitive tissue representation to support fine-resolution spatial and integrative analyses, it is critical to extend such tissue-based investigations to a 3D tissue space with spatially aligned serial tissue WSIs in different stains, such as Hematoxylin and Eosin (H&E) and Immunohistochemistry (IHC) biomarkers. However, such WSI registration is technically challenged by the overwhelming image scale, the complex histology structure change, and the significant difference in tissue appearances in different stains. The goal of this study is to register serial sections from multi-stain histopathology whole-slide image blocks. We propose a novel translation-based deep learning registration network CGNReg that spatially aligns serial WSIs stained in H&E and by IHC biomarkers without prior deformation information for the model training. First, synthetic IHC images are produced from H&E slides through a robust image synthesis algorithm. Next, the synthetic and the real IHC images are registered through a Fully Convolutional Network with multi-scaled deformable vector fields and a joint loss optimization. We perform the registration at the full image resolution, retaining the tissue details in the results. Evaluated with a dataset of 76 breast cancer patients with 1 H&E and 2 IHC serial WSIs for each patient, CGNReg presents promising performance as compared with multiple state-of-the-art systems in our evaluation. Our results suggest that CGNReg can produce promising registration results with serial WSIs in different stains, enabling integrative 3D tissue-based biomedical investigations.

CycleGAN for virtual stain transfer: Is seeing really believing?

Digital Pathology is an area prone to high variation due to multiple factors which can strongly affect diagnostic quality and visual appearance of the Whole-Slide-Images (WSIs). The state-of-the art methods to deal with such variation tend to address this through style-transfer inspired approaches. Usually, these solutions directly apply successful approaches from the literature, potentially with some task-related modifications. The majority of the obtained results are visually convincing, however, this paper shows that this is not a guarantee that such images can be directly used for either medical diagnosis or reducing domain shift.This article shows that slight modification in a stain transfer architecture, such as a choice of normalisation layer, while resulting in a variety of visually appealing results, surprisingly greatly effects the ability of a stain transfer model to reduce domain shift. By extensive qualitative and quantitative evaluations, we confirm that translations resulting from different stain transfer architectures are distinct from each other and from the real samples. Therefore conclusions made by visual inspection or pretrained model evaluation might be misleading.