1
|
Mao Y, Chen C, Wang Z, Cheng D, You P, Huang X, Zhang B, Zhao F. Generative adversarial networks with adaptive normalization for synthesizing T2-weighted magnetic resonance images from diffusion-weighted images. Front Neurosci 2022; 16:1058487. [PMID: 36452330 PMCID: PMC9704724 DOI: 10.3389/fnins.2022.1058487] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2022] [Accepted: 10/25/2022] [Indexed: 12/10/2023] Open
Abstract
Recently, attention has been drawn toward brain imaging technology in the medical field, among which MRI plays a vital role in clinical diagnosis and lesion analysis of brain diseases. Different sequences of MR images provide more comprehensive information and help doctors to make accurate clinical diagnoses. However, their costs are particularly high. For many image-to-image synthesis methods in the medical field, supervised learning-based methods require labeled datasets, which are often difficult to obtain. Therefore, we propose an unsupervised learning-based generative adversarial network with adaptive normalization (AN-GAN) for synthesizing T2-weighted MR images from rapidly scanned diffusion-weighted imaging (DWI) MR images. In contrast to the existing methods, deep semantic information is extracted from the high-frequency information of original sequence images, which are then added to the feature map in deconvolution layers as a modality mask vector. This image fusion operation results in better feature maps and guides the training of GANs. Furthermore, to better preserve semantic information against common normalization layers, we introduce AN, a conditional normalization layer that modulates the activations using the fused feature map. Experimental results show that our method of synthesizing T2 images has a better perceptual quality and better detail than the other state-of-the-art methods.
Collapse
Affiliation(s)
- Yanyan Mao
- College of Oceanography and Space Informatics, China University of Petroleum, Qingdao, China
- School of Computer Science and Technology, Shandong Business and Technology University, Yantai, China
| | - Chao Chen
- School of Computer Science and Technology, Shandong Business and Technology University, Yantai, China
| | - Zhenjie Wang
- College of Oceanography and Space Informatics, China University of Petroleum, Qingdao, China
| | - Dapeng Cheng
- School of Computer Science and Technology, Shandong Business and Technology University, Yantai, China
- Shandong Co-Innovation Center of Future Intelligent Computing, Yantai, China
| | - Panlu You
- School of Computer Science and Technology, Shandong Business and Technology University, Yantai, China
| | - Xingdan Huang
- School of Statistics, Shandong Business and Technology University, Yantai, China
| | - Baosheng Zhang
- School of Computer Science and Technology, Shandong Business and Technology University, Yantai, China
| | - Feng Zhao
- School of Computer Science and Technology, Shandong Business and Technology University, Yantai, China
- Shandong Co-Innovation Center of Future Intelligent Computing, Yantai, China
| |
Collapse
|
2
|
Wang T, Wang M, Zhu W, Wang L, Chen Z, Peng Y, Shi F, Zhou Y, Yao C, Chen X. Semi-MsST-GAN: A Semi-Supervised Segmentation Method for Corneal Ulcer Segmentation in Slit-Lamp Images. Front Neurosci 2022; 15:793377. [PMID: 35058743 PMCID: PMC8764146 DOI: 10.3389/fnins.2021.793377] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2021] [Accepted: 11/22/2021] [Indexed: 11/21/2022] Open
Abstract
Corneal ulcer is a common leading cause of corneal blindness. It is difficult to accurately segment corneal ulcers due to the following problems: large differences in the pathological shapes between point-flaky and flaky corneal ulcers, blurred boundary, noise interference, and the lack of sufficient slit-lamp images with ground truth. To address these problems, in this paper, we proposed a novel semi-supervised multi-scale self-transformer generative adversarial network (Semi-MsST-GAN) that can leverage unlabeled images to improve the performance of corneal ulcer segmentation in fluorescein staining of slit-lamp images. Firstly, to improve the performance of segmenting the corneal ulcer regions with complex pathological features, we proposed a novel multi-scale self-transformer network (MsSTNet) as the MsST-GAN generator, which can guide the model to aggregate the low-level weak semantic features with the high-level strong semantic information and adaptively learn the spatial correlation in feature maps. Then, to further improve the segmentation performance by leveraging unlabeled data, the semi-supervised approach based on the proposed MsST-GAN was explored to solve the problem of the lack of slit-lamp images with corresponding ground truth. The proposed Semi-MsST-GAN was comprehensively evaluated on the public SUSTech-SYSU dataset, which contains 354 labeled and 358 unlabeled fluorescein staining slit-lamp images. The results showed that, compared with other state-of-the-art methods, our proposed method achieves better performance with comparable efficiency.
Collapse
Affiliation(s)
- Tingting Wang
- Medical Image Processing, Analysis and Visualization (MIPAV) Laboratory, The School of Electronics and Information Engineering, Soochow University, Suzhou, China
| | - Meng Wang
- Medical Image Processing, Analysis and Visualization (MIPAV) Laboratory, The School of Electronics and Information Engineering, Soochow University, Suzhou, China
| | - Weifang Zhu
- Medical Image Processing, Analysis and Visualization (MIPAV) Laboratory, The School of Electronics and Information Engineering, Soochow University, Suzhou, China
| | - Lianyu Wang
- Medical Image Processing, Analysis and Visualization (MIPAV) Laboratory, The School of Electronics and Information Engineering, Soochow University, Suzhou, China
| | - Zhongyue Chen
- Medical Image Processing, Analysis and Visualization (MIPAV) Laboratory, The School of Electronics and Information Engineering, Soochow University, Suzhou, China
| | - Yuanyuan Peng
- Medical Image Processing, Analysis and Visualization (MIPAV) Laboratory, The School of Electronics and Information Engineering, Soochow University, Suzhou, China
| | - Fei Shi
- Medical Image Processing, Analysis and Visualization (MIPAV) Laboratory, The School of Electronics and Information Engineering, Soochow University, Suzhou, China
| | - Yi Zhou
- Medical Image Processing, Analysis and Visualization (MIPAV) Laboratory, The School of Electronics and Information Engineering, Soochow University, Suzhou, China
| | - Chenpu Yao
- Medical Image Processing, Analysis and Visualization (MIPAV) Laboratory, The School of Electronics and Information Engineering, Soochow University, Suzhou, China
| | - Xinjian Chen
- Medical Image Processing, Analysis and Visualization (MIPAV) Laboratory, The School of Electronics and Information Engineering, Soochow University, Suzhou, China
- The State Key Laboratory of Radiation Medicine and Protection, Soochow University, Suzhou, China
| |
Collapse
|
3
|
Fruend I. Constrained sampling from deep generative image models reveals mechanisms of human target detection. J Vis 2020; 20:32. [PMID: 32729908 PMCID: PMC7424951 DOI: 10.1167/jov.20.7.32] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Abstract
The first steps of visual processing are often described as a bank of oriented filters followed by divisive normalization. This approach has been tremendously successful at predicting contrast thresholds in simple visual displays. However, it is unclear to what extent this kind of architecture also supports processing in more complex visual tasks performed in naturally looking images. We used a deep generative image model to embed arc segments with different curvatures in naturalistic images. These images contain the target as part of the image scene, resulting in considerable appearance variation of target as well as background. Three observers localized arc targets in these images, with an average accuracy of 74.7%. Data were fit by several biologically inspired models, four standard deep convolutional neural networks (CNNs), and a five-layer CNN specifically trained for this task. Four models predicted observer responses particularly well; (1) a bank of oriented filters, similar to complex cells in primate area V1; (2) a bank of oriented filters followed by tuned gain control, incorporating knowledge about cortical surround interactions; (3) a bank of oriented filters followed by local normalization; and (4) the five-layer CNN. A control experiment with optimized stimuli based on these four models showed that the observers' data were best explained by model (2) with tuned gain control. These data suggest that standard models of early vision provide good descriptions of performance in much more complex tasks than what they were designed for, while general-purpose non linear models such as convolutional neural networks do not.
Collapse
Affiliation(s)
- Ingo Fruend
- Department of Psychology, Centre for Vision Research & Vision: Science to Application, York University, Toronto, ON, Canada
| |
Collapse
|