1
|
Meng X, Yu C, Zhang Z, Zhang X, Wang M. TG-Net: Using text prompts for improved skin lesion segmentation. Comput Biol Med 2024; 179:108819. [PMID: 38964245 DOI: 10.1016/j.compbiomed.2024.108819] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/19/2024] [Revised: 05/26/2024] [Accepted: 06/24/2024] [Indexed: 07/06/2024]
Abstract
Automatic skin segmentation is an efficient method for the early diagnosis of skin cancer, which can minimize the missed detection rate and treat early skin cancer in time. However, significant variations in texture, size, shape, the position of lesions, and obscure boundaries in dermoscopy images make it extremely challenging to accurately locate and segment lesions. To address these challenges, we propose a novel framework named TG-Net, which exploits textual diagnostic information to guide the segmentation of dermoscopic images. Specifically, TG-Net adopts a dual-stream encoder-decoder architecture. The dual-stream encoder comprises Res2Net for extracting image features and our proposed text attention (TA) block for extracting textual features. Through hierarchical guidance, textual features are embedded into the process of image feature extraction. Additionally, we devise a multi-level fusion (MLF) module to merge higher-level features and generate a global feature map as guidance for subsequent steps. In the decoding stage of the network, local features and the global feature map are utilized in three multi-scale reverse attention modules (MSRA) to produce the final segmentation results. We conduct extensive experiments on three publicly accessible datasets, namely ISIC 2017, HAM10000, and PH2. Experimental results demonstrate that TG-Net outperforms state-of-the-art methods, validating the reliability of our method. Source code is available at https://github.com/ukeLin/TG-Net.
Collapse
Affiliation(s)
- Xiangfu Meng
- School of Electronics and Information Engineering, Liaoning Technical University, Huludao, China
| | - Chunlin Yu
- School of Electronics and Information Engineering, Liaoning Technical University, Huludao, China.
| | - Zhichao Zhang
- School of Electronics and Information Engineering, Liaoning Technical University, Huludao, China
| | - Xiaoyan Zhang
- School of Electronics and Information Engineering, Liaoning Technical University, Huludao, China
| | - Meng Wang
- School of Electronics and Information Engineering, Liaoning Technical University, Huludao, China
| |
Collapse
|
2
|
Mao K, Li R, Cheng J, Huang D, Song Z, Liu Z. PL-Net: progressive learning network for medical image segmentation. Front Bioeng Biotechnol 2024; 12:1414605. [PMID: 38994123 PMCID: PMC11236745 DOI: 10.3389/fbioe.2024.1414605] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/09/2024] [Accepted: 05/30/2024] [Indexed: 07/13/2024] Open
Abstract
In recent years, deep convolutional neural network-based segmentation methods have achieved state-of-the-art performance for many medical analysis tasks. However, most of these approaches rely on optimizing the U-Net structure or adding new functional modules, which overlooks the complementation and fusion of coarse-grained and fine-grained semantic information. To address these issues, we propose a 2D medical image segmentation framework called Progressive Learning Network (PL-Net), which comprises Internal Progressive Learning (IPL) and External Progressive Learning (EPL). PL-Net offers the following advantages: 1) IPL divides feature extraction into two steps, allowing for the mixing of different size receptive fields and capturing semantic information from coarse to fine granularity without introducing additional parameters; 2) EPL divides the training process into two stages to optimize parameters and facilitate the fusion of coarse-grained information in the first stage and fine-grained information in the second stage. We conducted comprehensive evaluations of our proposed method on five medical image segmentation datasets, and the experimental results demonstrate that PL-Net achieves competitive segmentation performance. It is worth noting that PL-Net does not introduce any additional learnable parameters compared to other U-Net variants.
Collapse
Affiliation(s)
- Kunpeng Mao
- Chongqing City Management College, Chongqing, China
| | - Ruoyu Li
- College of Computer Science, Sichuan University, Chengdu, China
| | - Junlong Cheng
- College of Computer Science, Sichuan University, Chengdu, China
| | - Danmei Huang
- Chongqing City Management College, Chongqing, China
| | - Zhiping Song
- Chongqing University of Engineering, Chongqing, China
| | - ZeKui Liu
- Chongqing University of Engineering, Chongqing, China
| |
Collapse
|
3
|
Showrav TT, Hasan MK. Hi- gMISnet: generalized medical image segmentation using DWT based multilayer fusion and dual mode attention into high resolution pGAN. Phys Med Biol 2024; 69:115019. [PMID: 38593830 DOI: 10.1088/1361-6560/ad3cb3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/15/2024] [Accepted: 04/09/2024] [Indexed: 04/11/2024]
Abstract
Objective.Automatic medical image segmentation is crucial for accurately isolating target tissue areas in the image from background tissues, facilitating precise diagnoses and procedures. While the proliferation of publicly available clinical datasets led to the development of deep learning-based medical image segmentation methods, a generalized, accurate, robust, and reliable approach across diverse imaging modalities remains elusive.Approach.This paper proposes a novel high-resolution parallel generative adversarial network (pGAN)-based generalized deep learning method for automatic segmentation of medical images from diverse imaging modalities. The proposed method showcases better performance and generalizability by incorporating novel components such as partial hybrid transfer learning, discrete wavelet transform (DWT)-based multilayer and multiresolution feature fusion in the encoder, and a dual mode attention gate in the decoder of the multi-resolution U-Net-based GAN. With multi-objective adversarial training loss functions including a unique reciprocal loss for enforcing cooperative learning inpGANs, it further enhances the robustness and accuracy of the segmentation map.Main results.Experimental evaluations conducted on nine diverse publicly available medical image segmentation datasets, including PhysioNet ICH, BUSI, CVC-ClinicDB, MoNuSeg, GLAS, ISIC-2018, DRIVE, Montgomery, and PROMISE12, demonstrate the proposed method's superior performance. The proposed method achieves mean F1 scores of 79.53%, 88.68%, 82.50%, 93.25%, 90.40%, 94.19%, 81.65%, 98.48%, and 90.79%, respectively, on the above datasets, surpass state-of-the-art segmentation methods. Furthermore, our proposed method demonstrates robust multi-domain segmentation capabilities, exhibiting consistent and reliable performance. The assessment of the model's proficiency in accurately identifying small details indicates that the high-resolution generalized medical image segmentation network (Hi-gMISnet) is more precise in segmenting even when the target area is very small.Significance.The proposed method provides robust and reliable segmentation performance on medical images, and thus it has the potential to be used in a clinical setting for the diagnosis of patients.
Collapse
Affiliation(s)
- Tushar Talukder Showrav
- Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology (BUET), Dhaka, 1205, Bangladesh
| | - Md Kamrul Hasan
- Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology (BUET), Dhaka, 1205, Bangladesh
| |
Collapse
|
4
|
Din S, Mourad O, Serpedin E. LSCS-Net: A lightweight skin cancer segmentation network with densely connected multi-rate atrous convolution. Comput Biol Med 2024; 173:108303. [PMID: 38547653 DOI: 10.1016/j.compbiomed.2024.108303] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/29/2023] [Revised: 01/18/2024] [Accepted: 03/12/2024] [Indexed: 04/17/2024]
Abstract
The rising occurrence and notable public health consequences of skin cancer, especially of the most challenging form known as melanoma, have created an urgent demand for more advanced approaches to disease management. The integration of modern computer vision methods into clinical procedures offers the potential for enhancing the detection of skin cancer . The UNet model has gained prominence as a valuable tool for this objective, continuously evolving to tackle the difficulties associated with the inherent diversity of dermatological images. These challenges stem from diverse medical origins and are further complicated by variations in lighting, patient characteristics, and hair density. In this work, we present an innovative end-to-end trainable network crafted for the segmentation of skin cancer . This network comprises an encoder-decoder architecture, a novel feature extraction block, and a densely connected multi-rate Atrous convolution block. We evaluated the performance of the proposed lightweight skin cancer segmentation network (LSCS-Net) on three widely used benchmark datasets for skin lesion segmentation: ISIC 2016, ISIC 2017, and ISIC 2018. The generalization capabilities of LSCS-Net are testified by the excellent performance on breast cancer and thyroid nodule segmentation datasets. The empirical findings confirm that LSCS-net attains state-of-the-art results, as demonstrated by a significantly elevated Jaccard index.
Collapse
Affiliation(s)
- Sadia Din
- Electrical and Computer Engineering Program, Texas A&M University, Doha, Qatar.
| | | | - Erchin Serpedin
- Department of Electrical and Computer Engineering, Texas A&M University, College Station, TX, USA
| |
Collapse
|
5
|
Wang Y, Yang Z, Liu X, Li Z, Wu C, Wang Y, Jin K, Chen D, Jia G, Chen X, Ye J, Huang X. PGKD-Net: Prior-guided and Knowledge Diffusive Network for Choroid Segmentation. Artif Intell Med 2024; 150:102837. [PMID: 38553151 DOI: 10.1016/j.artmed.2024.102837] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/27/2023] [Revised: 03/01/2024] [Accepted: 03/03/2024] [Indexed: 04/02/2024]
Abstract
The thickness of the choroid is considered to be an important indicator of clinical diagnosis. Therefore, accurate choroid segmentation in retinal OCT images is crucial for monitoring various ophthalmic diseases. However, this is still challenging due to the blurry boundaries and interference from other lesions. To address these issues, we propose a novel prior-guided and knowledge diffusive network (PGKD-Net) to fully utilize retinal structural information to highlight choroidal region features and boost segmentation performance. Specifically, it is composed of two parts: a Prior-mask Guided Network (PG-Net) for coarse segmentation and a Knowledge Diffusive Network (KD-Net) for fine segmentation. In addition, we design two novel feature enhancement modules, Multi-Scale Context Aggregation (MSCA) and Multi-Level Feature Fusion (MLFF). The MSCA module captures the long-distance dependencies between features from different receptive fields and improves the model's ability to learn global context. The MLFF module integrates the cascaded context knowledge learned from PG-Net to benefit fine-level segmentation. Comprehensive experiments are conducted to evaluate the performance of the proposed PGKD-Net. Experimental results show that our proposed method achieves superior segmentation accuracy over other state-of-the-art methods. Our code is made up publicly available at: https://github.com/yzh-hdu/choroid-segmentation.
Collapse
Affiliation(s)
- Yaqi Wang
- College of Media Engineering, Communication University of Zhejiang, Hangzhou, China.
| | - Zehua Yang
- Hangzhou Dianzi University, Hangzhou, China.
| | - Xindi Liu
- Department of Ophthalmology, School of Medicine, The Second Affiliated Hospital of Zhejiang University, Hangzhou, China.
| | - Zhi Li
- Hangzhou Dianzi University, Hangzhou, China.
| | - Chengyu Wu
- Department of Mechanical, Electrical and Information Engineering, Shandong University, Weihai, China.
| | - Yizhen Wang
- Hangzhou Dianzi University, Hangzhou, China.
| | - Kai Jin
- Department of Ophthalmology, School of Medicine, The Second Affiliated Hospital of Zhejiang University, Hangzhou, China.
| | - Dechao Chen
- Hangzhou Dianzi University, Hangzhou, China.
| | | | | | - Juan Ye
- Department of Ophthalmology, School of Medicine, The Second Affiliated Hospital of Zhejiang University, Hangzhou, China.
| | | |
Collapse
|
6
|
Manh V, Jia X, Xue W, Xu W, Mei Z, Dong Y, Zhou J, Huang R, Ni D. An efficient framework for lesion segmentation in ultrasound images using global adversarial learning and region-invariant loss. Comput Biol Med 2024; 171:108137. [PMID: 38447499 DOI: 10.1016/j.compbiomed.2024.108137] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/23/2023] [Revised: 01/16/2024] [Accepted: 02/12/2024] [Indexed: 03/08/2024]
Abstract
Lesion segmentation in ultrasound images is an essential yet challenging step for early evaluation and diagnosis of cancers. In recent years, many automatic CNN-based methods have been proposed to assist this task. However, most modern approaches often lack capturing long-range dependencies and prior information making it difficult to identify the lesions with unfixed shapes, sizes, locations, and textures. To address this, we present a novel lesion segmentation framework that guides the model to learn the global information about lesion characteristics and invariant features (e.g., morphological features) of lesions to improve the segmentation in ultrasound images. Specifically, the segmentation model is guided to learn the characteristics of lesions from the global maps using an adversarial learning scheme with a self-attention-based discriminator. We argue that under such a lesion characteristics-based guidance mechanism, the segmentation model gets more clues about the boundaries, shapes, sizes, and positions of lesions and can produce reliable predictions. In addition, as ultrasound lesions have different textures, we embed this prior knowledge into a novel region-invariant loss to constrain the model to focus on invariant features for robust segmentation. We demonstrate our method on one in-house breast ultrasound (BUS) dataset and two public datasets (i.e., breast lesion (BUS B) and thyroid nodule from TNSCUI2020). Experimental results show that our method is specifically suitable for lesion segmentation in ultrasound images and can outperform the state-of-the-art approaches with Dice of 0.931, 0.906, and 0.876, respectively. The proposed method demonstrates that it can provide more important information about the characteristics of lesions for lesion segmentation in ultrasound images, especially for lesions with irregular shapes and small sizes. It can assist the current lesion segmentation models to better suit clinical needs.
Collapse
Affiliation(s)
- Van Manh
- Medical Ultrasound Image Computing (MUSIC) lab, School of Biomedical Engineering, Shenzhen University, Shenzhen, 518060, China
| | - Xiaohong Jia
- Department of Ultrasound Medicine, Ruijin Hospital, School of Medicine, Shanghai Jiaotong University, Shanghai, 200240, China
| | - Wufeng Xue
- Medical Ultrasound Image Computing (MUSIC) lab, School of Biomedical Engineering, Shenzhen University, Shenzhen, 518060, China
| | - Wenwen Xu
- Department of Ultrasound Medicine, Ruijin Hospital, School of Medicine, Shanghai Jiaotong University, Shanghai, 200240, China
| | - Zihan Mei
- Department of Ultrasound Medicine, Ruijin Hospital, School of Medicine, Shanghai Jiaotong University, Shanghai, 200240, China
| | - Yijie Dong
- Department of Ultrasound Medicine, Ruijin Hospital, School of Medicine, Shanghai Jiaotong University, Shanghai, 200240, China
| | - Jianqiao Zhou
- Department of Ultrasound Medicine, Ruijin Hospital, School of Medicine, Shanghai Jiaotong University, Shanghai, 200240, China.
| | - Ruobing Huang
- Medical Ultrasound Image Computing (MUSIC) lab, School of Biomedical Engineering, Shenzhen University, Shenzhen, 518060, China.
| | - Dong Ni
- Medical Ultrasound Image Computing (MUSIC) lab, School of Biomedical Engineering, Shenzhen University, Shenzhen, 518060, China.
| |
Collapse
|
7
|
Jiang Z, Wu Y, Huang L, Gu M. FDB-Net: Fusion double branch network combining CNN and transformer for medical image segmentation. JOURNAL OF X-RAY SCIENCE AND TECHNOLOGY 2024; 32:931-951. [PMID: 38848160 DOI: 10.3233/xst-230413] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/09/2024]
Abstract
BACKGROUND The rapid development of deep learning techniques has greatly improved the performance of medical image segmentation, and medical image segmentation networks based on convolutional neural networks and Transformer have been widely used in this field. However, due to the limitation of the restricted receptive field of convolutional operation and the lack of local fine information extraction ability of the self-attention mechanism in Transformer, the current neural networks with pure convolutional or Transformer structure as the backbone still perform poorly in medical image segmentation. METHODS In this paper, we propose FDB-Net (Fusion Double Branch Network, FDB-Net), a double branch medical image segmentation network combining CNN and Transformer, by using a CNN containing gnConv blocks and a Transformer containing Varied-Size Window Attention (VWA) blocks as the feature extraction backbone network, the dual-path encoder ensures that the network has a global receptive field as well as access to the target local detail features. We also propose a new feature fusion module (Deep Feature Fusion, DFF), which helps the image to simultaneously fuse features from two different structural encoders during the encoding process, ensuring the effective fusion of global and local information of the image. CONCLUSION Our model achieves advanced results in all three typical tasks of medical image segmentation, which fully validates the effectiveness of FDB-Net.
Collapse
Affiliation(s)
- Zhongchuan Jiang
- State Key Laboratory of Public Big Data, Guiyang, China
- College of Computer Science and Technology, Guizhou University, Guiyang, China
| | - Yun Wu
- State Key Laboratory of Public Big Data, Guiyang, China
- College of Computer Science and Technology, Guizhou University, Guiyang, China
| | - Lei Huang
- State Key Laboratory of Public Big Data, Guiyang, China
- College of Computer Science and Technology, Guizhou University, Guiyang, China
| | - Maohua Gu
- State Key Laboratory of Public Big Data, Guiyang, China
- College of Computer Science and Technology, Guizhou University, Guiyang, China
| |
Collapse
|
8
|
Zama S, Fujioka T, Yamaga E, Kubota K, Mori M, Katsuta L, Yashima Y, Sato A, Kawauchi M, Higuchi S, Kawanishi M, Ishiba T, Oda G, Nakagawa T, Tateishi U. Clinical Utility of Breast Ultrasound Images Synthesized by a Generative Adversarial Network. MEDICINA (KAUNAS, LITHUANIA) 2023; 60:14. [PMID: 38276048 PMCID: PMC10817540 DOI: 10.3390/medicina60010014] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/03/2023] [Revised: 12/10/2023] [Accepted: 12/18/2023] [Indexed: 01/27/2024]
Abstract
BACKGROUND AND OBJECTIVES This study compares the clinical properties of original breast ultrasound images and those synthesized by a generative adversarial network (GAN) to assess the clinical usefulness of GAN-synthesized images. MATERIALS AND METHODS We retrospectively collected approximately 200 breast ultrasound images for each of five representative histological tissue types (cyst, fibroadenoma, scirrhous, solid, and tubule-forming invasive ductal carcinomas) as training images. A deep convolutional GAN (DCGAN) image-generation model synthesized images of the five histological types. Two diagnostic radiologists (reader 1 with 13 years of experience and reader 2 with 7 years of experience) were given a reading test consisting of 50 synthesized and 50 original images (≥1-month interval between sets) to assign the perceived histological tissue type. The percentages of correct diagnoses were calculated, and the reader agreement was assessed using the kappa coefficient. RESULTS The synthetic and original images were indistinguishable. The correct diagnostic rates from the synthetic images for readers 1 and 2 were 86.0% and 78.0% and from the original images were 88.0% and 78.0%, respectively. The kappa values were 0.625 and 0.650 for the synthetic and original images, respectively. The diagnoses made from the DCGAN synthetic images and original images were similar. CONCLUSION The DCGAN-synthesized images closely resemble the original ultrasound images in clinical characteristics, suggesting their potential utility in clinical education and training, particularly for enhancing diagnostic skills in breast ultrasound imaging.
Collapse
Affiliation(s)
- Shu Zama
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Tomoyuki Fujioka
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Emi Yamaga
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Kazunori Kubota
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
- Department of Radiology, Dokkyo Medical University Saitama Medical Center, 2-1-50 Minami-koshigaya, Koshigaya 343-8555, Japan
| | - Mio Mori
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Leona Katsuta
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Yuka Yashima
- Department of Radiology, Dokkyo Medical University Saitama Medical Center, 2-1-50 Minami-koshigaya, Koshigaya 343-8555, Japan
| | - Arisa Sato
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Miho Kawauchi
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Subaru Higuchi
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Masaaki Kawanishi
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Toshiyuki Ishiba
- Department of Surgery, Breast Surgery, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Goshi Oda
- Department of Surgery, Breast Surgery, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Tsuyoshi Nakagawa
- Department of Surgery, Breast Surgery, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| | - Ukihide Tateishi
- Department of Diagnostic Radiology, Tokyo Medical and Dental University Hospital, 1-5-45, Yushima, Bunkyo-ku, Tokyo 113-8501, Japan
| |
Collapse
|
9
|
Strack C, Pomykala KL, Schlemmer HP, Egger J, Kleesiek J. "A net for everyone": fully personalized and unsupervised neural networks trained with longitudinal data from a single patient. BMC Med Imaging 2023; 23:174. [PMID: 37907876 PMCID: PMC10619304 DOI: 10.1186/s12880-023-01128-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/09/2022] [Accepted: 10/16/2023] [Indexed: 11/02/2023] Open
Abstract
BACKGROUND With the rise in importance of personalized medicine and deep learning, we combine the two to create personalized neural networks. The aim of the study is to show a proof of concept that data from just one patient can be used to train deep neural networks to detect tumor progression in longitudinal datasets. METHODS Two datasets with 64 scans from 32 patients with glioblastoma multiforme (GBM) were evaluated in this study. The contrast-enhanced T1w sequences of brain magnetic resonance imaging (MRI) images were used. We trained a neural network for each patient using just two scans from different timepoints to map the difference between the images. The change in tumor volume can be calculated with this map. The neural networks were a form of a Wasserstein-GAN (generative adversarial network), an unsupervised learning architecture. The combination of data augmentation and the network architecture allowed us to skip the co-registration of the images. Furthermore, no additional training data, pre-training of the networks or any (manual) annotations are necessary. RESULTS The model achieved an AUC-score of 0.87 for tumor change. We also introduced a modified RANO criteria, for which an accuracy of 66% can be achieved. CONCLUSIONS We show a novel approach to deep learning in using data from just one patient to train deep neural networks to monitor tumor change. Using two different datasets to evaluate the results shows the potential to generalize the method.
Collapse
Affiliation(s)
- Christian Strack
- Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), Girardetstraße 2, 45131, Essen, Germany.
- Division of Radiology, German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany.
- Medical Faculty Heidelberg, Heidelberg University, 69120, Heidelberg, Germany.
| | - Kelsey L Pomykala
- Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), Girardetstraße 2, 45131, Essen, Germany
| | - Heinz-Peter Schlemmer
- Division of Radiology, German Cancer Research Center (DKFZ), 69120, Heidelberg, Germany
- German Cancer Consortium (DKTK), Partner Site Essen, Hufelandstraße 55, 45147, Essen, Germany
| | - Jan Egger
- Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), Girardetstraße 2, 45131, Essen, Germany
- Cancer Research Center Cologne Essen (CCCE), University Medicine Essen, Hufelandstraße 55, 45147, Essen, Germany
| | - Jens Kleesiek
- Institute for AI in Medicine (IKIM), University Hospital Essen (AöR), Girardetstraße 2, 45131, Essen, Germany
- German Cancer Consortium (DKTK), Partner Site Essen, Hufelandstraße 55, 45147, Essen, Germany
- Cancer Research Center Cologne Essen (CCCE), University Medicine Essen, Hufelandstraße 55, 45147, Essen, Germany
- Department of Physics, TU Dortmund University, Otto-Hahn-Straße 4, D-44227, Dortmund, Germany
| |
Collapse
|
10
|
Chen L, Liao H, Kong W, Zhang D, Chen F. Anatomy preserving GAN for realistic simulation of intraoperative liver ultrasound images. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2023; 240:107642. [PMID: 37480644 DOI: 10.1016/j.cmpb.2023.107642] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/26/2023] [Revised: 05/31/2023] [Accepted: 06/02/2023] [Indexed: 07/24/2023]
Abstract
In ultrasound-guided liver surgery, the lack of large-scale intraoperative ultrasound images with important anatomical structures remains an obstacle hindering the successful application of AI to ultrasound guidance. In this case, intraoperative ultrasound (iUS) simulation should be conducted from preoperative magnetic resonance (pMR), which not only helps doctors understand the characteristics of iUS in advance, but also expands the iUS dataset from various imaging positions, thereby promoting the automatic iUS analysis in ultrasound guidance. Herein, a novel anatomy preserving generative adversarial network (ApGAN) framework was proposed to generate simulated intraoperative ultrasound (Sim-iUS) of liver with precise structure information from pMR. Specifically, the low-rank factors based bimodal fusion was first established focusing on the effective information of hepatic parenchyma. Then, a deformation field based correction module was introduced to learn and correct the slight structural distortion from surgical operations. Meanwhile, the multiple loss functions were designed to constrain the simulation of the content, structures, and style. Empirical results of clinical data showed that the proposed ApGAN obtained higher Structural Similarity (SSIM) of 0.74 and Fr´echet Inception Distance (FID) of 35.54 compared to existing methods. Furthermore, the average Hausdorff Distance (HD) error of the liver capsule structure was less than 0.25 mm, and the average relative (Euclidean Distance) ED error for polyps was 0.12 mm, indicating the high-level precision of this ApGAN in simulating the anatomical structures and focal areas.
Collapse
Affiliation(s)
- Lingyu Chen
- Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing University of Aeronautics and Astronautics, Nanjing, China; College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China
| | - Hongen Liao
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, China
| | - Wentao Kong
- Department of Ultrasound, Affiliated DrumTower Hospital, Medical School of Nanjing University, Nanjing 210008, China.
| | - Daoqiang Zhang
- Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing University of Aeronautics and Astronautics, Nanjing, China; College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China
| | - Fang Chen
- Key Laboratory of Brain-Machine Intelligence Technology, Ministry of Education, Nanjing University of Aeronautics and Astronautics, Nanjing, China; College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, Nanjing, China.
| |
Collapse
|
11
|
Huang X, Bajaj R, Li Y, Ye X, Lin J, Pugliese F, Ramasamy A, Gu Y, Wang Y, Torii R, Dijkstra J, Zhou H, Bourantas CV, Zhang Q. POST-IVUS: A perceptual organisation-aware selective transformer framework for intravascular ultrasound segmentation. Med Image Anal 2023; 89:102922. [PMID: 37598605 DOI: 10.1016/j.media.2023.102922] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/15/2022] [Revised: 07/06/2023] [Accepted: 08/01/2023] [Indexed: 08/22/2023]
Abstract
Intravascular ultrasound (IVUS) is recommended in guiding coronary intervention. The segmentation of coronary lumen and external elastic membrane (EEM) borders in IVUS images is a key step, but the manual process is time-consuming and error-prone, and suffers from inter-observer variability. In this paper, we propose a novel perceptual organisation-aware selective transformer framework that can achieve accurate and robust segmentation of the vessel walls in IVUS images. In this framework, temporal context-based feature encoders extract efficient motion features of vessels. Then, a perceptual organisation-aware selective transformer module is proposed to extract accurate boundary information, supervised by a dedicated boundary loss. The obtained EEM and lumen segmentation results will be fused in a temporal constraining and fusion module, to determine the most likely correct boundaries with robustness to morphology. Our proposed methods are extensively evaluated in non-selected IVUS sequences, including normal, bifurcated, and calcified vessels with shadow artifacts. The results show that the proposed methods outperform the state-of-the-art, with a Jaccard measure of 0.92 for lumen and 0.94 for EEM on the IVUS 2011 open challenge dataset. This work has been integrated into a software QCU-CMS2 to automatically segment IVUS images in a user-friendly environment.
Collapse
Affiliation(s)
- Xingru Huang
- School of Electronic Engineering and Computer Science, Queen Mary University of London, London, E3 4BL, UK; School of Communication Engineering, Hangzhou Dianzi University, Xiasha Higher Education Zone, Hangzhou, Zhejiang, China
| | - Retesh Bajaj
- Department of Cardiology, Barts Heart Centre, Barts Health NHS Trust, West Smithfield, London, EC1A 7BE, UK; Centre for Cardiovascular Medicine and Devices, William Harvey Research Institute, Queen Mary University of London, London, UK
| | - Yilong Li
- School of Electronic Engineering and Computer Science, Queen Mary University of London, London, E3 4BL, UK
| | - Xin Ye
- Zhejiang Provincial People's Hospital, 270 West Xueyuan Road, Wenzhou, Zhejiang, China
| | - Ji Lin
- School of Electronic Engineering and Computer Science, Queen Mary University of London, London, E3 4BL, UK
| | - Francesca Pugliese
- Department of Cardiology, Barts Heart Centre, Barts Health NHS Trust, West Smithfield, London, EC1A 7BE, UK; Centre for Cardiovascular Medicine and Devices, William Harvey Research Institute, Queen Mary University of London, London, UK
| | - Anantharaman Ramasamy
- Department of Cardiology, Barts Heart Centre, Barts Health NHS Trust, West Smithfield, London, EC1A 7BE, UK; Centre for Cardiovascular Medicine and Devices, William Harvey Research Institute, Queen Mary University of London, London, UK
| | - Yue Gu
- Zhejiang Institute of Mechanical and Electrical Engineering, Hangzhou, China
| | - Yaqi Wang
- College of Media Engineering, Communication University of Zhejiang, Hangzhou, China
| | - Ryo Torii
- Department of Mechanical Engineering, University College London, London, UK
| | | | - Huiyu Zhou
- School of Informatics, University of Leicester, University Road, Leicester, LE1 7RH, United Kingdom
| | - Christos V Bourantas
- Department of Cardiology, Barts Heart Centre, Barts Health NHS Trust, West Smithfield, London, EC1A 7BE, UK; Centre for Cardiovascular Medicine and Devices, William Harvey Research Institute, Queen Mary University of London, London, UK
| | - Qianni Zhang
- School of Electronic Engineering and Computer Science, Queen Mary University of London, London, E3 4BL, UK.
| |
Collapse
|
12
|
Zhou L, Liang L, Sheng X. GA-Net: Ghost convolution adaptive fusion skin lesion segmentation network. Comput Biol Med 2023; 164:107273. [PMID: 37562327 DOI: 10.1016/j.compbiomed.2023.107273] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/26/2023] [Revised: 06/30/2023] [Accepted: 07/16/2023] [Indexed: 08/12/2023]
Abstract
Automatic segmentation of skin lesions is a pivotal task in computer-aided diagnosis, playing a crucial role in the early detection and treatment of skin cancer. Despite the existence of numerous deep learning-based segmentation methods, the extraction of lesion features remains inadequate as a result of the segmentation process. Consequently, skin lesion image segmentation continues to face challenges regarding missing detailed information and inaccurate segmentation of the lesion region. In this paper, we propose a ghost convolution adaptive fusion network for skin lesion segmentation. First, the neural network incorporates a ghost module instead of the ordinary convolution layer, generating a rich skin lesion feature map for comprehensive target feature extraction. Subsequently, the network employs an adaptive fusion module and bilateral attention module to connect the encoding and decoding layers, facilitating the integration of shallow and deep network information. Moreover, multi-level output patterns are used for pixel prediction. Layer feature fusion effectively combines output features of different scales, thus improving image segmentation accuracy. The proposed network was extensively evaluated on three publicly available datasets: ISIC2016, ISIC2017, and ISIC2018. The experimental results demonstrated accuracies of 96.42%, 94.07%, and 95.03%, and kappa coefficients of 90.41%, 81.08%, and 86.96%, respectively. The overall performance of our network surpassed that of existing networks. Simulation experiments further revealed that the ghost convolution adaptive fusion network exhibited superior segmentation results for skin lesion images, offering new possibilities for the diagnosis of skin diseases.
Collapse
Affiliation(s)
- Longsong Zhou
- School of Electrical Engineering and Automation, Jiangxi University of Science and Technology, Ganzhou, Jiangxi, 341000, China; Jinguan Copper Branch of Tongling Nonferrous Metals Group Co, Ltd, Tongling, Anhui, 244100, China
| | - Liming Liang
- School of Electrical Engineering and Automation, Jiangxi University of Science and Technology, Ganzhou, Jiangxi, 341000, China
| | - Xiaoqi Sheng
- School of Computer Science and Engineering, South China University of Technology, Guangzhou, Guangdong, 510006, China.
| |
Collapse
|
13
|
Innani S, Dutande P, Baid U, Pokuri V, Bakas S, Talbar S, Baheti B, Guntuku SC. Generative adversarial networks based skin lesion segmentation. Sci Rep 2023; 13:13467. [PMID: 37596306 PMCID: PMC10439152 DOI: 10.1038/s41598-023-39648-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/23/2023] [Accepted: 07/28/2023] [Indexed: 08/20/2023] Open
Abstract
Skin cancer is a serious condition that requires accurate diagnosis and treatment. One way to assist clinicians in this task is using computer-aided diagnosis tools that automatically segment skin lesions from dermoscopic images. We propose a novel adversarial learning-based framework called Efficient-GAN (EGAN) that uses an unsupervised generative network to generate accurate lesion masks. It consists of a generator module with a top-down squeeze excitation-based compound scaled path, an asymmetric lateral connection-based bottom-up path, and a discriminator module that distinguishes between original and synthetic masks. A morphology-based smoothing loss is also implemented to encourage the network to create smooth semantic boundaries of lesions. The framework is evaluated on the International Skin Imaging Collaboration Lesion Dataset. It outperforms the current state-of-the-art skin lesion segmentation approaches with a Dice coefficient, Jaccard similarity, and accuracy of 90.1%, 83.6%, and 94.5%, respectively. We also design a lightweight segmentation framework called Mobile-GAN (MGAN) that achieves comparable performance as EGAN but with an order of magnitude lower number of training parameters, thus resulting in faster inference times for low compute resource settings.
Collapse
Affiliation(s)
- Shubham Innani
- Center of Excellence in Signal and Image Processing, Shri Guru Gobind Singhji Institute of Engineering and Technology, Nanded, Maharashtra, India.
| | - Prasad Dutande
- Center of Excellence in Signal and Image Processing, Shri Guru Gobind Singhji Institute of Engineering and Technology, Nanded, Maharashtra, India
| | - Ujjwal Baid
- Center of Excellence in Signal and Image Processing, Shri Guru Gobind Singhji Institute of Engineering and Technology, Nanded, Maharashtra, India
- Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, USA
| | - Venu Pokuri
- Department of Computer and Information Science, University of Pennsylvania, Philadelphia, USA
| | - Spyridon Bakas
- Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, USA
| | - Sanjay Talbar
- Center of Excellence in Signal and Image Processing, Shri Guru Gobind Singhji Institute of Engineering and Technology, Nanded, Maharashtra, India
| | - Bhakti Baheti
- Center of Excellence in Signal and Image Processing, Shri Guru Gobind Singhji Institute of Engineering and Technology, Nanded, Maharashtra, India
- Center for Biomedical Image Computing and Analytics, University of Pennsylvania, Philadelphia, USA
| | - Sharath Chandra Guntuku
- Department of Computer and Information Science, University of Pennsylvania, Philadelphia, USA
| |
Collapse
|
14
|
Yadav DP, Aljrees T, Kumar D, Kumar A, Singh KU, Singh T. Spatial attention-based residual network for human burn identification and classification. Sci Rep 2023; 13:12516. [PMID: 37532880 PMCID: PMC10397300 DOI: 10.1038/s41598-023-39618-0] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2023] [Accepted: 07/27/2023] [Indexed: 08/04/2023] Open
Abstract
Diagnosing burns in humans has become critical, as early identification can save lives. The manual process of burn diagnosis is time-consuming and complex, even for experienced doctors. Machine learning (ML) and deep convolutional neural network (CNN) models have emerged as the standard for medical image diagnosis. The ML-based approach typically requires handcrafted features for training, which may result in suboptimal performance. Conversely, DL-based methods automatically extract features, but designing a robust model is challenging. Additionally, shallow DL methods lack long-range feature dependency, decreasing efficiency in various applications. We implemented several deep CNN models, ResNeXt, VGG16, and AlexNet, for human burn diagnosis. The results obtained from these models were found to be less reliable since shallow deep CNN models need improved attention modules to preserve the feature dependencies. Therefore, in the proposed study, the feature map is divided into several categories, and the channel dependencies between any two channel mappings within a given class are highlighted. A spatial attention map is built by considering the links between features and their locations. Our attention-based model BuRnGANeXt50 kernel and convolutional layers are also optimized for human burn diagnosis. The earlier study classified the burn based on depth of graft and non-graft. We first classified the burn based on the degree. Subsequently, it is classified into graft and non-graft. Furthermore, the proposed model performance is evaluated on Burns_BIP_US_database. The sensitivity of the BuRnGANeXt50 is 97.22% and 99.14%, respectively, for classifying burns based on degree and depth. This model may be used for quick screening of burn patients and can be executed in the cloud or on a local machine. The code of the proposed method can be accessed at https://github.com/dhirujis02/Journal.git for the sake of reproducibility.
Collapse
Affiliation(s)
- D P Yadav
- Department of Computer Engineering and Applications, GLA University, Mathura, India
| | - Turki Aljrees
- Department College of Computer Sci. and Eng., University of Hafr Al-Batin, Hafar Al-Batin, 39524, Saudi Arabia
| | - Deepak Kumar
- Department of Computer Science, NIT Meghalaya, Shillong, India
| | - Ankit Kumar
- Department of Computer Engineering and Applications, GLA University, Mathura, India.
| | - Kamred Udham Singh
- School of Computing, Graphic Era Hill University, Dehradun, 248002, India
| | - Teekam Singh
- Department of Computer Science and Engineering, Graphic Era Deemed to be University, Dehradun, 248002, India
| |
Collapse
|
15
|
Mu J, Lin Y, Meng X, Fan J, Ai D, Chen D, Qiu H, Yang J, Gu Y. M-CSAFN: Multi-Color Space Adaptive Fusion Network for Automated Port-Wine Stains Segmentation. IEEE J Biomed Health Inform 2023; 27:3924-3935. [PMID: 37027679 DOI: 10.1109/jbhi.2023.3247479] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/24/2023]
Abstract
Automatic segmentation of port-wine stains (PWS) from clinical images is critical for accurate diagnosis and objective assessment of PWS. However, this is a challenging task due to the color heterogeneity, low contrast, and indistinguishable appearance of PWS lesions. To address such challenges, we propose a novel multi-color space adaptive fusion network (M-CSAFN) for PWS segmentation. First, a multi-branch detection model is constructed based on six typical color spaces, which utilizes rich color texture information to highlight the difference between lesions and surrounding tissues. Second, an adaptive fusion strategy is used to fuse complementary predictions, which address the significant differences within the lesions caused by color heterogeneity. Third, a structural similarity loss with color information is proposed to measure the detail error between predicted lesions and truth lesions. Additionally, a PWS clinical dataset consisting of 1413 image pairs was established for the development and evaluation of PWS segmentation algorithms. To verify the effectiveness and superiority of the proposed method, we compared it with other state-of-the-art methods on our collected dataset and four publicly available skin lesion datasets (ISIC 2016, ISIC 2017, ISIC 2018, and PH2). The experimental results show that our method achieves remarkable performance in comparison with other state-of-the-art methods on our collected dataset, achieving 92.29% and 86.14% on Dice and Jaccard metrics, respectively. Comparative experiments on other datasets also confirmed the reliability and potential capability of M-CSAFN in skin lesion segmentation.
Collapse
|
16
|
Yu Z, Yu L, Zheng W, Wang S. EIU-Net: Enhanced feature extraction and improved skip connections in U-Net for skin lesion segmentation. Comput Biol Med 2023; 162:107081. [PMID: 37301097 DOI: 10.1016/j.compbiomed.2023.107081] [Citation(s) in RCA: 6] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2022] [Revised: 04/25/2023] [Accepted: 05/27/2023] [Indexed: 06/12/2023]
Abstract
Skin lesion segmentation is a computer-aided diagnosis method for quantitative analysis of melanoma that can improve efficiency and accuracy. Although many methods based on U-Net have achieved tremendous success, they still cannot handle challenging tasks well due to weak feature extraction. In response to skin lesion segmentation, a novel method called EIU-Net is proposed to tackle the challenging task. To capture the local and global contextual information, we employ inverted residual blocks and an efficient pyramid squeeze attention (EPSA) block as the main encoders at different stages, while atrous spatial pyramid pooling (ASPP) is utilized after the last encoder and the soft-pool method is introduced for downsampling. Also, we propose a novel method named multi-layer fusion (MLF) module to effectively fuse the feature distributions and capture significant boundary information of skin lesions in different encoders to improve the performance of the network. Furthermore, a reshaped decoders fusion module is used to obtain multi-scale information by fusing feature maps of different decoders to improve the final results of skin lesion segmentation. To validate the performance of our proposed network, we compare it with other methods on four public datasets, including the ISIC 2016, ISIC 2017, ISIC 2018, and PH2 datasets. And the main metric Dice scores achieved by our proposed EIU-Net are 0.919, 0.855, 0.902, and 0.916 on the four datasets, respectively, outperforming other methods. Ablation experiments also demonstrate the effectiveness of the main modules in our proposed network. Our code is available at https://github.com/AwebNoob/EIU-Net.
Collapse
Affiliation(s)
- Zimin Yu
- Department of Computer Science and Engineering, School of Information Science and Engineering, Yunnan University, Kunming, 650504, Yunnan, China
| | - Li Yu
- Department of Computer Science and Engineering, School of Information Science and Engineering, Yunnan University, Kunming, 650504, Yunnan, China
| | - Weihua Zheng
- Department of Computer Science and Engineering, School of Information Science and Engineering, Yunnan University, Kunming, 650504, Yunnan, China
| | - Shunfang Wang
- Department of Computer Science and Engineering, School of Information Science and Engineering, Yunnan University, Kunming, 650504, Yunnan, China; The Key Lab of Intelligent Systems and Computing of Yunnan Province, Yunnan University, Kunming, Yunnan, China.
| |
Collapse
|
17
|
Huang Y, Jiao J, Yu J, Zheng Y, Wang Y. RsALUNet: A reinforcement supervision U-Net-based framework for multi-ROI segmentation of medical images. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2023.104743] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/27/2023]
|
18
|
Gong C, Jing C, Chen X, Pun CM, Huang G, Saha A, Nieuwoudt M, Li HX, Hu Y, Wang S. Generative AI for brain image computing and brain network computing: a review. Front Neurosci 2023; 17:1203104. [PMID: 37383107 PMCID: PMC10293625 DOI: 10.3389/fnins.2023.1203104] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/10/2023] [Accepted: 05/22/2023] [Indexed: 06/30/2023] Open
Abstract
Recent years have witnessed a significant advancement in brain imaging techniques that offer a non-invasive approach to mapping the structure and function of the brain. Concurrently, generative artificial intelligence (AI) has experienced substantial growth, involving using existing data to create new content with a similar underlying pattern to real-world data. The integration of these two domains, generative AI in neuroimaging, presents a promising avenue for exploring various fields of brain imaging and brain network computing, particularly in the areas of extracting spatiotemporal brain features and reconstructing the topological connectivity of brain networks. Therefore, this study reviewed the advanced models, tasks, challenges, and prospects of brain imaging and brain network computing techniques and intends to provide a comprehensive picture of current generative AI techniques in brain imaging. This review is focused on novel methodological approaches and applications of related new methods. It discussed fundamental theories and algorithms of four classic generative models and provided a systematic survey and categorization of tasks, including co-registration, super-resolution, enhancement, classification, segmentation, cross-modality, brain network analysis, and brain decoding. This paper also highlighted the challenges and future directions of the latest work with the expectation that future research can be beneficial.
Collapse
Affiliation(s)
- Changwei Gong
- Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
- Department of Computer Science, University of Chinese Academy of Sciences, Beijing, China
| | - Changhong Jing
- Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
- Department of Computer Science, University of Chinese Academy of Sciences, Beijing, China
| | - Xuhang Chen
- Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
- Department of Computer and Information Science, University of Macau, Macau, China
| | - Chi Man Pun
- Department of Computer and Information Science, University of Macau, Macau, China
| | - Guoli Huang
- Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| | - Ashirbani Saha
- Department of Oncology and School of Biomedical Engineering, McMaster University, Hamilton, ON, Canada
| | - Martin Nieuwoudt
- Institute for Biomedical Engineering, Stellenbosch University, Stellenbosch, South Africa
| | - Han-Xiong Li
- Department of Systems Engineering, City University of Hong Kong, Hong Kong, China
| | - Yong Hu
- Department of Orthopaedics and Traumatology, The University of Hong Kong, Hong Kong, China
| | - Shuqiang Wang
- Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
- Department of Computer Science, University of Chinese Academy of Sciences, Beijing, China
| |
Collapse
|
19
|
Mirikharaji Z, Abhishek K, Bissoto A, Barata C, Avila S, Valle E, Celebi ME, Hamarneh G. A survey on deep learning for skin lesion segmentation. Med Image Anal 2023; 88:102863. [PMID: 37343323 DOI: 10.1016/j.media.2023.102863] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/24/2022] [Revised: 02/01/2023] [Accepted: 05/31/2023] [Indexed: 06/23/2023]
Abstract
Skin cancer is a major public health problem that could benefit from computer-aided diagnosis to reduce the burden of this common disease. Skin lesion segmentation from images is an important step toward achieving this goal. However, the presence of natural and artificial artifacts (e.g., hair and air bubbles), intrinsic factors (e.g., lesion shape and contrast), and variations in image acquisition conditions make skin lesion segmentation a challenging task. Recently, various researchers have explored the applicability of deep learning models to skin lesion segmentation. In this survey, we cross-examine 177 research papers that deal with deep learning-based segmentation of skin lesions. We analyze these works along several dimensions, including input data (datasets, preprocessing, and synthetic data generation), model design (architecture, modules, and losses), and evaluation aspects (data annotation requirements and segmentation performance). We discuss these dimensions both from the viewpoint of select seminal works, and from a systematic viewpoint, examining how those choices have influenced current trends, and how their limitations should be addressed. To facilitate comparisons, we summarize all examined works in a comprehensive table as well as an interactive table available online3.
Collapse
Affiliation(s)
- Zahra Mirikharaji
- Medical Image Analysis Lab, School of Computing Science, Simon Fraser University, Burnaby V5A 1S6, Canada
| | - Kumar Abhishek
- Medical Image Analysis Lab, School of Computing Science, Simon Fraser University, Burnaby V5A 1S6, Canada
| | - Alceu Bissoto
- RECOD.ai Lab, Institute of Computing, University of Campinas, Av. Albert Einstein 1251, Campinas 13083-852, Brazil
| | - Catarina Barata
- Institute for Systems and Robotics, Instituto Superior Técnico, Avenida Rovisco Pais, Lisbon 1049-001, Portugal
| | - Sandra Avila
- RECOD.ai Lab, Institute of Computing, University of Campinas, Av. Albert Einstein 1251, Campinas 13083-852, Brazil
| | - Eduardo Valle
- RECOD.ai Lab, School of Electrical and Computing Engineering, University of Campinas, Av. Albert Einstein 400, Campinas 13083-952, Brazil
| | - M Emre Celebi
- Department of Computer Science and Engineering, University of Central Arkansas, 201 Donaghey Ave., Conway, AR 72035, USA.
| | - Ghassan Hamarneh
- Medical Image Analysis Lab, School of Computing Science, Simon Fraser University, Burnaby V5A 1S6, Canada.
| |
Collapse
|
20
|
Guo Q, Fang X, Wang L, Zhang E, Liu Z. Robust fusion for skin lesion segmentation of dermoscopic images. Front Bioeng Biotechnol 2023; 11:1057866. [PMID: 37020509 PMCID: PMC10069440 DOI: 10.3389/fbioe.2023.1057866] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2022] [Accepted: 02/21/2023] [Indexed: 03/22/2023] Open
Abstract
Robust skin lesion segmentation of dermoscopic images is still very difficult. Recent methods often take the combinations of CNN and Transformer for feature abstraction and multi-scale features for further classification. Both types of combination in general rely on some forms of feature fusion. This paper considers these fusions from two novel points of view. For abstraction, Transformer is viewed as the affinity exploration of different patch tokens and can be applied to attend CNN features in multiple scales. Consequently, a new fusion module, the Attention-based Transformer-And-CNN fusion module (ATAC), is proposed. ATAC augments the CNN features with more global contexts. For further classification, adaptively combining the information from multiple scales according to their contributions to object recognition is expected. Accordingly, a new fusion module, the GAting-based Multi-Scale fusion module (GAMS), is also introduced, which adaptively weights the information from multiple scales by the light-weighted gating mechanism. Combining ATAC and GAMS leads to a new encoder-decoder-based framework. In this method, ATAC acts as an encoder block to progressively abstract strong CNN features with rich global contexts attended by long-range relations, while GAMS works as an enhancement of the decoder to generate the discriminative features through adaptive fusion of multi-scale ones. This framework is especially good at lesions of varying sizes and shapes and of low contrasts and its performances are demonstrated with extensive experiments on public skin lesion segmentation datasets.
Collapse
Affiliation(s)
- Qingqing Guo
- School of Computer Science and Technology, Anhui University, Hefei, China
| | - Xianyong Fang
- School of Computer Science and Technology, Anhui University, Hefei, China
- *Correspondence: Xianyong Fang,
| | - Linbo Wang
- School of Computer Science and Technology, Anhui University, Hefei, China
| | - Enming Zhang
- Islet Pathophysiology, Department of Clinical Science, Lund University Diabetes Centre, Malmö, Sweden
| | - Zhengyi Liu
- School of Computer Science and Technology, Anhui University, Hefei, China
| |
Collapse
|
21
|
Chen J, Zhang J, Debattista K, Han J. Semi-Supervised Unpaired Medical Image Segmentation Through Task-Affinity Consistency. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:594-605. [PMID: 36219664 DOI: 10.1109/tmi.2022.3213372] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
Deep learning-based semi-supervised learning (SSL) algorithms are promising in reducing the cost of manual annotation of clinicians by using unlabelled data, when developing medical image segmentation tools. However, to date, most existing semi-supervised learning (SSL) algorithms treat the labelled images and unlabelled images separately and ignore the explicit connection between them; this disregards essential shared information and thus hinders further performance improvements. To mine the shared information between the labelled and unlabelled images, we introduce a class-specific representation extraction approach, in which a task-affinity module is specifically designed for representation extraction. We further cast the representation into two different views of feature maps; one is focusing on low-level context, while the other concentrates on structural information. The two views of feature maps are incorporated into the task-affinity module, which then extracts the class-specific representations to aid the knowledge transfer from the labelled images to the unlabelled images. In particular, a task-affinity consistency loss between the labelled images and unlabelled images based on the multi-scale class-specific representations is formulated, leading to a significant performance improvement. Experimental results on three datasets show that our method consistently outperforms existing state-of-the-art methods. Our findings highlight the potential of consistency between class-specific knowledge for semi-supervised medical image segmentation. The code and models are to be made publicly available at https://github.com/jingkunchen/TAC.
Collapse
|
22
|
Hasan MK, Ahamad MA, Yap CH, Yang G. A survey, review, and future trends of skin lesion segmentation and classification. Comput Biol Med 2023; 155:106624. [PMID: 36774890 DOI: 10.1016/j.compbiomed.2023.106624] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/20/2022] [Revised: 01/04/2023] [Accepted: 01/28/2023] [Indexed: 02/03/2023]
Abstract
The Computer-aided Diagnosis or Detection (CAD) approach for skin lesion analysis is an emerging field of research that has the potential to alleviate the burden and cost of skin cancer screening. Researchers have recently indicated increasing interest in developing such CAD systems, with the intention of providing a user-friendly tool to dermatologists to reduce the challenges encountered or associated with manual inspection. This article aims to provide a comprehensive literature survey and review of a total of 594 publications (356 for skin lesion segmentation and 238 for skin lesion classification) published between 2011 and 2022. These articles are analyzed and summarized in a number of different ways to contribute vital information regarding the methods for the development of CAD systems. These ways include: relevant and essential definitions and theories, input data (dataset utilization, preprocessing, augmentations, and fixing imbalance problems), method configuration (techniques, architectures, module frameworks, and losses), training tactics (hyperparameter settings), and evaluation criteria. We intend to investigate a variety of performance-enhancing approaches, including ensemble and post-processing. We also discuss these dimensions to reveal their current trends based on utilization frequencies. In addition, we highlight the primary difficulties associated with evaluating skin lesion segmentation and classification systems using minimal datasets, as well as the potential solutions to these difficulties. Findings, recommendations, and trends are disclosed to inform future research on developing an automated and robust CAD system for skin lesion analysis.
Collapse
Affiliation(s)
- Md Kamrul Hasan
- Department of Bioengineering, Imperial College London, UK; Department of Electrical and Electronic Engineering (EEE), Khulna University of Engineering & Technology (KUET), Khulna 9203, Bangladesh.
| | - Md Asif Ahamad
- Department of Electrical and Electronic Engineering (EEE), Khulna University of Engineering & Technology (KUET), Khulna 9203, Bangladesh.
| | - Choon Hwai Yap
- Department of Bioengineering, Imperial College London, UK.
| | - Guang Yang
- National Heart and Lung Institute, Imperial College London, UK; Cardiovascular Research Centre, Royal Brompton Hospital, UK.
| |
Collapse
|
23
|
Jiang Y, Dong J, Zhang Y, Cheng T, Lin X, Liang J. PCF-Net: Position and context information fusion attention convolutional neural network for skin lesion segmentation. Heliyon 2023; 9:e13942. [PMID: 36923881 PMCID: PMC10009446 DOI: 10.1016/j.heliyon.2023.e13942] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/11/2022] [Revised: 02/10/2023] [Accepted: 02/15/2023] [Indexed: 02/27/2023] Open
Abstract
Skin lesion segmentation is a crucial step in the process of skin cancer diagnosis and treatment. The variation in position, shape, size and edges of skin lesion areas poses a challenge for accurate segmentation of skin lesion areas through dermoscopic images. To meet these challenges, in this paper, using UNet as the baseline model, a convolutional neural network based on position and context information fusion attention is proposed, called PCF-Net. A novel two-branch attention mechanism is designed to aggregate Position and Context information, called Position and Context Information Aggregation Attention Module (PCFAM). A global context information complementary module (GCCM) was developed to obtain long-range dependencies. A multi-scale grouped dilated convolution feature extraction module (MSEM) was proposed to capture multi-scale feature information and place it in the bottleneck of UNet. On the ISIC2018 dataset, a large volume of ablation experiments demonstrated the superiority of PCF-Net for dermoscopic image segmentation after adding PCFAM, GCCM and MSEM. Compared with other state-of-the-art methods, the performance of PCF-Net achieves a competitive result in all metrics.
Collapse
Affiliation(s)
- Yun Jiang
- College of Computer Science and Engineering, Northwest Normal University, Lanzhou, China
| | - Jinkun Dong
- College of Computer Science and Engineering, Northwest Normal University, Lanzhou, China
| | - Yuan Zhang
- College of Computer Science and Engineering, Northwest Normal University, Lanzhou, China
| | - Tongtong Cheng
- College of Computer Science and Engineering, Northwest Normal University, Lanzhou, China
| | - Xin Lin
- College of Computer Science and Engineering, Northwest Normal University, Lanzhou, China
| | - Jing Liang
- College of Computer Science and Engineering, Northwest Normal University, Lanzhou, China
| |
Collapse
|
24
|
Wang Y, Su J, Xu Q, Zhong Y. A Collaborative Learning Model for Skin Lesion Segmentation and Classification. Diagnostics (Basel) 2023; 13:diagnostics13050912. [PMID: 36900056 PMCID: PMC10001355 DOI: 10.3390/diagnostics13050912] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/31/2023] [Revised: 02/19/2023] [Accepted: 02/24/2023] [Indexed: 03/06/2023] Open
Abstract
The automatic segmentation and classification of skin lesions are two essential tasks in computer-aided skin cancer diagnosis. Segmentation aims to detect the location and boundary of the skin lesion area, while classification is used to evaluate the type of skin lesion. The location and contour information of lesions provided by segmentation is essential for the classification of skin lesions, while the skin disease classification helps generate target localization maps to assist the segmentation task. Although the segmentation and classification are studied independently in most cases, we find meaningful information can be explored using the correlation of dermatological segmentation and classification tasks, especially when the sample data are insufficient. In this paper, we propose a collaborative learning deep convolutional neural networks (CL-DCNN) model based on the teacher-student learning method for dermatological segmentation and classification. To generate high-quality pseudo-labels, we provide a self-training method. The segmentation network is selectively retrained through classification network screening pseudo-labels. Specially, we obtain high-quality pseudo-labels for the segmentation network by providing a reliability measure method. We also employ class activation maps to improve the location ability of the segmentation network. Furthermore, we provide the lesion contour information by using the lesion segmentation masks to improve the recognition ability of the classification network. Experiments are carried on the ISIC 2017 and ISIC Archive datasets. The CL-DCNN model achieved a Jaccard of 79.1% on the skin lesion segmentation task and an average AUC of 93.7% on the skin disease classification task, which is superior to the advanced skin lesion segmentation methods and classification methods.
Collapse
Affiliation(s)
- Ying Wang
- School of Information Science and Engineering, University of Jinan, Jinan 250022, China
- Shandong Provincial Key Laboratory of Network Based Intelligent Computing, University of Jinan, Jinan 250022, China
| | - Jie Su
- School of Information Science and Engineering, University of Jinan, Jinan 250022, China
- Shandong Provincial Key Laboratory of Network Based Intelligent Computing, University of Jinan, Jinan 250022, China
- Correspondence: ; Tel.: +86-15054125550
| | - Qiuyu Xu
- School of Information Science and Engineering, University of Jinan, Jinan 250022, China
- Shandong Provincial Key Laboratory of Network Based Intelligent Computing, University of Jinan, Jinan 250022, China
| | - Yixin Zhong
- School of Information Science and Engineering, University of Jinan, Jinan 250022, China
- Artificial Intelligence Research Institute, University of Jinan, Jinan 250022, China
| |
Collapse
|
25
|
Qureshi I, Yan J, Abbas Q, Shaheed K, Riaz AB, Wahid A, Khan MWJ, Szczuko P. Medical image segmentation using deep semantic-based methods: A review of techniques, applications and emerging trends. INFORMATION FUSION 2023. [DOI: 10.1016/j.inffus.2022.09.031] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/08/2023]
|
26
|
Oliveira B, Torres HR, Morais P, Veloso F, Baptista AL, Fonseca JC, Vilaça JL. A multi-task convolutional neural network for classification and segmentation of chronic venous disorders. Sci Rep 2023; 13:761. [PMID: 36641527 PMCID: PMC9840616 DOI: 10.1038/s41598-022-27089-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/08/2022] [Accepted: 12/26/2022] [Indexed: 01/16/2023] Open
Abstract
Chronic Venous Disorders (CVD) of the lower limbs are one of the most prevalent medical conditions, affecting 35% of adults in Europe and North America. Due to the exponential growth of the aging population and the worsening of CVD with age, it is expected that the healthcare costs and the resources needed for the treatment of CVD will increase in the coming years. The early diagnosis of CVD is fundamental in treatment planning, while the monitoring of its treatment is fundamental to assess a patient's condition and quantify the evolution of CVD. However, correct diagnosis relies on a qualitative approach through visual recognition of the various venous disorders, being time-consuming and highly dependent on the physician's expertise. In this paper, we propose a novel automatic strategy for the joint segmentation and classification of CVDs. The strategy relies on a multi-task deep learning network, denominated VENet, that simultaneously solves segmentation and classification tasks, exploiting the information of both tasks to increase learning efficiency, ultimately improving their performance. The proposed method was compared against state-of-the-art strategies in a dataset of 1376 CVD images. Experiments showed that the VENet achieved a classification performance of 96.4%, 96.4%, and 97.2% for accuracy, precision, and recall, respectively, and a segmentation performance of 75.4%, 76.7.0%, 76.7% for the Dice coefficient, precision, and recall, respectively. The joint formulation increased the robustness of both tasks when compared to the conventional classification or segmentation strategies, proving its added value, mainly for the segmentation of small lesions.
Collapse
Affiliation(s)
- Bruno Oliveira
- Life and Health Sciences Research Institute (ICVS), School of Medicine, University of Minho, Campus de Gualtar, 4710-057, Braga, Portugal. .,ICVS/3B's - PT Government Associate Laboratory, Braga/Guimarães, Portugal. .,Algoritmi Center, School of Engineering, University of Minho, Guimarães, Portugal. .,2Ai - School of Technology, IPCA, Barcelos, Portugal. .,LASI-Associate Laboratory of Intelligent Systems, 4800-058, Guimarães, Portugal.
| | - Helena R Torres
- Life and Health Sciences Research Institute (ICVS), School of Medicine, University of Minho, Campus de Gualtar, 4710-057, Braga, Portugal.,ICVS/3B's - PT Government Associate Laboratory, Braga/Guimarães, Portugal.,Algoritmi Center, School of Engineering, University of Minho, Guimarães, Portugal.,2Ai - School of Technology, IPCA, Barcelos, Portugal.,LASI-Associate Laboratory of Intelligent Systems, 4800-058, Guimarães, Portugal
| | - Pedro Morais
- 2Ai - School of Technology, IPCA, Barcelos, Portugal.,LASI-Associate Laboratory of Intelligent Systems, 4800-058, Guimarães, Portugal
| | - Fernando Veloso
- Life and Health Sciences Research Institute (ICVS), School of Medicine, University of Minho, Campus de Gualtar, 4710-057, Braga, Portugal.,ICVS/3B's - PT Government Associate Laboratory, Braga/Guimarães, Portugal.,2Ai - School of Technology, IPCA, Barcelos, Portugal.,LASI-Associate Laboratory of Intelligent Systems, 4800-058, Guimarães, Portugal.,Department of Mechanical Engineering, School of Engineering, University of Minho, Guimarães, Portugal
| | | | - Jaime C Fonseca
- Algoritmi Center, School of Engineering, University of Minho, Guimarães, Portugal.,LASI-Associate Laboratory of Intelligent Systems, 4800-058, Guimarães, Portugal
| | - João L Vilaça
- 2Ai - School of Technology, IPCA, Barcelos, Portugal.,LASI-Associate Laboratory of Intelligent Systems, 4800-058, Guimarães, Portugal
| |
Collapse
|
27
|
Alenezi F, Armghan A, Polat K. A Novel Multi-Task Learning Network Based on Melanoma Segmentation and Classification with Skin Lesion Images. Diagnostics (Basel) 2023; 13:diagnostics13020262. [PMID: 36673072 PMCID: PMC9857507 DOI: 10.3390/diagnostics13020262] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/23/2022] [Revised: 01/04/2023] [Accepted: 01/09/2023] [Indexed: 01/13/2023] Open
Abstract
Melanoma is known worldwide as a malignant tumor and the fastest-growing skin cancer type. It is a very life-threatening disease with a high mortality rate. Automatic melanoma detection improves the early detection of the disease and the survival rate. In accordance with this purpose, we presented a multi-task learning approach based on melanoma recognition with dermoscopy images. Firstly, an effective pre-processing approach based on max pooling, contrast, and shape filters is used to eliminate hair details and to perform image enhancement operations. Next, the lesion region was segmented with a VGGNet model-based FCN Layer architecture using enhanced images. Later, a cropping process was performed for the detected lesions. Then, the cropped images were converted to the input size of the classifier model using the very deep super-resolution neural network approach, and the decrease in image resolution was minimized. Finally, a deep learning network approach based on pre-trained convolutional neural networks was developed for melanoma classification. We used the International Skin Imaging Collaboration, a publicly available dermoscopic skin lesion dataset in experimental studies. While the performance measures of accuracy, specificity, precision, and sensitivity, obtained for segmentation of the lesion region, were produced at rates of 96.99%, 92.53%, 97.65%, and 98.41%, respectively, the performance measures achieved rates for classification of 97.73%, 99.83%, 99.83%, and 95.67%, respectively.
Collapse
Affiliation(s)
- Fayadh Alenezi
- Department of Electrical Engineering, College of Engineering, Jouf University, Sakaka 72388, Saudi Arabia
- Correspondence: (F.A.); (K.P.)
| | - Ammar Armghan
- Department of Electrical Engineering, College of Engineering, Jouf University, Sakaka 72388, Saudi Arabia
| | - Kemal Polat
- Department of Electrical and Electronics Engineering, Bolu Abant Izzet Baysal University, Bolu 14280, Turkey
- Correspondence: (F.A.); (K.P.)
| |
Collapse
|
28
|
Celard P, Iglesias EL, Sorribes-Fdez JM, Romero R, Vieira AS, Borrajo L. A survey on deep learning applied to medical images: from simple artificial neural networks to generative models. Neural Comput Appl 2023; 35:2291-2323. [PMID: 36373133 PMCID: PMC9638354 DOI: 10.1007/s00521-022-07953-4] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/23/2022] [Accepted: 10/12/2022] [Indexed: 11/06/2022]
Abstract
Deep learning techniques, in particular generative models, have taken on great importance in medical image analysis. This paper surveys fundamental deep learning concepts related to medical image generation. It provides concise overviews of studies which use some of the latest state-of-the-art models from last years applied to medical images of different injured body areas or organs that have a disease associated with (e.g., brain tumor and COVID-19 lungs pneumonia). The motivation for this study is to offer a comprehensive overview of artificial neural networks (NNs) and deep generative models in medical imaging, so more groups and authors that are not familiar with deep learning take into consideration its use in medicine works. We review the use of generative models, such as generative adversarial networks and variational autoencoders, as techniques to achieve semantic segmentation, data augmentation, and better classification algorithms, among other purposes. In addition, a collection of widely used public medical datasets containing magnetic resonance (MR) images, computed tomography (CT) scans, and common pictures is presented. Finally, we feature a summary of the current state of generative models in medical image including key features, current challenges, and future research paths.
Collapse
Affiliation(s)
- P. Celard
- Computer Science Department, Universidade de Vigo, Escuela Superior de Ingeniería Informática, Campus Universitario As Lagoas, 32004 Ourense, Spain ,CINBIO - Biomedical Research Centre, Universidade de Vigo, Campus Universitario Lagoas-Marcosende, 36310 Vigo, Spain ,SING Research Group, Galicia Sur Health Research Institute (IIS Galicia Sur), SERGAS-UVIGO, Vigo, Spain
| | - E. L. Iglesias
- Computer Science Department, Universidade de Vigo, Escuela Superior de Ingeniería Informática, Campus Universitario As Lagoas, 32004 Ourense, Spain ,CINBIO - Biomedical Research Centre, Universidade de Vigo, Campus Universitario Lagoas-Marcosende, 36310 Vigo, Spain ,SING Research Group, Galicia Sur Health Research Institute (IIS Galicia Sur), SERGAS-UVIGO, Vigo, Spain
| | - J. M. Sorribes-Fdez
- Computer Science Department, Universidade de Vigo, Escuela Superior de Ingeniería Informática, Campus Universitario As Lagoas, 32004 Ourense, Spain ,CINBIO - Biomedical Research Centre, Universidade de Vigo, Campus Universitario Lagoas-Marcosende, 36310 Vigo, Spain ,SING Research Group, Galicia Sur Health Research Institute (IIS Galicia Sur), SERGAS-UVIGO, Vigo, Spain
| | - R. Romero
- Computer Science Department, Universidade de Vigo, Escuela Superior de Ingeniería Informática, Campus Universitario As Lagoas, 32004 Ourense, Spain ,CINBIO - Biomedical Research Centre, Universidade de Vigo, Campus Universitario Lagoas-Marcosende, 36310 Vigo, Spain ,SING Research Group, Galicia Sur Health Research Institute (IIS Galicia Sur), SERGAS-UVIGO, Vigo, Spain
| | - A. Seara Vieira
- Computer Science Department, Universidade de Vigo, Escuela Superior de Ingeniería Informática, Campus Universitario As Lagoas, 32004 Ourense, Spain ,CINBIO - Biomedical Research Centre, Universidade de Vigo, Campus Universitario Lagoas-Marcosende, 36310 Vigo, Spain ,SING Research Group, Galicia Sur Health Research Institute (IIS Galicia Sur), SERGAS-UVIGO, Vigo, Spain
| | - L. Borrajo
- Computer Science Department, Universidade de Vigo, Escuela Superior de Ingeniería Informática, Campus Universitario As Lagoas, 32004 Ourense, Spain ,CINBIO - Biomedical Research Centre, Universidade de Vigo, Campus Universitario Lagoas-Marcosende, 36310 Vigo, Spain ,SING Research Group, Galicia Sur Health Research Institute (IIS Galicia Sur), SERGAS-UVIGO, Vigo, Spain
| |
Collapse
|
29
|
Skeleton-based Tai Chi action segmentation using trajectory primitives and content. Neural Comput Appl 2022. [DOI: 10.1007/s00521-022-08185-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/28/2022]
|
30
|
Li Z, Koban KC, Schenck TL, Giunta RE, Li Q, Sun Y. Artificial Intelligence in Dermatology Image Analysis: Current Developments and Future Trends. J Clin Med 2022; 11:jcm11226826. [PMID: 36431301 PMCID: PMC9693628 DOI: 10.3390/jcm11226826] [Citation(s) in RCA: 32] [Impact Index Per Article: 16.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2022] [Revised: 10/24/2022] [Accepted: 10/28/2022] [Indexed: 11/22/2022] Open
Abstract
BACKGROUND Thanks to the rapid development of computer-based systems and deep-learning-based algorithms, artificial intelligence (AI) has long been integrated into the healthcare field. AI is also particularly helpful in image recognition, surgical assistance and basic research. Due to the unique nature of dermatology, AI-aided dermatological diagnosis based on image recognition has become a modern focus and future trend. Key scientific concepts of review: The use of 3D imaging systems allows clinicians to screen and label skin pigmented lesions and distributed disorders, which can provide an objective assessment and image documentation of lesion sites. Dermatoscopes combined with intelligent software help the dermatologist to easily correlate each close-up image with the corresponding marked lesion in the 3D body map. In addition, AI in the field of prosthetics can assist in the rehabilitation of patients and help to restore limb function after amputation in patients with skin tumors. THE AIM OF THE STUDY For the benefit of patients, dermatologists have an obligation to explore the opportunities, risks and limitations of AI applications. This study focuses on the application of emerging AI in dermatology to aid clinical diagnosis and treatment, analyzes the current state of the field and summarizes its future trends and prospects so as to help dermatologists realize the impact of new technological innovations on traditional practices so that they can embrace and use AI-based medical approaches more quickly.
Collapse
Affiliation(s)
- Zhouxiao Li
- Department of Plastic and Reconstructive Surgery, Shanghai 9th People’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200023, China
- Division of Hand, Plastic and Aesthetic Surgery, University Hospital, LMU Munich, 80339 Munich, Germany
| | | | - Thilo Ludwig Schenck
- Division of Hand, Plastic and Aesthetic Surgery, University Hospital, LMU Munich, 80339 Munich, Germany
| | - Riccardo Enzo Giunta
- Division of Hand, Plastic and Aesthetic Surgery, University Hospital, LMU Munich, 80339 Munich, Germany
| | - Qingfeng Li
- Department of Plastic and Reconstructive Surgery, Shanghai 9th People’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200023, China
- Correspondence: (Q.L.); (Y.S.)
| | - Yangbai Sun
- Department of Plastic and Reconstructive Surgery, Shanghai 9th People’s Hospital, Shanghai Jiao Tong University School of Medicine, Shanghai 200023, China
- Correspondence: (Q.L.); (Y.S.)
| |
Collapse
|
31
|
Boundary Aware U-Net for Medical Image Segmentation. ARABIAN JOURNAL FOR SCIENCE AND ENGINEERING 2022. [DOI: 10.1007/s13369-022-07431-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
|
32
|
Alam MJ, Mohammad MS, Hossain MAF, Showmik IA, Raihan MS, Ahmed S, Mahmud TI. S 2C-DeLeNet: A parameter transfer based segmentation-classification integration for detecting skin cancer lesions from dermoscopic images. Comput Biol Med 2022; 150:106148. [PMID: 36252363 DOI: 10.1016/j.compbiomed.2022.106148] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2022] [Revised: 09/08/2022] [Accepted: 09/24/2022] [Indexed: 11/28/2022]
Abstract
Dermoscopic images ideally depict pigmentation attributes on the skin surface which is highly regarded in the medical community for detection of skin abnormality, disease or even cancer. The identification of such abnormality, however, requires trained eyes and accurate detection necessitates the process being time-intensive. As such, computerized detection schemes have become quite an essential, especially schemes which adopt deep learning tactics. In this paper, a convolutional deep neural network, S2C-DeLeNet, is proposed, which (i) Performs segmentation procedure of lesion based regions with respect to the unaffected skin tissue from dermoscopic images using a segmentation sub-network, (ii) Classifies each image based on its medical condition type utilizing transferred parameters from the inherent segmentation sub-network. The architecture of the segmentation sub-network contains EfficientNet-B4 backbone in place of the encoder and the classification sub-network bears a 'Classification Feature Extraction' system which pulls trained segmentation feature maps towards lesion prediction. Inside the classification architecture, there have been designed, (i) A 'Feature Coalescing Module' in order to trail and mix each dimensional feature from both encoder and decoder, (ii) A '3D-Layer Residuals' block to create a parallel pathway of low-dimensional features with high variance for better classification. After fine-tuning on a publicly accessible dataset, a mean dice-score of 0.9494 during segmentation is procured which beats existing segmentation strategies and a mean accuracy of 0.9103 is obtained for classification which outperforms conventional and noted classifiers. Additionally, the already fine-tuned network demonstrates highly satisfactory results on other skin cancer segmentation datasets while cross-inference. Extensive experimentation is done to prove the efficacy of the network for not only dermoscopic images but also different medical modalities; which can show its potential in being a systematic diagnostic solution in the field of dermatology and possibly more.
Collapse
Affiliation(s)
- Md Jahin Alam
- Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology (BUET), Dhaka 1205, Bangladesh.
| | - Mir Sayeed Mohammad
- Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology (BUET), Dhaka 1205, Bangladesh.
| | - Md Adnan Faisal Hossain
- Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology (BUET), Dhaka 1205, Bangladesh
| | - Ishtiaque Ahmed Showmik
- Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology (BUET), Dhaka 1205, Bangladesh
| | - Munshi Sanowar Raihan
- Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology (BUET), Dhaka 1205, Bangladesh
| | - Shahed Ahmed
- Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology (BUET), Dhaka 1205, Bangladesh
| | - Talha Ibn Mahmud
- Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology (BUET), Dhaka 1205, Bangladesh
| |
Collapse
|
33
|
Gong C, Xue B, Jing C, He CH, Wu GC, Lei B, Wang S. Time-sequential graph adversarial learning for brain modularity community detection. MATHEMATICAL BIOSCIENCES AND ENGINEERING : MBE 2022; 19:13276-13293. [PMID: 36654046 DOI: 10.3934/mbe.2022621] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/17/2023]
Abstract
Brain community detection is an efficient method to represent the communities of brain networks. However, time-variable functions of the brain and the intricate brain community structure impose a great challenge on it. In this paper, a time-sequential graph adversarial learning (TGAL) framework is proposed to detect brain communities and characterize the structure of communities from brain networks. In the framework, a novel time-sequential graph neural network is designed as an encoder to extract efficient graph representations by spatio-temporal attention mechanism. Since it is difficult to capture the community structure, the measurable modularity loss is used to optimize by maximizing the modularity of the community. In addition, the framework employs an adversarial scheme to guide the learning of representation. The effectiveness of our model is shown through experiments on the real-world brain network datasets, and the great performance of brain community detection demonstrates the advantage of the proposed framework.
Collapse
Affiliation(s)
- Changwei Gong
- Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518060, China
- Department of Computer Science, University of Chinese Academy of Sciences, Beijing 100049, China
| | - Bing Xue
- Faculty of Computer science, University of Malaya, Malaya
| | - Changhong Jing
- Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518060, China
- Department of Computer Science, University of Chinese Academy of Sciences, Beijing 100049, China
| | - Chun-Hui He
- School of Civil Engineering, Xi'an University of Architecture and Technology, Xi'an 710055, China
| | - Guo-Cheng Wu
- Data Recovery Key Laboratory of Sichuan Province, College of Mathematics and Information Science, Neijiang Normal University, Neijiang 641100, China
| | - Baiying Lei
- School of Biomedical Engineering, Shenzhen University, Shenzhen 518060, China
| | - Shuqiang Wang
- Shenzhen Institutes of Advanced Technology, Chinese Academy of Sciences, Shenzhen 518060, China
| |
Collapse
|
34
|
Oloruntoba AI, Vestergaard T, Nguyen TD, Yu Z, Sashindranath M, Betz-Stablein B, Soyer HP, Ge Z, Mar V. Assessing the Generalizability of Deep Learning Models Trained on Standardized and Nonstandardized Images and Their Performance Against Teledermatologists: Retrospective Comparative Study. JMIR DERMATOLOGY 2022. [DOI: 10.2196/35150] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
Abstract
Background
Convolutional neural networks (CNNs) are a type of artificial intelligence that shows promise as a diagnostic aid for skin cancer. However, the majority are trained using retrospective image data sets with varying image capture standardization.
Objective
The aim of our study was to use CNN models with the same architecture—trained on image sets acquired with either the same image capture device and technique (standardized) or with varied devices and capture techniques (nonstandardized)—and test variability in performance when classifying skin cancer images in different populations.
Methods
In all, 3 CNNs with the same architecture were trained. CNN nonstandardized (CNN-NS) was trained on 25,331 images taken from the International Skin Imaging Collaboration (ISIC) using different image capture devices. CNN standardized (CNN-S) was trained on 177,475 MoleMap images taken with the same capture device, and CNN standardized number 2 (CNN-S2) was trained on a subset of 25,331 standardized MoleMap images (matched for number and classes of training images to CNN-NS). These 3 models were then tested on 3 external test sets: 569 Danish images, the publicly available ISIC 2020 data set consisting of 33,126 images, and The University of Queensland (UQ) data set of 422 images. Primary outcome measures were sensitivity, specificity, and area under the receiver operating characteristic curve (AUROC). Teledermatology assessments available for the Danish data set were used to determine model performance compared to teledermatologists.
Results
When tested on the 569 Danish images, CNN-S achieved an AUROC of 0.861 (95% CI 0.830-0.889) and CNN-S2 achieved an AUROC of 0.831 (95% CI 0.798-0.861; standardized models), with both outperforming CNN-NS (nonstandardized model; P=.001 and P=.009, respectively), which achieved an AUROC of 0.759 (95% CI 0.722-0.794). When tested on 2 additional data sets (ISIC 2020 and UQ), CNN-S (P<.001 and P<.001, respectively) and CNN-S2 (P=.08 and P=.35, respectively) still outperformed CNN-NS. When the CNNs were matched to the mean sensitivity and specificity of the teledermatologists on the Danish data set, the models’ resultant sensitivities and specificities were surpassed by the teledermatologists. However, when compared to CNN-S, the differences were not statistically significant (sensitivity: P=.10; specificity: P=.053). Performance across all CNN models as well as teledermatologists was influenced by image quality.
Conclusions
CNNs trained on standardized images had improved performance and, therefore, greater generalizability in skin cancer classification when applied to unseen data sets. This finding is an important consideration for future algorithm development, regulation, and approval.
Collapse
|
35
|
|
36
|
Wang S, Singh VK, Cheah E, Wang X, Li Q, Chou SH, Lehman CD, Kumar V, Samir AE. Stacked dilated convolutions and asymmetric architecture for U-Net-based medical image segmentation. Comput Biol Med 2022; 148:105891. [PMID: 35932729 PMCID: PMC9596264 DOI: 10.1016/j.compbiomed.2022.105891] [Citation(s) in RCA: 12] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/23/2022] [Revised: 06/25/2022] [Accepted: 07/16/2022] [Indexed: 12/18/2022]
Abstract
Deep learning has been widely utilized for medical image segmentation. The most commonly used U-Net and its variants often share two common characteristics but lack solid evidence for the effectiveness. First, each block (i.e., consecutive convolutions of feature maps of the same resolution) outputs feature maps from the last convolution, limiting the variety of the receptive fields. Second, the network has a symmetric structure where the encoder and the decoder paths have similar numbers of channels. We explored two novel revisions: a stacked dilated operation that outputs feature maps from multi-scale receptive fields to replace the consecutive convolutions; an asymmetric architecture with fewer channels in the decoder path. Two novel models were developed: U-Net using the stacked dilated operation (SDU-Net) and asymmetric SDU-Net (ASDU-Net). We used both publicly available and private datasets to assess the efficacy of the proposed models. Extensive experiments confirmed SDU-Net outperformed or achieved performance similar to the state-of-the-art while using fewer parameters (40% of U-Net). ASDU-Net further reduced the model parameters to 20% of U-Net with performance comparable to SDU-Net. In conclusion, the stacked dilated operation and the asymmetric structure are promising for improving the performance of U-Net and its variants.
Collapse
Affiliation(s)
- Shuhang Wang
- Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, 02114, MA, USA.
| | - Vivek Kumar Singh
- Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, 02114, MA, USA
| | - Eugene Cheah
- Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, 02114, MA, USA
| | - Xiaohong Wang
- Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, 02114, MA, USA
| | - Qian Li
- Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, 02114, MA, USA
| | - Shinn-Huey Chou
- Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, 02114, MA, USA
| | - Constance D Lehman
- Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, 02114, MA, USA
| | - Viksit Kumar
- Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, 02114, MA, USA
| | - Anthony E Samir
- Department of Radiology, Massachusetts General Hospital, Harvard Medical School, Boston, 02114, MA, USA
| |
Collapse
|
37
|
DGCU–Net: A new dual gradient-color deep convolutional neural network for efficient skin lesion segmentation. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2022.103829] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/01/2023]
|
38
|
Astaraki M, Smedby Ö, Wang C. Prior-aware autoencoders for lung pathology segmentation. Med Image Anal 2022; 80:102491. [DOI: 10.1016/j.media.2022.102491] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/13/2021] [Revised: 04/11/2022] [Accepted: 05/20/2022] [Indexed: 10/18/2022]
|
39
|
A Novel Approach to Skin Lesion Segmentation: Multipath Fusion Model with Fusion Loss. COMPUTATIONAL AND MATHEMATICAL METHODS IN MEDICINE 2022; 2022:2157322. [PMID: 35936380 PMCID: PMC9355768 DOI: 10.1155/2022/2157322] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/17/2022] [Revised: 06/17/2022] [Accepted: 06/24/2022] [Indexed: 11/30/2022]
Abstract
Segmentation of skin lesions plays a very important role in the early detection of skin cancer. However, indistinguishability due to various artifacts such as hair and contrast between normal skin and lesioned skin is an important challenge for specialist dermatologists. Computer-aided diagnostic systems using deep convolutional neural networks are gaining importance in order to cope with difficulties. This study focuses on deep learning-based fusion networks and fusion loss functions. For the automatic segmentation of skin lesions, U-Net (U-Net + ResNet 2D) with 2D residual blocks and 2D volumetric convolutional neural networks were fused for the first time in this study. Also, a new fusion loss function is proposed by combining Dice Loss (DL) and Focal Tversky Loss (FTL) to make the proposed fused model more robust. Of the 2594 image dataset, 20% is reserved for test data and 80% for training data. In test data training, a Jaccard score of 0.837 and a dice score of 0.918 were obtained. The proposed model was also scored on the ISIC 2018 Task 1 test images, whose ground truths were not shared. The proposed model performed well and achieved a Jaccard index of 0.800 and a dice score of 0.880 in the ISIC 2018 Task 1 test set. In addition, it has been observed that the new fused loss function obtained by fusing Focal Tversky Loss and Dice Loss functions in the proposed model increases the robustness of the model in the tests. The proposed new loss function fusion model has outstripped the cutting-edge approaches in the literature.
Collapse
|
40
|
Yue C, Ye M, Wang P, Huang D, Lu X. SRV-GAN: A generative adversarial network for segmenting retinal vessels. MATHEMATICAL BIOSCIENCES AND ENGINEERING : MBE 2022; 19:9948-9965. [PMID: 36031977 DOI: 10.3934/mbe.2022464] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
In the field of ophthalmology, retinal diseases are often accompanied by complications, and effective segmentation of retinal blood vessels is an important condition for judging retinal diseases. Therefore, this paper proposes a segmentation model for retinal blood vessel segmentation. Generative adversarial networks (GANs) have been used for image semantic segmentation and show good performance. So, this paper proposes an improved GAN. Based on R2U-Net, the generator adds an attention mechanism, channel and spatial attention, which can reduce the loss of information and extract more effective features. We use dense connection modules in the discriminator. The dense connection module has the characteristics of alleviating gradient disappearance and realizing feature reuse. After a certain amount of iterative training, the generated prediction map and label map can be distinguished. Based on the loss function in the traditional GAN, we introduce the mean squared error. By using this loss, we ensure that the synthetic images contain more realistic blood vessel structures. The values of area under the curve (AUC) in the retinal blood vessel pixel segmentation of the three public data sets DRIVE, CHASE-DB1 and STARE of the proposed method are 0.9869, 0.9894 and 0.9885, respectively. The indicators of this experiment have improved compared to previous methods.
Collapse
Affiliation(s)
- Chen Yue
- School of Medical Information, Wannan Medical College, Wuhu 241002, China
- Research Center of Health Big Data Mining and Applications, Wannan Medical College, Wuhu 241002, China
| | - Mingquan Ye
- School of Medical Information, Wannan Medical College, Wuhu 241002, China
- Research Center of Health Big Data Mining and Applications, Wannan Medical College, Wuhu 241002, China
| | - Peipei Wang
- School of Medical Information, Wannan Medical College, Wuhu 241002, China
- Research Center of Health Big Data Mining and Applications, Wannan Medical College, Wuhu 241002, China
| | - Daobin Huang
- School of Medical Information, Wannan Medical College, Wuhu 241002, China
- Research Center of Health Big Data Mining and Applications, Wannan Medical College, Wuhu 241002, China
| | - Xiaojie Lu
- School of Medical Information, Wannan Medical College, Wuhu 241002, China
- Research Center of Health Big Data Mining and Applications, Wannan Medical College, Wuhu 241002, China
| |
Collapse
|
41
|
Iqbal A, Sharif M, Yasmin M, Raza M, Aftab S. Generative adversarial networks and its applications in the biomedical image segmentation: a comprehensive survey. INTERNATIONAL JOURNAL OF MULTIMEDIA INFORMATION RETRIEVAL 2022; 11:333-368. [PMID: 35821891 PMCID: PMC9264294 DOI: 10.1007/s13735-022-00240-x] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/24/2021] [Revised: 03/16/2022] [Accepted: 05/24/2022] [Indexed: 05/13/2023]
Abstract
Recent advancements with deep generative models have proven significant potential in the task of image synthesis, detection, segmentation, and classification. Segmenting the medical images is considered a primary challenge in the biomedical imaging field. There have been various GANs-based models proposed in the literature to resolve medical segmentation challenges. Our research outcome has identified 151 papers; after the twofold screening, 138 papers are selected for the final survey. A comprehensive survey is conducted on GANs network application to medical image segmentation, primarily focused on various GANs-based models, performance metrics, loss function, datasets, augmentation methods, paper implementation, and source codes. Secondly, this paper provides a detailed overview of GANs network application in different human diseases segmentation. We conclude our research with critical discussion, limitations of GANs, and suggestions for future directions. We hope this survey is beneficial and increases awareness of GANs network implementations for biomedical image segmentation tasks.
Collapse
Affiliation(s)
- Ahmed Iqbal
- Department of Computer Science, COMSATS University Islamabad, Wah Campus, Pakistan
| | - Muhammad Sharif
- Department of Computer Science, COMSATS University Islamabad, Wah Campus, Pakistan
| | - Mussarat Yasmin
- Department of Computer Science, COMSATS University Islamabad, Wah Campus, Pakistan
| | - Mudassar Raza
- Department of Computer Science, COMSATS University Islamabad, Wah Campus, Pakistan
| | - Shabib Aftab
- Department of Computer Science, Virtual University of Pakistan, Lahore, Pakistan
| |
Collapse
|
42
|
Iqbal A, Sharif M, Khan MA, Nisar W, Alhaisoni M. FF-UNet: a U-Shaped Deep Convolutional Neural Network for Multimodal Biomedical Image Segmentation. Cognit Comput 2022. [DOI: 10.1007/s12559-022-10038-y] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/10/2023]
|
43
|
Alahmadi MD. Medical Image Segmentation with Learning Semantic and Global Contextual Representation. Diagnostics (Basel) 2022; 12:diagnostics12071548. [PMID: 35885454 PMCID: PMC9319384 DOI: 10.3390/diagnostics12071548] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/31/2022] [Revised: 06/18/2022] [Accepted: 06/20/2022] [Indexed: 11/16/2022] Open
Abstract
Automatic medical image segmentation is an essential step toward accurate diseases diagnosis and designing a follow-up treatment. This assistive method facilitates the cancer detection process and provides a benchmark to highlight the affected area. The U-Net model has become the standard design choice. Although the symmetrical structure of the U-Net model enables this network to encode rich semantic representation, the intrinsic locality of the CNN layers limits this network’s capability in modeling long-range contextual dependency. On the other hand, sequence to sequence Transformer models with a multi-head attention mechanism can enable them to effectively model global contextual dependency. However, the lack of low-level information stemming from the Transformer architecture limits its performance for capturing local representation. In this paper, we propose a two parallel encoder model, where in the first path the CNN module captures the local semantic representation whereas the second path deploys a Transformer module to extract the long-range contextual representation. Next, by adaptively fusing these two feature maps, we encode both representations into a single representative tensor to be further processed by the decoder block. An experimental study demonstrates that our design can provide rich and generic representation features which are highly efficient for a fine-grained semantic segmentation task.
Collapse
Affiliation(s)
- Mohammad D Alahmadi
- Department of Software Engineering, College of Computer Science and Engineering, University of Jeddah, Jeddah 23890, Saudi Arabia
| |
Collapse
|
44
|
Liu Y, Zhou J, Liu L, Zhan Z, Hu Y, Fu Y, Duan H. FCP-Net: A Feature-Compression-Pyramid Network Guided by Game-Theoretic Interactions for Medical Image Segmentation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2022; 41:1482-1496. [PMID: 34982679 DOI: 10.1109/tmi.2021.3140120] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Medical image segmentation is a crucial step in diagnosis and analysis of diseases for clinical applications. Deep convolutional neural network methods such as DeepLabv3+ have successfully been applied for medical image segmentation, but multi-level features are seldom integrated seamlessly into different attention mechanisms, and few studies have fully explored the interactions between medical image segmentation and classification tasks. Herein, we propose a feature-compression-pyramid network (FCP-Net) guided by game-theoretic interactions with a hybrid loss function (HLF) for the medical image segmentation. The proposed approach consists of segmentation branch, classification branch and interaction branch. In the encoding stage, a new strategy is developed for the segmentation branch by applying three modules, e.g., embedded feature ensemble, dilated spatial mapping and channel attention (DSMCA), and branch layer fusion. These modules allow effective extraction of spatial information, efficient identification of spatial correlation among various features, and fully integration of multi-receptive field features from different branches. In the decoding stage, a DSMCA module and a multi-scale feature fusion module are used to establish multiple skip connections for enhancing fusion features. Classification and interaction branches are introduced to explore the potential benefits of the classification information task to the segmentation task. We further explore the interactions of segmentation and classification branches from a game theoretic view, and design an HLF. Based on this HLF, the segmentation, classification and interaction branches can collaboratively learn and teach each other throughout the training process, thus applying the conjoint information between the segmentation and classification tasks and improving the generalization performance. The proposed model has been evaluated using several datasets, including ISIC2017, ISIC2018, REFUGE, Kvasir-SEG, BUSI, and PH2, and the results prove its competitiveness compared with other state-of-the-art techniques.
Collapse
|
45
|
Kaur R, GholamHosseini H, Sinha R, Lindén M. Automatic lesion segmentation using atrous convolutional deep neural networks in dermoscopic skin cancer images. BMC Med Imaging 2022; 22:103. [PMID: 35644612 PMCID: PMC9148511 DOI: 10.1186/s12880-022-00829-y] [Citation(s) in RCA: 14] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2021] [Accepted: 04/13/2022] [Indexed: 12/16/2022] Open
Abstract
BACKGROUND Melanoma is the most dangerous and aggressive form among skin cancers, exhibiting a high mortality rate worldwide. Biopsy and histopathological analysis are standard procedures for skin cancer detection and prevention in clinical settings. A significant step in the diagnosis process is the deep understanding of the patterns, size, color, and structure of lesions based on images obtained through dermatoscopes for the infected area. However, the manual segmentation of the lesion region is time-consuming because the lesion evolves and changes its shape over time, making its prediction challenging. Moreover, it is challenging to predict melanoma at the initial stage as it closely resembles other skin cancer types that are not malignant as melanoma; thus, automatic segmentation techniques are required to design a computer-aided system for accurate and timely detection. METHODS As deep learning approaches have gained significant attention in recent years due to their remarkable performance, therefore, in this work, we proposed a novel design of a convolutional neural network (CNN) framework based on atrous convolutions for automatic lesion segmentation. This architecture is built based on the concept of atrous/dilated convolutions which are effective for semantic segmentation. A deep neural network is designed from scratch employing several building blocks consisting of convolutional, batch normalization, leakyReLU layer, and fine-tuned hyperparameters contributing altogether towards higher performance. CONCLUSION The network was tested on three benchmark datasets provided by International Skin Imaging Collaboration (ISIC), i.e., ISIC 2016, ISIC 2017, and ISIC 2018. The experimental results showed that the proposed network achieved an average Jaccard index of 90.4% on ISIC 2016, 81.8% on ISIC 2017, and 89.1% on ISIC 2018 datasets, respectively which is recorded as higher than the top three winners of the ISIC challenge and other state-of-the-art methods. Also, the model successfully extracts lesions from the whole image in one pass in less time, requiring no pre-processing step. The conclusions yielded that network is accurate in performing lesion segmentation on adopted datasets.
Collapse
Affiliation(s)
- Ranpreet Kaur
- School of Engineering, Computer, and Mathematical Sciences, Auckland University of Technology, 55 Wellesley street, 1010, Auckland, New Zealand.
| | - Hamid GholamHosseini
- School of Engineering, Computer, and Mathematical Sciences, Auckland University of Technology, 55 Wellesley street, 1010, Auckland, New Zealand
| | - Roopak Sinha
- School of Engineering, Computer, and Mathematical Sciences, Auckland University of Technology, 55 Wellesley street, 1010, Auckland, New Zealand
| | - Maria Lindén
- School of Innovation Design and Engineering, Mälardalen University, Västerås, Sweden
| |
Collapse
|
46
|
Yamanakkanavar N, Choi JY, Lee B. Multiscale and Hierarchical Feature-Aggregation Network for Segmenting Medical Images. SENSORS (BASEL, SWITZERLAND) 2022; 22:3440. [PMID: 35591129 PMCID: PMC9104396 DOI: 10.3390/s22093440] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 03/31/2022] [Revised: 04/24/2022] [Accepted: 04/26/2022] [Indexed: 11/17/2022]
Abstract
We propose an encoder-decoder architecture using wide and deep convolutional layers combined with different aggregation modules for the segmentation of medical images. Initially, we obtain a rich representation of features that span from low to high levels and from small to large scales by stacking multiple k × k kernels, where each k × k kernel operation is split into k × 1 and 1 × k convolutions. In addition, we introduce two feature-aggregation modules-multiscale feature aggregation (MFA) and hierarchical feature aggregation (HFA)-to better fuse information across end-to-end network layers. The MFA module progressively aggregates features and enriches feature representation, whereas the HFA module merges the features iteratively and hierarchically to learn richer combinations of the feature hierarchy. Furthermore, because residual connections are advantageous for assembling very deep networks, we employ an MFA-based long residual connections to avoid vanishing gradients along the aggregation paths. In addition, a guided block with multilevel convolution provides effective attention to the features that were copied from the encoder to the decoder to recover spatial information. Thus, the proposed method using feature-aggregation modules combined with a guided skip connection improves the segmentation accuracy, achieving a high similarity index for ground-truth segmentation maps. Experimental results indicate that the proposed model achieves a superior segmentation performance to that obtained by conventional methods for skin-lesion segmentation, with an average accuracy score of 0.97 on the ISIC-2018, PH2, and UFBA-UESC datasets.
Collapse
Affiliation(s)
- Nagaraj Yamanakkanavar
- Department of Information and Communications Engineering, Chosun University, Gwangju 61452, Korea;
| | - Jae Young Choi
- Division of Computer & Electronic Systems Engineering, Hankuk University of Foreign Studies, Yongin 17035, Korea;
| | - Bumshik Lee
- Department of Information and Communications Engineering, Chosun University, Gwangju 61452, Korea;
| |
Collapse
|
47
|
Dense Convolutional Network and Its Application in Medical Image Analysis. BIOMED RESEARCH INTERNATIONAL 2022; 2022:2384830. [PMID: 35509707 PMCID: PMC9060995 DOI: 10.1155/2022/2384830] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 02/28/2022] [Accepted: 03/23/2022] [Indexed: 12/28/2022]
Abstract
Dense convolutional network (DenseNet) is a hot topic in deep learning research in recent years, which has good applications in medical image analysis. In this paper, DenseNet is summarized from the following aspects. First, the basic principle of DenseNet is introduced; second, the development of DenseNet is summarized and analyzed from five aspects: broaden DenseNet structure, lightweight DenseNet structure, dense unit, dense connection mode, and attention mechanism; finally, the application research of DenseNet in the field of medical image analysis is summarized from three aspects: pattern recognition, image segmentation, and object detection. The network structures of DenseNet are systematically summarized in this paper, which has certain positive significance for the research and development of DenseNet.
Collapse
|
48
|
Liu Q, Wang J, Zuo M, Cao W, Zheng J, Zhao H, Xie J. NCRNet: Neighborhood Context Refinement Network for skin lesion segmentation. Comput Biol Med 2022; 146:105545. [PMID: 35477048 DOI: 10.1016/j.compbiomed.2022.105545] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/24/2022] [Revised: 04/11/2022] [Accepted: 04/17/2022] [Indexed: 12/24/2022]
Abstract
Accurate skin lesion segmentation plays a fundamental role in computer-aided melanoma analysis. Recently, some FCN-based methods have been proposed and achieved promising results in lesion segmentation tasks. However, due to the variable shapes, different scales, noise interference, and ambiguous boundaries of skin lesions, the capabilities of lesion location and boundary delineation of these works are still insufficient. To overcome the above challenges, in this paper, we propose a novel Neighborhood Context Refinement Network (NCRNet) by using the coarse-to-fine strategy to achieve accurate skin lesion segmentation. The proposed NCRNet contains a shared encoder and two different but closely related decoders for locating the skin lesions and refining the lesion boundaries. Specifically, we first design the Parallel Attention Decoder (PAD), which can effectively extract and fuse the local detail information and global semantic information on multiple levels to locate skin lesions of different sizes and shapes. Then, based on the initial lesion location, we further design the Neighborhood Context Refinement Decoder (NCRD), aiming at leveraging the fine-grained multi-stage neighborhood context cues to refine the lesion boundaries continuously. Furthermore, the neighborhood-based deep supervision used in the NCRD can make the shared encoder pay more attention to the lesion boundary areas and promote convergence of the segmentation network. The public skin lesion segmentation dataset ISIC2017 is adopted to validate the effectiveness of the proposed NCRNet. Comprehensive experiments prove that the proposed NCRNet reaches the state-of-the-art performance than the other nine competitive methods and gets 78.62%, 86.55%, and 94.01% on Jaccard, Dice, and Accuracy, respectively.
Collapse
Affiliation(s)
- Qi Liu
- School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, 230026, China; Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, 215163, China
| | - Jingkun Wang
- School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, 230026, China; Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, 215163, China
| | - Mengying Zuo
- Department of Cardiology, Children's Hospital of Soochow University, Suzhou, 215003, China
| | - Weiwei Cao
- School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, 230026, China; Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, 215163, China.
| | - Jian Zheng
- School of Biomedical Engineering (Suzhou), Division of Life Sciences and Medicine, University of Science and Technology of China, Hefei, 230026, China; Suzhou Institute of Biomedical Engineering and Technology, Chinese Academy of Sciences, Suzhou, 215163, China; Jinan Guoke Medical Technology Development Co., Ltd, Jinan, 250101, China
| | - Hui Zhao
- The Wenzhou Third Clinical Institute Affiliated to Wenzhou Medical University, (Wenzhou People's Hospital), Wenzhou, 325000, China
| | - Jing Xie
- The Wenzhou Third Clinical Institute Affiliated to Wenzhou Medical University, (Wenzhou People's Hospital), Wenzhou, 325000, China.
| |
Collapse
|
49
|
Santos ESD, de M S Veras R, R T Aires K, M B F Portela H, Braz Junior G, Santos JD, Tavares JMR. Semi-automatic segmentation of skin lesions based on superpixels and hybrid texture information. Med Image Anal 2022; 77:102363. [DOI: 10.1016/j.media.2022.102363] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2021] [Revised: 12/13/2021] [Accepted: 01/10/2022] [Indexed: 10/19/2022]
|
50
|
Segmentation of ultrasound image sequences by combing a novel deep siamese network with a deformable contour model. Neural Comput Appl 2022. [DOI: 10.1007/s00521-022-07054-2] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|