1
|
Xu L, Zou Q. Semantic-Aware Fusion Network Based on Super-Resolution. SENSORS (BASEL, SWITZERLAND) 2024; 24:3665. [PMID: 38894455 PMCID: PMC11175180 DOI: 10.3390/s24113665] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/09/2024] [Revised: 06/02/2024] [Accepted: 06/04/2024] [Indexed: 06/21/2024]
Abstract
The aim of infrared and visible image fusion is to generate a fused image that not only contains salient targets and rich texture details, but also facilitates high-level vision tasks. However, due to the hardware limitations of digital cameras and other devices, there are more low-resolution images in the existing datasets, and low-resolution images are often accompanied by the problem of losing details and structural information. At the same time, existing fusion algorithms focus too much on the visual quality of the fused images, while ignoring the requirements of high-level vision tasks. To address the above challenges, in this paper, we skillfully unite the super-resolution network, fusion network and segmentation network, and propose a super-resolution-based semantic-aware fusion network. First, we design a super-resolution network based on a multi-branch hybrid attention module (MHAM), which aims to enhance the quality and details of the source image, enabling the fusion network to integrate the features of the source image more accurately. Then, a comprehensive information extraction module (STDC) is designed in the fusion network to enhance the network's ability to extract finer-grained complementary information from the source image. Finally, the fusion network and segmentation network are jointly trained to utilize semantic loss to guide the semantic information back to the fusion network, which effectively improves the performance of the fused images on high-level vision tasks. Extensive experiments show that our method is more effective than other state-of-the-art image fusion methods. In particular, our fused images not only have excellent visual perception effects, but also help to improve the performance of high-level vision tasks.
Collapse
Affiliation(s)
- Lingfeng Xu
- School of Microelectronics, Tianjin University, Tianjin 300072, China;
| | - Qiang Zou
- School of Microelectronics, Tianjin University, Tianjin 300072, China;
- Tianjin International Joint Research Center for Internet of Things, Tianjin 300072, China
- Tianjin Key Laboratory of Imaging and Sensing Microelectronic Technology, Tianjin University, Tianjin 300072, China
| |
Collapse
|
2
|
Su D, Luo J, Fei C. An Efficient and Rapid Medical Image Segmentation Network. IEEE J Biomed Health Inform 2024; 28:2979-2990. [PMID: 38457317 DOI: 10.1109/jbhi.2024.3374780] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/10/2024]
Abstract
Accurate medical image segmentation is an essential part of the medical image analysis process that provides detailed quantitative metrics. In recent years, extensions of classical networks such as UNet have achieved state-of-the-art performance on medical image segmentation tasks. However, the high model complexity of these networks limits their applicability to devices with constrained computational resources. To alleviate this problem, we propose a shallow hierarchical Transformer for medical image segmentation, called SHFormer. By decreasing the number of transformer blocks utilized, the model complexity of SHFormer can be reduced to an acceptable level. To improve the learned attention while keeping the structure lightweight, we propose a spatial-channel connection module. This module separately learns attention in the spatial and channel dimensions of the feature while interconnecting them to produce more focused attention. To keep the decoder lightweight, the MLP-D module is proposed to progressively fuse multi-scale features in which channels are aligned using Multi-Layer Perceptron (MLP) and spatial information is fused by convolutional blocks. We first validated the performance of SHFormer on the ISIC-2018 dataset. Compared to the latest network, SHFormer exhibits comparable performance with 15 times fewer parameters, 30 times lower computational complexity and 5 times higher inference efficiency. To test the generalizability of SHFormer, we introduced the polyp dataset for additional testing. SHFormer achieves comparable segmentation accuracy to the latest network while having lower computational overhead.
Collapse
|
3
|
Du H, Wang J, Liu M, Wang Y, Meijering E. SwinPA-Net: Swin Transformer-Based Multiscale Feature Pyramid Aggregation Network for Medical Image Segmentation. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024; 35:5355-5366. [PMID: 36121961 DOI: 10.1109/tnnls.2022.3204090] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
The precise segmentation of medical images is one of the key challenges in pathology research and clinical practice. However, many medical image segmentation tasks have problems such as large differences between different types of lesions and similar shapes as well as colors between lesions and surrounding tissues, which seriously affects the improvement of segmentation accuracy. In this article, a novel method called Swin Pyramid Aggregation network (SwinPA-Net) is proposed by combining two designed modules with Swin Transformer to learn more powerful and robust features. The two modules, named dense multiplicative connection (DMC) module and local pyramid attention (LPA) module, are proposed to aggregate the multiscale context information of medical images. The DMC module cascades the multiscale semantic feature information through dense multiplicative feature fusion, which minimizes the interference of shallow background noise to improve the feature expression and solves the problem of excessive variation in lesion size and type. Moreover, the LPA module guides the network to focus on the region of interest by merging the global attention and the local attention, which helps to solve similar problems. The proposed network is evaluated on two public benchmark datasets for polyp segmentation task and skin lesion segmentation task as well as a clinical private dataset for laparoscopic image segmentation task. Compared with existing state-of-the-art (SOTA) methods, the SwinPA-Net achieves the most advanced performance and can outperform the second-best method on the mean Dice score by 1.68%, 0.8%, and 1.2% on the three tasks, respectively.
Collapse
|
4
|
Yue G, Zhuo G, Yan W, Zhou T, Tang C, Yang P, Wang T. Boundary uncertainty aware network for automated polyp segmentation. Neural Netw 2024; 170:390-404. [PMID: 38029720 DOI: 10.1016/j.neunet.2023.11.050] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/20/2023] [Revised: 07/15/2023] [Accepted: 11/22/2023] [Indexed: 12/01/2023]
Abstract
Recently, leveraging deep neural networks for automated colorectal polyp segmentation has emerged as a hot topic due to the favored advantages in evading the limitations of visual inspection, e.g., overwork and subjectivity. However, most existing methods do not pay enough attention to the uncertain areas of colonoscopy images and often provide unsatisfactory segmentation performance. In this paper, we propose a novel boundary uncertainty aware network (BUNet) for precise and robust colorectal polyp segmentation. Specifically, considering that polyps vary greatly in size and shape, we first adopt a pyramid vision transformer encoder to learn multi-scale feature representations. Then, a simple yet effective boundary exploration module (BEM) is proposed to explore boundary cues from the low-level features. To make the network focus on the ambiguous area where the prediction score is biased to neither the foreground nor the background, we further introduce a boundary uncertainty aware module (BUM) that explores error-prone regions from the high-level features with the assistance of boundary cues provided by the BEM. Through the top-down hybrid deep supervision, our BUNet implements coarse-to-fine polyp segmentation and finally localizes polyp regions precisely. Extensive experiments on five public datasets show that BUNet is superior to thirteen competing methods in terms of both effectiveness and generalization ability.
Collapse
Affiliation(s)
- Guanghui Yue
- National-Reginoal Key Technology Engineering Laboratory for Medical Ultrasound, Guangdong Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Marshall Laboratory of Biomedical Engineering, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen 518060, China
| | - Guibin Zhuo
- National-Reginoal Key Technology Engineering Laboratory for Medical Ultrasound, Guangdong Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Marshall Laboratory of Biomedical Engineering, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen 518060, China
| | - Weiqing Yan
- School of Computer and Control Engineering, Yantai University, Yantai 264005, China
| | - Tianwei Zhou
- College of Management, Shenzhen University, Shenzhen 518060, China.
| | - Chang Tang
- School of Computer Science, China University of Geosciences, Wuhan 430074, China
| | - Peng Yang
- National-Reginoal Key Technology Engineering Laboratory for Medical Ultrasound, Guangdong Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Marshall Laboratory of Biomedical Engineering, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen 518060, China
| | - Tianfu Wang
- National-Reginoal Key Technology Engineering Laboratory for Medical Ultrasound, Guangdong Key Laboratory of Biomedical Measurements and Ultrasound Imaging, Marshall Laboratory of Biomedical Engineering, School of Biomedical Engineering, Shenzhen University Medical School, Shenzhen University, Shenzhen 518060, China
| |
Collapse
|
5
|
Shen W, Wang Y, Liu M, Wang J, Ding R, Zhang Z, Meijering E. Branch Aggregation Attention Network for Robotic Surgical Instrument Segmentation. IEEE TRANSACTIONS ON MEDICAL IMAGING 2023; 42:3408-3419. [PMID: 37342952 DOI: 10.1109/tmi.2023.3288127] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/23/2023]
Abstract
Surgical instrument segmentation is of great significance to robot-assisted surgery, but the noise caused by reflection, water mist, and motion blur during the surgery as well as the different forms of surgical instruments would greatly increase the difficulty of precise segmentation. A novel method called Branch Aggregation Attention network (BAANet) is proposed to address these challenges, which adopts a lightweight encoder and two designed modules, named Branch Balance Aggregation module (BBA) and Block Attention Fusion module (BAF), for efficient feature localization and denoising. By introducing the unique BBA module, features from multiple branches are balanced and optimized through a combination of addition and multiplication to complement strengths and effectively suppress noise. Furthermore, to fully integrate the contextual information and capture the region of interest, the BAF module is proposed in the decoder, which receives adjacent feature maps from the BBA module and localizes the surgical instruments from both global and local perspectives by utilizing a dual branch attention mechanism. According to the experimental results, the proposed method has the advantage of being lightweight while outperforming the second-best method by 4.03%, 1.53%, and 1.34% in mIoU scores on three challenging surgical instrument datasets, respectively, compared to the existing state-of-the-art methods. Code is available at https://github.com/SWT-1014/BAANet.
Collapse
|
6
|
Nie J, Xie J, Cao J, Pang Y. Context and detail interaction network for stereo rain streak and raindrop removal. Neural Netw 2023; 166:215-224. [PMID: 37515901 DOI: 10.1016/j.neunet.2023.07.013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2023] [Revised: 06/23/2023] [Accepted: 07/11/2023] [Indexed: 07/31/2023]
Abstract
Recently stereo image deraining has attracted lots of attention due to its superiority of abundant information from cross views. Exploring interaction information across stereo views is the key to improving the performance of stereo image deraining. In this paper, we design a general coarse-to-fine deraining framework for stereo rain streak and raindrop removal, called CDINet, comprising a stereo rain removal subnet and a stereo detail recovery subnet to restore images progressively. Two types of interaction modules are devised to explore interaction information for rain removal and detail recovery, respectively. Specifically, a global context interaction module is proposed to learn long-range dependencies of stereo images and remove rain by utilizing stereo structural information. A local detail interaction module is designed to model local contextual correlation, which aims at restoring the detail information by using neighborhood information from cross views. Extensive experiments are conducted on the two datasets including a synthetic rain streak removal dataset (RainKITTI) and a real raindrop removal dataset (Stereo Waterdrop), which demonstrates that our method sets new state-of-the-art deraining performance in terms of both quantitative and qualitative metrics with faster speed.
Collapse
Affiliation(s)
- Jing Nie
- School of Microelectronics and Communication Engineering, Chongqing University, Chongqing, 400044, China; Shanghai Artificial Intelligence Laboratory, Shanghai, 200232, China.
| | - Jin Xie
- School of Big Data and Software Engineering, Chongqing University, Chongqing, 400044, China; Shanghai Artificial Intelligence Laboratory, Shanghai, 200232, China.
| | - Jiale Cao
- School of Electrical and Information Engineering, Tianjin University, Tianjin, 300072, China.
| | - Yanwei Pang
- School of Electrical and Information Engineering, Tianjin University, Tianjin, 300072, China.
| |
Collapse
|
7
|
Zhu H, Wu H, Wang X, He D, Liu Z, Pan X. DPACFuse: Dual-Branch Progressive Learning for Infrared and Visible Image Fusion with Complementary Self-Attention and Convolution. SENSORS (BASEL, SWITZERLAND) 2023; 23:7205. [PMID: 37631742 PMCID: PMC10458385 DOI: 10.3390/s23167205] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/06/2023] [Revised: 08/03/2023] [Accepted: 08/04/2023] [Indexed: 08/27/2023]
Abstract
Infrared and visible image fusion aims to generate a single fused image that not only contains rich texture details and salient objects, but also facilitates downstream tasks. However, existing works mainly focus on learning different modality-specific or shared features, and ignore the importance of modeling cross-modality features. To address these challenges, we propose Dual-branch Progressive learning for infrared and visible image fusion with a complementary self-Attention and Convolution (DPACFuse) network. On the one hand, we propose Cross-Modality Feature Extraction (CMEF) to enhance information interaction and the extraction of common features across modalities. In addition, we introduce a high-frequency gradient convolution operation to extract fine-grained information and suppress high-frequency information loss. On the other hand, to alleviate the CNN issues of insufficient global information extraction and computation overheads of self-attention, we introduce the ACmix, which can fully extract local and global information in the source image with a smaller computational overhead than pure convolution or pure self-attention. Extensive experiments demonstrated that the fused images generated by DPACFuse not only contain rich texture information, but can also effectively highlight salient objects. Additionally, our method achieved approximately 3% improvement over the state-of-the-art methods in MI, Qabf, SF, and AG evaluation indicators. More importantly, our fused images enhanced object detection and semantic segmentation by approximately 10%, compared to using infrared and visible images separately.
Collapse
Affiliation(s)
- Huayi Zhu
- School of Computer and Information Security, Guilin University of Electronic Science and Technology, Guilin 541004, China
| | - Heshan Wu
- School of Computer and Information Security, Guilin University of Electronic Science and Technology, Guilin 541004, China
| | - Xiaolong Wang
- School of Computer and Information Security, Guilin University of Electronic Science and Technology, Guilin 541004, China
| | - Dongmei He
- School of Computer and Information Security, Guilin University of Electronic Science and Technology, Guilin 541004, China
| | - Zhenbing Liu
- School of Artificial Intelligence, Guilin University of Electronic Science and Technology, Guilin 541004, China
| | - Xipeng Pan
- School of Computer and Information Security, Guilin University of Electronic Science and Technology, Guilin 541004, China
| |
Collapse
|
8
|
Chen Z, Sun Y, Bi X, Yue J. Lightweight image de-snowing: A better trade-off between network capacity and performance. Neural Netw 2023; 165:896-908. [PMID: 37441907 DOI: 10.1016/j.neunet.2023.06.029] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2022] [Revised: 06/11/2023] [Accepted: 06/24/2023] [Indexed: 07/15/2023]
Abstract
The single image de-snowing task is an essential topic in computer vision, as images captured on snowy days degrade the performance of current vision-based intelligent systems. Existing methods build complex network structures with numerous parameters to pursue continuous performance improvement. Nonetheless, they generally ignore the negative impact of large memory consumption in real applications. This paper aims to address the above problem by making a trade-off between network capacity and performance. We propose two novel networks suitable for different application scenarios. For devices with small memory and requiring fast inference speed, we propose an extremely lightweight recursive network (XLRNet). XLRNet is constructed by a single recursive strategy and two novel lightweight modules. For devices with large memory and pursuing better de-snowing performance, we propose a coupled lightweight dual recursive network (CLDRNet). CLDRNet cascades two XLRNets by a novel dual recursive strategy and a novel dual coupled LSTM module (DC-LSTM). Extensive experiments demonstrate the effectiveness and superiority of our two models on three synthetic datasets and real-world datasets.
Collapse
Affiliation(s)
- Zheng Chen
- College of Information and Communication Engineering, Harbin Engineering University, Harbin 150001, China.
| | - Yiwen Sun
- Institute for AI, Peking University, Beijing 100871, China.
| | - Xiaojun Bi
- Key Laboratory of Ethnic Language Intelligent Analysis and Security Governance of MOE, Minzu University of China, Beijing, China; School of Information Engineering, Minzu University of China, Beijing 100081, China.
| | - Jianyu Yue
- College of Information and Communication Engineering, Harbin Engineering University, Harbin 150001, China.
| |
Collapse
|
9
|
Liu H, Ma M, Wang M, Chen Z, Zhao Y. SCFusion: Infrared and Visible Fusion Based on Salient Compensation. ENTROPY (BASEL, SWITZERLAND) 2023; 25:985. [PMID: 37509931 PMCID: PMC10378341 DOI: 10.3390/e25070985] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/01/2023] [Revised: 06/20/2023] [Accepted: 06/23/2023] [Indexed: 07/30/2023]
Abstract
The aim of infrared and visible image fusion is to integrate the complementary information of the two modalities for high-quality fused images. However, many deep learning fusion algorithms have not considered the characteristics of infrared images in low-light scenes, leading to the problems of weak texture details, low contrast of infrared targets and poor visual perception in the existing methods. Therefore, in this paper, we propose a salient compensation-based fusion method that makes sufficient use of the characteristics of infrared and visible images to generate high-quality fused images under low-light conditions. First, we design a multi-scale edge gradient module (MEGB) in the texture mainstream to adequately extract the texture information of the dual input of infrared and visible images; on the other hand, the salient tributary is pre-trained by salient loss to obtain the saliency map based on the salient dense residual module (SRDB) to extract salient features, which is supplemented in the process of overall network training. We propose the spatial bias module (SBM) to fuse global information with local information. Finally, extensive comparison experiments with existing methods show that our method has significant advantages in describing target features and global scenes, the effectiveness of the proposed module is demonstrated by ablation experiments. In addition, we also verify the facilitation of this paper's method for high-level vision on a semantic segmentation task.
Collapse
Affiliation(s)
- Haipeng Liu
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, China
| | - Meiyan Ma
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, China
| | - Meng Wang
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, China
- Yunnan Province Key Laboratory of Computer, Kunming University of Science and Technology, Kunming 650500, China
| | - Zhaoyu Chen
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, China
| | - Yibo Zhao
- Faculty of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, China
| |
Collapse
|
10
|
Dong S, Fan Z, Chen Y, Chen K, Qin M, Zeng M, Lu X, Zhou G, Gao X, Liu JM. Performance estimation for the memristor-based computing-in-memory implementation of extremely factorized network for real-time and low-power semantic segmentation. Neural Netw 2023; 160:202-215. [PMID: 36657333 DOI: 10.1016/j.neunet.2023.01.008] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2022] [Revised: 12/05/2022] [Accepted: 01/09/2023] [Indexed: 01/15/2023]
Abstract
Nowadays many semantic segmentation algorithms have achieved satisfactory accuracy on von Neumann platforms (e.g., GPU), but the speed and energy consumption have not meet the high requirements of certain edge applications like autonomous driving. To tackle this issue, it is of necessity to design an efficient lightweight semantic segmentation algorithm and then implement it on emerging hardware platforms with high speed and energy efficiency. Here, we first propose an extremely factorized network (EFNet) which can learn multi-scale context information while preserving rich spatial information with reduced model complexity. Experimental results on the Cityscapes dataset show that EFNet achieves an accuracy of 68.0% mean intersection over union (mIoU) with only 0.18M parameters, at a speed of 99 frames per second (FPS) on a single RTX 3090 GPU. Then, to further improve the speed and energy efficiency, we design a memristor-based computing-in-memory (CIM) accelerator for the hardware implementation of EFNet. It is shown by the simulation in DNN+NeuroSim V2.0 that the memristor-based CIM accelerator is ∼63× (∼4.6×) smaller in area, at most ∼9.2× (∼1000×) faster, and ∼470× (∼2400×) more energy-efficient than the RTX 3090 GPU (the Jetson Nano embedded development board), although its accuracy slightly decreases by 1.7% mIoU. Therefore, the memristor-based CIM accelerator has great potential to be deployed at the edge to implement lightweight semantic segmentation models like EFNet. This study showcases an algorithm-hardware co-design to realize real-time and low-power semantic segmentation at the edge.
Collapse
Affiliation(s)
- Shuai Dong
- Institute for Advanced Materials, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China; Guangdong Provincial Key Laboratory of Optical Information Materials and Technology, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China
| | - Zhen Fan
- Institute for Advanced Materials, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China; Guangdong Provincial Key Laboratory of Optical Information Materials and Technology, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China.
| | - Yihong Chen
- Institute for Advanced Materials, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China
| | - Kaihui Chen
- Institute for Advanced Materials, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China
| | - Minghui Qin
- Institute for Advanced Materials, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China
| | - Min Zeng
- Institute for Advanced Materials, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China
| | - Xubing Lu
- Institute for Advanced Materials, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China
| | - Guofu Zhou
- Guangdong Provincial Key Laboratory of Optical Information Materials and Technology, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China; National Center for International Research on Green Optoelectronics, South China Normal University, Guangzhou, 510006, China
| | - Xingsen Gao
- Institute for Advanced Materials, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China
| | - Jun-Ming Liu
- Laboratory of Solid State Microstructures and Innovation Center of Advanced Microstructures, Nanjing University, Nanjing, 210093, China
| |
Collapse
|
11
|
BFMNet: Bilateral feature fusion network with multi-scale context aggregation for real-time semantic segmentation. Neurocomputing 2023. [DOI: 10.1016/j.neucom.2022.11.084] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
|
12
|
Kim H, Wang C, Byun H, Hu W, Kim S, Jiao Q, Lee TH. Variable three-term conjugate gradient method for training artificial neural networks. Neural Netw 2023; 159:125-136. [PMID: 36565690 DOI: 10.1016/j.neunet.2022.12.001] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/10/2022] [Revised: 09/24/2022] [Accepted: 12/02/2022] [Indexed: 12/13/2022]
Abstract
Artificial neural networks (ANNs) have been widely adopted as general computational tools both in computer science as well as many other engineering fields. Stochastic gradient descent (SGD) and adaptive methods such as Adam are popular as robust optimization algorithms used to train the ANNs. However, the effectiveness of these algorithms is limited because they calculate a search direction based on a first-order gradient. Although higher-order gradient methods such as Newton's method have been proposed, they require the Hessian matrix to be semi-definite, and its inversion incurs a high computational cost. Therefore, in this paper, we propose a variable three-term conjugate gradient (VTTCG) method that approximates the Hessian matrix to enhance search direction and uses a variable step size to achieve improved convergence stability. To evaluate the performance of the VTTCG method, we train different ANNs on benchmark image classification and generation datasets. We also conduct a similar experiment in which a grasp generation and selection convolutional neural network (GGS-CNN) is trained to perform intelligent robotic grasping. After considering a simulated environment, we also test the GGS-CNN with a physical grasping robot. The experimental results show that the performance of the VTTCG method is superior to that of four conventional methods, including SGD, Adam, AMSGrad, and AdaBelief.
Collapse
Affiliation(s)
- Hansu Kim
- Department of Automotive Engineering, Hanyang University, Seoul 04763, Republic of Korea; BK21 Four Education and Research Program for Automotive-Software Convergence, Hanyang University, Seoul 04763, Republic of Korea
| | - Chuxuan Wang
- State Key Laboratory of Fluid Power and Mechatronic Systems, Zhejiang University, Hangzhou 310058, People's Republic of China
| | - Hyoseok Byun
- Department of Automotive Engineering, Hanyang University, Seoul 04763, Republic of Korea
| | - Weifei Hu
- State Key Laboratory of Fluid Power and Mechatronic Systems, Zhejiang University, Hangzhou 310058, People's Republic of China.
| | - Sanghyuk Kim
- Department of Automotive Engineering (Automotive-Computer Convergence), Hanyang University, Seoul 04763, Republic of Korea
| | - Qing Jiao
- State Key Laboratory of Fluid Power and Mechatronic Systems, Zhejiang University, Hangzhou 310058, People's Republic of China
| | - Tae Hee Lee
- Department of Automotive Engineering, Hanyang University, Seoul 04763, Republic of Korea.
| |
Collapse
|
13
|
A Spatial–Spectral Joint Attention Network for Change Detection in Multispectral Imagery. REMOTE SENSING 2022. [DOI: 10.3390/rs14143394] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Change detection determines and evaluates changes by comparing bi-temporal images, which is a challenging task in the remote-sensing field. To better exploit the high-level features, deep-learning-based change-detection methods have attracted researchers’ attention. Most deep-learning-based methods only explore the spatial–spectral features simultaneously. However, we assume the key spatial-change areas should be more important, and attention should be paid to the specific bands which can best reflect the changes. To achieve this goal, we propose the spatial–spectral joint attention network (SJAN). Compared with traditional methods, SJAN introduces the spatial–spectral attention mechanism to better explore the key changed areas and the key separable bands. To be more specific, a novel spatial-attention module is designed to extract the spatially key regions first. Secondly, the spectral-attention module is developed to adaptively focus on the separable bands of land-cover materials. Finally, a novel objective function is proposed to help the model to measure the similarity of learned spatial–spectral features from both spectrum amplitude and angle perspectives. The proposed SJAN is validated on three benchmark datasets. Comprehensive experiments have been conducted to demonstrate the effectiveness of the proposed SJAN.
Collapse
|
14
|
Meng Y, Xu H, Ma Z, Zhou J, Hui D. Detail-semantic guide network based on spatial attention for surface defect detection with fewer samples. APPL INTELL 2022. [DOI: 10.1007/s10489-022-03671-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
15
|
Reliable Label-Supervised Pixel Attention Mechanism for Weakly Supervised Building Segmentation in UAV Imagery. REMOTE SENSING 2022. [DOI: 10.3390/rs14133196] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Building segmentation for Unmanned Aerial Vehicle (UAV) imagery usually requires pixel-level labels, which are time-consuming and expensive to collect. Weakly supervised semantic segmentation methods for image-level labeling have recently achieved promising performance in natural scenes, but there have been few studies on UAV remote sensing imagery. In this paper, we propose a reliable label-supervised pixel attention mechanism for building segmentation in UAV imagery. Our method is based on the class activation map. However, classification networks tend to capture discriminative parts of the object and are insensitive to over-activation; therefore, class activation maps cannot directly guide segmentation network training. To overcome these challenges, we first design a Pixel Attention Module that captures rich contextual relationships, which can further mine more discriminative regions, in order to obtain a modified class activation map. Then, we use the initial seeds generated by the classification network to synthesize reliable labels. Finally, we design a reliable label loss, which is defined as the sum of the pixel-level differences between the reliable labels and the modified class activation map. Notably, the reliable label loss can handle over-activation. The preceding steps can significantly improve the quality of the pseudo-labels. Experiments on our home-made UAV data set indicate that our method can achieve 88.8% mIoU on the test set, outperforming previous state-of-the-art weakly supervised methods.
Collapse
|
16
|
Wang D, Xiang S, Zhou Y, Mu J, Zhou H, Irampaye R. Multiple-Attention Mechanism Network for Semantic Segmentation. SENSORS 2022; 22:s22124477. [PMID: 35746258 PMCID: PMC9228958 DOI: 10.3390/s22124477] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 03/23/2022] [Revised: 06/07/2022] [Accepted: 06/11/2022] [Indexed: 12/04/2022]
Abstract
Contextual information and the dependencies between dimensions is vital in image semantic segmentation. In this paper, we propose a multiple-attention mechanism network (MANet) for semantic segmentation in a very effective and efficient way. Concretely, the contributions are as follows: (1) a novel dual-attention mechanism for capturing feature dependencies in spatial and channel dimensions, where the adjacent position attention captures the dependencies between pixels well; (2) a new cross-dimensional interactive attention feature fusion module, which strengthens the fusion of fine location structure information in low-level features and category semantic information in high-level features. We conduct extensive experiments on semantic segmentation benchmarks including PASCAL VOC 2012 and Cityscapes datasets. Our MANet achieves the mIoU scores of 75.5% and 72.8% on PASCAL VOC 2012 and Cityscapes datasets, respectively. The effectiveness of the network is higher than the previous popular semantic segmentation networks under the same conditions.
Collapse
Affiliation(s)
- Dongli Wang
- School of Automation and Electronic Information, Xiangtan University, Xiangtan 411105, China; (D.W.); (S.X.)
| | - Shengliang Xiang
- School of Automation and Electronic Information, Xiangtan University, Xiangtan 411105, China; (D.W.); (S.X.)
| | - Yan Zhou
- School of Automation and Electronic Information, Xiangtan University, Xiangtan 411105, China; (D.W.); (S.X.)
- Correspondence:
| | - Jinzhen Mu
- Shanghai Aerospace Control Technology Institute, Shanghai 201109, China;
| | - Haibin Zhou
- School of Mathematics and Computational Science, Xiangtan University, Xiangtan 411105, China; (H.Z.); (R.I.)
| | - Richard Irampaye
- School of Mathematics and Computational Science, Xiangtan University, Xiangtan 411105, China; (H.Z.); (R.I.)
| |
Collapse
|
17
|
Co-attention dictionary network for weakly-supervised semantic segmentation. Neurocomputing 2022. [DOI: 10.1016/j.neucom.2021.11.046] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/20/2022]
|
18
|
Lee J, Kim SB. Uncertainty-aware hierarchical segment-channel attention mechanism for reliable and interpretable multichannel signal classification. Neural Netw 2022; 150:68-86. [DOI: 10.1016/j.neunet.2022.02.019] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2021] [Revised: 07/28/2021] [Accepted: 02/23/2022] [Indexed: 11/24/2022]
|
19
|
RELAXNet: Residual efficient learning and attention expected fusion network for real-time semantic segmentation. Neurocomputing 2022. [DOI: 10.1016/j.neucom.2021.12.003] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
|
20
|
Zhang X, Du B, Wu Z, Wan T. LAANet: lightweight attention-guided asymmetric network for real-time semantic segmentation. Neural Comput Appl 2022. [DOI: 10.1007/s00521-022-06932-z] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
|
21
|
Hyperspectral Image Classification across Different Datasets: A Generalization to Unseen Categories. REMOTE SENSING 2021. [DOI: 10.3390/rs13091672] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
With the rapid developments of hyperspectral imaging, the cost of collecting hyperspectral data has been lower, while the demand for reliable and detailed hyperspectral annotations has been much more substantial. However, limited by the difficulties of labelling annotations, most existing hyperspectral image (HSI) classification methods are trained and evaluated on a single hyperspectral data cube. It brings two significant challenges. On the one hand, many algorithms have reached a nearly perfect classification accuracy, but their trained models are hard to generalize to other datasets. On the other hand, since different hyperspectral datasets are usually not collected in the same scene, different datasets will contain different classes. To address these issues, in this paper, we propose a new paradigm for HSI classification, which is training and evaluating separately across different hyperspectral datasets. It is of great help to labelling hyperspectral data. However, it has rarely been studied in the hyperspectral community. In this work, we utilize a three-phase scheme, including feature embedding, feature mapping, and label reasoning. More specifically, we select a pair of datasets acquired by the same hyperspectral sensor, and the classifier learns from one dataset and then evaluated it on the other. Inspired by the latest advances in zero-shot learning, we introduce label semantic representation to establish associations between seen categories in the training set and unseen categories in the testing set. Extensive experiments on two pairs of datasets with different comparative methods have shown the effectiveness and potential of zero-shot learning in HSI classification.
Collapse
|