1
|
Lu K, Cheng J, Li H, Ouyang T. MFAFNet: A Lightweight and Efficient Network with Multi-Level Feature Adaptive Fusion for Real-Time Semantic Segmentation. SENSORS (BASEL, SWITZERLAND) 2023; 23:6382. [PMID: 37514676 PMCID: PMC10384613 DOI: 10.3390/s23146382] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/05/2023] [Revised: 07/04/2023] [Accepted: 07/11/2023] [Indexed: 07/30/2023]
Abstract
Currently, real-time semantic segmentation networks are intensely demanded in resource-constrained practical applications, such as mobile devices, drones and autonomous driving systems. However, most of the current popular approaches have difficulty in obtaining sufficiently large receptive fields, and they sacrifice low-level details to improve inference speed, leading to decreased segmentation accuracy. In this paper, a lightweight and efficient multi-level feature adaptive fusion network (MFAFNet) is proposed to address this problem. Specifically, we design a separable asymmetric reinforcement non-bottleneck module, which designs a parallel structure to extract short- and long-range contextual information and use optimized convolution to increase the inference speed. In addition, we propose a feature adaptive fusion module that effectively balances feature maps with multiple resolutions to reduce the loss of spatial detail information. We evaluate our model with state-of-the-art real-time semantic segmentation methods on the Cityscapes and Camvid datasets. Without any pre-training and post-processing, our MFAFNet has only 1.27 M parameters, while achieving accuracies of 75.9% and 69.9% mean IoU with speeds of 60.1 and 82.6 FPS on the Cityscapes and Camvid test sets, respectively. The experimental results demonstrate that the proposed method achieves an excellent trade-off between inference speed, segmentation accuracy and model size.
Collapse
Affiliation(s)
- Kai Lu
- School of Cyberspace Security (School of Cryptology), Hainan University, Haikou 570228, China
- Department of Public Safety Technology, Hainan Vocational College of Political Science and Law, Haikou 571100, China
| | - Jieren Cheng
- School of Computer Science and Technology, Hainan University, Haikou 570228, China
| | - Hua Li
- School of Cyberspace Security (School of Cryptology), Hainan University, Haikou 570228, China
| | - Tianyu Ouyang
- Faculty of Engineering, The University of Sydney, Camperdown, NSW 2006, Australia
| |
Collapse
|
2
|
Dong S, Fan Z, Chen Y, Chen K, Qin M, Zeng M, Lu X, Zhou G, Gao X, Liu JM. Performance estimation for the memristor-based computing-in-memory implementation of extremely factorized network for real-time and low-power semantic segmentation. Neural Netw 2023; 160:202-215. [PMID: 36657333 DOI: 10.1016/j.neunet.2023.01.008] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/01/2022] [Revised: 12/05/2022] [Accepted: 01/09/2023] [Indexed: 01/15/2023]
Abstract
Nowadays many semantic segmentation algorithms have achieved satisfactory accuracy on von Neumann platforms (e.g., GPU), but the speed and energy consumption have not meet the high requirements of certain edge applications like autonomous driving. To tackle this issue, it is of necessity to design an efficient lightweight semantic segmentation algorithm and then implement it on emerging hardware platforms with high speed and energy efficiency. Here, we first propose an extremely factorized network (EFNet) which can learn multi-scale context information while preserving rich spatial information with reduced model complexity. Experimental results on the Cityscapes dataset show that EFNet achieves an accuracy of 68.0% mean intersection over union (mIoU) with only 0.18M parameters, at a speed of 99 frames per second (FPS) on a single RTX 3090 GPU. Then, to further improve the speed and energy efficiency, we design a memristor-based computing-in-memory (CIM) accelerator for the hardware implementation of EFNet. It is shown by the simulation in DNN+NeuroSim V2.0 that the memristor-based CIM accelerator is ∼63× (∼4.6×) smaller in area, at most ∼9.2× (∼1000×) faster, and ∼470× (∼2400×) more energy-efficient than the RTX 3090 GPU (the Jetson Nano embedded development board), although its accuracy slightly decreases by 1.7% mIoU. Therefore, the memristor-based CIM accelerator has great potential to be deployed at the edge to implement lightweight semantic segmentation models like EFNet. This study showcases an algorithm-hardware co-design to realize real-time and low-power semantic segmentation at the edge.
Collapse
Affiliation(s)
- Shuai Dong
- Institute for Advanced Materials, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China; Guangdong Provincial Key Laboratory of Optical Information Materials and Technology, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China
| | - Zhen Fan
- Institute for Advanced Materials, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China; Guangdong Provincial Key Laboratory of Optical Information Materials and Technology, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China.
| | - Yihong Chen
- Institute for Advanced Materials, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China
| | - Kaihui Chen
- Institute for Advanced Materials, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China
| | - Minghui Qin
- Institute for Advanced Materials, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China
| | - Min Zeng
- Institute for Advanced Materials, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China
| | - Xubing Lu
- Institute for Advanced Materials, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China
| | - Guofu Zhou
- Guangdong Provincial Key Laboratory of Optical Information Materials and Technology, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China; National Center for International Research on Green Optoelectronics, South China Normal University, Guangzhou, 510006, China
| | - Xingsen Gao
- Institute for Advanced Materials, South China Academy of Advanced Optoelectronics, South China Normal University, Guangzhou, 510006, China
| | - Jun-Ming Liu
- Laboratory of Solid State Microstructures and Innovation Center of Advanced Microstructures, Nanjing University, Nanjing, 210093, China
| |
Collapse
|
3
|
Zhang C, Xu F, Wu C, Xu C. Rethinking 1D convolution for lightweight semantic segmentation. Front Neurorobot 2023; 17:1119231. [PMID: 36845064 PMCID: PMC9947531 DOI: 10.3389/fnbot.2023.1119231] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/08/2022] [Accepted: 01/26/2023] [Indexed: 02/11/2023] Open
Abstract
Lightweight semantic segmentation promotes the application of semantic segmentation in tiny devices. The existing lightweight semantic segmentation network (LSNet) has the problems of low precision and a large number of parameters. In response to the above problems, we designed a full 1D convolutional LSNet. The tremendous success of this network is attributed to the following three modules: 1D multi-layer space module (1D-MS), 1D multi-layer channel module (1D-MC), and flow alignment module (FA). The 1D-MS and the 1D-MC add global feature extraction operations based on the multi-layer perceptron (MLP) idea. This module uses 1D convolutional coding, which is more flexible than MLP. It increases the global information operation, improving features' coding ability. The FA module fuses high-level and low-level semantic information, which solves the problem of precision loss caused by the misalignment of features. We designed a 1D-mixer encoder based on the transformer structure. It performed fusion encoding of the feature space information extracted by the 1D-MS module and the channel information extracted by the 1D-MC module. 1D-mixer obtains high-quality encoded features with very few parameters, which is the key to the network's success. The attention pyramid with FA (AP-FA) uses an AP to decode features and adds a FA module to solve the problem of feature misalignment. Our network requires no pre-training and only needs a 1080Ti GPU for training. It achieved 72.6 mIoU and 95.6 FPS on the Cityscapes dataset and 70.5 mIoU and 122 FPS on the CamVid dataset. We ported the network trained on the ADE2K dataset to mobile devices, and the latency of 224 ms proves the application value of the network on mobile devices. The results on the three datasets prove that the network generalization ability we designed is powerful. Compared to state-of-the-art lightweight semantic segmentation algorithms, our designed network achieves the best balance between segmentation accuracy and parameters. The parameters of LSNet are only 0.62 M, which is currently the network with the highest segmentation accuracy within 1 M parameters.
Collapse
Affiliation(s)
- Chunyu Zhang
- Faculty of Robot Science and Engineering, Northeastern University, Shenyang, China,*Correspondence: Chunyu Zhang,
| | - Fang Xu
- Shenyang Siasun Robot & Automation Company Ltd., Shenyang, China
| | - Chengdong Wu
- Faculty of Robot Science and Engineering, Northeastern University, Shenyang, China
| | - Chenglong Xu
- College of Intelligent Systems Science and Engineering, Harbin Engineering University, Harbin, China
| |
Collapse
|
5
|
Luo D, Kang H, Long J, Zhang J, Liu X, Quan T. GDN: Guided Down-sampling Network for Real-time Semantic Segmentation. Neurocomputing 2022. [DOI: 10.1016/j.neucom.2022.11.075] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022]
|
6
|
Huang L, Dai S, He Z. Few-shot object detection with dense-global feature interaction and dual-contrastive learning. APPL INTELL 2022. [DOI: 10.1007/s10489-022-04243-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/27/2022]
|
7
|
Cai Y, Li L, Wang D, Liu X. MFNet: Multi-level fusion aware feature pyramid based multi-view stereo network for 3D reconstruction. APPL INTELL 2022. [DOI: 10.1007/s10489-022-03754-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/01/2022]
|