1
|
Pan J, Zhong S, Yue T, Yin Y, Tang Y. Multi-Task Foreground-Aware Network with Depth Completion for Enhanced RGB-D Fusion Object Detection Based on Transformer. SENSORS (BASEL, SWITZERLAND) 2024; 24:2374. [PMID: 38610585 PMCID: PMC11014210 DOI: 10.3390/s24072374] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/08/2024] [Revised: 03/29/2024] [Accepted: 04/04/2024] [Indexed: 04/14/2024]
Abstract
Fusing multiple sensor perceptions, specifically LiDAR and camera, is a prevalent method for target recognition in autonomous driving systems. Traditional object detection algorithms are limited by the sparse nature of LiDAR point clouds, resulting in poor fusion performance, especially for detecting small and distant targets. In this paper, a multi-task parallel neural network based on the Transformer is constructed to simultaneously perform depth completion and object detection. The loss functions are redesigned to reduce environmental noise in depth completion, and a new fusion module is designed to enhance the network's perception of the foreground and background. The network leverages the correlation between RGB pixels for depth completion, completing the LiDAR point cloud and addressing the mismatch between sparse LiDAR features and dense pixel features. Subsequently, we extract depth map features and effectively fuse them with RGB features, fully utilizing the depth feature differences between foreground and background to enhance object detection performance, especially for challenging targets. Compared to the baseline network, improvements of 4.78%, 8.93%, and 15.54% are achieved in the difficult indicators for cars, pedestrians, and cyclists, respectively. Experimental results also demonstrate that the network achieves a speed of 38 fps, validating the efficiency and feasibility of the proposed method.
Collapse
Affiliation(s)
- Jiasheng Pan
- School of Computer Engineering and Science, Shanghai University, No. 99 Shangda Road, Shanghai 200444, China;
| | - Songyi Zhong
- School of Mechatronic Engineering and Automation, Shanghai University, No. 99 Shangda Road, Shanghai 200444, China;
- School of Artificial Intelligence, Shanghai University, No. 99 Shangda Road, Shanghai 200444, China; (Y.Y.); (Y.T.)
| | - Tao Yue
- School of Mechatronic Engineering and Automation, Shanghai University, No. 99 Shangda Road, Shanghai 200444, China;
| | - Yankun Yin
- School of Artificial Intelligence, Shanghai University, No. 99 Shangda Road, Shanghai 200444, China; (Y.Y.); (Y.T.)
| | - Yanhao Tang
- School of Artificial Intelligence, Shanghai University, No. 99 Shangda Road, Shanghai 200444, China; (Y.Y.); (Y.T.)
| |
Collapse
|
2
|
FUN-SIS: A Fully UNsupervised approach for Surgical Instrument Segmentation. Med Image Anal 2023; 85:102751. [PMID: 36716700 DOI: 10.1016/j.media.2023.102751] [Citation(s) in RCA: 5] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2022] [Revised: 10/03/2022] [Accepted: 01/10/2023] [Indexed: 01/21/2023]
Abstract
Automatic surgical instrument segmentation of endoscopic images is a crucial building block of many computer-assistance applications for minimally invasive surgery. So far, state-of-the-art approaches completely rely on the availability of a ground-truth supervision signal, obtained via manual annotation, thus expensive to collect at large scale. In this paper, we present FUN-SIS, a Fully-UNsupervised approach for binary Surgical Instrument Segmentation. FUN-SIS trains a per-frame segmentation model on completely unlabelled endoscopic videos, by solely relying on implicit motion information and instrument shape-priors. We define shape-priors as realistic segmentation masks of the instruments, not necessarily coming from the same dataset/domain as the videos. The shape-priors can be collected in various and convenient ways, such as recycling existing annotations from other datasets. We leverage them as part of a novel generative-adversarial approach, allowing to perform unsupervised instrument segmentation of optical-flow images during training. We then use the obtained instrument masks as pseudo-labels in order to train a per-frame segmentation model; to this aim, we develop a learning-from-noisy-labels architecture, designed to extract a clean supervision signal from these pseudo-labels, leveraging their peculiar noise properties. We validate the proposed contributions on three surgical datasets, including the MICCAI 2017 EndoVis Robotic Instrument Segmentation Challenge dataset. The obtained fully-unsupervised results for surgical instrument segmentation are almost on par with the ones of fully-supervised state-of-the-art approaches. This suggests the tremendous potential of the proposed method to leverage the great amount of unlabelled data produced in the context of minimally invasive surgery.
Collapse
|
3
|
Zhuge M, Fan DP, Liu N, Zhang D, Xu D, Shao L. Salient Object Detection via Integrity Learning. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2023; 45:3738-3752. [PMID: 35666793 DOI: 10.1109/tpami.2022.3179526] [Citation(s) in RCA: 7] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Although current salient object detection (SOD) works have achieved significant progress, they are limited when it comes to the integrity of the predicted salient regions. We define the concept of integrity at both a micro and macro level. Specifically, at the micro level, the model should highlight all parts that belong to a certain salient object. Meanwhile, at the macro level, the model needs to discover all salient objects in a given image. To facilitate integrity learning for SOD, we design a novel Integrity Cognition Network (ICON), which explores three important components for learning strong integrity features. 1) Unlike existing models, which focus more on feature discriminability, we introduce a diverse feature aggregation (DFA) component to aggregate features with various receptive fields (i.e., kernel shape and context) and increase feature diversity. Such diversity is the foundation for mining the integral salient objects. 2) Based on the DFA features, we introduce an integrity channel enhancement (ICE) component with the goal of enhancing feature channels that highlight the integral salient objects, while suppressing the other distracting ones. 3) After extracting the enhanced features, the part-whole verification (PWV) method is employed to determine whether the part and whole object features have strong agreement. Such part-whole agreements can further improve the micro-level integrity for each salient object. To demonstrate the effectiveness of our ICON, comprehensive experiments are conducted on seven challenging benchmarks. Our ICON outperforms the baseline methods in terms of a wide range of metrics. Notably, our ICON achieves ∼ 10% relative improvement over the previous best model in terms of average false negative ratio (FNR), on six datasets. Codes and results are available at: https://github.com/mczhuge/ICON.
Collapse
|
4
|
Seong H, Hyun J, Kim E. Video Object Segmentation Using Kernelized Memory Network With Multiple Kernels. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2023; 45:2595-2612. [PMID: 35353695 DOI: 10.1109/tpami.2022.3163375] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Semi-supervised video object segmentation (VOS) is to predict the segment of a target object in a video when a ground truth segmentation mask for the target is given in the first frame. Recently, space-time memory networks (STM) have received significant attention as a promising approach for semi-supervised VOS. However, an important point has been overlooked in applying STM to VOS: The solution (=STM) is non-local, but the problem (=VOS) is predominantly local. To solve this mismatch between STM and VOS, we propose new VOS networks called kernelized memory network (KMN) and KMN with multiple kernels (KMN M). Our networks conduct not only Query-to-Memory matching but also Memory-to-Query matching. In Memory-to-Query matching, a kernel is employed to reduce the degree of non-localness of the STM. In addition, we present a Hide-and-Seek strategy in pre-training to handle occlusions effectively. The proposed networks surpass the state-of-the-art results on standard benchmarks by a significant margin (+4% in JM on DAVIS 2017 test-dev set). The runtimes of our proposed KMN and KMN M on DAVIS 2016 validation set are 0.12 and 0.13 seconds per frame, respectively, and the two networks have similar computation times to STM.
Collapse
|
5
|
A multi-source feature extraction network for salient object detection. Neural Comput Appl 2023. [DOI: 10.1007/s00521-022-08172-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/04/2023]
|
6
|
Liu JJ, Hou Q, Liu ZA, Cheng MM. PoolNet+: Exploring the Potential of Pooling for Salient Object Detection. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2023; 45:887-904. [PMID: 34982676 DOI: 10.1109/tpami.2021.3140168] [Citation(s) in RCA: 9] [Impact Index Per Article: 9.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
We explore the potential of pooling techniques on the task of salient object detection by expanding its role in convolutional neural networks. In general, two pooling-based modules are proposed. A global guidance module (GGM) is first built based on the bottom-up pathway of the U-shape architecture, which aims to guide the location information of the potential salient objects into layers at different feature levels. A feature aggregation module (FAM) is further designed to seamlessly fuse the coarse-level semantic information with the fine-level features in the top-down pathway. We can progressively refine the high-level semantic features with these two modules and obtain detail enriched saliency maps. Experimental results show that our proposed approach can locate the salient objects more accurately with sharpened details and substantially improve the performance compared with the existing state-of-the-art methods. Besides, our approach is fast and can run at a speed of 53 FPS when processing a 300 ×400 image. To make our approach better applied to mobile applications, we take MobileNetV2 as our backbone and re-tailor the structure of our pooling-based modules. Our mobile version model achieves a running speed of 66 FPS yet still performs better than most existing state-of-the-art methods. To verify the generalization ability of the proposed method, we apply it to the edge detection, RGB-D salient object detection, and camouflaged object detection tasks, and our method achieves better results than the corresponding state-of-the-art methods of these three tasks. Code can be found at http://mmcheng.net/poolnet/.
Collapse
|
7
|
STI-Net: Spatiotemporal Integration Network for Video Saliency Detection. Inf Sci (N Y) 2023. [DOI: 10.1016/j.ins.2023.01.106] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/30/2023]
|
8
|
Risnandar. DeSa COVID-19: Deep salient COVID-19 image-based quality assessment. JOURNAL OF KING SAUD UNIVERSITY. COMPUTER AND INFORMATION SCIENCES 2022; 34:9501-9512. [PMID: 38620925 PMCID: PMC8647162 DOI: 10.1016/j.jksuci.2021.11.013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/23/2021] [Revised: 11/14/2021] [Accepted: 11/16/2021] [Indexed: 04/17/2024]
Abstract
This study offers an advanced method to evaluate the coronavirus disease 2019 (COVID-19) image quality. The salient COVID-19 image map is incorporated with the deep convolutional neural network (DCNN), namely DeSa COVID-19, which exerts the n-convex method for the full-reference image quality assessment (FR-IQA). The glaring outcomes substantiate that DeSa COVID-19 and the recommended DCNN architecture can convey a remarkable accomplishment on the COVID-chestxray and the COVID-CT datasets, respectively. The salient COVID-19 image map is also gauged in the minuscule COVID-19 image patches. The exploratory results attest that DeSa COVID-19 and the recommended DCNN methods are very good accomplishment compared with other advanced methods on COVID-chestxray and COVID-CT datasets, respectively. The recommended DCNN also acquires the enhanced outgrowths faced with several advanced full-reference-medical-image-quality-assessment (FR-MIQA) techniques in the fast fading (FF), blocking artifact (BA), white noise Gaussian (WG), JPEG, and JPEG2000 (JP2K) in the distorted and undistorted COVID-19 images. The Spearman's rank order correlation coefficient (SROCC) and the linear correlation coefficient (LCC) appraise the recommended DCNN and DeSa COVID-19 fulfillment which are compared the recent FR-MIQA methods. The DeSa COVID-19 evaluation outshines 2.63 % and 2.62 % higher compared the recommended DCNN, and 28.53 % and 29.01 % esteem all of advanced FR-MIQAs methods on SROCC and LCC measures, respectively. The shift add operations of trigonometric, logarithmic, and exponential functions are mowed down in the computational complexity of the DeSa COVID-19 and the recommended DCNN. The DeSa COVID-19 more superior the recommended DCNN and also the other recent full-reference medical image quality assessment methods.
Collapse
Affiliation(s)
- Risnandar
- The Intelligent Systems Research Group, School of Computing, Telkom University, Jl. Telekomunikasi No. 1, Terusan Buahbatu-Dayeuhkolot, Bandung, West Java 40257 Indonesia
- The Computer Vision Research Group, the Research Center for Informatics, Indonesian Institute of Sciences (LIPI) and the National Research and Innovation Agency (BRIN), Republic of Indonesia, Jl. Sangkuriang/Cisitu No.21/154D LIPI Building 20th, 3rd Floor, Bandung, West Java, 40135 Indonesia
| |
Collapse
|
9
|
Jia XZ, DongYe CL, Peng YJ, Zhao WX, Liu TD. MRBENet: A Multiresolution Boundary Enhancement Network for Salient Object Detection. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2022; 2022:7780756. [PMID: 36262601 PMCID: PMC9576351 DOI: 10.1155/2022/7780756] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/04/2022] [Revised: 09/08/2022] [Accepted: 09/24/2022] [Indexed: 11/17/2022]
Abstract
Salient Object Detection (SOD) simulates the human visual perception in locating the most attractive objects in the images. Existing methods based on convolutional neural networks have proven to be highly effective for SOD. However, in some cases, these methods cannot satisfy the need of both accurately detecting intact objects and maintaining their boundary details. In this paper, we present a Multiresolution Boundary Enhancement Network (MRBENet) that exploits edge features to optimize the location and boundary fineness of salient objects. We incorporate a deeper convolutional layer into the backbone network to extract high-level semantic features and indicate the location of salient objects. Edge features of different resolutions are extracted by a U-shaped network. We designed a Feature Fusion Module (FFM) to fuse edge features and salient features. Feature Aggregation Module (FAM) based on spatial attention performs multiscale convolutions to enhance salient features. The FFM and FAM allow the model to accurately locate salient objects and enhance boundary fineness. Extensive experiments on six benchmark datasets demonstrate that the proposed method is highly effective and improves the accuracy of salient object detection compared with state-of-the-art methods.
Collapse
Affiliation(s)
- Xing-Zhao Jia
- College of Computer Science and Engineering, Shandong University of Science and Technology, Qingdao 266590, China
| | - Chang-Lei DongYe
- College of Computer Science and Engineering, Shandong University of Science and Technology, Qingdao 266590, China
| | - Yan-Jun Peng
- College of Computer Science and Engineering, Shandong University of Science and Technology, Qingdao 266590, China
| | - Wen-Xiu Zhao
- College of Computer Science and Engineering, Shandong University of Science and Technology, Qingdao 266590, China
| | - Tian-De Liu
- College of Computer Science and Engineering, Shandong University of Science and Technology, Qingdao 266590, China
| |
Collapse
|
10
|
Boosting Few-shot visual recognition via saliency-guided complementary attention. Neurocomputing 2022. [DOI: 10.1016/j.neucom.2022.08.028] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022]
|
11
|
Complementary Segmentation of Primary Video Objects with Reversible Flows. APPLIED SCIENCES-BASEL 2022. [DOI: 10.3390/app12157781] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Segmenting primary objects in a video is an important yet challenging problem in intelligent video surveillance, as it exhibits various levels of foreground/background ambiguities. To reduce such ambiguities, we propose a novel formulation via exploiting foreground and background context as well as their complementary constraint. Under this formulation, a unified objective function is further defined to encode each cue. For implementation, we design a complementary segmentation network (CSNet) with two separate branches, which can simultaneously encode the foreground and background information along with joint spatial constraints. The CSNet is trained on massive images with manually annotated salient objects in an end-to-end manner. By applying CSNet on each video frame, the spatial foreground and background maps can be initialized. To enforce temporal consistency effectively and efficiently, we divide each frame into superpixels and construct a neighborhood reversible flow that reflects the most reliable temporal correspondences between superpixels in far-away frames. With such a flow, the initialized foregroundness and backgroundness can be propagated along the temporal dimension so that primary video objects gradually pop out and distractors are well suppressed. Extensive experimental results on three video datasets show that the proposed approach achieves impressive performance in comparisons with 22 state-of-the-art models.
Collapse
|
12
|
Yan T, Huang X, Zhao Q. Hierarchical Superpixel Segmentation by Parallel CRTrees Labeling. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2022; 31:4719-4732. [PMID: 35797313 DOI: 10.1109/tip.2022.3187563] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
This paper proposes a hierarchical superpixel segmentation by representing an image as a hierarchy of 1-nearest neighbor (1-NN) graphs with pixels/superpixels denoting the graph vertices. The 1-NN graphs are built from the pixel/superpixel adjacent matrices to ensure connectivity. To determine the next-level superpixel hierarchy, inspired by FINCH clustering, the weakly connected components (WCCs) of the 1-NN graph are labeled as superpixels. We reveal that the WCCs of a 1-NN graph consist of a forest of cycle-root-trees (CRTrees). The forest-like structure inspires us to propose a two-stage parallel CRTrees labeling which first links the child vertices to the cycle-roots and then labels all the vertices by the cycle-roots. We also propose an inter-inner superpixel distance penalization and a Lab color lightness penalization base on the property that the distance of a CRTree decreases monotonically from the child to root vertices. Experiments show the parallel CRTrees labeling is several times faster than recent advanced sequential and parallel connected components labeling algorithms. The proposed hierarchical superpixel segmentation has comparable performance to the best performer ETPS (state-of-the-arts) on the BSDS500, NYUV2, and Fash datasets. At the same time, it can achieve 200FPS for 480P video streams.
Collapse
|
13
|
Liu Y, Cheng MM, Zhang XY, Nie GY, Wang M. DNA: Deeply Supervised Nonlinear Aggregation for Salient Object Detection. IEEE TRANSACTIONS ON CYBERNETICS 2022; 52:6131-6142. [PMID: 33531332 DOI: 10.1109/tcyb.2021.3051350] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Recent progress on salient object detection mainly aims at exploiting how to effectively integrate multiscale convolutional features in convolutional neural networks (CNNs). Many popular methods impose deep supervision to perform side-output predictions that are linearly aggregated for final saliency prediction. In this article, we theoretically and experimentally demonstrate that linear aggregation of side-output predictions is suboptimal, and it only makes limited use of the side-output information obtained by deep supervision. To solve this problem, we propose deeply supervised nonlinear aggregation (DNA) for better leveraging the complementary information of various side-outputs. Compared with existing methods, it: 1) aggregates side-output features rather than predictions and 2) adopts nonlinear instead of linear transformations. Experiments demonstrate that DNA can successfully break through the bottleneck of the current linear approaches. Specifically, the proposed saliency detector, a modified U-Net architecture with DNA, performs favorably against state-of-the-art methods on various datasets and evaluation metrics without bells and whistles.
Collapse
|
14
|
Tan Z, Gu X. Co-saliency detection with intra-group two-stage group semantics propagation and inter-group contrastive learning. Knowl Based Syst 2022. [DOI: 10.1016/j.knosys.2022.109356] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/28/2022]
|
15
|
Zhang Q, Kang Z, Xu Z, Huang S, Fu H. Spaks: Self-paced multiple kernel subspace clustering with feature smoothing regularization. Knowl Based Syst 2022. [DOI: 10.1016/j.knosys.2022.109500] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/17/2022]
|
16
|
Wei L, Lang C, Liang L, Feng S, Wang T, Chen S. Weakly Supervised Video Object Segmentation via Dual-attention Cross-branch Fusion. ACM T INTEL SYST TEC 2022. [DOI: 10.1145/3506716] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
Abstract
Recently, concerning the challenge of collecting large-scale explicitly annotated videos, weakly supervised video object segmentation (WSVOS) using video tags has attracted much attention. Existing WSVOS approaches follow a general pipeline including two phases, i.e., a pseudo masks generation phase and a refinement phase. To explore the intrinsic property and correlation buried in the video frames, most of them focus on the later phase by introducing optical flow as temporal information to provide more supervision. However, these optical flow-based studies are greatly affected by illumination and distortion and lack consideration of the discriminative capacity of multi-level deep features. In this article, with the goal of capturing more effective temporal information and investigating a temporal information fusion strategy accordingly, we propose a unified WSVOS model by adopting a two-branch architecture with a multi-level cross-branch fusion strategy, named as dual-attention cross-branch fusion network (DACF-Net). Concretely, the two branches of DACF-Net, i.e., a temporal prediction subnetwork (TPN) and a spatial segmentation subnetwork (SSN), are used for extracting temporal information and generating predicted segmentation masks, respectively. To perform the cross-branch fusion between TPN and SSN, we propose a dual-attention fusion module that can be plugged into the SSN flexibly. We also pose a cross-frame coherence loss (CFCL) to achieve smooth segmentation results by exploiting the coherence of masks produced by TPN and SSN. Extensive experiments demonstrate the effectiveness of proposed approach compared with the state-of-the-arts on two challenging datasets, i.e., Davis-2016 and YouTube-Objects.
Collapse
Affiliation(s)
- Lili Wei
- the Beijing Key Laboratory of Traffic Data Analysis and Mining, School of Computerand Information Technology, Beijing Jiaotong University, Beijing, China
| | - Congyan Lang
- the Beijing Key Laboratory of Traffic Data Analysis and Mining, School of Computerand Information Technology, Beijing Jiaotong University, Beijing, China
| | - Liqian Liang
- the Beijing Key Laboratory of Traffic Data Analysis and Mining, School of Computerand Information Technology, Beijing Jiaotong University, Beijing, China
| | - Songhe Feng
- the Beijing Key Laboratory of Traffic Data Analysis and Mining, School of Computerand Information Technology, Beijing Jiaotong University, Beijing, China
| | - Tao Wang
- the Beijing Key Laboratory of Traffic Data Analysis and Mining, School of Computerand Information Technology, Beijing Jiaotong University, Beijing, China
| | - Shidi Chen
- the Beijing Key Laboratory of Traffic Data Analysis and Mining, School of Computerand Information Technology, Beijing Jiaotong University, Beijing, China
| |
Collapse
|
17
|
Yang J, Wang L, Li Y. Feature Refine Network for Salient Object Detection. SENSORS (BASEL, SWITZERLAND) 2022; 22:s22124490. [PMID: 35746271 PMCID: PMC9228599 DOI: 10.3390/s22124490] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 05/09/2022] [Revised: 05/27/2022] [Accepted: 06/09/2022] [Indexed: 06/12/2023]
Abstract
Different feature learning strategies have enhanced performance in recent deep neural network-based salient object detection. Multi-scale strategy and residual learning strategies are two types of multi-scale learning strategies. However, there are still some problems, such as the inability to effectively utilize multi-scale feature information and the lack of fine object boundaries. We propose a feature refined network (FRNet) to overcome the problems mentioned, which includes a novel feature learning strategy that combines the multi-scale and residual learning strategies to generate the final saliency prediction. We introduce the spatial and channel 'squeeze and excitation' blocks (scSE) at the side outputs of the backbone. It allows the network to concentrate more on saliency regions at various scales. Then, we propose the adaptive feature fusion module (AFFM), which efficiently fuses multi-scale feature information in order to predict superior saliency maps. Finally, to supervise network learning of more information on object boundaries, we propose a hybrid loss that contains four fundamental losses and combines properties of diverse losses. Comprehensive experiments demonstrate the effectiveness of the FRNet on five datasets, with competitive results when compared to other relevant approaches.
Collapse
|
18
|
Wang W, Lai Q, Fu H, Shen J, Ling H, Yang R. Salient Object Detection in the Deep Learning Era: An In-Depth Survey. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2022; 44:3239-3259. [PMID: 33434124 DOI: 10.1109/tpami.2021.3051099] [Citation(s) in RCA: 54] [Impact Index Per Article: 27.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/27/2023]
Abstract
As an essential problem in computer vision, salient object detection (SOD) has attracted an increasing amount of research attention over the years. Recent advances in SOD are predominantly led by deep learning-based solutions (named deep SOD). To enable in-depth understanding of deep SOD, in this paper, we provide a comprehensive survey covering various aspects, ranging from algorithm taxonomy to unsolved issues. In particular, we first review deep SOD algorithms from different perspectives, including network architecture, level of supervision, learning paradigm, and object-/instance-level detection. Following that, we summarize and analyze existing SOD datasets and evaluation metrics. Then, we benchmark a large group of representative SOD models, and provide detailed analyses of the comparison results. Moreover, we study the performance of SOD algorithms under different attribute settings, which has not been thoroughly explored previously, by constructing a novel SOD dataset with rich attribute annotations covering various salient object types, challenging factors, and scene categories. We further analyze, for the first time in the field, the robustness of SOD models to random input perturbations and adversarial attacks. We also look into the generalization and difficulty of existing SOD datasets. Finally, we discuss several open issues of SOD and outline future research directions. All the saliency prediction maps, our constructed dataset with annotations, and codes for evaluation are publicly available at https://github.com/wenguanwang/SODsurvey.
Collapse
|
19
|
Spatiotemporal context-aware network for video salient object detection. Neural Comput Appl 2022. [DOI: 10.1007/s00521-022-07330-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|
20
|
Atehortúa A, Romero E, Garreau M. Characterization of motion patterns by a spatio-temporal saliency descriptor in cardiac cine MRI. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2022; 218:106714. [PMID: 35263659 DOI: 10.1016/j.cmpb.2022.106714] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/29/2021] [Revised: 02/03/2022] [Accepted: 02/23/2022] [Indexed: 06/14/2023]
Abstract
BACKGROUND AND OBJECTIVE Abnormalities of the heart motion reveal the presence of a disease. However, a quantitative interpretation of the motion is still a challenge due to the complex dynamics of the heart. This work proposes a quantitative characterization of regional cardiac motion patterns in cine magnetic resonance imaging (MRI) by a novel spatio-temporal saliency descriptor. METHOD The strategy starts by dividing the cardiac sequence into a progression of scales which are in due turn mapped to a feature space of regional orientation changes, mimicking the multi-resolution decomposition of oriented primitive changes of visual systems. These changes are estimated as the difference between a particular time and the rest of the sequence. This decomposition is then temporarily and regionally integrated for a particular orientation and then for the set of different orientations. A final spatio-temporal 4D saliency map is obtained as the summation of the previously integrated information for the available scales. The saliency dispersion of this map was computed in standard cardiac locations as a measure of the regional motion pattern and was applied to discriminate control and hypertrophic cardiomyopathy (HCM) subjects during the diastolic phase. RESULTS Salient motion patterns were estimated from an experimental set, which consisted of 3D sequences acquired by MRI from 108 subjects (33 control, 35 HCM, 20 dilated cardiomyopathy (DCM), and 20 myocardial infarction (MINF) from heterogeneous datasets). HCM and control subjects were classified by an SVM that learned the salient motion patterns estimated from the presented strategy, by achieving a 94% AUC. In addition, statistical differences (test t-student, p<0.05) were found among groups of disease in the septal and anterior ventricular segments at both the ED and ES, with salient motion characteristics aligned with existing knowledge on the diseases. CONCLUSIONS Regional wall motion abnormality in the apical, anterior, basal, and inferior segments was associated with the saliency dispersion in HCM, DCM, and MINF compared to healthy controls during the systolic and diastolic phases. This saliency analysis may be used to detect subtle changes in heart function.
Collapse
Affiliation(s)
- Angélica Atehortúa
- Universidad Nacional de Colombia, Bogotá, Colombia; Univ Rennes, Inserm, LTSI UMR 1099, Rennes F-35000, France
| | | | | |
Collapse
|
21
|
Liu X, Li D. Binary segmentation based on visual attention consistency under background-change. Appl Soft Comput 2022. [DOI: 10.1016/j.asoc.2022.108738] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/01/2023]
|
22
|
Kaur J, Singh W. Tools, techniques, datasets and application areas for object detection in an image: a review. MULTIMEDIA TOOLS AND APPLICATIONS 2022; 81:38297-38351. [PMID: 35493415 PMCID: PMC9033309 DOI: 10.1007/s11042-022-13153-y] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 04/16/2021] [Revised: 02/24/2022] [Accepted: 04/10/2022] [Indexed: 06/14/2023]
Abstract
Object detection is one of the most fundamental and challenging tasks to locate objects in images and videos. Over the past, it has gained much attention to do more research on computer vision tasks such as object classification, counting of objects, and object monitoring. This study provides a detailed literature review focusing on object detection and discusses the object detection techniques. A systematic review has been followed to summarize the current research work's findings and discuss seven research questions related to object detection. Our contribution to the current research work is (i) analysis of traditional, two-stage, one-stage object detection techniques, (ii) Dataset preparation and available standard dataset, (iii) Annotation tools, and (iv) performance evaluation metrics. In addition, a comparative analysis has been performed and analyzed that the proposed techniques are different in their architecture, optimization function, and training strategies. With the remarkable success of deep neural networks in object detection, the performance of the detectors has improved. Various research challenges and future directions for object detection also has been discussed in this research paper.
Collapse
Affiliation(s)
- Jaskirat Kaur
- Department of Computer Science, Punjabi University, Patiala, India
| | - Williamjeet Singh
- Department of Computer Science and Engineering, Punjabi University, Patiala, India
| |
Collapse
|
23
|
Saliency detection based on hybrid artificial bee colony and firefly optimization. Pattern Anal Appl 2022. [DOI: 10.1007/s10044-022-01063-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/18/2022]
|
24
|
A novel spatiotemporal attention enhanced discriminative network for video salient object detection. APPL INTELL 2022. [DOI: 10.1007/s10489-021-02649-z] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/27/2022]
|
25
|
Lu X, Wang W, Shen J, Crandall D, Luo J. Zero-Shot Video Object Segmentation With Co-Attention Siamese Networks. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2022; 44:2228-2242. [PMID: 33232224 DOI: 10.1109/tpami.2020.3040258] [Citation(s) in RCA: 17] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
We introduce a novel network, called CO-attention siamese network (COSNet), to address the zero-shot video object segmentation task in a holistic fashion. We exploit the inherent correlation among video frames and incorporate a global co-attention mechanism to further improve the state-of-the-art deep learning based solutions that primarily focus on learning discriminative foreground representations over appearance and motion in short-term temporal segments. The co-attention layers in COSNet provide efficient and competent stages for capturing global correlations and scene context by jointly computing and appending co-attention responses into a joint feature space. COSNet is a unified and end-to-end trainable framework where different co-attention variants can be derived for capturing diverse properties of the learned joint feature space. We train COSNet with pairs (or groups) of video frames, and this naturally augments training data and allows increased learning capacity. During the segmentation stage, the co-attention model encodes useful information by processing multiple reference frames together, which is leveraged to infer the frequently reappearing and salient foreground objects better. Our extensive experiments over three large benchmarks demonstrate that COSNet outperforms the current alternatives by a large margin. Our implementations are available at https://github.com/carrierlxk/COSNet.
Collapse
|
26
|
RoiSeg: An Effective Moving Object Segmentation Approach Based on Region-of-Interest with Unsupervised Learning. APPLIED SCIENCES-BASEL 2022. [DOI: 10.3390/app12052674] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/27/2023]
Abstract
Traditional video object segmentation often has low detection speed and inaccurate results due to the jitter caused by the pan-and-tilt or hand-held devices. Deep neural network (DNN) has been widely adopted to address these problems; however, it relies on a large number of annotated data and high-performance computing units. Therefore, DNN is not suitable for some special scenarios (e.g., no prior knowledge or powerful computing ability). In this paper, we propose RoiSeg, an effective moving object segmentation approach based on Region-of-Interest (ROI), which utilizes unsupervised learning method to achieve automatic segmentation of moving objects. Specifically, we first hypothesize that the central n × n pixels of images act as the ROI to represent the features of the segmented moving object. Second, we pool the ROI to a central point of the foreground to simplify the segmentation problem into a classification problem based on ROI. Third but not the least, we implement a trajectory-based classifier and an online updating mechanism to address the classification problem and the compensation of class imbalance, respectively. We conduct extensive experiments to evaluate the performance of RoiSeg and the experimental results demonstrate that RoiSeg is more accurate and faster compared with other segmentation algorithms. Moreover, RoiSeg not only effectively handles ambient lighting changes, fog, salt and pepper noise, but also has a good ability to deal with camera jitter and windy scenes.
Collapse
|
27
|
Joint learning affinity matrix and representation matrix for robust low-rank multi-kernel clustering. APPL INTELL 2022. [DOI: 10.1007/s10489-021-02974-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022]
|
28
|
Lane Mark Detection with Pre-Aligned Spatial-Temporal Attention. SENSORS 2022; 22:s22030794. [PMID: 35161542 PMCID: PMC8838618 DOI: 10.3390/s22030794] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/14/2021] [Revised: 01/14/2022] [Accepted: 01/18/2022] [Indexed: 02/04/2023]
Abstract
Lane mark detection plays an important role in autonomous driving under structural environments. Many deep learning-based lane mark detection methods have been put forward in recent years. However, most of current methods limit their solutions within one single image and do not make use of the de facto successive image input during the driving scene, which may lead to inferior performance in some challenging scenarios such as occlusion, shadows, and lane mark degradation. To address the issue, we propose a novel lane mark detection network which takes pre-aligned multiple successive frames as inputs to produce more stable predictions. A Spatial-Temporal Attention Module (STAM) is designed in the network to adaptively aggregate the feature information of history frames to the current frame. Various structure of the STAM is also studied to ensure the best performance. Experiments on Tusimple and ApolloScape datasets show that our method can effectively improve lane mark detection and achieve state-of-the-art performance.
Collapse
|
29
|
Zhai Y, Fan DP, Yang J, Borji A, Shao L, Han J, Wang L. Bifurcated Backbone Strategy for RGB-D Salient Object Detection. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2021; 30:8727-8742. [PMID: 34613915 DOI: 10.1109/tip.2021.3116793] [Citation(s) in RCA: 12] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Multi-level feature fusion is a fundamental topic in computer vision. It has been exploited to detect, segment and classify objects at various scales. When multi-level features meet multi-modal cues, the optimal feature aggregation and multi-modal learning strategy become a hot potato. In this paper, we leverage the inherent multi-modal and multi-level nature of RGB-D salient object detection to devise a novel Bifurcated Backbone Strategy Network (BBS-Net). Our architecture, is simple, efficient, and backbone-independent. In particular, first, we propose to regroup the multi-level features into teacher and student features using a bifurcated backbone strategy (BBS). Second, we introduce a depth-enhanced module (DEM) to excavate informative depth cues from the channel and spatial views. Then, RGB and depth modalities are fused in a complementary way. Extensive experiments show that BBS-Net significantly outperforms 18 state-of-the-art (SOTA) models on eight challenging datasets under five evaluation measures, demonstrating the superiority of our approach (~4% improvement in S-measure vs . the top-ranked model: DMRA). In addition, we provide a comprehensive analysis on the generalization ability of different RGB-D datasets and provide a powerful training set for future research. The complete algorithm, benchmark results, and post-processing toolbox are publicly available at https://github.com/zyjwuyan/BBS-Net.
Collapse
|
30
|
|
31
|
Multi-pathway feature integration network for salient object detection. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2021.08.082] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
|
32
|
Liang Y, Qin G, Sun M, Qin J, Yan J, Zhang Z. Semantic and detail collaborative learning network for salient object detection. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2021.08.037] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
|
33
|
|
34
|
|
35
|
Huang Z, Chen HX, Zhou T, Yang YZ, Liu BY. Multi-level cross-modal interaction network for RGB-D salient object detection. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2021.04.053] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|
36
|
Bu X, Sun Y, Wang J, Liu K, Liang J, Jin G, Chung TS. Weakly supervised video object segmentation initialized with referring expression. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2020.06.129] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
|
37
|
Human scanpath estimation based on semantic segmentation guided by common eye fixation behaviors. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2020.07.121] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/18/2023]
|
38
|
Learning spatial-channel regularization jointly with correlation filter for visual tracking. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2020.04.146] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/22/2022]
|
39
|
Zhu X, Li Y, Fu H, Fan X, Shi Y, Lei J. RGB-D salient object detection via cross-modal joint feature extraction and low-bound fusion loss. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2020.05.110] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
|
40
|
Lin G, Zhao S, Shen J. Video person re-identification with global statistic pooling and self-attention distillation. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2020.05.111] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
|
41
|
|
42
|
Ning X, Gong K, Li W, Zhang L. JWSAA: Joint weak saliency and attention aware for person re-identification. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2020.05.106] [Citation(s) in RCA: 57] [Impact Index Per Article: 19.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
|
43
|
|
44
|
Chen Z, Cong R, Xu Q, Huang Q. DPANet: Depth Potentiality-Aware Gated Attention Network for RGB-D Salient Object Detection. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2021; 30:7012-7024. [PMID: 33141667 DOI: 10.1109/tip.2020.3028289] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
There are two main issues in RGB-D salient object detection: (1) how to effectively integrate the complementarity from the cross-modal RGB-D data; (2) how to prevent the contamination effect from the unreliable depth map. In fact, these two problems are linked and intertwined, but the previous methods tend to focus only on the first problem and ignore the consideration of depth map quality, which may yield the model fall into the sub-optimal state. In this paper, we address these two issues in a holistic model synergistically, and propose a novel network named DPANet to explicitly model the potentiality of the depth map and effectively integrate the cross-modal complementarity. By introducing the depth potentiality perception, the network can perceive the potentiality of depth information in a learning-based manner, and guide the fusion process of two modal data to prevent the contamination occurred. The gated multi-modality attention module in the fusion process exploits the attention mechanism with a gate controller to capture long-range dependencies from a cross-modal perspective. Experimental results compared with 16 state-of-the-art methods on 8 datasets demonstrate the validity of the proposed approach both quantitatively and qualitatively. https://github.com/JosephChenHub/DPANet.
Collapse
|
45
|
|
46
|
Wang W, Shen J, Lu X, Hoi SCH, Ling H. Paying Attention to Video Object Pattern Understanding. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2021; 43:2413-2428. [PMID: 31940522 DOI: 10.1109/tpami.2020.2966453] [Citation(s) in RCA: 25] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
This paper conducts a systematic study on the role of visual attention in video object pattern understanding. By elaborately annotating three popular video segmentation datasets (DAVIS 16, Youtube-Objects, and SegTrack V2) with dynamic eye-tracking data in the unsupervised video object segmentation (UVOS) setting. For the first time, we quantitatively verified the high consistency of visual attention behavior among human observers, and found strong correlation between human attention and explicit primary object judgments during dynamic, task-driven viewing. Such novel observations provide an in-depth insight of the underlying rationale behind video object pattens. Inspired by these findings, we decouple UVOS into two sub-tasks: UVOS-driven Dynamic Visual Attention Prediction (DVAP) in spatiotemporal domain, and Attention-Guided Object Segmentation (AGOS) in spatial domain. Our UVOS solution enjoys three major advantages: 1) modular training without using expensive video segmentation annotations, instead, using more affordable dynamic fixation data to train the initial video attention module and using existing fixation-segmentation paired static/image data to train the subsequent segmentation module; 2) comprehensive foreground understanding through multi-source learning; and 3) additional interpretability from the biologically-inspired and assessable attention. Experiments on four popular benchmarks show that, even without using expensive video object mask annotations, our model achieves compelling performance compared with state-of-the-arts and enjoys fast processing speed (10 fps on a single GPU). Our collected eye-tracking data and algorithm implementations have been made publicly available at https://github.com/wenguanwang/AGS.
Collapse
|
47
|
|
48
|
Towards accurate RGB-D saliency detection with complementary attention and adaptive integration. Neurocomputing 2021. [DOI: 10.1016/j.neucom.2020.12.125] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
|
49
|
Dong X, Shen J, Wang W, Shao L, Ling H, Porikli F. Dynamical Hyperparameter Optimization via Deep Reinforcement Learning in Tracking. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2021; 43:1515-1529. [PMID: 31796388 DOI: 10.1109/tpami.2019.2956703] [Citation(s) in RCA: 22] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/10/2023]
Abstract
Hyperparameters are numerical pre-sets whose values are assigned prior to the commencement of a learning process. Selecting appropriate hyperparameters is often critical for achieving satisfactory performance in many vision problems, such as deep learning-based visual object tracking. However, it is often difficult to determine their optimal values, especially if they are specific to each video input. Most hyperparameter optimization algorithms tend to search a generic range and are imposed blindly on all sequences. In this paper, we propose a novel dynamical hyperparameter optimization method that adaptively optimizes hyperparameters for a given sequence using an action-prediction network leveraged on continuous deep Q-learning. Since the observation space for object tracking is significantly more complex than those in traditional control problems, existing continuous deep Q-learning algorithms cannot be directly applied. To overcome this challenge, we introduce an efficient heuristic strategy to handle high dimensional state space, while also accelerating the convergence behavior. The proposed algorithm is applied to improve two representative trackers, a Siamese-based one and a correlation-filter-based one, to evaluate its generalizability. Their superior performances on several popular benchmarks are clearly demonstrated. Our source code is available at https://github.com/shenjianbing/dqltracking.
Collapse
|
50
|
Xu M, Fu P, Liu B, Li J. Multi-Stream Attention-Aware Graph Convolution Network for Video Salient Object Detection. IEEE TRANSACTIONS ON IMAGE PROCESSING : A PUBLICATION OF THE IEEE SIGNAL PROCESSING SOCIETY 2021; 30:4183-4197. [PMID: 33822725 DOI: 10.1109/tip.2021.3070200] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/12/2023]
Abstract
Recent advances in deep convolution neural networks (CNNs) boost the development of video salient object detection (SOD), and many remarkable deep-CNNs video SOD models have been proposed. However, many existing deep-CNNs video SOD models still suffer from coarse boundaries of the salient object, which may be attributed to the loss of high-frequency information. The traditional graph-based video SOD models can preserve object boundaries well by conducting superpixels/supervoxels segmentation in advance, but they perform weaker in highlighting the whole object than the latest deep-CNNs models, limited by heuristic graph clustering algorithms. To tackle this problem, we find a new way to address this issue under the framework of graph convolution networks (GCNs), taking advantage of graph model and deep neural network. Specifically, a superpixel-level spatiotemporal graph is first constructed among multiple frame-pairs by exploiting the motion cues implied in the frame-pairs. Then the graph data is imported into the devised multi-stream attention-aware GCN, where a novel Edge-Gated graph convolution (GC) operation is proposed to boost the saliency information aggregation on the graph data. A novel attention module is designed to encode the spatiotemporal sematic information via adaptive selection of graph nodes and fusion of the static-specific and the motion-specific graph embedding. Finally, a smoothness-aware regularization term is proposed to enhance the uniformity of salient object. Graph nodes (superpixels) inherently belonging to the same class will be ideally clustered together in the learned embedding space. Extensive experiments have been conducted on three widely used datasets. Compared with fourteen state-of-the-art video SOD models, our proposed method can well retain the salient object boundaries and possess a strong learning ability, which shows that this work is a good practice for designing GCNs for video SOD.
Collapse
|