Improved region proposal network for enhanced few-shot object detection

Despite significant success of deep learning in object detection tasks, the standard training of deep neural networks requires access to a substantial quantity of annotated images across all classes. Data annotation is an arduous and time-consuming endeavor, particularly when dealing with infrequent objects. Few-shot object detection (FSOD) methods have emerged as a solution to the limitations of classic object detection approaches based on deep learning. FSOD methods demonstrate remarkable performance by achieving robust object detection using a significantly smaller amount of training data. A challenge for FSOD is that instances from novel classes that do not belong to the fixed set of training classes appear in the background and the base model may pick them up as potential objects. These objects behave similarly to label noise because they are classified as one of the training dataset classes, leading to FSOD performance degradation. We develop a semi-supervised algorithm to detect and then utilize these unlabeled novel objects as positive samples during the FSOD training stage to improve FSOD performance. Specifically, we develop a hierarchical ternary classification region proposal network (HTRPN) to localize the potential unlabeled novel objects and assign them new objectness labels to distinguish these objects from the base training dataset classes. Our improved hierarchical sampling strategy for the region proposal network (RPN) also boosts the perception ability of the object detection model for large objects. We test our approach and COCO and PASCAL VOC baselines that are commonly used in FSOD literature. Our experimental results indicate that our method is effective and outperforms the existing state-of-the-art (SOTA) FSOD methods. Our implementation is provided as a supplement to support reproducibility of the results https://github.com/zshanggu/HTRPN.1.

Intervening on few-shot object detection based on the front-door criterion

Most few-shot object detection methods aim to utilize the learned generalizable knowledge from base categories to identify instances of novel categories. The fundamental assumption of these approaches is that the model can acquire sufficient transferable knowledge through the learning of base categories. However, our motivating experiments reveal a phenomenon that the model is overfitted to the data of base categories. To discuss the impact of this phenomenon on detection from a causal perspective, we develop a Structural Causal Model involving two key variables, causal generative factors and spurious generative factors. Both variables are derived from the base categories. Generative factors are latent variables or features that are used to control image generation. Causal generative factors are general generative factors that directly influence the generation process, while spurious generative factors are specific to certain categories, specifically the base categories in the problem we are analyzing. We recognize that the essence of the few-shot object detection methods lies in modeling the statistic dependence between novel object instances and their corresponding categories determined by the causal generative factors, while the set of spurious generative factors serves as a confounder in the modeling process. To mitigate the misleading impact of the spurious generative factors, we propose the Front-door Regulator guided by the front-door criterion. Front-door Regulator consists of two plug-and-play regularization terms, namely Semantic Grouping and Semantic Decoupling. We substantiate the effectiveness of our proposed method through experiments conducted on multiple benchmark datasets.