1
|
Zhu Z, Nan L, Xie H, Chen H, Wang J, Wei M, Qin J. CSDN: Cross-Modal Shape-Transfer Dual-Refinement Network for Point Cloud Completion. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 2024; 30:3545-3563. [PMID: 37018698 DOI: 10.1109/tvcg.2023.3236061] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
How will you repair a physical object with some missings? You may imagine its original shape from previously captured images, recover its overall (global) but coarse shape first, and then refine its local details. We are motivated to imitate the physical repair procedure to address point cloud completion. To this end, we propose a cross-modal shape-transfer dual-refinement network (termed CSDN), a coarse-to-fine paradigm with images of full-cycle participation, for quality point cloud completion. CSDN mainly consists of "shape fusion" and "dual-refinement" modules to tackle the cross-modal challenge. The first module transfers the intrinsic shape characteristics from single images to guide the geometry generation of the missing regions of point clouds, in which we propose IPAdaIN to embed the global features of both the image and the partial point cloud into completion. The second module refines the coarse output by adjusting the positions of the generated points, where the local refinement unit exploits the geometric relation between the novel and the input points by graph convolution, and the global constraint unit utilizes the input image to fine-tune the generated offset. Different from most existing approaches, CSDN not only explores the complementary information from images but also effectively exploits cross-modal data in the whole coarse-to-fine completion procedure. Experimental results indicate that CSDN performs favorably against twelve competitors on the cross-modal benchmark.
Collapse
|
2
|
Zhu P, Para WR, Fruhstuck A, Femiani J, Wonka P. Large-Scale Architectural Asset Extraction from Panoramic Imagery. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 2022; 28:1301-1316. [PMID: 32746280 DOI: 10.1109/tvcg.2020.3010694] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
We present a system to extract architectural assets from large-scale collections of panoramic imagery. We automatically rectify and crop parts of the panoramic image that contain dominant planes, and then use object detection to extract assets such as façades and windows. We also provide various tools to identify attributes of the assets to determine the asset quality and index the assets for search. In addition, we propose a User Interface (UI) to visualize and query assets. Finally, we present applications for urban modeling and texture synthesis.
Collapse
|
3
|
|
4
|
Zhang R, Lin L, Wang G, Wang M, Zuo W. Hierarchical Scene Parsing by Weakly Supervised Learning with Image Descriptions. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2019; 41:596-610. [PMID: 29993474 DOI: 10.1109/tpami.2018.2799846] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
This paper investigates a fundamental problem of scene understanding: how to parse a scene image into a structured configuration (i.e., a semantic object hierarchy with object interaction relations). We propose a deep architecture consisting of two networks: i) a convolutional neural network (CNN) extracting the image representation for pixel-wise object labeling and ii) a recursive neural network (RsNN) discovering the hierarchical object structure and the inter-object relations. Rather than relying on elaborative annotations (e.g., manually labeled semantic maps and relations), we train our deep model in a weakly-supervised learning manner by leveraging the descriptive sentences of the training images. Specifically, we decompose each sentence into a semantic tree consisting of nouns and verb phrases, and apply these tree structures to discover the configurations of the training images. Once these scene configurations are determined, then the parameters of both the CNN and RsNN are updated accordingly by back propagation. The entire model training is accomplished through an Expectation-Maximization method. Extensive experiments show that our model is capable of producing meaningful scene configurations and achieving more favorable scene labeling results on two benchmarks (i.e., PASCAL VOC 2012 and SYSU-Scenes) compared with other state-of-the-art weakly-supervised deep learning methods. In particular, SYSU-Scenes contains more than 5,000 scene images with their semantic sentence descriptions, which is created by us for advancing research on scene parsing.
Collapse
|
5
|
Park S, Nie BX, Zhu SC, Nie BX, Zhu SC, Park S, Nie BX. Attribute And-Or Grammar for Joint Parsing of Human Pose, Parts and Attributes. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2018; 40:1555-1569. [PMID: 28749346 DOI: 10.1109/tpami.2017.2731842] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/07/2023]
Abstract
This paper presents an attribute and-or grammar (A-AOG) model for jointly inferring human body pose and human attributes in a parse graph with attributes augmented to nodes in the hierarchical representation. In contrast to other popular methods in the current literature that train separate classifiers for poses and individual attributes, our method explicitly represents the decomposition and articulation of body parts, and account for the correlations between poses and attributes. The A-AOG model is an amalgamation of three traditional grammar formulations: (i) Phrase structure grammar representing the hierarchical decomposition of the human body from whole to parts; (ii) Dependency grammar modeling the geometric articulation by a kinematic graph of the body pose; and (iii) Attribute grammar accounting for the compatibility relations between different parts in the hierarchy so that their appearances follow a consistent style. The parse graph outputs human detection, pose estimation, and attribute prediction simultaneously, which are intuitive and interpretable. We conduct experiments on two tasks on two datasets, and experimental results demonstrate the advantage of joint modeling in comparison with computing poses and attributes independently. Furthermore, our model obtains better performance over existing methods for both pose estimation and attribute prediction tasks.
Collapse
|
6
|
Liu X, Zhao Y, Zhu SC. Single-View 3D Scene Reconstruction and Parsing by Attribute Grammar. IEEE TRANSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE 2018; 40:710-725. [PMID: 28368817 DOI: 10.1109/tpami.2017.2689007] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/07/2023]
Abstract
In this paper, we present an attribute grammar for solving two coupled tasks: i) parsing a 2D image into semantic regions; and ii) recovering the 3D scene structures of all regions. The proposed grammar consists of a set of production rules, each describing a kind of spatial relation between planar surfaces in 3D scenes. These production rules are used to decompose an input image into a hierarchical parse graph representation where each graph node indicates a planar surface or a composite surface. Different from other stochastic image grammars, the proposed grammar augments each graph node with a set of attribute variables to depict scene-level global geometry, e.g., camera focal length, or local geometry, e.g., surface normal, contact lines between surfaces. These geometric attributes impose constraints between a node and its off-springs in the parse graph. Under a probabilistic framework, we develop a Markov Chain Monte Carlo method to construct a parse graph that optimizes the 2D image recognition and 3D scene reconstruction purposes simultaneously. We evaluated our method on both public benchmarks and newly collected datasets. Experiments demonstrate that the proposed method is capable of achieving state-of-the-art scene reconstruction of a single image.
Collapse
|
7
|
John Joseph FJ, Auwatanamongkol S. A crowding multi-objective genetic algorithm for image parsing. Neural Comput Appl 2016. [DOI: 10.1007/s00521-015-2000-2] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
|
8
|
|
9
|
Xu M, Wang Z, Petrou M. Tower of Knowledge for scene interpretation: A survey. Pattern Recognit Lett 2014. [DOI: 10.1016/j.patrec.2014.02.009] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/27/2022]
|
10
|
|
11
|
Ogiela L. Cognitive informatics in image semantics description, identification and automatic pattern understanding. Neurocomputing 2013. [DOI: 10.1016/j.neucom.2013.06.001] [Citation(s) in RCA: 28] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
|
12
|
Battaglia FP, Borensztajn G, Bod R. Structured cognition and neural systems: from rats to language. Neurosci Biobehav Rev 2012; 36:1626-39. [PMID: 22537592 DOI: 10.1016/j.neubiorev.2012.04.004] [Citation(s) in RCA: 20] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2011] [Revised: 04/01/2012] [Accepted: 04/10/2012] [Indexed: 10/28/2022]
Abstract
Much of animal and human cognition is compositional in nature: higher order, complex representations are formed by (rule-governed) combination of more primitive representations. We review here some of the evidence for compositionality in perception and memory, motivating an approach that takes ideas and techniques from computational linguistics to model aspects of structural representation in cognition. We summarize some recent developments in our work that, on the one hand, use algorithms from computational linguistics to model memory consolidation and the formation of semantic memory, and on the other hand use insights from the neurobiology of memory to develop a neurally inspired model of syntactic parsing that improves over existing (not cognitively motivated) models in computational linguistics. These two theoretical studies highlight interesting analogies between language acquisition, semantic memory and memory consolidation, and suggest possible neural mechanisms, implemented in computational algorithms that may underlie memory consolidation.
Collapse
Affiliation(s)
- Francesco P Battaglia
- Center for Neuroscience - Swammerdam Institute for Life Sciences, Universiteit van Amsterdam, Science Park 904 1098XH, Amsterdam, The Netherlands.
| | | | | |
Collapse
|
13
|
Grompone von Gioi R, Delon J, Morel JM. The collaboration of grouping laws in vision. ACTA ACUST UNITED AC 2012; 106:266-83. [PMID: 22343519 DOI: 10.1016/j.jphysparis.2011.12.002] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/23/2011] [Accepted: 12/16/2011] [Indexed: 11/29/2022]
Abstract
Gestalt theory gives a list of geometric grouping laws that could in principle give a complete account of human image perception. Based on an extensive thesaurus of clever graphical images, this theory discusses how grouping laws collaborate, and conflict toward a global image understanding. Unfortunately, as shown in the bibliographical analysis herewith, the attempts to formalize the grouping laws in computer vision and psychophysics have at best succeeded to compute individual partial structures (or partial gestalts), such as alignments or symmetries. Nevertheless, we show here that a never formalized clever Gestalt experimental procedure, the Nachzeichnung suggests a numerical set up to implement and test the collaboration of partial gestalts. The new computational procedure proposed here analyzes a digital image, and performs a numerical simulation that we call Nachtanz or Gestaltic dance. In this dance, the analyzed digital image is gradually deformed in a random way, but maintaining the detected partial gestalts. The resulting dancing images should be perceptually indistinguishable if and only if the grouping process was complete. Like the Nachzeichnung, the Nachtanz permits a visual exploration of the degrees of freedom still available to a figure after all partial groups (or gestalts) have been detected. In the new proposed procedure, instead of drawing themselves, subjects will be shown samples of the automatic Gestalt dances and required to evaluate if the figures are similar. Several numerical preliminary results with this new Gestaltic experimental setup are thoroughly discussed.
Collapse
|
14
|
XU MAI, PETROU MARIA, LU JIANHUA. LEARNING LOGIC RULES FOR THE TOWER OF KNOWLEDGE USING MARKOV LOGIC NETWORKS. INT J PATTERN RECOGN 2011. [DOI: 10.1142/s0218001411008610] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
Abstract
In this paper, we propose a novel logic-rule learning approach for the Tower of Knowledge (ToK) architecture, based on Markov logic networks, for scene interpretation. This approach is in the spirit of the recently proposed Markov logic networks for machine learning. Its purpose is to learn the soft-constraint logic rules for labeling the components of a scene. In our approach, FOIL (First Order Inductive Learner) is applied to learn the logic rules for MLN and then gradient ascent search is utilized to compute weights attached to each rule for softening the rules. This approach also benefits from the architecture of ToK, in reasoning whether a component in a scene has the right characteristics in order to fulfil the functions a label implies, from the logic point of view. One significant advantage of the proposed approach, rather than the previous versions of ToK, is its automatic logic learning capability such that the manual insertion of logic rules is not necessary. Experiments of labeling the identified components in buildings, for building scene interpretation, illustrate the promise of this approach.
Collapse
Affiliation(s)
- MAI XU
- Department of Electrical Engineering, Tsinghua University, P. R. China
| | - MARIA PETROU
- Department of Electrical and Electronic Engineering, Imperial College London, Exhibition Road, London, SW7 2AZ, UK
| | - JIANHUA LU
- Department of Electrical Engineering, Tsinghua University, P. R. China
| |
Collapse
|
15
|
|