Part-Based Representations of Visual Shape and Implications for Visual Cognition

Image Collage on Arbitrary Shape via Shape-Aware Slicing and Optimization

Image collage is a very useful tool for visualizing an image collection. Most of the existing methods and commercial applications for generating image collages are designed on simple shapes, such as rectangular and circular layouts. This greatly limits the use of image collages in some artistic and creative settings. Although there are some methods that can generate irregularly-shaped image collages, they often suffer from severe image overlapping and excessive blank space. This prevents such methods from being effective information communication tools. In this article, we present a shape slicing algorithm and an optimization scheme that can create image collages of arbitrary shapes in an informative and visually pleasing manner given an input shape and an image collection. To overcome the challenge of irregular shapes, we propose a novel algorithm, called Shape-Aware Slicing, which partitions the input shape into cells based on medial axis and binary slicing tree. Shape-Aware Slicing,which is designed specifically for irregular shapes, takes human perception and shape structure into account to generate visually pleasing partitions. Then, the layout is optimized by analyzing input images with the goal of maximizing the total salient regions of the images. To evaluate our method, we conduct extensive experiments and compare our results against previous work. The evaluations show that our proposed algorithm can efficiently arrange image collections on irregular shapes and create visually superior results than prior work and existing commercial tools.

Curious Objects: How Visual Complexity Guides Attention and Engagement

Some things look more complex than others. For example, a crenulate and richly organized leaf may seem more complex than a plain stone. What is the nature of this experience-and why do we have it in the first place? Here, we explore how object complexity serves as an efficiently extracted visual signal that the object merits further exploration. We algorithmically generated a library of geometric shapes and determined their complexity by computing the cumulative surprisal of their internal skeletons-essentially quantifying the "amount of information" within each shape-and then used this approach to ask new questions about the perception of complexity. Experiments 1-3 asked what kind of mental process extracts visual complexity: a slow, deliberate, reflective process (as when we decide that an object is expensive or popular) or a fast, effortless, and automatic process (as when we see that an object is big or blue)? We placed simple and complex objects in visual search arrays and discovered that complex objects were easier to find among simple distractors than simple objects are among complex distractors-a classic search asymmetry indicating that complexity is prioritized in visual processing. Next, we explored the function of complexity: Why do we represent object complexity in the first place? Experiments 4-5 asked subjects to study serially presented objects in a self-paced manner (for a later memory test); subjects dwelled longer on complex objects than simple objects-even when object shape was completely task-irrelevant-suggesting a connection between visual complexity and exploratory engagement. Finally, Experiment 6 connected these implicit measures of complexity to explicit judgments. Collectively, these findings suggest that visual complexity is extracted efficiently and automatically, and even arouses a kind of "perceptual curiosity" about objects that encourages subsequent attentional engagement.

A Simple Pipeline for Coherent Grid Maps

Grid maps are spatial arrangements of simple tiles (often squares or hexagons), each of which represents a spatial element. They are an established, effective way to show complex data per spatial element, using visual encodings within each tile ranging from simple coloring to nested small-multiples visualizations. An effective grid map is coherent with the underlying geographic space: the tiles maintain the contiguity, neighborhoods and identifiability of the corresponding spatial elements, while the grid map as a whole maintains the global shape of the input. Of particular importance are salient local features of the global shape which need to be represented by tiles assigned to the appropriate spatial elements. State-of-the-art techniques can adequately deal only with simple cases, such as close-to-uniform spatial distributions or global shapes that have few characteristic features. We introduce a simple fully-automated 3-step pipeline for computing coherent grid maps. Each step is a well-studied problem: shape decomposition based on salient features, tile-based Mosaic Cartograms, and point-set matching. Our pipeline is a seamless composition of existing techniques for these problems and results in high-quality grid maps. We provide an implementation, demonstrate the efficacy of our approach on various complex datasets, and compare it to the state-of-the-art.

Parts beget parts: Bootstrapping hierarchical object representations through visual statistical learning

Previous research has shown that humans are able to acquire statistical regularities among shape parts that form various spatial configurations, via exposure to these configurations without any task or feedback. The present study extends this approach of visual statistical learning to examine whether prior knowledge of parts, acquired in a separate learning context, facilitates acquisition of multi-layer hierarchical representations of objects. After participants had learned to encode a shape-pair as a chunk into memory, they viewed cluttered scenes containing multiple shape chunks. One of the larger configurations was constructed by combining the learned shape-pair with an unfamiliar, complementary shape-pair. Although the complementary shape-pair had never been presented separately during learning, it was remembered better than other shape pairs that were parts of larger configurations. The greater perceived familiarity of the complementary shape-pair depended on the encoding strength of the previously learned shape-pair. This "parts-beget-parts" effect suggests that statistical learning, in combination with prior knowledge, can represent objects as a coherent whole and also as a spatial configuration of parts by bootstrapping multi-layer hierarchical structures.

Diagnostic parts are not exclusive in the search template for real-world object categories

Visual search can be aided by a search template: a preparatory representation of relevant target features. But which features are relevant in complex, real-world category search? Previous research suggests that this template must be flexible to account for variations in naturalistic stimulus properties such as size and occlusion, and that shapes of diagnostic parts of objects are a likely candidate. Here, in three experiments, we systematically evaluated the contribution of diagnostic object parts and whole object shape to the category-level search template. Our hypothesis was that features that better match the active search template will capture attention during search more strongly than partially-matching features. Results showed that while whole objects captured attention reliably and globally across the visual field, diagnostic parts failed to do so in all three experiments. This suggests that whole object shape is a necessary component of the category-level search template.

Shape Similarity Assessment Method for Coastline Generalization

Although shape similarity is one fundamental element in coastline generalization quality, its related research is still inadequate. Consistent with the hierarchical pattern of shape recognition, the Dual-side Bend Forest Shape Representation Model is presented by reorganizing the coastline into bilateral bend forests, which are made of continuous root-bends based on Constrained Delaunay Triangulation and Convex Hull. Subsequently, the shape contribution ratio of each level in the model is expressed by its area distribution in the model. Then, the shape similarity assessment is conducted on the model in a top–down layer by layer pattern. Contrast experiments are conducted among the presented method and the Length Ratio, Hausdorff Distance and Turning Function, showing the improvements of the presented method over the others, including (1) the hierarchical shape representation model can distinguish shape features of different layers on dual-side effectively, which is consistent with shape recognition, (2) its usability and stability among coastlines and scales, and (3) it is sensitive to changes in main shape features caused by coastline generalization.

Temporal properties of amodal completion: Influences of knowledge

We studied the influence of knowledge in the interpretation of partly occluded objects. In the past decades, amodal completion has often been studied by using abstract, meaningless outlines of rather stylistic, geometric shapes. It has been recognized that smooth continuation of partly occluded contours behind an occluding surface is a strong completion tendency. In the current study we contrast this structurally driven completion tendency with knowledge driven tendencies. We used a set of partly occluded well-known objects for which structure-based completions and knowledge-based completions resulted in either the same or different interpretations. We adopted the behavioural primed matching paradigm to measure differential priming effects due to these completion tendencies. Our results implied differential temporal properties for structure-based and knowledge-based effects during perception of partly occluded objects. Interestingly, knowledge has an influence as early as 150 ms after the onset of the prime.

Object Category Understanding via Eye Fixations on Freehand Sketches

The study of eye gaze fixations on photographic images is an active research area. In contrast, the image sub-category of freehand sketches has not received as much attention for such studies. In this paper, we analyze the results of a free-viewing gaze fixation study conducted on 3904 freehand sketches distributed across 160 object categories. Our analysis shows that fixation sequences exhibit marked consistency within a sketch, across sketches of a category and even across suitably grouped sets of categories. This multi-level consistency is remarkable given the variability in depiction and extreme image content sparsity that characterizes hand-drawn object sketches. In this paper, we show that the multi-level consistency in the fixation data can be exploited to 1) predict a test sketch's category given only its fixation sequence and 2) build a computational model which predicts part-labels underlying fixations on objects. We hope that our findings motivate the community to deem sketch-like representations worthy of gaze-based studies vis-a-vis photographic images.

Early Computation of Contour Curvature and Part Structure: Evidence from Holes

We used holes to study unilateral border ownership and in particular the information carried by the sign of the curvature along the contour (ie the difference between convex and concave regions). When people perceive a hole, its shape has a reversed curvature polarity (ie a changed sign of curvature) compared to the same region perceived as an object. Bertamini (2001 Perception30 1295–1310), and Bertamini and Croucher (2003 Cognition87 33–54) suggested and found evidence to support the hypothesis that, because convex regions are perceived as parts, positional information is more readily available for convex regions. Therefore a change is predicted when a given region is perceived as either a hole or a figure. We confirm that finding in this study, using holes defined by binocular disparity. We conclude that a change from figure to hole always reverses the encoding of curvature polarity. In turn, polarity obligatorily affects perceived part structure and the processing of position.

The Role of Part Structure in the Perceptual Localization of a Shape

The process of object localization may be accomplished with respect to a particular reference location, such as the center of gravity, COG (eg Vishwanath and Kowler, 2003 Vision Research43 1637 – 1653). Here, we investigated how part structure affects an object's reference location. The reference location was evaluated with a measure of the illusory displacement of an internal target element embedded within a larger object (Morgan et al, 1990 Vision Research30 1793 – 1810). To examine whether the reference location is different for shapes with part structure, two shapes were tested: circle (small and large; no part structure) and bell (shape with two parts, one larger than the other). Results were examined with respect to two predictions: either the location of an object is based on its shape as a whole, disregarding part structure (ie a single, overall COG), or the parts are processed separately (different COGs).

With the circles, the results showed a systematic illusory displacement of the internal target toward the COG. With the bell, the illusion was significantly weaker than with both circles—even though the main part of the bell had the same size as the small circle, and its horizontal axis had the same extent as the large circle. Moreover, the distance judgments for the bell were consistent with a (weaker) reference point being located at the COG of the larger part, rather than at the COG of the entire bell. These results show that the part structure of a shape plays a role in the representation of its location, and that for complex shapes the perceived location of an embedded element depends more on the parts within which it is embedded, rather than on the whole shape.

A computational model of the short-cut rule for 2D shape decomposition

We propose a new 2D shape decomposition method based on the short-cut rule. The short-cut rule originates from cognition research, and states that the human visual system prefers to partition an object into parts using the shortest possible cuts. We propose and implement a computational model for the short-cut rule and apply it to the problem of shape decomposition. The model we proposed generates a set of cut hypotheses passing through the points on the silhouette, which represent the negative minima of curvature. We then show that most part-cut hypotheses can be eliminated by analysis of local properties of each. Finally, the remaining hypotheses are evaluated in ascending length order, which guarantees that of any pair of conflicting cuts only the shortest will be accepted. We demonstrate that, compared with state-of-the-art shape decomposition methods, the proposed approach achieves decomposition results, which better correspond to human intuition as revealed in psychological experiments.

Minimum near-convex shape decomposition

Shape decomposition is a fundamental problem for part-based shape representation. We propose the minimum near-convex decomposition (MNCD) to decompose arbitrary shapes into minimum number of "near-convex" parts. The near-convex shape decomposition is formulated as a discrete optimization problem by minimizing the number of nonintersecting cuts. Two perception rules are imposed as constraints into our objective function to improve the visual naturalness of the decomposition. With the degree of near-convexity a user-specified parameter, our decomposition is robust to local distortions and shape deformation. The optimization can be efficiently solved via binary integer linear programming. Both theoretical analysis and experiment results show that our approach outperforms the state-of-the-art results without introducing redundant parts and thus leads to robust shape representation.

Processing convexity and concavity along a 2-D contour: figure-ground, structural shape, and attention

Interest in convexity has a long history in vision science. For smooth contours in an image, it is possible to code regions of positive (convex) and negative (concave) curvature, and this provides useful information about solid shape. We review a large body of evidence on the role of this information in perception of shape and in attention. This includes evidence from behavioral, neurophysiological, imaging, and developmental studies. A review is necessary to analyze the evidence on how convexity affects (1) separation between figure and ground, (2) part structure, and (3) attention allocation. Despite some broad agreement on the importance of convexity in these areas, there is a lack of consensus on the interpretation of specific claims--for example, on the contribution of convexity to metric depth and on the automatic directing of attention to convexities or to concavities. The focus is on convexity and concavity along a 2-D contour, not convexity and concavity in 3-D, but the important link between the two is discussed. We conclude that there is good evidence for the role of convexity information in figure-ground organization and in parsing, but other, more specific claims are not (yet) well supported.

Part perception in infancy: sensitivity to the short-cut rule

Research indicates that object perception involves the decomposition of images into parts. A critical principle that governs part decomposition by adults is the short-cut rule, which states that, all else being equal, the visual system parses objects using the shortest possible cuts. We examined whether 6.5-month-olds' parsing of images also follows the short-cut rule. Infants in the experimental conditions were habituated to cross shapes and then tested for their preference between segregated patterns produced using long cuts versus short cuts. Infants in the control conditions were directly tested with the segregated patterns. Infants in the experimental conditions exhibited a greater novelty preference for the long-cut over the short-cut patterns than did those in the control conditions, thereby indicating that they are more likely to segregate cross shapes using short cuts rather than long cuts. This sensitivity to the short-cut rule was evident when two alternative parameters, part area and protrusion, were controlled in Experiments 1 and 2, respectively. Thus, a critical principle that governs part segregation in adulthood is operational by 6.5 months of age.

Identification of everyday objects on the basis of fragmented outline versions

Although Attneave (1954 Psychological Review 61 183 193) and Biederman (1987 Psychological Review 94 115-147) have argued that curved contour segments are most important in shape perception, Kennedy and Domander (1985 Perception 14 367-370) showed that fragmented object contours are better identifiable when straight segments are shown. We used the set of line drawings published by Snodgrass and Vanderwart (1980 Journal of Experimental Psychology: Human Learning and Memory 6 174-215), to make outline versions that could be used to investigate this issue with a larger and more heterogeneous stimulus set. Fragments were placed either around the 'salient' points or around the midpoints (points midway between two salient points), creating curved versus relatively straight fragments when the original outline was fragmented (experiment 1), or angular and straight fragments when straight-line versions were fragmented (experiment 2). We manipulated fragment length in each experiment except the last one, in which we presented only selected points (experiment 3). While fragmented versions were on average more identifiable when straight fragments were shown, certain objects were more identifiable when the curved segments or the angles were shown. A tentative explanation of these results is presented in terms of an advantage for straight segments during grouping processes for outlines with high part salience, and an advantage for curved segments during matching processes for outlines with low part salience.

Who owns the contour of a visual hole?

Holes are useful in the study of shape, contour curvature, and border ownership. Several authors have suggested that holes have figural or quasi-figural status. I discuss three criteria to test the evidence that holes behave more like figures than like ground: (i) holes perceived as such; (ii) similar performance for holes and figures; (iii) different performance for holes and other ground regions. Using these criteria, I review the literature and conclude that holes do not have figural status in relation to border ownership. I also argue that holes are ideal stimuli to study figure-ground organisation.

Bayesian estimation of the shape skeleton

Skeletal representations of shape have attracted enormous interest ever since their introduction by Blum [Blum H (1973) J Theor Biol 38:205-287], because of their potential to provide a compact, but meaningful, shape representation, suitable for both neural modeling and computational applications. But effective computation of the shape skeleton remains a notorious unsolved problem; existing approaches are extremely sensitive to noise and give counterintuitive results with simple shapes. In conventional approaches, the skeleton is defined by a geometric construction and computed by a deterministic procedure. We introduce a Bayesian probabilistic approach, in which a shape is assumed to have "grown" from a skeleton by a stochastic generative process. Bayesian estimation is used to identify the skeleton most likely to have produced the shape, i.e., that best "explains" it, called the maximum a posteriori skeleton. Even with natural shapes with substantial contour noise, this approach provides a robust skeletal representation whose branches correspond to the natural parts of the shape.

Visual search for a circular region perceived as a figure versus as a hole: Evidence of the importance of part structure

To study contour curvature polarity, we compared strictly convex regions (circular figures) with strictly concave regions (circular holes). We tested for an asymmetry between visual searches for concavities and those for convexities. We found that providing a preview of the background benefited search for concavities (holes) more than it did search for convexities (figures) and that for convex figures, nearer targets were responded to more quickly. Importantly, however, we failed to find any support for the hypothesis that concave targets are inherently more salient. We conclude that previous findings in the literature, which have been taken to indicate preferential processing of concavities, due to their increased salience, are more likely the result of an early computation of part structure based on concavities.

Segmentation of object outlines into parts: a large-scale integrative study

In this study, a large number of observers (N=201) were asked to segment a collection of outlines derived from line drawings of everyday objects (N=88). This data set was then used as a benchmark to evaluate current models of object segmentation. All of the previously proposed rules of segmentation were found supported in our results. For example, minima of curvature (i.e. locations along the contour where negative curvature takes an extreme value) were often used as segmentation points. The second point of a pair connected by a segmentation line often depended on more global shape characteristics such as proximity, collinearity, symmetry, and elongation. Based on these results, a framework is presented in which all of the previously proposed (and now empirically validated) segmentation rules or rules for part formation are integrated.

Detection of change in shape: an advantage for concavities

Shape representation was studied using a change detection task. Observers viewed two individual shapes in succession, either identical or one a slightly altered version of the other, and reported whether they detected a change. We found a dramatic advantage for concave compared to convex changes of equal magnitude. Observers were more accurate when a concavity along the contour was introduced, or removed, compared to a convexity. This result sheds light on the underlying representation of visual shape, and in particular the central role played by part-boundaries. Moreover, this finding shows how change detection methodology can serve as a useful tool in studying the specific form of visual representations.

Early computation of part structure: evidence from visual search

The visual system represents object shapes in terms of intermediate-level parts. The minima rule proposes that the visual system uses negative minima of curvature to define boundaries between parts. We used visual search to test whether part structures consistent with the minima rule are computed preattentively--or at least, rapidly and early in visual processing. The results of Experiments 1 and 2 showed that whereas the search for a non-minima-segmented shape is fast and efficient among minima-segmented shapes, the reverse search is slow and inefficient. This asymmetry is expected if parsing at negative minima occurs obligatorily. The results of Experiments 3 and 4 showed that although both minima- and non-minima-segmented shapes pop out among unsegmented shapes, the search for minima-segmented shapes is significantly slower. Together, these results demonstrate that the visual system segments shapes into parts, using negative minima of curvature, and that it does so rapidly in early stages of visual processing.