Contour curvature polarity and surface interpolation

Trailblazers in Neuroscience: Using compositionality to understand how parts combine in whole objects

A fundamental question for any visual system is whether its image representation can be understood in terms of its components. Decomposing any image into components is challenging because there are many possible decompositions with no common dictionary, and enumerating them leads to a combinatorial explosion. Even in perception, many objects are readily seen as containing parts, but there are many exceptions. These exceptions include objects that are not perceived as containing parts, properties like symmetry that cannot be localized to any single part, and also special categories like words and faces whose perception is widely believed to be holistic. Here, I describe a novel approach we have used to address these issues and evaluate compositionality at the behavioral and neural levels. The key design principle is to create a large number of objects by combining a small number of pre-defined components in all possible ways. This allows for building component-based models that explain whole objects using a combination of these components. Importantly, any systematic error in model fits can be used to detect the presence of emergent or holistic properties. Using this approach, we have found that whole object representations are surprisingly predictable from their components, that some components are preferred to others in perception, and that emergent properties can be discovered or explained using compositional models. Thus, compositionality is a powerful approach for understanding how whole objects relate to their parts.

Mid-level Priming by Completion vs. Mosaic Solutions

We report two experiments on the role of mid-level processes in image segmentation and completion. In the primed matching task of Experiment 1, a cue→prime sequence was presented before the imperative stimulus consisting of target shapes with positive versus negative contour curvature polarity and one versus two axes of mirror symmetry. Priming shapes were included in two composite occlusion displays with the same T-junction information and different geometric features supporting a distinct balance between completion and mosaic solutions. A cue, either congruent or incongruent with targets, preceded the presentation of the composite priming display. Matching performance was affected by primes in the expected direction, while cue congruency participated only in a marginally significant three-way interaction, and prime duration had no effect. In Experiment 2, the cue→prime sequence was replaced by a fixation cross to control for the priming effect obtained in Experiment 1. The study confirmed that contour connectability and curvature polarity are effective structural factors capable of competing with symmetry in mid-level image segmentation and completion processes.

Perceptual mechanisms underlying amodal surface integration of 3-D stereoscopic stimuli

The visual system can represent a partially occluded 3-D surface from images of separated surface segments. The underlying amodal surface integration process accomplishes this by amodally extending each surface segment behind the occluder (amodal surface extension) and integrating the extended surfaces to form a whole surface representation. We conducted five experiments to investigate how depth cues, such as binocular disparity, half-occlusion, and monocular depth cues (T-junctions and L-junctions), contribute to amodal surface extension, and how the geometrical relationship and image similarity among the surface segments affect surface integration. This was achieved by having observers adjust the stereoscopic depth and slant of a comparison stimulus to match those of the tested 3-D stimulus. We found that both binocular disparity and half-occlusion cues are used to determine border-ownership assignment of surface segments and for amodal surface extension. We also found that separated surface segments need to have the same luminance contrast-polarity for them to be integrated as a whole surface. Finally, we found that having the same motion direction, minimum misalignment between boundary contours, and proximity among separated segments facilitate their integration. Overall, our findings reveal a set of "perceptual factors" for amodal surface integration, which arguably reflects our visual system's built-in knowledge of the regularities in natural scenes.

Investigating shape representation using sensitivity to part- and axis-based transformations

Part- and axis-based approaches organize shape representations in terms of simple parts and their spatial relationships. Shape transformations that alter qualitative part structure have been shown to be more detectable than those that preserve it. We compared sensitivity to various transformations that change quantitative properties of parts and their spatial relationships, while preserving qualitative part structure. Shape transformations involving changes in length, width, curvature, orientation and location were applied to a small part attached to a larger base of a two-part shape. Increment thresholds were estimated for each transformation using a 2IFC procedure. Thresholds were converted into common units of shape difference to enable comparisons across transformations. Higher sensitivity was consistently found for transformations involving a parameter of a single part (length, width, curvature) than those involving spatial relations between two parts (relative orientation and location), suggesting a single-part superiority effect. Moreover, sensitivity to shifts in part location - a biomechanically implausible shape transformation - was consistently poorest. The influence of region-based geometry was investigated via stereoscopic manipulation of figure and ground. Sensitivity was compared across positive parts (protrusions) and negative parts (indentations) for transformations involving a change in orientation or location. For changes in part orientation (biomechanically plausible), sensitivity was better for positive than negative parts; whereas for changes in part location (biomechanically implausible), no systematic difference was observed.

Created unequal: Temporal dynamics of modal and amodal boundary interpolation

In this study we manipulate the distribution of contrast polarity reversals in inducing configurations to create novel variants of modal and amodal completion. The novel variants, better equated in their geometric and photometric characteristics offer a superior way to probe similarities and differences in the temporal dynamics that underlie different forms of perceptual completion. We use dot localisation to directly compare the spatial characteristics of modally and amodally interpolated contours at presentation durations ranging from 120 to 300ms and find robust differences in the spatiotemporal formation of modally and amodally completed boundaries. Modally completed contours are localised more accurately and with better spatial precision across all presentation durations. Our results challenge the assumption that the boundary interpolation system depends solely on the geometrical relatability of inducing fragments and suggest that boundary interpolation depends on the spatial distribution of local luminance relationships. As an alternative to the strong version of the identity hypothesis, we propose that modal and amodal completion are mediated by different mechanisms, triggered by particular configurations of contrast polarity.

Processing convexity and concavity along a 2-D contour: figure-ground, structural shape, and attention

Interest in convexity has a long history in vision science. For smooth contours in an image, it is possible to code regions of positive (convex) and negative (concave) curvature, and this provides useful information about solid shape. We review a large body of evidence on the role of this information in perception of shape and in attention. This includes evidence from behavioral, neurophysiological, imaging, and developmental studies. A review is necessary to analyze the evidence on how convexity affects (1) separation between figure and ground, (2) part structure, and (3) attention allocation. Despite some broad agreement on the importance of convexity in these areas, there is a lack of consensus on the interpretation of specific claims--for example, on the contribution of convexity to metric depth and on the automatic directing of attention to convexities or to concavities. The focus is on convexity and concavity along a 2-D contour, not convexity and concavity in 3-D, but the important link between the two is discussed. We conclude that there is good evidence for the role of convexity information in figure-ground organization and in parsing, but other, more specific claims are not (yet) well supported.

The "side" matters: how configurality is reflected in completion

The perception of figure-ground organization is a highly context-sensitive phenomenon. Accumulating evidence suggests that the so-called completion phenomenon is tightly linked to this figure-ground organization. While many computational models have applied borderline completion algorithms based on the detection of boundary alignments, we point out the problems of this approach. We hypothesize that completion is a result of computing the figure-ground organization. Specifically, the global interactions in the neural network activate the "border-ownership" sensitive neurons at the location where no luminance contrast is given and this activation corresponds to the perception of illusory contours. The implications of this result to the general property of emerging Gestalt percepts are discussed.

A century of Gestalt psychology in visual perception: I. Perceptual grouping and figure-ground organization

In 1912, Max Wertheimer published his paper on phi motion, widely recognized as the start of Gestalt psychology. Because of its continued relevance in modern psychology, this centennial anniversary is an excellent opportunity to take stock of what Gestalt psychology has offered and how it has changed since its inception. We first introduce the key findings and ideas in the Berlin school of Gestalt psychology, and then briefly sketch its development, rise, and fall. Next, we discuss its empirical and conceptual problems, and indicate how they are addressed in contemporary research on perceptual grouping and figure-ground organization. In particular, we review the principles of grouping, both classical (e.g., proximity, similarity, common fate, good continuation, closure, symmetry, parallelism) and new (e.g., synchrony, common region, element and uniform connectedness), and their role in contour integration and completion. We then review classic and new image-based principles of figure-ground organization, how it is influenced by past experience and attention, and how it relates to shape and depth perception. After an integrated review of the neural mechanisms involved in contour grouping, border ownership, and figure-ground perception, we conclude by evaluating what modern vision science has offered compared to traditional Gestalt psychology, whether we can speak of a Gestalt revival, and where the remaining limitations and challenges lie. A better integration of this research tradition with the rest of vision science requires further progress regarding the conceptual and theoretical foundations of the Gestalt approach, which is the focus of a second review article.

Volume completion in 4.5-month-old infants

In this study, we examined 4.5-month-old infants' visual completion of self-occluding three-dimensional objects. A previous study on this topic reported that 6-month-old, but not 4-month-old infants extrapolate a convex, symmetric prism from a limited view of its surfaces (Soska & Johnson, 2008). As of yet, studies on the development of amodal completion of three-dimensional, self-occluding objects are scarce. Given 4-month-old infants' abilities to derive three-dimensional shape from a variety of visual cues, three-dimensional amodal completion may well depend on the perceptual strength of three-dimensionality in the stimulus displays. The first experiments (1A and 1B) tested this hypothesis by means of a habituation paradigm and showed that 4.5-month-old infants are indeed able to amodally complete the back of a self-occluding object when sufficient three-dimensional cues are available. Further support for volume completion in 4.5-month-old infants was found in a second experiment, again using a habituation paradigm, that measured perceived connectedness between two visually separated, self-occluding, three-dimensional objects.

Animal Recognition and Eye Movements

We assessed the importance of outline contour and individual features in mediating the recognition of animals by examining response times and eye movements in an animal-object decision task (i.e., deciding whether or not an object was an animal that may be encountered in real life). There were shorter latencies for animals as compared with nonanimals and performance was similar for shaded line drawings and silhouettes, suggesting that important information for recognition lies in the outline contour. The most salient information in the outline contour was around the head, followed by the lower torso and leg regions. We also observed effects of object orientation and argue that the usefulness of the head and lower torso/leg regions is consistent with a role for the object axis in recognition.

Surface completion affected by luminance contrast polarity and common motion

Our visual system ably integrates the visible parts of a partially occluded surface with the occluded parts (amodal surface completion), mainly by relying on the surface boundary contours of the image. Less known, is whether the visual system also utilizes surface feature information, such as luminance contrast polarity, for surface completion. We conducted three experiments to investigate this issue. Experiment 1 found that when visible segments of a partially occluded rectangle with the same luminance contrast polarity move behind an occluding surface, observers perceive the visible segments as part of the occluded rectangle moving cohesively behind the occluding surface. However, when the visible segments have opposite luminance contrast polarity, the global motion of the segments is barely perceived, suggesting a failure of amodal surface integration. Experiment 2 revealed that this same luminance contrast polarity constraint applies to amodal surface integration of a display without an explicit occluding surface image. Experiment 3 showed that both the shape and luminance contrast polarity of the visible segments of the partially occluded rectangle affect amodal surface completion. Together, these findings demonstrate that luminance contrast polarity, along with surface boundary contour, are important cues for amodal surface integration.

Approximation, torsion, and amodally-completed surfaces

Consider a stereoscopic display simulating two rectangular patches, the lower frontoparallel and the upper slanted around the vertical axis. When the two patches are amodally completed and appear as the unoccluded parts of a smooth surface partially hidden by a foreground frontoparallel surface, either real or illusory, their relative slant is underestimated with respect to a baseline condition in which they are perceived as separate rectangles. Slant assimilation was studied in three experiments using with- vs. without-occluder displays and two methods, slant matching and speeded classification of twist direction. In Experiments 1 and 2 we found slant assimilation in with-occluder displays and slant contrast in without-occluder displays. In Experiment 3 we isolated a component of slant assimilation attributable to the mere presence of the occluder. Twist classification performance was impaired even when edge geometry hindered amodal completion, but the performance loss was larger when surface patches were amodally completed. To minimize the required amount of torsion, input fragments are misperceived, indicating that in limiting conditions amodal completion is mediated by approximation rather than interpolation. Slant assimilation decreases as twist angle increases, up to a limit above which the visual system does not support the formation of a smooth amodal surface with torsion.

Rapid Figure – Ground Responses to Stereograms Reveal an Advantage for a Convex Foreground

Convexity has long been recognised as a factor that affects figure – ground segmentation, even when pitted against other factors such as symmetry [Kanizsa and Gerbino, 1976 Art and Artefacts Ed.M Henle (New York: Springer) pp 25–32], It is accepted in the literature that the difference between concave and convex contours is important for the visual system, and that there is a prior expectation favouring convexities as figure. We used bipartite stimuli and a simple task in which observers had to report whether the foreground was on the left or the right. We report objective evidence that supports the idea that convexity affects figure – ground assignment, even though our stimuli were not pictorial in that depth order was specified unambiguously by binocular disparity.

Who owns the contour of a visual hole?

Holes are useful in the study of shape, contour curvature, and border ownership. Several authors have suggested that holes have figural or quasi-figural status. I discuss three criteria to test the evidence that holes behave more like figures than like ground: (i) holes perceived as such; (ii) similar performance for holes and figures; (iii) different performance for holes and other ground regions. Using these criteria, I review the literature and conclude that holes do not have figural status in relation to border ownership. I also argue that holes are ideal stimuli to study figure-ground organisation.

Amodal completion and visual holes (static and moving)

Occlusion is a frequent occurrence in a cluttered world of opaque objects. Often information about the shape of partly occluded objects can be gathered from the visible portion of the object and in particular its contours. Here we address the case where a region of a surface is visible exclusively through an aperture (visual hole). We make several observations about the grouping of surface regions visible through holes, and the appearance of moving objects and holes. These observations support the view that holes are shape properties of the object-with-hole.

Surface geometry influences the shape of illusory contours

Geometric and neural models of illusory-contour (IC) synthesis currently use only local contour geometry to derive the shape of ICs. Work on the visual representation of shape, by contrast, points to the importance of both contour and surface geometry. We investigated the influence of surface-based geometric factors on IC shape. The local geometry of inducing-contour pairs was equated in stereoscopic IC displays, and the shape of the enclosed surface was varied by manipulating sign of curvature, cross-axial shape width, and medial-axis geometry. IC shapes were measured using a parametric shape-adjustment task (Experiment 1) and a dot-adjustment task (Experiment 2). Both methods revealed large influences of surface geometry. ICs enclosing locally concave regions were perceived to be systematically more angular than those enclosing locally convex regions. Importantly, the influence of sign of curvature was modulated significantly by shape width and medial-axis geometry: IC shape difference between convex and concave inducers was greater for narrow shapes than wider ones, and greater for shapes with straight axis and symmetric contours (diamond versus bowtie), than those with curved axis and parallel contours (bent tubes). Even at the level of illusory "contours," there is a contribution of region-based geometry which is sensitive to nonlocal shape properties involving medial geometry and part decomposition. Models of IC synthesis must incorporate the role of nonlocal region-based geometric factors in a way that parallels their role in organizing visual shape representation more generally.

Visual search for a circular region perceived as a figure versus as a hole: Evidence of the importance of part structure

To study contour curvature polarity, we compared strictly convex regions (circular figures) with strictly concave regions (circular holes). We tested for an asymmetry between visual searches for concavities and those for convexities. We found that providing a preview of the background benefited search for concavities (holes) more than it did search for convexities (figures) and that for convex figures, nearer targets were responded to more quickly. Importantly, however, we failed to find any support for the hypothesis that concave targets are inherently more salient. We conclude that previous findings in the literature, which have been taken to indicate preferential processing of concavities, due to their increased salience, are more likely the result of an early computation of part structure based on concavities.

Role of a circle’s center in visual interpolation

Basic geometric patterns like straight lines and circles seem fundamental to human perception and mental imagery. In this study we examined subjects' ability to interpolate circular curves-to derive the whole circle from an arc of 180 degrees or less. Specifically, we tested how the center point is utilized during such visual interpolation. Naturally, a mechanism that interpolates by extending the curvature of the visible arc will be unaffected by the presence or absence of the center point. On the other hand, a mechanism that achieves the same end by completing the circle from estimates of the center and radius will be significantly aided by the presence of the center. We found that when the visible arc was long (180 degrees), presenting the circle's center did not affect the precision with which subjects localized the invisible section. However, when the visible arc was relatively short (90 degrees or 45 degrees), displaying the center point significantly increased spatial precision. Thus, both computational mechanisms appear to exist.

Visual interpolation is not scale invariant

According to the scale-dependence hypothesis, the visual interpolation of contour fragments depends on the retinal separation of endpoints: as the retinal size of a partially occluded angle increases, the interpolated contour gradually deviates from the shortest connecting path and approaches the shape of the unoccluded angle. In the field model, as the retinal size increases the strength of good continuation increases while the strength of the minimal-path tendency decreases. To test the scale-dependence hypothesis--as well as other hypotheses connected to inclusion, support-ratio dependence, and extended relatability--we ran two experiments using the probe localization technique. Stimuli were regular polygons with rectilinear contours bounding symmetrically occluded angles. Retinal size was manipulated by changing viewing distance. Observers were asked to judge if a probe, briefly superposed on the occlusion region, was inside or outside the amodally completed angle. Retinal size strongly influenced the penetration of interpolated trajectories in the predicted direction. However, support ratio and interpolated angle size interacted with retinal size, consistently with the idea that unification factors are effective within a spatial window. We modified the field model to include the size of such a window as a new parameter and generated model-based trajectories that fitted empirical data closely.

Detection of change in shape and its relation to part structure

Using a change detection paradigm (Barenholtz, E., Cohen, E. H., Feldman, J., & Singh, M. (2003). Detection of change in shape: An advantage for concavities. Cognition, 89(1) 1-9), we measured sensitivity to the changes of shapes and in particular the difference between detecting a new convex or concave vertex. We conclude that concave vertices per se are not more salient, but changes in the sequence of convexities and concavities along a contour are salient. We argue that these changes are likely to signal a change in perceived part structure.