Spatio-temporal priority revisited: the role of feature identity and similarity for object correspondence in apparent motion

The role of color in transsaccadic object correspondence

With each saccade, visual information is disrupted, and the visual system is tasked with establishing object correspondence between the presaccadic and postsaccadic representations of the saccade target. There is substantial evidence that the visual system consults spatiotemporal continuity when determining object correspondence across saccades. The evidence for surface feature continuity, however, is mixed. Surface features that are integral to the saccade target object's identity (e.g., shape and contrast polarity) are informative of object continuity, but features that may only imply the state of the object (e.g., orientation) are ignored. The present study tested whether color information is consulted to determine transsaccadic object continuity. We used two variations of the intrasaccadic target displacement task. In Experiments 1 and 2, participants reported the direction of the target displacement. In Experiments 3 and 4, they instead reported whether they detected any target movement. In all experiments, we manipulated the saccade target's continuity by removing it briefly (i.e., blanking) and by changing its color. We found that large color changes can disrupt stability and increase sensitivity to displacements for both direction and movement reports, although not as strongly as long blank durations (250 ms). Interestingly, even smaller color changes, but not blanking, reduced response biases. These results indicate that disrupting surface feature continuity may impact the process of transsaccadic object correspondence more strongly than spatiotemporal disruptions by both increasing the sensitivity and decreasing the response bias.

Self-reference modulates the perception of visual apparent motion

Visual apparent motion is a perceptual illusion where sequentially presented static stimuli containing no physically continuous motion are perceived as moving. In the current study, we examined whether and how self-reference, as a typical high-level information processing, could modulate perceptual categorization of the apparent motion in Ternus display, even when self-reference is task-irrelevant. Two frames were consecutively presented, with the first frame consisting of two identical stimuli (e.g., two rectangles) on the leftmost and the middle positions and the second frame consisting of two stimuli on the middle and the rightmost positions. Depending on the inter-stimulus interval (ISI) between the two frames, the display could be perceived as showing Element Motion (EM), with the peripheral stimulus moving from one side to the other while the middle stimulus remains stationary or flashes briefly at the middle position, or Group Motion (GM), with both stimuli appearing to move as a whole. Participants were tested in this configuration and then learned to associate different labels (Self, Friend, Stranger) with geometric shapes (Circle, Rectangle, Triangle). They were tested again in the new configuration. Results showed that after association (vs. before association), participants were more likely to perceive the Ternus display of self-associated shapes as GM, but this effect did not appear for friend-associated or stranger-associated shapes. Self-referential processing spatially "glues" the two stimuli in a frame with the concept of "Self," leading to a more dominant percept of GM.

Does the construction retrieval account apply to cross‐modal inhibition of return in semantic context?

The traditional attentional reorienting hypothesis is insufficient to explain spatial and nonspatial inhibition of return (IOR). Therefore, a construction retrieval account that includes the influence of top-down attentional sets has been proposed and can explain both spatial and nonspatial IOR. However, it remains unknown whether the construction retrieval account can be applied to non-surface features of stimuli, as well as whether its construction and retrieval mechanisms are supra-modal. The present study manipulated semantic feature congruency and spatial location congruency between the prime and the target in cross-modal audio-visual and visual-audio experimental conditions, respectively, by orthogonally combining spatial and nonspatial IOR paradigms. Our results showed that there was an interaction between semantic feature congruency and spatial location congruency controlled by the attentional sets, and that this interaction was consistent in cross-modal audio-visual and visual-audio conditions. These results suggest that the construction retrieval account can be applied to abstract semantic features and that its construction and retrieval mechanisms are supra-modal. The present study extends the application scope of the construction retrieval account and promotes the interpretation of IOR under a unified theoretical framework.

Exploring the effectiveness of auditory, visual, and audio-visual sensory cues in a multiple object tracking environment

Maintaining object correspondence among multiple moving objects is an essential task of the perceptual system in many everyday life activities. A substantial body of research has confirmed that observers are able to track multiple target objects amongst identical distractors based only on their spatiotemporal information. However, naturalistic tasks typically involve the integration of information from more than one modality, and there is limited research investigating whether auditory and audio-visual cues improve tracking. In two experiments, we asked participants to track either five target objects or three versus five target objects amongst similarly indistinguishable distractor objects for 14 s. During the tracking interval, the target objects bounced occasionally against the boundary of a centralised orange circle. A visual cue, an auditory cue, neither or both coincided with these collisions. Following the motion interval, the participants were asked to indicate all target objects. Across both experiments and both set sizes, our results indicated that visual and auditory cues increased tracking accuracy although visual cues were more effective than auditory cues. Audio-visual cues, however, did not increase tracking performance beyond the level of purely visual cues for both high and low load conditions. We discuss the theoretical implications of our findings for multiple object tracking as well as for the principles of multisensory integration.

Perception of Happening: How the Brain Deals with the No-History Problem

In physics, the temporal dimension has units of infinitesimally brief duration. Given this, how is it possible to perceive things, such as motion, music, and vibrotactile stimulation, that involve extension across many units of time? To address this problem, it is proposed that there is what is termed an "information construct of happening" (ICOH), a simultaneous representation of recent, temporally differentiated perceptual information on the millisecond time scale. The main features of the ICOH are (i) time marking, semantic labeling of all information in the ICOH with ordinal temporal information and distance from what is informationally identified as the present moment, (ii) vector informational features that specify kind, direction, and rate of change for every feature in a percept, and (iii) connectives, information relating vector informational features at adjacent temporal locations in the ICOH. The ICOH integrates products of perceptual processing with recent historical information in sensory memory on the subsecond time scale. Perceptual information about happening in informational sensory memory is encoded in semantic form that preserves connected semantic trails of vector and timing information. The basic properties of the ICOH must be supported by a general and widespread timing mechanism that generates ordinal and interval timing information and it is suggested that state-dependent networks may suffice for that purpose. Happening, therefore, is perceived at a moment and is constituted by an information structure of connected recent historical information.

Features integrate along a motion trajectory when object integrity is preserved

Information about a moving object is usually poor at each retinotopic location because photoreceptor activation is short, noisy, and affected by shadows, reflections of other objects, and so on. Integration across the motion trajectory may yield a much better estimate about the objects’ features. Using the sequential metacontrast paradigm, we have shown previously that features, indeed, integrate along a motion trajectory in a long-lasting window of unconscious processing. In the sequential metacontrast paradigm, a percept of two diverging streams is elicited by the presentation of a central line followed by a sequence of flanking pairs of lines. When several lines are spatially offset, the offsets integrate mandatorily for several hundreds of milliseconds along the motion trajectory of the streams. We propose that, within these long-lasting windows, stimuli are first grouped based on Gestalt principles of grouping. These processes establish reference frames that are used to attribute features. Features are then integrated following their respective reference frame. Here using occlusion and bouncing effects, we show that indeed such grouping operations are in place. We found that features integrate only when the spatiotemporal integrity of the object is preserved. Moreover, when several moving objects are present, only features belonging to the same object integrate. Overall, our results show that feature integration is a deliberate strategy of the brain and long-lasting windows of processing can be seen as periods of sense making.

Illusory size determines the perception of ambiguous apparent motion

The visual system constructs perceptions based on ambiguous information. For motion perception, the correspondence problem arises, i.e., the question of which object went where. We asked at which level of processing correspondence is solved – lower levels based on information that is directly available in the retinal input or higher levels based on information that has been abstracted beyond the input directly available at the retina? We used a Ponzo-like illusion to manipulate the perceived size and separations of elements in an ambiguous apparent motion display. Specifically, we presented Ternus displays – for which the type of motion that is perceived depends on how correspondence is resolved – at apparently different distances from the viewer using pictorial depth cues. We found that the perception of motion depended on the apparent depth of the displays, indicating that correspondence processes utilize information that is produced at higher-level processes.

How Do Humans Perform in Multiple Object Tracking With Unstable Features

In real-world scenarios, objects’ surface features sometimes change as they move, impairing the continuity of objects. However, it is still unknown how our visual system adapts to this dynamic change. Hence, the present study investigated the role of feature changes in attentive tracking through a modified multiple object tracking (MOT) task. The feature heterogeneity and feature stability were manipulated in two experiments. The results from Experiment 1 showed that the tracking performance under feature-changed condition was lower than that under the feature-fixed condition only when the objects were four colors grouped or all unique, suggesting that the performance decrease was moderated by the feature heterogeneity. In Experiment 2, we further examined this effect by manipulating the frequency of feature change. The results showed that when the target set was one color or two colors grouped (the color grouping for the distractor set corresponded with it), the tracking performance decreased significantly as the feature-change frequency increased. However, this trend was not the case when the objects were of the same color or eight unique colors. In addition, a relatively consistent effect appeared both in Experiments 1 and 2. When objects have unique features, the tracking performance decreased significantly as the increase of feature heterogeneity in each frequency of feature changes. Taken together, we concluded that unstable features could be utilized in attentive tracking, and the extent to which the observers relied on surface feature information to assist tracking depended on the level of feature heterogeneity and the frequency of feature change.

Effects of Transient and Nontransient Changes of Surface Feature on Object Correspondence

Object correspondence is a fundamental problem in perception. Classic theories hold that the computation of correspondence is solely based on spatiotemporal information. Recent research showed that surface features also play an important role. However, the surface features of objects in many studies did not change throughout a trial. This study investigated the effect of feature change on object correspondence using the object-reviewing paradigm. Two moving objects underwent transient feature changes on color dimension (Experiment 1A) or a combination of three dimensions (Experiment 2A). Moreover, the objects moved behind four occluders to make the feature change nontransient (Experiments 1B and 2B). Object-specific preview benefits were reduced or eliminated when feature changes were transient, but the benefits were not affected when feature changes were nontransient. The effects of transient versus nontransient changes of surface feature in object correspondence are discussed.

How voluntary spatial attention influences feature biases in object correspondence

Our visual system is able to establish associations between corresponding images across space and time and to maintain the identity of objects, even though the information our retina receives is ambiguous. It has been shown that lower level factors-as, for example, spatiotemporal proximity-can affect this correspondence problem. In addition, higher level factors-as, for example, semantic knowledge-can influence correspondence, suggesting that correspondence might also be solved at a higher object-based level of processing, which could be mediated by attention. To test this hypothesis, we instructed participants to voluntarily direct their attention to individual elements in the Ternus display. In this ambiguous apparent motion display, three elements are aligned next to each other and shifted by one position from one frame to the next. This shift can be either perceived as all elements moving together (group motion) or as one element jumping across the others (element motion). We created a competitive Ternus display, in which the color of the elements was manipulated in such a way that the percept was biased toward element motion for one color and toward group motion for another color. If correspondence can be established at an object-based level, attending toward one of the biased elements should increase the likelihood that this element determines the correspondence solution and thereby that the biased motion is perceived. Our results were in line with this hypothesis providing support for an object-based correspondence process that is based on a one-to-one mapping of the most similar elements mediated via attention.

What was that object? On the role of identity information in the formation of object files and conscious object perception

Object files are a psychological representation that allows the human brain to keep track of objects, as they move and change across time. The question regarding what information is used to individuate versus update object files has been the focus of considerable scientific debate. Historically, the role of an object's spatiotemporal history was emphasised, whereas more recent work has demonstrated a key contribution from surface features, such as colour. The purpose of the present study was to investigate the role of identity-level information in the formation and individuation of object files, and how it compares to the contribution of featural information. Using a modified spatial repetition-blindness paradigm, across four experiments, there was convergent evidence that surface features contribute to the formation of object files, whereas the role of identity information was at best much smaller and less reliable than the clear contribution from surface features, and the most parsimonious explanation is that it was not present at all.

Decoding information about dynamically occluded objects in visual cortex

During dynamic occlusion, an object passes behind an occluding surface and then later reappears. Even when completely occluded from view, such objects are experienced as continuing to exist or persist behind the occluder even though they are no longer visible. The contents and neural basis of this persistent representation remain poorly understood. Questions remain as to whether there is information maintained about the object itself (i.e. its shape or identity) or non-object-specific information such as its position or velocity as it is tracked behind an occluder, as well as which areas of visual cortex represent such information. Recent studies have found that early visual cortex is activated by "invisible" objects during visual imagery and by unstimulated regions along the path of apparent motion, suggesting that some properties of dynamically occluded objects may also be neurally represented in early visual cortex. We applied functional magnetic resonance imaging in human subjects to examine representations within visual cortex during dynamic occlusion. For gradually occluded, but not for instantly disappearing objects, there was an increase in activity in early visual cortex (V1, V2, and V3). This activity was spatially-specific, corresponding to the occluded location in the visual field. However, the activity did not encode enough information about object identity to discriminate between different kinds of occluded objects (circles vs. stars) using MVPA. In contrast, object identity could be decoded in spatially-specific subregions of higher-order, topographically organized areas such as ventral, lateral, and temporal occipital areas (VO, LO, and TO) as well as the functionally defined LOC and hMT+. These results suggest that early visual cortex may only represent the dynamically occluded object's position or motion path, while later visual areas represent object-specific information.

Ambiguity in Tactile Apparent Motion Perception

Background

In von Schiller’s Stroboscopic Alternative Motion (SAM) stimulus two visually presented diagonal dot pairs, located on the corners of an imaginary rectangle, alternate with each other and induce either horizontal, vertical or, rarely, rotational motion percepts. SAM motion perception can be described by a psychometric function of the dot aspect ratio (“AR”, i.e. the relation between vertical and horizontal dot distances). Further, with equal horizontal and vertical dot distances (AR = 1) perception is biased towards vertical motion. In a series of five experiments, we presented tactile SAM versions and studied the role of AR and of different reference frames for the perception of tactile apparent motion.

Methods

We presented tactile SAM stimuli and varied the ARs, while participants reported the perceived motion directions. Pairs of vibration stimulators were attached to the participants’ forearms and stimulator distances were varied within and between forearms. We compared straight and rotated forearm conditions with each other in order to disentangle the roles of exogenous and endogenous reference frames.

Results

Increasing the tactile SAM’s AR biased perception towards vertical motion, but the effect was weak compared to the visual modality. We found no horizontal disambiguation, even for very small tactile ARs. A forearm rotation by 90° kept the vertical bias, even though it was now coupled with small ARs. A 45° rotation condition with crossed forearms, however, evoked a strong horizontal motion bias.

Discussion

Existing approaches to explain the visual SAM bias fail to explain the current tactile results. Particularly puzzling is the strong horizontal bias in the crossed-forearm conditions. In the case of tactile apparent motion, there seem to be no fixed priority rule for perceptual disambiguation. Rather the weighting of available evidence seems to depend on the degree of stimulus ambiguity, the current situation and on the perceptual strategy of the individual observer.

The nature of altered vision near the hands: evidence for the magnocellular enhancement account from object correspondence through occlusion

A growing body of evidence indicates that the perception of visual stimuli is altered when they occur near the observer's hands, relative to other locations in space (see Brockmole, Davoli, Abrams, & Witt, 2013, for a review). Several accounts have been offered to explain the pattern of performance across different tasks. These have typically focused on attentional explanations (attentional prioritization and detailed attentional evaluation of stimuli in near-hand space), but more recently, it has been suggested that near-hand space enjoys enhanced magnocellular (M) input. Here we differentiate between the attentional and M-cell accounts, via a task that probes the roles of position consistency and color consistency in determining dynamic object correspondence through occlusion. We found that placing the hands near the visual display made observers use only position consistency, and not color, in determining object correspondence through occlusion, which is consistent with the fact that M cells are relatively insensitive to color. In contrast, placing observers' hands far from the stimuli allowed both color and position contribute. This provides evidence in favor of the M-cell enhancement account of altered vision near the hands.

Illumination frame of reference in the object-reviewing paradigm: A case of luminance and lightness

The present study combines the object-reviewing paradigm (Kahneman, Treisman, & Gibbs, 1992) with the checkershadow illusion (Adelson, 1995) to contrast the effects of objects' luminance versus lightness on the object-specific preview benefit. To this end, we manipulated objects' luminance and the amount of illumination given by an informative background scene in experiments. In line with previous studies (Moore, Stephens, & Hein, 2010), there was no object-specific preview benefit when objects were presented on a uniformly colored background and luminance switched between objects. In contrast, when objects were presented on the checkershadow illusion background which provided an explanation for the luminance switch, a reliable object-specific preview benefit was observed. This suggests that object correspondence as measured by the object-reviewing paradigm can be influenced by scene-induced, perceived lightness of objects' surfaces. We replicated this finding and moreover showed that the scene context only influences the object-specific preview benefit if the objects are perceived as part of the background scene.

Substituting objects from consciousness: a review of object substitution masking

Object substitution masking (OSM) occurs when a sparse (e.g., four-dot), temporally trailing mask obscures the visibility of a briefly presented target. Here, we review theories of OSM: those that propose that OSM reflects the interplay between feedforward and feedback/reentrant neural processes, those that predict that feedforward processing alone gives rise to the phenomenon, and theories that focus on cognitive explanations, such as object updating. We discuss how each of these theories accommodates key findings from the OSM literature. In addition, we examine the relationship between OSM and other visual-cognitive phenomena, including object correspondence through occlusion, change blindness, metacontrast masking, backward masking, and visual short-term memory. Finally, we examine the level of processing at which OSM impairs target perception. Collectively, OSM appears to reflect the conditions under which the brain confuses two visual events for one when they are encoded with low spatiotemporal resolution, due to processing resources being otherwise occupied.

Surface feature congruency effects in the object-reviewing paradigm are dependent on task memory demands

Perception of object continuity depends on establishing correspondence between objects viewed across disruptions in visual information. The role of spatiotemporal information in guiding object continuity is well documented; the role of surface features, however, is controversial. Some researchers have shown an object-specific preview benefit (OSPB)-a standard index of object continuity-only when correspondence could be based on an object's spatiotemporal information, whereas others have found color-based OSPB, suggesting that surface features can also guide object continuity. This study shows that surface feature-based OSPB is dependent on the task memory demands. When the task involved letters and matching just one target letter to the preview ones, no color congruency effect was found under spatiotemporal discontinuity and spatiotemporal ambiguity (Experiments 1-3), indicating that the absence of feature-based OSPB cannot be accounted for by salient spatiotemporal discontinuity. When the task involved complex shapes and matching two target shapes to the preview ones, color-based OSPB was obtained. Critically, however, when a visual working memory task was performed concurrently with the matching task, the presence of a nonspatial (but not a spatial) working memory load eliminated the color-based OSPB (Experiments 4 and 5). These results suggest that the surface feature congruency effects that are observed in the object-reviewing paradigm (with the matching task) reflect memory-based strategies that participants use to solve a memory-demanding task; therefore, they are not reliable measures of online object continuity and cannot be taken as evidence for the role of surface features in establishing object correspondence.

An object-mediated updating account of insensitivity to transsaccadic change

Recent evidence has suggested that relatively precise information about the location and visual form of a saccade target object is retained across a saccade. However, this information appears to be available for report only when the target is removed briefly, so that the display is blank when the eyes land. We hypothesized that the availability of precise target information is dependent on whether a post-saccade object is mapped to the same object representation established for the presaccade target. If so, then the post-saccade features of the target overwrite the presaccade features, a process of object mediated updating in which visual masking is governed by object continuity. In two experiments, participants' sensitivity to the spatial displacement of a saccade target was improved when that object changed surface feature properties across the saccade, consistent with the prediction of the object-mediating updating account. Transsaccadic perception appears to depend on a mechanism of object-based masking that is observed across multiple domains of vision. In addition, the results demonstrate that surface-feature continuity contributes to visual stability across saccades.