Object correspondence across brief occlusion is established on the basis of both spatiotemporal and surface feature cues

Multiple object tracking with extended occlusions

In everyday life, we often view objects through a limited aperture (e.g., soccer players on TV or cars slipping into our blind spot on a busy road), where objects often move out of view and reappear in a different place later. We modelled this situation in a series of multiple object tracking (MOT) experiments, in which we introduced a cover on the edges of the observed area and manipulated its width. This method introduced systematic occlusions, which were longer than those used in previous MOT studies. Experiment 1 (N = 50) showed that tracking under such conditions is possible, although difficult. An item-level analysis confirmed that people made more errors in targets that were covered longer and more often. In Experiment 2 (N = 50), we manipulated the tracking workload and found that the participants were less affected by the cover when the tracking load was low. In Experiment 3 (N = 50), we asked the participants to keep track of the objects' identities (multiple identity tracking [MIT]). Although MIT is subjectively more demanding, memorising identities improved performance in the most difficult cover conditions. Contrary to previous reports, we also found that even partial occlusions negatively affected tracking.

The role of color in transsaccadic object correspondence

With each saccade, visual information is disrupted, and the visual system is tasked with establishing object correspondence between the presaccadic and postsaccadic representations of the saccade target. There is substantial evidence that the visual system consults spatiotemporal continuity when determining object correspondence across saccades. The evidence for surface feature continuity, however, is mixed. Surface features that are integral to the saccade target object's identity (e.g., shape and contrast polarity) are informative of object continuity, but features that may only imply the state of the object (e.g., orientation) are ignored. The present study tested whether color information is consulted to determine transsaccadic object continuity. We used two variations of the intrasaccadic target displacement task. In Experiments 1 and 2, participants reported the direction of the target displacement. In Experiments 3 and 4, they instead reported whether they detected any target movement. In all experiments, we manipulated the saccade target's continuity by removing it briefly (i.e., blanking) and by changing its color. We found that large color changes can disrupt stability and increase sensitivity to displacements for both direction and movement reports, although not as strongly as long blank durations (250 ms). Interestingly, even smaller color changes, but not blanking, reduced response biases. These results indicate that disrupting surface feature continuity may impact the process of transsaccadic object correspondence more strongly than spatiotemporal disruptions by both increasing the sensitivity and decreasing the response bias.

The go no-go reviewing paradigm: A reliable method for measuring perceptual object-file updating

The object-file framework explains how object continuity is maintained as objects move: it stipulates that when we focus our attention on an object, we automatically retrieve this object's recent history. Supporting evidence comes from the object-specific preview benefit (OSPB): participants are faster to name a target letter when the same letter appeared in the same versus a different object in a preceding (preview) display. Although this framework has been very influential, replicating the OSPB has proved difficult, presumably because observers could ignore the preview display. To address this problem, a modified object-reviewing paradigm was suggested, which became the standard paradigm: participants are required to report whether the target letter matches one of the preview display's letters. However, as this paradigm makes retrieval of the object's history task-relevant, it is a useful method for studying the structure of object representations for memory but does not capture the automaticity of the object-reviewing process, which is the heart of the object-file account of perception. Here, we suggest an alternative go/no-go object-reviewing paradigm that is specifically tailored to study object-files for perception: it requires participants to attend to the preview display, yet does not require explicit retrieval of the object history. Using our new paradigm, we reliably replicate the OSPB. As a proof of concept, we revisit the persistence of the OSPB, previously investigated with the modified paradigm.

The best game in town: The reemergence of the language-of-thought hypothesis across the cognitive sciences

Mental representations remain the central posits of psychology after many decades of scrutiny. However, there is no consensus about the representational format(s) of biological cognition. This paper provides a survey of evidence from computational cognitive psychology, perceptual psychology, developmental psychology, comparative psychology, and social psychology, and concludes that one type of format that routinely crops up is the language-of-thought (LoT). We outline six core properties of LoTs: (i) discrete constituents; (ii) role-filler independence; (iii) predicate-argument structure; (iv) logical operators; (v) inferential promiscuity; and (vi) abstract content. These properties cluster together throughout cognitive science. Bayesian computational modeling, compositional features of object perception, complex infant and animal reasoning, and automatic, intuitive cognition in adults all implicate LoT-like structures. Instead of regarding LoT as a relic of the previous century, researchers in cognitive science and philosophy-of-mind must take seriously the explanatory breadth of LoT-based architectures. We grant that the mind may harbor many formats and architectures, including iconic and associative structures as well as deep-neural-network-like architectures. However, as computational/representational approaches to the mind continue to advance, classical compositional symbolic structures - that is, LoTs - only prove more flexible and well-supported over time.

Does the construction retrieval account apply to cross‐modal inhibition of return in semantic context?

The traditional attentional reorienting hypothesis is insufficient to explain spatial and nonspatial inhibition of return (IOR). Therefore, a construction retrieval account that includes the influence of top-down attentional sets has been proposed and can explain both spatial and nonspatial IOR. However, it remains unknown whether the construction retrieval account can be applied to non-surface features of stimuli, as well as whether its construction and retrieval mechanisms are supra-modal. The present study manipulated semantic feature congruency and spatial location congruency between the prime and the target in cross-modal audio-visual and visual-audio experimental conditions, respectively, by orthogonally combining spatial and nonspatial IOR paradigms. Our results showed that there was an interaction between semantic feature congruency and spatial location congruency controlled by the attentional sets, and that this interaction was consistent in cross-modal audio-visual and visual-audio conditions. These results suggest that the construction retrieval account can be applied to abstract semantic features and that its construction and retrieval mechanisms are supra-modal. The present study extends the application scope of the construction retrieval account and promotes the interpretation of IOR under a unified theoretical framework.

Perception of Happening: How the Brain Deals with the No-History Problem

In physics, the temporal dimension has units of infinitesimally brief duration. Given this, how is it possible to perceive things, such as motion, music, and vibrotactile stimulation, that involve extension across many units of time? To address this problem, it is proposed that there is what is termed an "information construct of happening" (ICOH), a simultaneous representation of recent, temporally differentiated perceptual information on the millisecond time scale. The main features of the ICOH are (i) time marking, semantic labeling of all information in the ICOH with ordinal temporal information and distance from what is informationally identified as the present moment, (ii) vector informational features that specify kind, direction, and rate of change for every feature in a percept, and (iii) connectives, information relating vector informational features at adjacent temporal locations in the ICOH. The ICOH integrates products of perceptual processing with recent historical information in sensory memory on the subsecond time scale. Perceptual information about happening in informational sensory memory is encoded in semantic form that preserves connected semantic trails of vector and timing information. The basic properties of the ICOH must be supported by a general and widespread timing mechanism that generates ordinal and interval timing information and it is suggested that state-dependent networks may suffice for that purpose. Happening, therefore, is perceived at a moment and is constituted by an information structure of connected recent historical information.

Capturing the objects of vision with neural networks

Human visual perception carves a scene at its physical joints, decomposing the world into objects, which are selectively attended, tracked and predicted as we engage our surroundings. Object representations emancipate perception from the sensory input, enabling us to keep in mind that which is out of sight and to use perceptual content as a basis for action and symbolic cognition. Human behavioural studies have documented how object representations emerge through grouping, amodal completion, proto-objects and object files. By contrast, deep neural network models of visual object recognition remain largely tethered to sensory input, despite achieving human-level performance at labelling objects. Here, we review related work in both fields and examine how these fields can help each other. The cognitive literature provides a starting point for the development of new experimental tasks that reveal mechanisms of human object perception and serve as benchmarks driving the development of deep neural network models that will put the object into object recognition.

Intrasaccadic motion streaks jump-start gaze correction

Rapid eye movements (saccades) incessantly shift objects across the retina. To establish object correspondence, the visual system is thought to match surface features of objects across saccades. Here, we show that an object's intrasaccadic retinal trace-a signal previously considered unavailable to visual processing-facilitates this match making. Human observers made saccades to a cued target in a circular stimulus array. Using high-speed visual projection, we swiftly rotated this array during the eyes' flight, displaying continuous intrasaccadic target motion. Observers' saccades landed between the target and a distractor, prompting secondary saccades. Independently of the availability of object features, which we controlled tightly, target motion increased the rate and reduced the latency of gaze-correcting saccades to the initial presaccadic target, in particular when the target's stimulus features incidentally gave rise to efficient motion streaks. These results suggest that intrasaccadic visual information informs the establishment of object correspondence and jump-starts gaze correction.

Task-specific engagement of object-based and space-based attention with spatiotemporally defined objects

We used a form of ambiguous apparent motion known as Ternus motion to isolate the effects of object-based and space-based attention, and to explore functional differences between them. Two frames of horizontally aligned disks that were shifted by one position between frames were temporally separated by either a short or a long inter-stimulus interval (ISI). Short ISI displays were perceived as element motion where one disk appeared to jump across the other two. Long ISI displays were perceived as group motion where all three disks appeared to move together. Because element and group motion imply mutually exclusive object structures, adding stimuli (e.g., a small gap) to one disk in each frame created conditions of orthogonal object and location status (same or different), depending on ISI. We used two tasks with different functional demands, an identification task (Experiments 1 and 3a) in which observers responded to a single attribute of the final stimulus, and a comparison task (Experiments 2 and 3b) in which observers compared two attributes across two stimuli. Reliable object-specific effects occurred only with the comparison task, whereas location-specific effects occurred with both tasks. These results confirm that attention can be directed to objects separately from spatial locations and vice versa, and, moreover, that object-based and space-based attention are engaged differently depending on the processing demands of the task.

Effects of Transient and Nontransient Changes of Surface Feature on Object Correspondence

Object correspondence is a fundamental problem in perception. Classic theories hold that the computation of correspondence is solely based on spatiotemporal information. Recent research showed that surface features also play an important role. However, the surface features of objects in many studies did not change throughout a trial. This study investigated the effect of feature change on object correspondence using the object-reviewing paradigm. Two moving objects underwent transient feature changes on color dimension (Experiment 1A) or a combination of three dimensions (Experiment 2A). Moreover, the objects moved behind four occluders to make the feature change nontransient (Experiments 1B and 2B). Object-specific preview benefits were reduced or eliminated when feature changes were transient, but the benefits were not affected when feature changes were nontransient. The effects of transient versus nontransient changes of surface feature in object correspondence are discussed.

Feature-based guidance of attention during post-saccadic selection

Current models of trans-saccadic perception propose that, after a saccade, the saccade target object must be localized among objects near the landing position. However, the nature of the attentional mechanisms supporting this process is currently under debate. In the present study, we tested whether surface properties of the saccade target object automatically bias post-saccadic selection using a variant of the visual search task. Participants executed a saccade to a shape-singleton target in a circular array. During this primary saccade, the array sometimes rotated so that the eyes landed between the target and an adjacent distractor, requiring gaze correction. In addition, each object in the array had an incidental color value. On Switch trials, the target and adjacent distractor switched colors. The accuracy and latency of gaze correction to the target (measures that provide a direct index of target localization) were compared with a control condition in which no color switch occurred (No-switch trials). Gaze correction to the target was substantially impaired in the Switch condition. This result was obtained even when participants had substantial incentive to avoid encoding the color of the saccade target. In addition, similar effects were observed when the roles of the two feature dimensions (color and shape) were reversed. The results indicate that saccade target features are automatically encoded before a saccade, are retained in visual working memory across the saccade, and instantiate a feature-based selection operation when the eyes land, biasing attention toward objects that match target features.

Feature-based guidance of attention by visual working memory is applied independently of remembered object location

Visual working memory (VWM) has been implicated both in the online representation of object tokens (in the object-file framework) and in the top-down guidance of attention during visual search, implementing a feature template. It is well established that object representations in VWM are structured by location, with access to the content of VWM modulated by position consistency. In the present study, we examined whether this property generalizes to the guidance of attention. Specifically, in two experiments, we probed whether the guidance of spatial attention from features in VWM is modulated by the position of the object from which these features were encoded. Participants remembered an object with an incidental color. Items in a subsequent search array could match either the color of the remembered object, the location, or both. Robust benefits of color match (when the matching item was the target) and costs (when the matching items was a distractor) were observed. Critically, the magnitude of neither effect was influenced by spatial correspondence. The results demonstrate that features in VWM influence attentional priority maps in a manner that does not necessarily inherit the spatial structure of the object representations in which those features are maintained.

What was that object? On the role of identity information in the formation of object files and conscious object perception

Object files are a psychological representation that allows the human brain to keep track of objects, as they move and change across time. The question regarding what information is used to individuate versus update object files has been the focus of considerable scientific debate. Historically, the role of an object's spatiotemporal history was emphasised, whereas more recent work has demonstrated a key contribution from surface features, such as colour. The purpose of the present study was to investigate the role of identity-level information in the formation and individuation of object files, and how it compares to the contribution of featural information. Using a modified spatial repetition-blindness paradigm, across four experiments, there was convergent evidence that surface features contribute to the formation of object files, whereas the role of identity information was at best much smaller and less reliable than the clear contribution from surface features, and the most parsimonious explanation is that it was not present at all.

Calibration of peripheral perception of shape with and without saccadic eye movements

The cortical representations of a visual object differ radically across saccades. Several studies claim that the visual system adapts the peripheral percept to better match the subsequent foveal view. Recently, Herwig, Weiß, and Schneider (2015, Annals of the New York Academy of Sciences, 1339(1), 97-105) found that the perception of shape demonstrates a saccade-dependent learning effect. Here, we ask whether this learning actually requires saccades. We replicated Herwig et al.'s (2015) study and introduced a fixation condition. In a learning phase, participants were exposed to objects whose shape systematically changed during a saccade, or during a displacement from peripheral to foveal vision (without a saccade). In a subsequent test, objects were perceived as less (more) curved if they previously changed from more circular (triangular) in the periphery to more triangular (circular) in the fovea. Importantly, this pattern was seen both with and without saccades. We then tested whether a variable delay between the presentations of the peripheral and foveal objects would affect their association-hypothetically weakening it at longer delays. Again, we found that shape judgments depended on the changes experienced during the learning phase and that they were similar in both the saccade and fixation conditions. Surprisingly, they were not affected by the delay between the peripheral and foveal presentations over the range we tested. These results suggest that a general associative process, independent of saccade execution, contributes to the perception of shape across viewpoints.

Binding object features to locations: Does the "spatial congruency bias" update with object movement?

One of the fundamental challenges of visual cognition is how our visual systems combine information about an object's features with its spatial location. A recent phenomenon related to object-location binding, the "spatial congruency bias," revealed that two objects are more likely to be perceived as having the same identity or features if they appear in the same spatial location, versus if the second object appears in a different location. The spatial congruency bias suggests that irrelevant location information is automatically encoded with and bound to other object properties, biasing perceptual judgments. Here we further explored this new phenomenon and its role in object-location binding by asking what happens when an object moves to a new location: Is the spatial congruency bias sensitive to spatiotemporal contiguity cues, or does it remain linked to the original object location? Across four experiments, we found that the spatial congruency bias remained strongly linked to the original object location. However, under certain circumstances-for instance, when the first object paused and remained visible for a brief time after the movement-the congruency bias was found at both the original location and the updated location. These data suggest that the spatial congruency bias is based more on low-level visual information than on spatiotemporal contiguity cues, and reflects a type of object-location binding that is primarily tied to the original object location and that may only update to the object's new location if there is time for the features to be re-encoded and rebound following the movement.

Representing dynamic stimulus information during occlusion

Human observers maintain a representation of the visual features of objects when they become occluded. This representation facilitates the interpretation of occluded events and allows us to quickly identify objects upon reappearing. Here we investigated whether visual features that change over time are also represented during occlusion. To answer this question we used an illusion from the time perception domain in which the perceived duration of an event increases as its temporal frequency content increases. In the first experiment we demonstrate temporal frequency induced modulation of duration both when the object remains visible as well as when it becomes temporarily occluded. Additionally, we demonstrate that time dilation for temporarily occluded objects cannot be explained by modulations of duration as a result of pre- and post-occlusion presentation of the object. In a second experiment, we corroborate this finding by demonstrating that modulation of the perceived duration of occluded events depends on the expected temporal frequency content of the object during occlusion. Together these results demonstrate that the dynamic properties of an object are represented during occlusion. We conclude that the representations of occluded objects contain a wide range of features derived from the period when the object was still visible, including information about both the static and dynamic properties of the object.

Aesthetics by Numbers: Links between Perceived Texture Qualities and Computed Visual Texture Properties

Our world is filled with texture. For the human visual system, this is an important source of information for assessing environmental and material properties. Indeed-and presumably for this reason-the human visual system has regions dedicated to processing textures. Despite their abundance and apparent relevance, only recently the relationships between texture features and high-level judgments have captured the interest of mainstream science, despite long-standing indications for such relationships. In this study, we explore such relationships, as these might be used to predict perceived texture qualities. This is relevant, not only from a psychological/neuroscience perspective, but also for more applied fields such as design, architecture, and the visual arts. In two separate experiments, observers judged various qualities of visual textures such as beauty, roughness, naturalness, elegance, and complexity. Based on factor analysis, we find that in both experiments, ~75% of the variability in the judgments could be explained by a two-dimensional space, with axes that are closely aligned to the beauty and roughness judgments. That a two-dimensional judgment space suffices to capture most of the variability in the perceived texture qualities suggests that observers use a relatively limited set of internal scales on which to base various judgments, including aesthetic ones. Finally, for both of these judgments, we determined the relationship with a large number of texture features computed for each of the texture stimuli. We find that the presence of lower spatial frequencies, oblique orientations, higher intensity variation, higher saturation, and redness correlates with higher beauty ratings. Features that captured image intensity and uniformity correlated with roughness ratings. Therefore, a number of computational texture features are predictive of these judgments. This suggests that perceived texture qualities-including the aesthetic appreciation-are sufficiently universal to be predicted-with reasonable accuracy-based on the computed feature content of the textures.

The nature of altered vision near the hands: evidence for the magnocellular enhancement account from object correspondence through occlusion

A growing body of evidence indicates that the perception of visual stimuli is altered when they occur near the observer's hands, relative to other locations in space (see Brockmole, Davoli, Abrams, & Witt, 2013, for a review). Several accounts have been offered to explain the pattern of performance across different tasks. These have typically focused on attentional explanations (attentional prioritization and detailed attentional evaluation of stimuli in near-hand space), but more recently, it has been suggested that near-hand space enjoys enhanced magnocellular (M) input. Here we differentiate between the attentional and M-cell accounts, via a task that probes the roles of position consistency and color consistency in determining dynamic object correspondence through occlusion. We found that placing the hands near the visual display made observers use only position consistency, and not color, in determining object correspondence through occlusion, which is consistent with the fact that M cells are relatively insensitive to color. In contrast, placing observers' hands far from the stimuli allowed both color and position contribute. This provides evidence in favor of the M-cell enhancement account of altered vision near the hands.

Illumination frame of reference in the object-reviewing paradigm: A case of luminance and lightness

The present study combines the object-reviewing paradigm (Kahneman, Treisman, & Gibbs, 1992) with the checkershadow illusion (Adelson, 1995) to contrast the effects of objects' luminance versus lightness on the object-specific preview benefit. To this end, we manipulated objects' luminance and the amount of illumination given by an informative background scene in experiments. In line with previous studies (Moore, Stephens, & Hein, 2010), there was no object-specific preview benefit when objects were presented on a uniformly colored background and luminance switched between objects. In contrast, when objects were presented on the checkershadow illusion background which provided an explanation for the luminance switch, a reliable object-specific preview benefit was observed. This suggests that object correspondence as measured by the object-reviewing paradigm can be influenced by scene-induced, perceived lightness of objects' surfaces. We replicated this finding and moreover showed that the scene context only influences the object-specific preview benefit if the objects are perceived as part of the background scene.

A touchy subject: advancing the modulated visual pathways account of altered vision near the hand

A growing body of evidence demonstrates that human vision operates differently in the space near and on the hands; for example, early findings in this literature reported that rapid onsets are detected faster near the hands, and that objects are searched more thoroughly. These and many other effects were attributed to enhanced attention via the recruitment of bimodal visual-tactile neurons representing the hand and near-hand space. However, recent research supports an alternative account: stimuli near the hands are preferentially processed by the action-oriented magnocellular visual pathway at the expense of processing in the parvocellular pathway. This Modulated Visual Pathways (MVP) account of altered vision near the hands describes a hand position-dependent trade-off between the two main retinal-cortical visual pathways between the eye and brain. The MVP account explains past findings and makes new predictions regarding near-hand vision supported by new research.

Substituting objects from consciousness: a review of object substitution masking

Object substitution masking (OSM) occurs when a sparse (e.g., four-dot), temporally trailing mask obscures the visibility of a briefly presented target. Here, we review theories of OSM: those that propose that OSM reflects the interplay between feedforward and feedback/reentrant neural processes, those that predict that feedforward processing alone gives rise to the phenomenon, and theories that focus on cognitive explanations, such as object updating. We discuss how each of these theories accommodates key findings from the OSM literature. In addition, we examine the relationship between OSM and other visual-cognitive phenomena, including object correspondence through occlusion, change blindness, metacontrast masking, backward masking, and visual short-term memory. Finally, we examine the level of processing at which OSM impairs target perception. Collectively, OSM appears to reflect the conditions under which the brain confuses two visual events for one when they are encoded with low spatiotemporal resolution, due to processing resources being otherwise occupied.

Sensory memory of structure-from-motion is shape-specific

Perceptual priming can stabilize the phenomenal appearance of multistable visual displays (Leopold, Wilke, Maier, & Logothetis, Nature Neuroscience, 5, 605-609, 2002). Prior exposure to such displays induces a sensory memory of their appearance, which persists over long intervals and intervening stimulation, and which facilitates renewed perception of the same appearance. Here, we investigated perceptual priming for the apparent rotation in depth of ambiguous structure-from-motion (SFM) displays. Specifically, we generated SFM objects with different three-dimensional shapes and presented them in random order and with intervening blank periods. To assess perceptual priming, we established the probability that a perceived direction of rotation would persist between successive objects. In general, persistence was greatest between identical objects, intermediate between similar objects, and negligible between dissimilar objects. These results demonstrate unequivocally that sensory memory for apparent rotation is specific to three-dimensional shape, contrary to previous reports (e.g., Maier, Wilke, Logothetis, & Leopold, Current Biology, 13, 1076-1085, 2003). Because persistence did not depend on presentation order for any pair of objects, it provides a commutative measure for the similarity of object shapes. However, it is not clear exactly which features or aspects of object shape determine similarity. At least, we did not find simple, low-level features (such as volume overlap, heterogeneity, or rotational symmetry) that could have accounted for all observations. Accordingly, it seems that sensory memory of SFM (which underlies priming of ambiguous rotation) engages higher-level representations of object surface and shape.

Surface feature congruency effects in the object-reviewing paradigm are dependent on task memory demands

Perception of object continuity depends on establishing correspondence between objects viewed across disruptions in visual information. The role of spatiotemporal information in guiding object continuity is well documented; the role of surface features, however, is controversial. Some researchers have shown an object-specific preview benefit (OSPB)-a standard index of object continuity-only when correspondence could be based on an object's spatiotemporal information, whereas others have found color-based OSPB, suggesting that surface features can also guide object continuity. This study shows that surface feature-based OSPB is dependent on the task memory demands. When the task involved letters and matching just one target letter to the preview ones, no color congruency effect was found under spatiotemporal discontinuity and spatiotemporal ambiguity (Experiments 1-3), indicating that the absence of feature-based OSPB cannot be accounted for by salient spatiotemporal discontinuity. When the task involved complex shapes and matching two target shapes to the preview ones, color-based OSPB was obtained. Critically, however, when a visual working memory task was performed concurrently with the matching task, the presence of a nonspatial (but not a spatial) working memory load eliminated the color-based OSPB (Experiments 4 and 5). These results suggest that the surface feature congruency effects that are observed in the object-reviewing paradigm (with the matching task) reflect memory-based strategies that participants use to solve a memory-demanding task; therefore, they are not reliable measures of online object continuity and cannot be taken as evidence for the role of surface features in establishing object correspondence.

Evidence for scene-based motion correspondence

To maintain stable object representations as our eyes or the objects themselves move, the visual system must determine how newly sampled information relates to existing object representations. To solve this correspondence problem, the visual system uses not only spatiotemporal information (e.g., the spatial and temporal proximity between elements), but also feature information (e.g., the similarity in size or luminance between elements). Here we asked whether motion correspondence relies solely on image-based feature information, or whether it is influenced by scene-based information (e.g., the perceived sizes of surfaces or the perceived illumination conditions). We manipulated scene-based information separately from image-based information in the Ternus display, an ambiguous apparent-motion display, and found that scene-based information influences how motion correspondence is resolved, indicating that theories of motion correspondence that are based on "scene-blind" mechanisms are insufficient.

Bottlenecks of motion processing during a visual glance: the leaky flask model

Where do the bottlenecks for information and attention lie when our visual system processes incoming stimuli? The human visual system encodes the incoming stimulus and transfers its contents into three major memory systems with increasing time scales, viz., sensory (or iconic) memory, visual short-term memory (VSTM), and long-term memory (LTM). It is commonly believed that the major bottleneck of information processing resides in VSTM. In contrast to this view, we show major bottlenecks for motion processing prior to VSTM. In the first experiment, we examined bottlenecks at the stimulus encoding stage through a partial-report technique by delivering the cue immediately at the end of the stimulus presentation. In the second experiment, we varied the cue delay to investigate sensory memory and VSTM. Performance decayed exponentially as a function of cue delay and we used the time-constant of the exponential-decay to demarcate sensory memory from VSTM. We then decomposed performance in terms of quality and quantity measures to analyze bottlenecks along these dimensions. In terms of the quality of information, two thirds to three quarters of the motion-processing bottleneck occurs in stimulus encoding rather than memory stages. In terms of the quantity of information, the motion-processing bottleneck is distributed, with the stimulus-encoding stage accounting for one third of the bottleneck. The bottleneck for the stimulus-encoding stage is dominated by the selection compared to the filtering function of attention. We also found that the filtering function of attention is operating mainly at the sensory memory stage in a specific manner, i.e., influencing only quantity and sparing quality. These results provide a novel and more complete understanding of information processing and storage bottlenecks for motion processing.

Automatic feature-based grouping during multiple object tracking

Contour interpolation automatically binds targets with distractors to impair multiple object tracking (Keane, Mettler, Tsoi, & Kellman, 2011). Is interpolation special in this regard or can other features produce the same effect? To address this question, we examined the influence of eight features on tracking: color, contrast polarity, orientation, size, shape, depth, interpolation, and a combination (shape, color, size). In each case, subjects tracked 4 of 8 objects that began as undifferentiated shapes, changed features as motion began (to enable grouping), and returned to their undifferentiated states before halting. We found that intertarget grouping improved performance for all feature types except orientation and interpolation (Experiment 1 and Experiment 2). Most importantly, target-distractor grouping impaired performance for color, size, shape, combination, and interpolation. The impairments were, at times, large (>15% decrement in accuracy) and occurred relative to a homogeneous condition in which all objects had the same features at each moment of a trial (Experiment 2), and relative to a "diversity" condition in which targets and distractors had different features at each moment (Experiment 3). We conclude that feature-based grouping occurs for a variety of features besides interpolation, even when irrelevant to task instructions and contrary to the task demands, suggesting that interpolation is not unique in promoting automatic grouping in tracking tasks. Our results also imply that various kinds of features are encoded automatically and in parallel during tracking.

Prediction processes during multiple object tracking (MOT): involvement of dorsal and ventral premotor cortices

BACKGROUND

The multiple object tracking (MOT) paradigm is a cognitive task that requires parallel tracking of several identical, moving objects following nongoal-directed, arbitrary motion trajectories.

AIMS

The current study aimed to investigate the employment of prediction processes during MOT. As an indicator for the involvement of prediction processes, we targeted the human premotor cortex (PM). The PM has been repeatedly implicated to serve the internal modeling of future actions and action effects, as well as purely perceptual events, by means of predictive feedforward functions.

MATERIALS AND METHODS

Using functional magnetic resonance imaging (fMRI), BOLD activations recorded during MOT were contrasted with those recorded during the execution of a cognitive control task that used an identical stimulus display and demanded similar attentional load. A particular effort was made to identify and exclude previously found activation in the PM-adjacent frontal eye fields (FEF).

RESULTS

We replicated prior results, revealing occipitotemporal, parietal, and frontal areas to be engaged in MOT.

DISCUSSION

The activation in frontal areas is interpreted to originate from dorsal and ventral premotor cortices. The results are discussed in light of our assumption that MOT engages prediction processes.

CONCLUSION

We propose that our results provide first clues that MOT does not only involve visuospatial perception and attention processes, but prediction processes as well.

The strategic retention of task-relevant objects in visual working memory

The serial and spatially extended nature of many real-world visual tasks suggests the need for control over the content of visual working memory (VWM). We examined the management of VWM in a task that required participants to prioritize individual objects for retention during scene viewing. There were 5 principal findings: (a) Strategic retention of task-relevant objects was effective and was dissociable from the current locus of visual attention; (b) strategic retention was implemented by protection from interference rather than by preferential encoding; (c) this prioritization was flexibly transferred to a new object as task demands changed; (d) no-longer-relevant items were efficiently eliminated from VWM; and (e) despite this level of control, attended and fixated objects were consolidated into VWM regardless of task relevance. These results are consistent with a model of VWM control in which each fixated object is automatically encoded into VWM, replacing a portion of the content in VWM. However, task-relevant objects can be selectively protected from replacement.

Forgetting what was where: the fragility of object-location binding

Although we frequently take advantage of memory for objects locations in everyday life, understanding how an object’s identity is bound correctly to its location remains unclear. Here we examine how information about object identity, location and crucially object-location associations are differentially susceptible to forgetting, over variable retention intervals and memory load. In our task, participants relocated objects to their remembered locations using a touchscreen. When participants mislocalized objects, their reports were clustered around the locations of other objects in the array, rather than occurring randomly. These ‘swap’ errors could not be attributed to simple failure to remember either the identity or location of the objects, but rather appeared to arise from failure to bind object identity and location in memory. Moreover, such binding failures significantly contributed to decline in localization performance over retention time. We conclude that when objects are forgotten they do not disappear completely from memory, but rather it is the links between identity and location that are prone to be broken over time.

An object-mediated updating account of insensitivity to transsaccadic change

Recent evidence has suggested that relatively precise information about the location and visual form of a saccade target object is retained across a saccade. However, this information appears to be available for report only when the target is removed briefly, so that the display is blank when the eyes land. We hypothesized that the availability of precise target information is dependent on whether a post-saccade object is mapped to the same object representation established for the presaccade target. If so, then the post-saccade features of the target overwrite the presaccade features, a process of object mediated updating in which visual masking is governed by object continuity. In two experiments, participants' sensitivity to the spatial displacement of a saccade target was improved when that object changed surface feature properties across the saccade, consistent with the prediction of the object-mediating updating account. Transsaccadic perception appears to depend on a mechanism of object-based masking that is observed across multiple domains of vision. In addition, the results demonstrate that surface-feature continuity contributes to visual stability across saccades.

Spatio-temporal priority revisited: the role of feature identity and similarity for object correspondence in apparent motion

We live in a dynamic environment in which objects change location over time. To maintain stable object representations the visual system must determine how newly sampled information relates to existing object representations, the correspondence problem. Spatiotemporal information is clearly an important factor that the visual system takes into account when solving the correspondence problem, but is feature information irrelevant as some theories suggest? The Ternus display provides a context in which to investigate solutions to the correspondence problem. Two sets of three horizontally aligned disks, shifted by one position, were presented in alternation. Depending on how correspondence is resolved, these displays are perceived either as one disk "jumping" from one end of the row to the other (element motion) or as a set of three disks shifting back and forth together (group motion). We manipulated a feature (e.g., color) of the disks such that, if features were taken into account by the correspondence process, it would bias the resolution of correspondence toward one version or the other. Features determined correspondence, whether they were luminance-defined or not. Moreover, correspondence could be established on the basis of similarity, when features were not identical between alternations. Finally, the stronger the feature information supported a certain correspondence solution the more it dominated spatiotemporal information.

The Role of Surface Feature Continuity in Object-based Inhibition of Return

The contribution of surface feature continuity to object-based inhibition of return (IOR) was tested in three experiments. Participants executed a saccade to a previously fixated or unfixated colored disk after the object had moved to a new location. Object-based IOR was observed as lengthened saccade latency to a previously fixated object. The consistency of surface feature (color) and spatiotemporal information was manipulated to examine the feature used to define the persisting objects to which inhibition is assigned. If the two objects traded colors during motion, object-based IOR was reliably reduced (Experiment 2), suggesting a role for surface feature properties in defining the objects of object-based IOR. However, if the two objects changed to new colors during motion, object-based IOR was preserved (Experiment 1), and color consistency was not sufficient to support object continuity across a salient spatiotemporal discontinuity (Experiment 3). These results suggest that surface feature consistency plays a significant role in defining object persistence for the purpose of IOR, although surface features may be weighted less strongly than spatiotemporal features in this domain.

Trajectory recognition as the basis for object individuation: a functional model of object file instantiation and object-token encoding

The perception of persisting visual objects is mediated by transient intermediate representations, object files, that are instantiated in response to some, but not all, visual trajectories. The standard object file concept does not, however, provide a mechanism sufficient to account for all experimental data on visual object persistence, object tracking, and the ability to perceive spatially disconnected stimuli as continuously existing objects. Based on relevant anatomical, functional, and developmental data, a functional model is constructed that bases visual object individuation on the recognition of temporal sequences of apparent center-of-mass positions that are specifically identified as trajectories by dedicated "trajectory recognition networks" downstream of the medial-temporal motion-detection area. This model is shown to account for a wide range of data, and to generate a variety of testable predictions. Individual differences in the recognition, abstraction, and encoding of trajectory information are expected to generate distinct object persistence judgments and object recognition abilities. Dominance of trajectory information over feature information in stored object tokens during early infancy, in particular, is expected to disrupt the ability to re-identify human and other individuals across perceptual episodes, and lead to developmental outcomes with characteristics of autism spectrum disorders.

Features, as well as space and time, guide object persistence

What role do surface features (e.g., color) play in the establishment and maintenance of episodic representations of objects (object files)? Mitroff and Alvarez (2007) showed that stimuli that were linked by a continuous spatiotemporal history yielded object-specific preview benefits--a standard index of object files-whereas stimuli linked only by shared surface features did not. Here, it is shown that abruptly changing the features of an object that has been established on the basis of spatiotemporal history can disrupt object-specific preview benefits (Experiments 1 and 2). Moreover, under some conditions, feature match alone can give rise to the preview benefits (Experiment 3). These results indicate that surface features, as well as spatiotemporal factors, play an important role in establishing and maintaining episodic object representations.

Binding objects to locations: the relationship between object files and visual working memory

The relationship between object files and visual working memory (VWM) was investigated in a new paradigm combining features of traditional VWM experiments (color change detection) and object-file experiments (memory for the properties of moving objects). Object-file theory was found to account for a key component of object-position binding in VWM: With motion, color memory came to be associated with the new locations of objects. However, robust binding to the original locations was found despite clear evidence that the objects had moved. This latter binding appears to constitute a scene-based component in VWM, which codes object location relative to the abstract spatial configuration of the display and is largely insensitive to the dynamic properties of objects.

Differential-Activation Theory Can Account for the Ternus Display: Rejoinder to Petersik

The Ternus display is a bistable apparent-motion display that has captivated researchers for decades. Recently, Odic and Pratt (2008, Perception37 1790–1804) provided evidence against the well-known two-process theory of Braddick and Adlard [1978, Visual Psychophysics and Physiology (New York: Academic Press) pp 417–426] and provided an explanatory framework using the differential-activation theory of Gilroy et al (2001, Perception & Psychophysics63 847–861). A comment by Petersik (this issue) challenges the methodology and theoretical implications of Odic and Pratt, and claims that the two-process distinction still has a role to play in the Ternus display. In this rejoinder, we examine the main points made by Petersik and expand on the differential-activation theory and its applicability to the Ternus display.