How do we track invisible objects?

Multiple object tracking with extended occlusions

In everyday life, we often view objects through a limited aperture (e.g., soccer players on TV or cars slipping into our blind spot on a busy road), where objects often move out of view and reappear in a different place later. We modelled this situation in a series of multiple object tracking (MOT) experiments, in which we introduced a cover on the edges of the observed area and manipulated its width. This method introduced systematic occlusions, which were longer than those used in previous MOT studies. Experiment 1 (N = 50) showed that tracking under such conditions is possible, although difficult. An item-level analysis confirmed that people made more errors in targets that were covered longer and more often. In Experiment 2 (N = 50), we manipulated the tracking workload and found that the participants were less affected by the cover when the tracking load was low. In Experiment 3 (N = 50), we asked the participants to keep track of the objects' identities (multiple identity tracking [MIT]). Although MIT is subjectively more demanding, memorising identities improved performance in the most difficult cover conditions. Contrary to previous reports, we also found that even partial occlusions negatively affected tracking.

Tracking multiple fish

Although the Multiple Object Tracking (MOT) task is a widely used experimental method for studying divided attention, tracking objects in the real world usually looks different. For example, in the real world, objects are usually clearly distinguishable from each other and also possess different movement patterns. One such case is tracking groups of creatures, such as tracking fish in an aquarium. We used movies of fish in an aquarium and measured general tracking performance in this task (Experiment 1). In Experiment 2, we compared tracking accuracy within-subjects in fish tracking, tracking typical MOT stimuli, and in a third condition using standard MOT uniform objects which possessed movement patterns similar to the real fish. This third condition was added to further examine the impact of different motion characteristics on tracking performance. Results within a Bayesian framework showed that tracking real fish shares similarities with tracking simple objects in a typical laboratory MOT task. Furthermore, we observed a close relationship between performance in both laboratory MOT tasks (typical and fish-like) and real fish tracking, suggesting that the commonly used laboratory MOT task possesses a good level of ecological validity.

The Functional Visual Field(s) in simple visual search

During a visual search for a target among distractors, observers do not fixate every location in the search array. Rather processing is thought to occur within a Functional Visual Field (FVF) surrounding each fixation. We argue that there are three questions that can be asked at each fixation and that these imply three different senses of the FVF. 1) Can I identify what is at location XY? This defines a resolution FVF. 2) To what shall I attend during this fixation? This defines an Attentional FVF. 3) Where should I fixate next? This defines an Exploratory FVF. We examine FVFs 2&3 using eye movements in visual search. In three Experiments, we collected eye movements during visual search for the target letter T among distractor letter Ls (Exps 1 and 3) or for a color X orientation conjunction (Exp 2). Saccades that do not go to the target can be used to define the Exploratory FVF. The saccade that goes to the target can be used to define the Attentional FVF since the target was probably covertly detected during the prior fixation. The Exploratory FVF is larger than the Attentional FVF for all three experiments. Interestingly, the probability that the next saccade would go to the target was always well below 1.0, even when the current fixation was close to the target and well within any reasonable estimate of the FVF. Measuring search-based Exploratory and Attentional FVFs sheds light on how we can miss clearly visible targets.

Attentional capture in multiple object tracking

Attentional processes are generally assumed to be involved in multiple object tracking (MOT). The attentional capture paradigm is regularly used to study conditions of attentional control. It has up to now not been used to assess influences of sudden onset distractor stimuli in MOT. We investigated whether attentional capture does occur in MOT: Are onset distractors processed at all in dynamic attentional tasks? We found that sudden onset distractors were effective in lowering probe detection, thus demonstrating attentional capture. Tracking performance as dependent measure was not affected. The attentional capture effect persisted in conditions of higher tracking load (Experiment 2) and was dramatically increased in lower presentation frequency of the onset distractor (Experiment 3). Tracking performance was shown to suffer only when onset distractors were presented serially with very short time gaps in between, thus effectively disturbing re-engaging attention on the tracking set (Experiment 4). We discuss that rapid dis- and re-engagement of the attention process on target objects and an additional more basic process that continuously provides location information allow managing strong disruptions of attention during tracking.

Effectiveness of "rescue saccades" on the accuracy of tracking multiple moving targets: An eye-tracking study on the effects of target occlusions

Occlusion is one of the main challenges in tracking multiple moving objects. In almost all real-world scenarios, a moving object or a stationary obstacle occludes targets partially or completely for a short or long time during their movement. A previous study (Zelinsky & Todor, 2010) reported that subjects make timely saccades toward the object in danger of being occluded. Observers make these so-called “rescue saccades” to prevent target swapping. In this study, we examined whether these saccades are helpful. To this aim, we used as the stimuli recorded videos from natural movement of zebrafish larvae swimming freely in a circular container. We considered two main types of occlusion: object-object occlusions that naturally exist in the videos, and object-occluder occlusions created by adding a stationary doughnut-shape occluder in some videos. Four different scenarios were studied: (1) no occlusions, (2) only object-object occlusions, (3) only object-occluder occlusion, or (4) both object-object and object-occluder occlusions. For each condition, two set sizes (two and four) were applied. Participants’ eye movements were recorded during tracking, and rescue saccades were extracted afterward. The results showed that rescue saccades are helpful in handling object-object occlusions but had no reliable effect on tracking through object-occluder occlusions. The presence of occlusions generally increased visual sampling of the scenes; nevertheless, tracking accuracy declined due to occlusion.

Multiple-target tracking in human and machine vision

Humans are able to track multiple objects at any given time in their daily activities—for example, we can drive a car while monitoring obstacles, pedestrians, and other vehicles. Several past studies have examined how humans track targets simultaneously and what underlying behavioral and neural mechanisms they use. At the same time, computer-vision researchers have proposed different algorithms to track multiple targets automatically. These algorithms are useful for video surveillance, team-sport analysis, video analysis, video summarization, and human–computer interaction. Although there are several efficient biologically inspired algorithms in artificial intelligence, the human multiple-target tracking (MTT) ability is rarely imitated in computer-vision algorithms. In this paper, we review MTT studies in neuroscience and biologically inspired MTT methods in computer vision and discuss the ways in which they can be seen as complementary.

Multiple-target tracking (MTT) is a challenging task vital for both a human’s daily life and for many artificial intelligent systems, such as those used for urban traffic control. Neuroscientists are interested in discovering the underlying neural mechanisms that successfully exploit cognitive resources, e.g., spatial attention or memory, during MTT. Computer-vision specialists aim to develop powerful MTT algorithms based on advanced models or data-driven computational methods. In this paper, we review MTT studies from both communities and discuss how findings from cognitive studies can inspire developers to construct higher performing MTT algorithms. Moreover, some directions have been proposed through which MTT algorithms could raise new questions in the cognitive science domain, and answering them can shed light on neural processes underlying MTT.

Contribution of Visuospatial and Motion-Tracking to Invisible Motion

People experience an object's motion even when it is occluded. We investigate the processing of invisible motion in three experiments. Observers saw a moving circle passing behind an invisible, irregular hendecagonal polygon and had to respond as quickly as possible when the target had “just reappeared” from behind the occluder. Without explicit cues allowing the end of each of the eight hidden trajectories to be predicted (length ranging between 4.7 and 5 deg), we found as expected, if visuospatial attention was involved, anticipation errors, providing that information on pre-occluder motion was available. This indicates that the observers, rather than simply responding when they saw the target, tended to anticipate its reappearance (Experiment 1). The new finding is that, with a fixation mark indicating the center of the invisible trajectory, a linear relationship between the physical and judged occlusion duration is found, but not without it (Experiment 2) or with a fixation mark varying in position from trial to trial (Experiment 3). We interpret the role of central fixation in the differences in distinguishing trajectories smaller than 0.3 deg, by suggesting that it reflects spatiotemporal computation and motion-tracking. These two mechanisms allow visual imagery to form of the point symmetrical to that of the disappearance, with respect to fixation, and then for the occluded moving target to be tracked up to this point.

Extrapolation occurs in multiple object tracking when eye movements are controlled

There is much debate regarding the types of information observers use to track moving objects. Howe and Holcombe (Journal of Vision 12(13): 1-10, 2012) recently reported evidence that observers employ extrapolation while tracking. However, their study is potentially confounded because it did not control for eye movements. As eye movements can aid extrapolation, it is unclear whether extrapolation can still occur in multiple object tracking (MOT) when eye movements are eliminated. In the current study, we addressed this question using an eye tracker to ensure that fixation was always maintained on a central fixation point while observers performed a tracking task. In the predictable condition, objects always travelled along linear paths. In the unpredictable condition, objects randomly changed direction every 300-600 ms. If observers employ extrapolation, we would expect performance to be greater in the former condition than in the latter condition. Our results showed that observers did indeed perform better in the predictable condition than in the unpredictable condition, at least when tracking just two objects (Experiments 1, 3, and 4). Extrapolation occurred less when tracking loads increased or when the objects moved more slowly (Experiment 2).

Aging and audio-visual and multi-cue integration in motion

The perception of naturalistic events relies on the ability to integrate information from multiple sensory systems, an ability that may change with healthy aging. When two objects move toward and then past one another, their trajectories are perceptually ambiguous: the objects may seem to stream past one another or bounce off one another. Previous research showed that auditory or visual events that occur at the time of disks' coincidence could bias the percept toward bouncing or streaming. We exploited this malleable percept to assay age-related changes in the integration of multiple inter- and intra-modal cues. The disks' relative luminances were manipulated to produce stimuli strongly favoring either bouncing or streaming, or to produce ambiguous motion (equal luminances). A sharp sound coincident with the disks' overlap increased both groups' perception of bouncing, but did so significantly less for older subjects. An occluder's impact on motion perception varied with its duration: a long duration occluder promoted streaming in both groups; a brief occluder promoted bouncing in younger subjects, but not older ones. Control experiments demonstrated that the observed differences between younger and older subjects resulted from neither age-related changes in retinal illuminance nor age-related changes in hearing, pointing to weakened inter- and intra-modal integration with aging. These changes could contribute to previously demonstrated age-related perceptual and memory deficits.

Swapping or dropping? Electrophysiological measures of difficulty during multiple object tracking

In the multiple object tracking task, participants are asked to keep targets separate from identical distractors as all items move randomly. It is well known that simple manipulations such as object speed and number of distractors dramatically alter the number of targets that are successfully tracked, but very little is known about what causes this variation in performance. One possibility is that participants tend to lose track of objects (dropping) more frequently under these conditions. Another is that the tendency to confuse a target with a distractor increases (swapping). These two mechanisms have very different implications for the attentional architecture underlying tracking. However, behavioral data alone cannot differentiate between these possibilities. In the current study, we used an electrophysiological marker of the number of items being actively tracked to assess which type of errors tended to occur during speed and distractor load manipulations. Our neural measures suggest that increased distractor load led to an increased likelihood of confusing targets with distractors while increased speed led to an increased chance of a target item being dropped. Behavioral experiments designed to test this novel prediction support this assertion.

Visually Tracking and Localizing Expanding and Contracting Objects

The maintenance of attention on moving objects is required for cognition to reliably engage with the visual world. Theories of object tracking need to explain on which patterns of visual stimulation one can easily maintain attention and on which patterns one cannot. A previous study has shown that it is easier to track rigid objects than objects that expand and contract along their direction of motion, in a manner that resembles a substance pouring from one location to another (vanMarle and Scholl 2003 Psychological Science14 498–504). Here we investigate six possible explanations for this finding and find evidence supporting two of them. Our results show that, first, objects that expand and contract tend to overlap and crowd each other more, and this increases tracking difficulty. Second, expansion and contraction make it harder to localize objects, even when there is only a single target to attend to, and this may also increase tracking difficulty. Currently, there is no theory of object tracking that can account for the second finding.

A simple proximity heuristic allows tracking of multiple objects through occlusion

Moving objects in the world present a challenge to the visual system, in that they often move in and out of view as they are occluded by other surfaces. Nevertheless, the ability to track multiple objects through periods of occlusion is surprisingly robust. Here, we identify a simple heuristic that underlies this ability: Pre- and postocclusion views of objects are linked together solely by their spatial proximity. Tracking through occlusion was always improved when the postocclusion instances reappeared closer to the preocclusion views. Strikingly, this was true even when objects' previous trajectories predicted different reappearance locations and when objects reappeared "too close," from invisible "slits" in empty space, rather than from more distant occluder contours. Tracking through occlusion appears to rely only on spatial proximity, and not on encoding heading information, likely reappearance locations, or the visible structure of occluders.

Evidence for a shared mechanism used in multiple-object tracking and subitizing

It has been proposed that the mechanism that supports the ability to keep track of multiple moving objects also supports subitizing--the ability to quickly and accurately enumerate a small set of objects. To test this hypothesis, we investigated the effects on subitizing when human observers were required to perform a multiple object tracking task and an enumeration task simultaneously. In three experiments, participants (Exp. 1, N = 24; Exp. 2, N = 11; Exp. 3, N = 37) enumerated sets of zero to nine squares that were flashed while they tracked zero, two, or four moving discs. The results indicated that the number of items participants could subitize decreased by one for each item they tracked. No such pattern was seen when the enumeration task was paired with an equally difficult, but nonvisual, working memory task. These results suggest that a shared visual mechanism supports multiple object tracking and subitizing.

Spatial Reference in Multiple Object Tracking

Spatial reference in multiple object tracking is available from configurations of dynamic objects and static reference objects. In three experiments, we studied the use of spatial reference in tracking and in relocating targets after abrupt scene rotations. Observers tracked 1, 2, 3, 4, and 6 targets in 3D scenes, in which white balls moved on a square floor plane. The floor plane was either visible thus providing static spatial reference or it was invisible. Without scene rotations, the configuration of dynamic objects provided sufficient spatial reference and static spatial reference was not advantageous. In contrast, with abrupt scene rotations of 20°, static spatial reference supported in relocating targets. A wireframe floor plane lacking local visual detail was as effective as a checkerboard. Individually colored geometric forms as static reference objects provided no additional benefit either, even if targets were centered on these forms at the abrupt scene rotation. Individualizing the dynamic objects themselves by color for a brief interval around the abrupt scene rotation, however, did improve performance. We conclude that attentional tracking of moving targets proceeds within dynamic configurations but detached from static local background.

Multiple-Object Tracking

The Multiple-Object Tracking paradigm has most commonly been utilized to investigate how subsets of targets can be tracked from among a set of identical objects. Recently, this research has been extended to examine the function of featural information when tracking is of objects that can be individuated. We report on a study whose findings suggest that, while participants can only hold featural information for roughly two targets this task does not affect tracking performance detrimentally and points to a discontinuity between the cognitive processes that subserve spatial location and featural information.

Tracking planets and moons: mechanisms of object tracking revealed with a new paradigm

People can attend to and track multiple moving objects over time. Cognitive theories of this ability emphasize location information and differ on the importance of motion information. Results from several experiments have shown that increasing object speed impairs performance, although speed was confounded with other properties such as proximity of objects to one another. Here, we introduce a new paradigm to study multiple object tracking in which object speed and object proximity were manipulated independently. Like the motion of a planet and moon, each target-distractor pair rotated about both a common local point as well as the center of the screen. Tracking performance was strongly affected by object speed even when proximity was controlled. Additional results suggest that two different mechanisms are used in object tracking--one sensitive to speed and proximity and the other sensitive to the number of distractors. These observations support models of object tracking that include information about object motion and reject models that use location alone.

The role of visual attention in multiple object tracking: evidence from ERPs

We examined the role of visual attention in the multiple object tracking (MOT) task by measuring the amplitude of the N1 component of the event-related potential (ERP) to probe flashes presented on targets, distractors, or empty background areas. We found evidence that visual attention enhances targets and suppresses distractors (Experiment 1 & 3). However, we also found that when tracking load was light (two targets and two distractors), accurate tracking could be carried out without any apparent contribution from the visual attention system (Experiment 2). Our results suggest that attentional selection during MOT is flexibly determined by task demands as well as tracking load and that visual attention may not always be necessary for accurate tracking.

Direction information in multiple object tracking is limited by a graded resource

Is multiple object tracking (MOT) limited by a fixed set of structures (slots), a limited but divisible resource, or both? Here, we answer this question by measuring the precision of the direction representation for tracked targets. The signature of a limited resource is a decrease in precision as the square root of the tracking load. The signature of fixed slots is a fixed precision. Hybrid models predict a rapid decrease to asymptotic precision. In two experiments, observers tracked moving disks and reported target motion direction by adjusting a probe arrow. We derived the precision of representation of correctly tracked targets using a mixture distribution analysis. Precision declined with target load according to the square-root law up to six targets. This finding is inconsistent with both pure and hybrid slot models. Instead, directional information in MOT appears to be limited by a continuously divisible resource.

Multiple-object tracking: enhanced visuospatial representations as a result of experience

Research has demonstrated that individuals who routinely engage in complex visuospatial tasks (e.g., radar operators) show an enhanced ability to track multiple randomly moving targets. This study examined tracking expertise using members of a University Officer Training Corps (OTCs) who regularly engage in tasks requiring good dynamic spatial cognition. As expected, the results show that OTCs have enhanced tracking ability relative to other undergraduates. More importantly, they support the idea that, while one set of executive processes are involved in the moment-by-moment updating of the visuospatial representations necessary for dynamic, multiple-object tracking, other processes are activated when whole object sets disappear simultaneously, to create a long-term memory trace of the objects' locations at the moment of their disappearance. Expertise only arose in the former processes, but was lost after a short decay period, such as occurred with a delayed response.

Looking at the center of the targets helps multiple object tracking

The ability to move our gaze to locations of interest facilitates interactions in everyday life. Where do participants direct gaze when multiple locations are of interest simultaneously? We previously demonstrated that, when tracking several moving targets amidst distractors in a multiple object tracking (MOT) task, participants primarily looked at a central point in between the targets (H. M. Fehd & A. E. Seiffert, 2008). This strategy of center-looking is in contrast to a target-looking strategy where participants would saccade from target to target. Here we investigated what factors influence the use of center-looking as well as its effectiveness. By decreasing object speed, we determined that center-looking is not a result of avoiding costly eye movements during tracking. Decreasing object size showed that peripheral visibility is necessary for tracking, but that center-looking continues up to the limits of peripheral visibility. Further analysis revealed that participants often engaged in both target-looking and center-looking by switching gaze from the center to targets and back again. Directly comparing participants' performance when they either did or did not include center-looking along with target-looking revealed that center-looking facilitates tracking performance. These results suggest that there is value in looking at the center that relates directly to the process of tracking multiple objects.

Influence of depth cues on multiple objects tracking in 3D scene

Multiple-object-tracking tasks require an observer to track a group of identical objects moving in 2D space. The current study was conducted in an attempt to examine object tracking in 3D space. We were interested in testing influence of classical depth cues (texture gradients, relative size and contrast) on tracking. In Experiment 1 we varied the presence of these depth cues while subjects were tracking four (out of eight) identical, moving objects. Texture gradient, a cue related to scene layout, did not influence object tracking. Experiment 2 was designed to clarify the differences between contrast and relative size effects. Results revealed that contrast was a more effective cue for multiple object tracking in 3D scenes. The effect of occlusion was also examined. Several occluders, presented in the scene, were occasionally masking the targets. Tracking was more successful when occluders were arranged in different depth planes, mimicking more natural conditions. Increasing the number of occlusions led to poorer performance.

The role of location and motion information in the tracking and recovery of moving objects

Observers in a multiple object tracking task can track about four to five independently moving targets among several moving distractors, even if all of the stimuli disappear for a 300-msec gap. How observers reacquire targets following such a gap reveals what kind of information they can maintain for targets. Previous research has suggested that participants maintain minimal information about a set of moving objects--namely, just their present spatial locations. We report five new experiments that demonstrate retention of location information for at least four objects, and extrapolated motion information for around two objects.

Perceptual completion in a dynamic scene: an investigation with an ambiguous motion paradigm

In this study we employed the streaming-bouncing stimulus to investigate aspects of dynamic occlusion, e.g., of objects that temporarily move under occlusion while covertly being tracked. Two occluders, both either luminance-defined or invisible (virtual), were placed on the trajectories of the moving objects in the streaming-bouncing stimulus. We found that the bouncing percept was dominant when the objects moved under luminance-defined occluders but not when they moved under virtual occluders. Perceived motion direction thus varied with occluder visibility. The results seem to suggest that perceptual completion of a moving object interferes with constant motion processing of the same object.

Multiple object juggling: Changing what is tracked during extended multiple object tracking

The multiple object tracking (MOT) task has been a useful tool for studying the deployment of limited-capacity visual resources over time. Since it involves sustained attention to multiple objects, this task is a promising model for real-world visual cognition. However, real-world tasks differ in two critical ways from standard laboratory MOT designs. First, in real-world tracking, it is unusual for the set of tracked items to be identified all at once and to remain unchanged over time. Second, real-world tracking tasks may need to be sustained over a period of minutes, and not mere seconds. How well is MOT performance maintained over extended periods of time? In four experiments, we demonstrate that observers can dynamically "juggle" objects in and out of the tracked set with little apparent cost, and can sustain this performance for up to 10 min at a time. This performance requires implicit or explicit feedback. In the absence of feedback, performance tracking drops steadily over the course of several minutes.

Tracking unique objects

Is content addressable in the representation that subserves performance in multiple-object-tracking (MOT) experiments? We devised an MOT variant that featured unique, nameable objects (cartoon animals) as stimuli. There were two possible response modes: standard, in which observers were asked to report the locations of all target items, and specific, in which observers had to report the location of a particular object (e.g., "Where is the zebra?"). A measure of capacity derived from accuracy allowed for comparisons of the results between conditions. We found that capacity in the specific condition (1.4 to 2.6 items across several experiments) was always reliably lower than capacity in the standard condition (2.3 to 3.4 items). Observers could locate specific objects, indicating a content-addressable representation. However, capacity differences between conditions, as well as differing responses to the experimental manipulations, suggest that there may be two separate systems involved in tracking, one carrying only positional information, and one carrying identity information as well.