Studying visual attention using the multiple object tracking paradigm: A tutorial review

A hidden Markov model for analyzing eye-tracking of moving objects : Case study in a sustained attention paradigm

Eye-tracking provides an opportunity to generate and analyze high-density data relevant to understanding cognition. However, while events in the real world are often dynamic, eye-tracking paradigms are typically limited to assessing gaze toward static objects. In this study, we propose a generative framework, based on a hidden Markov model (HMM), for using eye-tracking data to analyze behavior in the context of multiple moving objects of interest. We apply this framework to analyze data from a recent visual object tracking task paradigm, TrackIt, for studying selective sustained attention in children. Within this paradigm, we present two validation experiments to show that the HMM provides a viable approach to studying eye-tracking data with moving stimuli, and to illustrate the benefits of the HMM approach over some more naive possible approaches. The first experiment utilizes a novel 'supervised' variant of TrackIt, while the second compares directly with judgments made by human coders using data from the original TrackIt task. Our results suggest that the HMM-based method provides a robust analysis of eye-tracking data with moving stimuli, both for adults and for children as young as 3.5-6 years old.

Effectiveness of "rescue saccades" on the accuracy of tracking multiple moving targets: An eye-tracking study on the effects of target occlusions

Occlusion is one of the main challenges in tracking multiple moving objects. In almost all real-world scenarios, a moving object or a stationary obstacle occludes targets partially or completely for a short or long time during their movement. A previous study (Zelinsky & Todor, 2010) reported that subjects make timely saccades toward the object in danger of being occluded. Observers make these so-called “rescue saccades” to prevent target swapping. In this study, we examined whether these saccades are helpful. To this aim, we used as the stimuli recorded videos from natural movement of zebrafish larvae swimming freely in a circular container. We considered two main types of occlusion: object-object occlusions that naturally exist in the videos, and object-occluder occlusions created by adding a stationary doughnut-shape occluder in some videos. Four different scenarios were studied: (1) no occlusions, (2) only object-object occlusions, (3) only object-occluder occlusion, or (4) both object-object and object-occluder occlusions. For each condition, two set sizes (two and four) were applied. Participants’ eye movements were recorded during tracking, and rescue saccades were extracted afterward. The results showed that rescue saccades are helpful in handling object-object occlusions but had no reliable effect on tracking through object-occluder occlusions. The presence of occlusions generally increased visual sampling of the scenes; nevertheless, tracking accuracy declined due to occlusion.

How Do Humans Perform in Multiple Object Tracking With Unstable Features

In real-world scenarios, objects’ surface features sometimes change as they move, impairing the continuity of objects. However, it is still unknown how our visual system adapts to this dynamic change. Hence, the present study investigated the role of feature changes in attentive tracking through a modified multiple object tracking (MOT) task. The feature heterogeneity and feature stability were manipulated in two experiments. The results from Experiment 1 showed that the tracking performance under feature-changed condition was lower than that under the feature-fixed condition only when the objects were four colors grouped or all unique, suggesting that the performance decrease was moderated by the feature heterogeneity. In Experiment 2, we further examined this effect by manipulating the frequency of feature change. The results showed that when the target set was one color or two colors grouped (the color grouping for the distractor set corresponded with it), the tracking performance decreased significantly as the feature-change frequency increased. However, this trend was not the case when the objects were of the same color or eight unique colors. In addition, a relatively consistent effect appeared both in Experiments 1 and 2. When objects have unique features, the tracking performance decreased significantly as the increase of feature heterogeneity in each frequency of feature changes. Taken together, we concluded that unstable features could be utilized in attentive tracking, and the extent to which the observers relied on surface feature information to assist tracking depended on the level of feature heterogeneity and the frequency of feature change.

Dynamic visual attention characteristics and their relationship to match performance in skilled basketball players

Background

Dynamic visual attention is important in basketball because it may affect the performance of players and thus the match outcome. The goals of this study were to investigate the difference in dynamic visual attention characteristics between highly skilled basketball players and nonathletic college students and to explore the relationship between visual attention and game-related performance among the basketball players.

Methods

In total, 24 highly skilled basketball players and 24 nonathletic college students participated in a multiple object tracking task. The task was conducted so that either the number of targets that were visually tracked or the speed at which a given number of tracked targets moved was altered to examine the difference in dynamic visual attention characteristics between the basketball players and nonathletic college students. The relationship between visual tracking speed (VTS) and game-related statistics, including assists, steals, mistakes, fouls and points scored recorded for every match during the season, was assessed among the basketball players by using Pearson correlations.

Results

A significant main effect of target tracking load was observed (P < 0.001), with visual tracking performance significantly decreased as target number increased. In addition, the speed at which the targets moved had a significant effect on visual tracking performance (P < 0.001), with tracking performance significantly decreased as target speed increased. However, no significant difference was observed in the abilities of basketball players and nonathletic college students to simultaneously track up to six targets. By contrast, a significant interaction between group and target speed was found (P < 0.001), with the visual tracking accuracy of basketball players significantly greater than that of college students at the higher target speeds examined (P < 0.001). Among basketball players, there were positive, large, and statistically significant correlations in the accuracy in VTS trials and the number of assists (P < 0.001) and between the accuracy in VTS trials and the number of steals (P < 0.001).

Conclusion

The advantage of skilled basketball players to handle dynamic visual information in a multiple object tracking task was not attributable to the target number but to the target speed. Those athletes with greater dynamic visual attention were more likely to successfully assist or to steal the ball, enhancing performance of the athlete as well as contributing to a more successful team match. These findings may inform basketball training programs to improve player and team performances during matches.

Selection enables enhancement: An integrated model of object tracking

The diversity of research on visual attention and multiple-object tracking presents challenges for anyone hoping to develop a unified account. One key challenge is identifying the attentional limitations that give rise to competition among targets during tracking. To address this challenge, we present a computational model of object tracking that relies on two attentional mechanisms: serial selection and parallel enhancement. Selection picks out an object for further processing, whereas enhancement increases sensitivity to stimuli in regions where objects have been selected previously. In this model, multiple target locations can be tracked in parallel via enhancement, whereas a single target can be selected so that additional information beyond its location can be processed. In simulations of two psychological experiments, we demonstrate that spatial competition during enhancement and temporal competition for selection can explain a range of findings on multiple-object tracking, and we argue that the interaction between selection and enhancement captured in the model is critical to understanding attention more broadly.

Individual differences in multiple object tracking, attentional cueing, and age account for variability in the capacity of audiovisual integration

There has been a recent increase in individual differences research within the field of audiovisual perception (Spence & Squire, 2003, Current Biology, 13(13), R519-R521), and furthering the understanding of audiovisual integration capacity with an individual differences approach is an important facet within this line of research. Across four experiments, participants were asked to complete an audiovisual integration capacity task (cf. Van der Burg, Awh, & Olivers, 2013, Psychological Science, 24(3), 345-351; Wilbiks & Dyson, 2016, PLOS ONE 11(12), e0168304; 2018, Journal of Experimental Psychology: Human Perception and Performance, 44(6), 871-884), along with differing combinations of additional perceptual tasks. Experiment 1 employed a multiple object tracking task and a visual working memory task. Experiment 2 compared performance on the capacity task with that of the Attention Network Test. Experiment 3 examined participants' focus in space through a Navon task and vigilance through time. Having completed this exploratory work, in Experiment 4 we collected data again from the tasks that were found to correlate significantly across the first three experiments and entered them into a regression model to predict capacity. The current research provides a preliminary explanation of the vast individual differences seen in audiovisual integration capacity in previous research, showing that by considering an individual's multiple object tracking span, focus in space, and attentional factors, we can account for up to 34.3% of the observed variation in capacity. Future research should seek to examine higher-level differences between individuals that may contribute to audiovisual integration capacity, including neurodevelopmental and mental health differences.

Integration of Position and Predictive Motion Signals in Aging Vision

We examined the effect of aging on the integration of position and motion signals, which is essential for tracking visual objects, using the motion-induced position shift (MIPS) phenomenon. We first measured the MIPS and bias in speed perception at three eccentricities. Both young and older adults showed the increasing MIPS and decreasing perceived speed as the eccentricity increased, which is consistent with previous literature. More importantly, we found that the mean MIPS was 2.87 times larger in older adults, and the response variability in position tasks showed a larger difference between age groups compared with the difference in speed tasks. We then measured the MIPS across stimulus durations. Temporal changes in the MIPS showed similar patterns in young and older adults in that the MIPS initially peaked at around 60 ms and approached an asymptote. We further analyzed the changes in response variability across stimulus durations to estimate sensory noise and propagation noise separately and found that only sensory noise was significantly larger in older adults. The overall results suggest that the increased MIPS in older adults is due to the increased dependency on predictive motion signals to compensate for the relatively imprecise position signals, which in turn implies that older adults would depend more on the motion signals to track objects.

Evaluating spatiotemporal integration of shape cues

Prior work has shown that humans can successfully identify letters that are constructed with a sparse array of dots, wherein the dot pattern reflects the strokes that would normally be used to fashion a given letter. In the present work the dots were briefly displayed, one at a time in sequence, varying the spatial order in which they were shown. A forward sequence was spatially ordered as though one were passing a stroke across the dots to connect them. Experiments compared this baseline condition to the following three conditions: a) the dot sequence was spatially ordered, but in the reverse direction from how letter strokes might normally be written; b) the dots in each stroke of the letter were displayed in a random order; c) the sequence of displayed dots were chosen for display from any location in the letter. Significant differences were found between the baseline condition and all three of the comparison conditions, with letter recognition being far worse for the random conditions than for conditions that provided consistent spatial ordering of dot sequences. These findings show that spatial order is critical for integration of shape cues that have been sequentially displayed.

Multiple object-tracking isolates feedback-specific load in attention and learning

Feedback is beneficial for learning. Nevertheless, it remains unclear whether (i) feedback draws attentional resources when integrated and (ii) the benefits of feedback for learning can be demonstrated using an attention-based task. We therefore (i) isolated feedback-specific load from task-specific load via individual differences in attention resource capacity and (ii) examined the effect of trial-by-trial feedback (i.e., present vs. absent) on learning a multiple object-tracking (MOT) paradigm. We chose MOT because it is a robust measure of attention resource capacity. In Study 1 participants tracked one (i.e., lowest attentional load condition) through four target items (i.e., highest load condition) among eight total items. One group (n = 32) received trial-by-trial feedback whereas the other group (n = 32) did not. The absence of feedback resulted in better MOT performance compared with the presence of feedback. Moreover, the difference in MOT capability between groups increased as the task-specific attentional load increased. These findings suggest that feedback integration requires attentional resources. Study 2 examined whether the absence (n = 19) or presence (n = 19) of feedback affects learning on the same MOT task across four testing days. When holding task-specific load constant, improvement in MOT was greater with feedback than without. Although this study is the first to isolate feedback-specific load in attention with MOT, more evidence is needed to demonstrate how the benefits of feedback translate to improvement on an attention-based task. These findings encourage future research to further explore the interaction between feedback, attention and learning.

Examining the roles of working memory and visual attention in multiple object tracking expertise

When tracking multiple moving targets among visually similar distractors, human observers are capable of distributing attention over several spatial locations. It is unclear, however, whether capacity limitations or perceptual-cognitive abilities are responsible for the development of expertise in multiple object tracking. Across two experiments, we examined the role of working memory and visual attention in tracking expertise. In Experiment 1, individuals who regularly engaged in object tracking sports (soccer and rugby) displayed improved tracking performance, relative to non-tracking sports (swimming, rowing, running) (p = 0.02, ηp2 = 0.163), but no differences in gaze strategy (ps > 0.31). In Experiment 2, participants trained on an adaptive object tracking task showed improved tracking performance (p = 0.005, d = 0.817), but no changes in gaze strategy (ps > 0.07). They did, however, show significant improvement in a working memory transfer task (p < 0.001, d = 0.970). These findings indicate that the development of tracking expertise is more closely linked to processing capacity limits than perceptual-cognitive strategies.

Multiple-target tracking in human and machine vision

Humans are able to track multiple objects at any given time in their daily activities—for example, we can drive a car while monitoring obstacles, pedestrians, and other vehicles. Several past studies have examined how humans track targets simultaneously and what underlying behavioral and neural mechanisms they use. At the same time, computer-vision researchers have proposed different algorithms to track multiple targets automatically. These algorithms are useful for video surveillance, team-sport analysis, video analysis, video summarization, and human–computer interaction. Although there are several efficient biologically inspired algorithms in artificial intelligence, the human multiple-target tracking (MTT) ability is rarely imitated in computer-vision algorithms. In this paper, we review MTT studies in neuroscience and biologically inspired MTT methods in computer vision and discuss the ways in which they can be seen as complementary.

Multiple-target tracking (MTT) is a challenging task vital for both a human’s daily life and for many artificial intelligent systems, such as those used for urban traffic control. Neuroscientists are interested in discovering the underlying neural mechanisms that successfully exploit cognitive resources, e.g., spatial attention or memory, during MTT. Computer-vision specialists aim to develop powerful MTT algorithms based on advanced models or data-driven computational methods. In this paper, we review MTT studies from both communities and discuss how findings from cognitive studies can inspire developers to construct higher performing MTT algorithms. Moreover, some directions have been proposed through which MTT algorithms could raise new questions in the cognitive science domain, and answering them can shed light on neural processes underlying MTT.

Attention in visually typical and amblyopic children

Amblyopia is a cortical visual disorder caused by unequal visual input to the brain from the two eyes during development. Amblyopes show reduced visual acuity and contrast sensitivity and abnormal binocularity, as well as more “global” perceptual losses, such as figure-ground segregation and global form integration. Currently, there is no consensus on the neural basis for these higher-order perceptual losses. One contributing factor could be that amblyopes have deficiencies in attention, such that the attentional processes that control the selection of information favor the better eye. Previous studies in amblyopic adults are conflicting as to whether attentional deficits exist. However, studies where intact attentional ability has been shown to exist were conducted in adults; it is possible that it was acquired through experience. To test this hypothesis, we studied attentional processing in amblyopic children. We examined covert endogenous attention using a classical spatial cueing paradigm in amblyopic and visually typical 5- to 10-year old children. We found that all children, like adults, independently of visual condition, benefited from attentional cueing: They performed significantly better on trials with an informative (valid) cue than with the uninformative (neutral) cue. Response latencies were also significantly shorter for the valid cue condition. No statistically significant difference was found between the performance of the amblyopic and the visually typical children or between dominant and nondominant eyes of all children. The results showed that covert spatial attention is intact in amblyopic and visually typical children and is therefore not likely to account for higher-order perceptual losses in amblyopic children.

Multiple identity tracking strategies vary by age: An ERP study

Top-down modulation underlies our ability to focus attention on task-relevant stimuli and ignore irrelevant distractions. Although age-related differences in neural correlates of top-down modulation have been investigated in multiple studies using variety of tasks (Gazzaley et al., 2005; Störmer et al., 2013), the effect of age on top-down modulation in a multiple identity tracking (MIT) task is still unknown. Thus, we investigated age-related differences in the MIT task by employing event-related potentials (ERPs). Participants tracked ten uniquely colored disks, two of which were randomly designated as targets at the beginning of each trial; the targets moved among four stationary distractors (serving as ERP baseline) and four moving distractors. Each type of stimulus was probed during the trial to capture differential patterns of brain activation. Tracking performance was similar across age groups. ERP data showed that younger adults performed the MIT task by enhancing the unique identities associated with targets relative to distractors through feature-based tracking. Older adults showed a pattern of distractor suppression engaging both location- and feature-based tracking strategies. Thus, our findings suggest that compared to younger adults, older adults engage greater levels of neural activity to achieve the same level of performance. These findings are discussed in light of theories of cognitive aging.

Distinct Neural Mechanisms Meet Challenges in Dynamic Visual Attention due to Either Load or Object Spacing

When engaged in dynamic visuospatial tasks, the brain copes with perceptual and cognitive processing challenges. During multiple-object tracking (MOT), the number of objects to be tracked (i.e., load) imposes attentional demands, but so does spatial interference from irrelevant objects (i.e., close encounters). Presently, it is not clear whether the effect of load on accuracy solely depends on the number of close encounters. If so, the same cognitive and physiological mechanisms deal with increasing load by preparing for and dealing with spatial interference. However, this has never been directly tested. Such knowledge is important to understand the neurophysiology of dynamic visual attention and resolve conflicting views within visual cognition concerning sources of capacity limitations. We varied the processing challenge in MOT task in two ways: the number of targets and the minimum spatial proximity between targets and distractors. In a first experiment, we measured task-induced pupil dilations and saccades during MOT. In a separate cohort, we measured fMRI activity. In both cohorts, increased load and close encounters (i.e., close spatial proximity) led to reduced accuracy in an additive manner. Load was associated with pupil dilations, whereas close encounters were not. Activity in dorsal attentional areas and frequency of saccades were proportionally larger both with higher levels of load and close encounters. Close encounters recruited additionally ventral attentional areas that may reflect orienting mechanisms. The activity in two brainstem nuclei, ventral tegmental area/substantia nigra and locus coeruleus, showed clearly dissociated patterns. Our results constitute convergent evidence indicating that different mechanisms underlie processing challenges due to load and object spacing.

Goal-directed unequal attention allocation during multiple object tracking

In standard multiple object tracking (MOT) tasks the relative importance of the targets being tracked is equal. This is atypical of everyday situations in which an individual may need to prioritize one target relative to another and so allocate attention unequally. We report three experiments that examined whether participants could unequally split attention using a modified MOT task in which target priority was manipulated. Specifically, we examined the effect of priority on participants’ magnitude of error and used a distribution mixture analysis to investigate how priority affected both participants’ probability of losing an item and tracking precision. Experiment 1 (trajectory tracking) revealed a higher magnitude of error and higher proportion of guessing for low- compared with high-priority targets. Experiments 2 (trajectory tracking) and 3 (position tracking) examined how fine-grained this ability is by manipulating target priority at finer increments. In line with Experiment 1, results from both these experiments indicated that participants could split attention unequally. There was some evidence that participants could allocate attention unequally at fine increments, but this was less conclusive. Taken together, these experiments demonstrate participants’ ability to distribute attention unequally across multiple moving objects but suggest some limitation with the flexibility of attention allocation.

How selective attention affects the detection of motion changes with peripheral vision in MOT

In sports, peripheral vision is expected to play an important role in tasks that demand distributed attention and motion-change detection. By using the Multiple-Object-Tracking (MOT) task, these demands were simulated in a well-controlled laboratory environment. Participants tracked four target out of ten moving objects (6 distractors) and pressed a button when one of the ten objects stopped. Detection rates for tracked targets were compared to detection rates of non-tracked distractors at eccentricities between 5° and 25°. The study's aim was to test how the location of attention affects peripheral motion detection. The results show a large attention effect because target stops were detected in 89 % and distractor stops only in 55 % of the trials. Distractor stops were more likely detected when they occurred closer to the fovea while target stops were detected at all eccentricities. That means, orienting attention at target objects facilitates the peripheral detection of their motion changes in monitoring tasks. Having distractors closer to the fovea increases the chance to also detect motion changes of unattended objects. On a theoretical level, results support a tracking mechanism with object-based attention, serial covert attention shifts and flexible but limited attentional resources. On a practical level, sports' experts should use their extensive knowledge to locate attention on most-relevant objects and reduce the eccentricity to other objects to detect motion changes of attended and unattended objects.

Eye Behavior During Multiple Object Tracking and Multiple Identity Tracking

We review all published eye-tracking studies to date that have used eye movements to examine multiple object (MOT) or multiple identity tracking (MIT). In both tasks, observers dynamically track multiple moving objects. In MOT the objects are identical, whereas in MIT they have distinct identities. In MOT, observers prefer to fixate on blank space, which is often the center of gravity formed by the moving targets (centroid). In contrast, in MIT observers have a strong preference for the target-switching strategy, presumably to refresh and maintain identity-location bindings for the targets. To account for the qualitative differences between MOT and MIT, two mechanisms have been posited, a position tracking (MOT) and an identity tracking (MOT & MIT) mechanism. Eye-tracking studies of MOT have also demonstrated that observers execute rescue saccades toward targets in danger of becoming occluded or are about to change direction after a collision. Crowding attracts the eyes close to it in order to increase visual acuity for the crowded objects to prevent target loss. It is suggested that future studies should concentrate more on MIT, as MIT more closely resembles tracking in the real world.

Two-year-olds succeed at MIT: Multiple identity tracking in 20- and 25-month-old infants

Infants' ability to remember objects and their locations emerges during the first year of life. However, not much is known about infants' ability to track objects' identities in a dynamic environment. Here, we tailored the delayed match retrieval eye-tracking paradigm to study infants' ability to track two object identities during occlusion-an infant version of multiple identity tracking (MIT). Delayed match retrieval uses virtual "cards" as stimuli that are first shown face up, exposing to-be-remembered information, and then turned face down, occluding it. Here, cards were subject to movement during the face-down occlusion period. We used complex non-nameable objects as card faces to discourage verbal rehearsal. In three experiments (N = 110), we compared infants' ability to track object identities when two previously exposed cards were static (Experiment 1), were moved into new positions along the same trajectory (Experiment 2), or were moved along different trajectories (Experiment 3) while face down. We found that 20-month-olds could remember two object identities when static; however, it was not until 25 months of age that infants could track when movement was introduced. Our results show that the ability to track multiple identities in visual working memory is present by 25 months.

Tracking within-category colors is easier: Color categories modulate location processing in a dynamic visual task

The categorical perception (CP) of color describes the phenomenon that colors across categories (e.g., blue and green) are more discriminable than within-category colors (e.g., green) even when the perceptual distance is controlled. While most studies are conducted in a static visual scene, the current study investigated the effect of color categories when tracking multiple colored objects within two experiments. The targets or distractors were either from the same color category or from two different categories, and the perceptual distance was controlled across all conditions. We found that location tracking was facilitated when the targets or the distractors shared the same color category in the tracking task, which required location tracking and color memorization at the same time, compared with when they were from two different categories (Experiment 1). By contrast, when location tracking was the sole demand and no explicit color processing was required, the target CP effects, but not the distractor CP effects, persisted (Experiment 2). We conclude that color categories can dynamically modulate preattention visual processing when color working memory is involved, but that attention is necessary to produce CP effects when color is irrelevant to the task.

Team ball sport participation is associated with performance in two sustained visual attention tasks: Position monitoring and target identification in rapid serial visual presentation streams

We investigated the relationship between participation in team ball sports and performance in two sustained spatiotemporal attention tasks-a position monitoring variant of the multiple object tracking (MOT) task and target identification in the rapid serial visual presentation (RSVP) task. Thirty participants were asked about the frequency of their participation in team ball sports and undertook both the MOT task and RSVP task. In the MOT task, participants viewed an array of eight discs as they moved unpredictably for 3-8s before disappearing. On each trial, a subset of these were marked as targets for tracking, meaning that participants attempted to keep track of their positions on the screen with as much precision as possible. At the end of each trial, participants reported the final perceived position of a queried target. In the RSVP task, a stream of letters was presented at the center of the screen and participants attempted to report either one or both of two target letters embedded in the stream. Participation in team ball sports was associated with superior performance in both tasks: in the MOT task, individuals reporting more time spent playing team ball sports on a weekly basis were able to report the positions of targets with greater precision, and in the RSVP task they reported targets more accurately. The current findings add to the literature of somewhat mixed findings regarding the extent to which participation in sports may be associated with superior visual attention abilities.

Mindful breath awareness meditation facilitates efficiency gains in brain networks: A steady-state visually evoked potentials study

The beneficial effects of mindfulness-based therapeutic interventions have stimulated a rapidly growing body of scientific research into underlying psychological processes. Resulting evidence indicates that engaging with mindfulness meditation is associated with increased performance on a range of cognitive tasks. However, the mechanisms promoting these improvements require further investigation. We studied changes in behavioural performance of 34 participants during a multiple object tracking (MOT) task that taps core cognitive processes, namely sustained selective visual attention and spatial working memory. Concurrently, we recorded the steady-state visually evoked potential (SSVEP), an EEG signal elicited by the continuously flickering moving objects, and indicator of attentional engagement. Participants were tested before and after practicing eight weeks of mindful breath awareness meditation or progressive muscle relaxation as active control condition. The meditation group improved their MOT-performance and exhibited a reduction of SSVEP amplitudes, whereas no such changes were observed in the relaxation group. Neither group changed in self-reported positive affect and mindfulness, while a marginal increase in negative affect was observed in the mindfulness group. This novel way of combining MOT and SSVEP provides the important insight that mindful breath awareness meditation may lead to refinements of attention networks, enabling more efficient use of attentional resources.