The influence of clutter on real-world scene search: Evidence from search efficiency and eye movements

Impact of the motion and visual complexity of the background on players' performance in video game-like displays

The visual interfaces of virtual environments such as video games often show scenes where objects are superimposed on a moving background. Three experiments were designed to better understand the impact of the complexity and/or overall motion of two types of visual backgrounds often used in video games on the detection and use of superimposed, stationary items. The impact of background complexity and motion was assessed during two typical video game tasks: a relatively complex visual search task and a classic, less demanding shooting task. Background motion impaired participants' performance only when they performed the shooting game task, and only when the simplest of the two backgrounds was used. In contrast, and independently of background motion, performance on both tasks was impaired when the complexity of the background increased. Eye movement recordings demonstrated that most of the findings reflected the impact of low-level features of the two backgrounds on gaze control.

Looking for trouble: a description of oculomotor search strategies during live CCTV operation

Recent research has begun to address how CCTV operators in the modern control room attempt to search for crime (e.g., Howard et al., 2011). However, an often-neglected element of the CCTV task is that the operators have at their disposal a multiplexed wall of scenes, and a single spot-monitor on which they can select any of these feeds for inspection. Here we examined how 2 trained CCTV operators used these sources of information to search from crime during a morning, afternoon, and night-time shift. We found that they spent surprisingly little time viewing the multiplex wall, instead preferentially spending most of their time searching on the single-scene spot-monitor. Such search must require a sophisticated understanding of the surveilled environment, as the operators must make their selection of which screen to view based on their prediction of where crime is likely to occur. This seems to be reflected in the difference in the screens that they selected to view at different times of the day. For example, night-clubs received close monitoring at night, but were seldom viewed in mid-morning. Such narrowing of search based on a contextual understanding of an environment is not a new idea (e.g., Torralba et al., 2006), and appears to contribute to operator's selection strategy. This research prompts new questions regarding the nature of representation that operators have of their environment, and how they might develop expectation-based search strategies to countermand the demands of the large influx of visual information. Future research should ensure not to neglect examination of operator behavior "in the wild" (Hutchins, 1995a), as such insights are difficult to gain from laboratory based paradigms alone.

Where's Wally: the influence of visual salience on referring expression generation

REFERRING EXPRESSION GENERATION (REG) PRESENTS THE CONVERSE PROBLEM TO VISUAL SEARCH: given a scene and a specified target, how does one generate a description which would allow somebody else to quickly and accurately locate the target?Previous work in psycholinguistics and natural language processing has failed to find an important and integrated role for vision in this task. That previous work, which relies largely on simple scenes, tends to treat vision as a pre-process for extracting feature categories that are relevant to disambiguation. However, the visual search literature suggests that some descriptions are better than others at enabling listeners to search efficiently within complex stimuli. This paper presents a study testing whether participants are sensitive to visual features that allow them to compose such "good" descriptions. Our results show that visual properties (salience, clutter, area, and distance) influence REG for targets embedded in images from the Where's Wally? books. Referring expressions for large targets are shorter than those for smaller targets, and expressions about targets in highly cluttered scenes use more words. We also find that participants are more likely to mention non-target landmarks that are large, salient, and in close proximity to the target. These findings identify a key role for visual salience in language production decisions and highlight the importance of scene complexity for REG.

Parsing eye-tracking data of variable quality to provide accurate fixation duration estimates in infants and adults

Researchers studying infants' spontaneous allocation of attention have traditionally relied on hand-coding infants' direction of gaze from videos; these techniques have low temporal and spatial resolution and are labor intensive. Eye-tracking technology potentially allows for much more precise measurement of how attention is allocated at the subsecond scale, but a number of technical and methodological issues have given rise to caution about the quality and reliability of high temporal resolution data obtained from infants. We present analyses suggesting that when standard dispersal-based fixation detection algorithms are used to parse eye-tracking data obtained from infants, the results appear to be heavily influenced by interindividual variations in data quality. We discuss the causes of these artifacts, including fragmentary fixations arising from flickery or unreliable contact with the eyetracker and variable degrees of imprecision in reported position of gaze. We also present new algorithms designed to cope with these problems by including a number of new post hoc verification checks to identify and eliminate fixations that may be artifactual. We assess the results of our algorithms by testing their reliability using a variety of methods and on several data sets. We contend that, with appropriate data analysis methods, fixation duration can be a reliable and stable measure in infants. We conclude by discussing ways in which studying fixation durations during unconstrained orienting may offer insights into the relationship between attention and learning in naturalistic settings.

Eye Tracking Metrics: A Toolbox for Assessing the Effects of Clutter on Attention Allocation

Display clutter is a problem that affects operators in various data-rich environments. Clutter measurement techniques such as image processing and performance measures can provide an estimate of clutter but are largely not suited to tracing the effects of clutter on the dynamic allocation of attention. Eye tracking is a promising process-oriented tool that can help assess in real-time the attentional costs associated with the different aspects of clutter. In this experiment, we investigated which of a number of eye tracking metrics in the literature are sensitive to clutter. Twenty-two participants were asked to look for a target in static and dynamic images that were classified as either high or low in clutter. Response time and error rate were recorded, and an eye tracker was used to compute the identified eye tracking metrics. Results showed that, in both the static and dynamic conditions, a large number of eye tracking metrics were significantly affected by an increase in clutter. This suggests that eye tracking can be used to supplement other clutter measurement techniques by providing information about dynamic attention allocation.

Effects of Superposition on Oculomotor Guidance and Target Recognition

Searching for targets in complex scenes like X-ray baggage screening is demanding. One cause of impaired search performance is the superimposed objects of X-ray images. In this study, we compared searching for threat items in a superimposed display and a sparse display. Using both behavioral and eye movement measures, we explored the effects of superposition on search course. As expected, superposition affected accuracy rates and response times. Results showed searching in the superimposed display caused longer fixation durations, and the effect was pronounced throughout the search trial. We also found longer search initiation time and target verification time in superposition, suggesting that the initial planning of oculomotor guidance and target recognition was influenced.

TAM: Explaining off-object fixations and central fixation tendencies as effects of population averaging during search

Understanding how patterns are selected for both recognition and action, in the form of an eye movement, is essential to understanding the mechanisms of visual search. It is argued that selecting a pattern for fixation is time consuming-requiring the pruning of a population of possible saccade vectors to isolate the specific movement to the potential target. To support this position, two experiments are reported showing evidence for off-object fixations, where fixations land between objects rather than directly on objects, and central fixations, where initial saccades land near the center of scenes. Both behaviors were modeled successfully using TAM (Target Acquisition Model; Zelinsky, 2008). TAM interprets these behaviors as expressions of population averaging occurring at different times during saccade target selection. A large population early during search results in the averaging of the entire scene and a central fixation; a smaller population later during search results in averaging between groups of objects and off-object fixations.

Search performance with discrete-cell stimulus arrays: filtered naturalistic images and probabilistic markers

In order to partially fill the gap between search studies with artificial and naturalistic stimuli, experiments with segmented and spatially filtered images of real stimuli in the discrete cells of search arrays were conducted. RT × set size functions obtained with the spatially filtered arrays were compared with those obtained with geometric and other types of arrays. With the aid of Fourier analysis, components of target certainty/uncertainty, target-distractor similarity/dissimilarity, and distractor homogeneity/heterogeneity were evaluated for their effects on search performance and efficiency. The comparative effects of valid/invalid and ambiguous/unambiguous markers of search items were included in the experiments to probe the potential contributions of top-down and bottom-up control to search with naturalistic stimuli. Visual coding in the form of spatial frequency passbands and top-down preferences for distractor search were used to explain the results.

The impact of map orientation and generalisation on congestion decisions: a comparison of schematic-egocentric and topographic-allocentric maps

Map information for drivers is usually presented in an allocentric-topographic form (as with printed maps) or in an egocentric-schematic form (as with road signs). The advent of new variable message boards on UK motorways raises the possibility of presenting road maps to reflect congestion ahead. Should these maps be allocentric-topographic or egocentric-schematic? This was assessed in an eye tracking study, with participants viewing maps of a motorway network in order to identify whether any congestion was relevant to their intended route. The schematic-egocentric maps were responded to most accurately with shorter fixation durations suggesting easier processing. In particular, the driver's entrance and intended exit from the map were attended to more in the allocentric maps. Individual differences in mental rotation ability also seem to contribute to poor performance on allocentric maps. The results favour schematic-egocentric maps for roadside congestion information, but also provide theoretical insights into map-rotation and individual differences. Statement of Relevance: This study informs designers and policy makers about optimum representations of traffic congestion on roadside variable message signs and, furthermore, demonstrates that individual differences contribute to problems with processing certain sign types. Schematic-egocentric representations of a motorway network produced the best results, as noted in behavioural and eye movement measures.

Change detection in complex scenes: hemispheric contribution and the role of perceptual and semantic factors

The perceptual salience and semantic relevance of objects for the meaning of a scene were evaluated with multiple criteria and then manipulated in a change-detection experiment that used an original combination of one-shot and tachistoscopic divided-visual-field paradigms to study behavioural hemispheric asymmetry. Coloured drawings that depicted meaningful situations were presented centrally and very briefly (120 ms) and only the changes were lateralised by adding an object in the right or in the left visual hemifield. High salience and high relevance improved both response times (RTs) and accuracy, although the overall contribution of salience was greater than that of relevance. Moreover, only for low-salience changes did relevance affect speed. RTs were shorter when a change occurred in the left visual hemifield, suggesting a right-hemisphere advantage for detection of visual change. Also, men responded faster than women. The theoretical and methodological implications are discussed.

Visual search performance in the autism spectrum II: the radial frequency search task with additional segmentation cues

The Embedded Figures Test (EFT) requires detecting a shape within a complex background and individuals with autism or high Autism-spectrum Quotient (AQ) scores are faster and more accurate on this task than controls. This research aimed to uncover the visual processes producing this difference. Previously we developed a search task using radial frequency (RF) patterns with controllable amounts of target/distracter overlap on which high AQ participants showed more efficient search than low AQ observers. The current study extended the design of this search task by adding two lines which traverse the display on random paths sometimes intersecting target/distracters, other times passing between them. As with the EFT, these lines segment and group the display in ways that are task irrelevant. We tested two new groups of observers and found that while RF search was slowed by the addition of segmenting lines for both groups, the high AQ group retained a consistent search advantage (reflected in a shallower gradient for reaction time as a function of set size) over the low AQ group. Further, the high AQ group were significantly faster and more accurate on the EFT compared to the low AQ group. That is, the results from the present RF search task demonstrate that segmentation and grouping created by intersecting lines does not further differentiate the groups and is therefore unlikely to be a critical factor underlying the EFT performance difference. However, once again, we found that superior EFT performance was associated with shallower gradients on the RF search task.

Exploring the perceptual causes of search set-size effects in complex scenes

We explored search set-size effects in the context of scenes, where objects are difficult to delineate and define. Observers searched for a tank target in landscape scenes populated by boulder, shrub, and tree distractors, which varied in their color and size similarity to the target. Scenes contained either 75 or 125 distractors. In two experiments we found that adding boulder or shrub distractors that were similar to the target in color and in size produced a consistent set-size effect, but adding distractors that were similar to the target only in size did not. Tree distractors, which were larger than the target, also produced a set-size effect regardless of whether they had a target-similar color (experiment 1) or not (experiment 2). In a third experiment we varied target-tree color similarity and tree color heterogeneity and found no change in search efficiency. We interpret these data as evidence for two independent sources of set-size effects in scenes, one subject to target--distractor similarity relationships (shrubs/boulders) and the other reflecting overall visual clutter (trees), with these latter clutter effects definable by edge content.

Searching in the dark: cognitive relevance drives attention in real-world scenes

We investigated whether the deployment of attention in scenes is better explained by visual salience or by cognitive relevance. In two experiments, participants searched for target objects in scene photographs. The objects appeared in semantically appropriate locations but were not visually salient within their scenes. Search was fast and efficient, with participants much more likely to look to the targets than to the salient regions. This difference was apparent from the first fixation and held regardless of whether participants were familiar with the visual form of the search targets. In the majority of trials, salient regions were not fixated. The critical effects were observed for all 24 participants across the two experiments. We outline a cognitive relevance framework to account for the control of attention and fixation in scenes.

A new step towards understanding Embedded Figures Test performance in the autism spectrum: the radial frequency search task

The Embedded Figure Test (EFT) requires locating a simple shape embedded within a background of overlapping target-irrelevant scene elements. Observers with autism, or those with high levels of autistic-like traits, typically outperform matched comparison groups on the EFT. This research investigated the critical visual properties which give rise to this improved performance. The EFT is a search task and so here a radial frequency (RF) search task was created to directly explore efficacy of visual search and also the influence of element overlap on performance. In all conditions, the task was to detect whether the target RF3 (a triangular shape chosen for its visual properties) was present among a number of distracter RF4 (a square shape) patterns. The conditions employed were: 'singles', where all the patterns were spatially discrete, 'pairs', where two overlapping elements formed each cluster, and 'quads', comprising four overlapping elements per cluster. Compared to students scoring low on the Autism Spectrum Quotient (AQ; n=27), those scoring high on the AQ (n=23) were faster on the EFT and also significantly less influenced by increasing set size of the stimulus array in all RF search task conditions. However, the group difference in RF search performance was unaffected by the amount of stimulus overlap. Thus a simple search task is sufficient to detect a performance advantage associated with higher levels of autistic traits and has the advantages of a solid footing in visual theory and being readily repeatable for the purpose of assessing performance variability and change with interventions.