Guided search for triple conjunctions

Realistic About Reference Production: Testing the Effects of Domain Size and Saturation

Experiments on visually grounded, definite reference production often manipulate simple visual scenes in the form of grids filled with objects, for example, to test how speakers are affected by the number of objects that are visible. Regarding the latter, it was found that speech onset times increase along with domain size, at least when speakers refer to nonsalient target objects that do not pop out of the visual domain. This finding suggests that even in the case of many distractors, speakers perform object-by-object scans of the visual scene. The current study investigates whether this systematic processing strategy can be explained by the simplified nature of the scenes that were used, and if different strategies can be identified for photo-realistic visual scenes. In doing so, we conducted a preregistered experiment that manipulated domain size and saturation; replicated the measures of speech onset times; and recorded eye movements to measure speakers' viewing strategies more directly. Using controlled photo-realistic scenes, we find (1) that speech onset times increase linearly as more distractors are present; (2) that larger domains elicit relatively fewer fixation switches back and forth between the target and its distractors, mainly before speech onset; and (3) that speakers fixate the target relatively less often in larger domains, mainly after speech onset. We conclude that careful object-by-object scans remain the dominant strategy in our photo-realistic scenes, to a limited extent combined with low-level saliency mechanisms. A relevant direction for future research would be to employ less controlled photo-realistic stimuli that do allow for interpretation based on context.

Control over attentional capture within 170 ms by long-term memory control settings: Evidence from the N2pc

Observers adopt attentional control settings (ACSs) based on their goals that guide the capture of attention: Searched-for stimuli capture attention, and stimuli that are not searched for do not. While previous behavioural research indicates that observers can adopt long-term memory (LTM) ACSs (Giammarco et al. Visual Cognition, 24, 78-101, 2016), it seems surprising that representations in LTM could guide attention quickly enough to control attentional capture. To assess the claim that LTM ACSs exert control over early attentional orienting, we recorded electroencephalography while participants studied and searched for 30 target objects in an attention cueing task. Participants reported the studied target and ignored the preceding cues. To control for perceptual evoked responses, on each trial we presented two cue objects (one studied and one nonstudied). Even though participants were instructed to ignore the cues, studied cues produced the N2pc event-related potential, indicating early attentional orienting that was preferentially directed towards the studied cue versus the nonstudied cue. Critically, the N2pc was detectable within 170 ms, confirming that LTM ACSs rapidly control early capture. We propose an update to contemporary models of attentional capture to account for rapid attentional guidance by LTM ACSs.

Assessing the allocation of attention during visual search using digit-tracking, a calibration-free alternative to eye tracking

Digit-tracking, a simple, calibration-free technique, has proven to be a good alternative to eye tracking in vision science. Participants view stimuli superimposed by Gaussian blur on a touchscreen interface and slide a finger across the display to locally sharpen an area the size of the foveal region just at the finger's position. Finger movements are recorded as an indicator of eye movements and attentional focus. Because of its simplicity and portability, this system has many potential applications in basic and applied research. Here we used digit-tracking to investigate visual search and replicated several known effects observed using different types of search arrays. Exploration patterns measured with digit-tracking during visual search of natural scenes were comparable to those previously reported for eye-tracking and constrained by similar saliency. Therefore, our results provide further evidence for the validity and relevance of digit-tracking for basic and applied research on vision and attention.

The structure of illusory conjunctions reveals hierarchical binding of multipart objects

The world around us is filled with complex objects, full of color, motion, shape, and texture, and these features seem to be represented separately in the early visual system. Anne Treisman pointed out that binding these separate features together into coherent conscious percepts is a serious challenge, and she argued that selective attention plays a critical role in this process. Treisman also showed that, consistent with this view, outside the focus of attention we suffer from illusory conjunctions: misperceived pairings of features into objects. Here we used Treisman's logic to study the structure of pre-attentive representations of multipart, multicolor objects, by exploring the patterns of illusory conjunctions that arise outside the focus of attention. We found consistent evidence of some pre-attentive binding of colors to their parts, and weaker evidence of binding multiple colors of the same object. The extent to which such hierarchical binding occurs seems to depend on the geometric structure of multipart objects: Objects whose parts are easier to separate seem to exhibit greater pre-attentive binding. Together, these results suggest that representations outside the focus of attention are not entirely a "shapeless bundles of features," but preserve some meaningful object structure.

Categorical grouping is not required for guided conjunction search

Knowledge of target features can guide attention in many conjunction searches in a top-down manner. For example, in search of a red vertical line among blue vertical and red horizontal lines, observers can guide attention toward all red items and all vertical items. In typical conjunction searches, distractors often form perceptually vivid, categorical groups of identical objects. This could favor the efficient search via guidance of attention to these "segmentable" groups. Can attention be guided if the distractors are not neatly segmentable (e.g., if colors vary continuously from red through purple to blue)? We tested search for conjunctions of color × orientation (Experiments 1, 3, 4, 5) or length × orientation (Experiment 2). In segmentable conditions, distractors could form two clear groups (e.g., blue steep and red flat). In non-segmentable conditions, distractors varied smoothly from red to blue and/or steep to flat; thus, discouraging grouping and increasing overall heterogeneity. We found that the efficiency of conjunction search was reasonably high and unaffected by segmentability. The same lack of segmentability had a detrimental effect on feature search (Experiment 4) and on conjunction search, if target information was limited to one feature (e.g., find the odd item in the red set, "subset search," Experiment 3). Guidance in conjunction search may not require grouping and segmentation cues that are very important in other tasks like texture discrimination. Our results support an idea of simultaneous, parallel top-down guidance by multiple features and argue against models suggesting sequential guidance by each feature in turn.

Scene memory and spatial inhibition in visual search : A neural dynamic process model and new experimental evidence

Any object-oriented action requires that the object be first brought into the attentional foreground, often through visual search. Outside the laboratory, this would always take place in the presence of a scene representation acquired from ongoing visual exploration. The interaction of scene memory with visual search is still not completely understood. Feature integration theory (FIT) has shaped both research on visual search, emphasizing the scaling of search times with set size when searches entail feature conjunctions, and research on visual working memory through the change detection paradigm. Despite its neural motivation, there is no consistently neural process account of FIT in both its dimensions. We propose such an account that integrates (1) visual exploration and the building of scene memory, (2) the attentional detection of visual transients and the extraction of search cues, and (3) visual search itself. The model uses dynamic field theory in which networks of neural dynamic populations supporting stable activation states are coupled to generate sequences of processing steps. The neural architecture accounts for basic findings in visual search and proposes a concrete mechanism for the integration of working memory into the search process. In a behavioral experiment, we address the long-standing question of whether both the overall speed and the efficiency of visual search can be improved by scene memory. We find both effects and provide model fits of the behavioral results. In a second experiment, we show that the increase in efficiency is fragile, and trace that fragility to the resetting of spatial working memory.

On Visually-Grounded Reference Production: Testing the Effects of Perceptual Grouping and 2D/3D Presentation Mode

When referring to a target object in a visual scene, speakers are assumed to consider certain distractor objects to be more relevant than others. The current research predicts that the way in which speakers come to a set of relevant distractors depends on how they perceive the distance between the objects in the scene. It reports on the results of two language production experiments, in which participants referred to target objects in photo-realistic visual scenes. Experiment 1 manipulated three factors that were expected to affect perceived distractor distance: two manipulations of perceptual grouping (region of space and type similarity), and one of presentation mode (2D vs. 3D). In line with most previous research on visually-grounded reference production, an offline measure of visual attention was taken here: the occurrence of overspecification with color. The results showed effects of region of space and type similarity on overspecification, suggesting that distractors that are perceived as being in the same group as the target are more often considered relevant distractors than distractors in a different group. Experiment 2 verified this suggestion with a direct measure of visual attention, eye tracking, and added a third manipulation of grouping: color similarity. For region of space in particular, the eye movements data indeed showed patterns in the expected direction: distractors within the same region as the target were fixated more often, and longer, than distractors in a different region. Color similarity was found to affect overspecification with color, but not gaze duration or the number of distractor fixations. Also the expected effects of presentation mode (2D vs. 3D) were not convincingly borne out by the data. Taken together, these results provide direct evidence for the close link between scene perception and language production, and indicate that perceptual grouping principles can guide speakers in determining the distractor set during reference production.

What is a preattentive feature?

The concept of a preattentive feature has been central to vision and attention research for about half a century. A preattentive feature is a feature that guides attention in visual search and that cannot be decomposed into simpler features. While that definition seems straightforward, there is no simple diagnostic test that infallibly identifies a preattentive feature. This paper briefly reviews the criteria that have been proposed and illustrates some of the difficulties of definition.

Attentional guidance varies with display density

The aim of the present study was to investigate how display density affects attentional guidance in heterogeneous search displays. In Experiment 1 we presented observers with heterogeneous sparse and dense search displays which were adaptively changed over the course of the experiment using genetic algorithms. We generated random displays, and based upon fastest search times, the displays that allowed most efficient search were selected to generate new displays for the next generations, thus revealing which properties facilitated or inhibited target search across display densities. The results showed that the prevalence of distractors sharing the target color was substantially reduced over generations in sparse displays. Dense displays also evolved to contain less distractors sharing the target color but only when the orientation of the distractors resembled the target orientation. More importantly, spatial analyses revealed that changes across generations occurred across all areas in sparse displays but were confined to occur around the target location only in dense displays. In Experiment 2, in which we used a factorial design, we showed that the presence of potentially interfering distractors in the target area affected search in dense displays but not in sparse displays. Together the results suggest that the role of salience-driven attentional guidance is larger in dense than sparse displays even in the absence of display homogeneity.

Search and Match Task: Development of a Taskified Match-3 Puzzle Game to Assess and Practice Visual Search

Background

Visual search declines with aging, dementia, and brain injury and is linked to limitations in everyday activities. Recent studies suggest that visual search can be improved with practice using computerized visual search tasks and puzzle video games. For practical use, it is important that visual search ability can be assessed and practiced in a controlled and adaptive way. However, commercial puzzle video games make it hard to control task difficulty, and there are little means to collect performance data.

Objective

The aim of this study was to develop and initially validate the search and match task (SMT) that combines an enjoyable tile-matching match-3 puzzle video game with features of the visual search paradigm (taskified game). The SMT was designed as a single-target visual search task that allows control over task difficulty variables and collection of performance data.

Methods

The SMT is played on a grid-based (width × height) puzzle board, filled with different types of colored polygons. A wide range of difficulty levels was generated by combinations of 3 task variables over a range from 4 to 8 including height and width of the puzzle board (set size) and the numbers of tile types (distractor heterogeneity). For each difficulty level, large numbers of playable trials were pregenerated using Python. Each trial consists of 4 consecutive puzzle boards, where the goal of the task is to find a target tile configuration (search) on the puzzle board and swap 2 adjacent tiles to create a line of 3 identical tiles (match). For each puzzle board, there is exactly 1 possible match (single target search). In a user study with 28 young adults (aged 18 to 31 years), 13 older (aged 64 to 79 years) and 11 oldest (aged 86 to 98 years) adults played the long (young and older adults) or short version (oldest adults) of the difficulty levels of the SMT. Participants rated their perception and the usability of the task and completed neuropsychological tests that measure cognitive domains engaged by the puzzle game.

Results

Results from the user study indicate that the target search time is associated with set size, distractor heterogeneity, and age. Results further indicate that search performance is associated with general cognitive ability, selective and divided attention, visual search, and visuospatial and pattern recognition ability.

Conclusions

Overall, this study shows that an everyday puzzle game–based task can be experimentally controlled, is enjoyable and user-friendly, and permits data collection to assess visual search and cognitive abilities. Further research is needed to evaluate the potential of the SMT game to assess and practice visual search ability in an enjoyable and adaptive way. A PsychoPy version of the SMT is freely available for researchers.

Early Visual Processing of Feature Saliency Tasks: A Review of Psychophysical Experiments

The visual system is constantly bombarded with information originating from the outside world, but it is unable to process all the received information at any given time. In fact, the most salient parts of the visual scene are chosen to be processed involuntarily and immediately after the first glance along with endogenous signals in the brain. Vision scientists have shown that the early visual system, from retina to lateral geniculate nucleus (LGN) and then primary visual cortex, selectively processes the low-level features of the visual scene. Everything we perceive from the visual scene is based on these feature properties and their subsequent combination in higher visual areas. Different experiments have been designed to investigate the impact of these features on saliency and understand the relative visual mechanisms. In this paper, we review the psychophysical experiments which have been published in the last decades to indicate how the low-level salient features are processed in the early visual cortex and extract the most important and basic information of the visual scene. Important and open questions are discussed in this review as well and one might pursue these questions to investigate the impact of higher level features on saliency in complex scenes or natural images.

Predicting Search Performance in Heterogeneous Visual Search Scenes with Real-World Objects

Previous work in our lab has demonstrated that efficient visual search with a fixed target has a reaction time by set size function that is best characterized by logarithmic curves. Further, the steepness of these logarithmic curves is determined by the similarity between target and distractor items (Buetti et al., 2016). A theoretical account of these findings was proposed, namely that a parallel, unlimited capacity, exhaustive processing architecture is underlying such data. Here, we conducted two experiments to expand these findings to a set of real-world stimuli, in both homogeneous and heterogeneous search displays. We used computational simulations of this architecture to identify a way to predict RT performance in heterogeneous search using parameters estimated from homogeneous search data. Further, by examining the systematic deviation from our predictions in the observed data, we found evidence that early visual processing for individual items is not independent. Instead, items in homogeneous displays seemed to facilitate each other’s processing by a multiplicative factor. These results challenge previous accounts of heterogeneity effects in visual search, and demonstrate the explanatory and predictive power of an approach that combines computational simulations and behavioral data to better understand performance in visual search.

Five Factors that Guide Attention in Visual Search

How do we find what we are looking for? Fundamental limits on visual processing mean that even when the desired target is in our field of view, we often need to search, because it is impossible to recognize everything at once. Searching involves directing attention to objects that might be the target. This deployment of attention is not random. It is guided to the most promising items and locations by five factors discussed here: Bottom-up salience, top-down feature guidance, scene structure and meaning, the previous history of search over time scales from msec to years, and the relative value of the targets and distractors. Modern theories of search need to specify how all five factors combine to shape search behavior. An understanding of the rules of guidance can be used to improve the accuracy and efficiency of socially-important search tasks, from security screening to medical image perception.

Understanding visual attention in childhood: Insights from a new visual foraging task

A recently developed visual foraging task, involving multiple targets of different types, can provide a rich and dynamic picture of visual attention performance. We measured the foraging performance of 66 children aged 4–7 years, along with measures of two conceptually related constructs, self-regulation and verbal working memory. Our results show that foraging patterns of young children differ from adult patterns. Children have difficulty with foraging for two target types, not only when they are defined by a conjunction of features but, unlike adults, also when they forage simultaneously for two target types that are distinguished from distractors by a single feature. Importantly, such feature/conjunction differences between adults and children are not seen in more traditional single-target visual search tasks. Interestingly, the foraging patterns of the youngest children were slightly more adult-like than of the oldest ones, which may suggest that older children attempt to use strategies that they have not yet fully mastered. The older children were, however, able to complete more trials, during both feature and conjunction foraging. Self-regulation and verbal working memory did not seem to affect foraging strategies, but both were connected with faster and more efficient foraging. We propose that our visual foraging paradigm is a promising avenue for studying the development of visual cognitive abilities.

Competing Distractors Facilitate Visual Search in Heterogeneous Displays

In the present study, we examine how observers search among complex displays. Participants were asked to search for a big red horizontal line among 119 distractor lines of various sizes, orientations and colours, leading to 36 different feature combinations. To understand how people search in such a heterogeneous display, we evolved the search display by using a genetic algorithm (Experiment 1). The best displays (i.e., displays corresponding to the fastest reaction times) were selected and combined to create new, evolved displays. Search times declined over generations. Results show that items sharing the same colour and orientation as the target disappeared over generations, implying they interfered with search, but items sharing the same colour and were 12.5° different in orientation only interfered if they were also the same size. Furthermore, and inconsistent with most dominant visual search theories, we found that non-red horizontal distractors increased over generations, indicating that these distractors facilitated visual search while participants were searching for a big red horizontally oriented target. In Experiments 2 and 3, we replicated these results using conventional, factorial experiments. Interestingly, in Experiment 4, we found that this facilitation effect was only present when the displays were very heterogeneous. While current models of visual search are able to successfully describe search in homogeneous displays, our results challenge the ability of these models to describe visual search in heterogeneous environments.

Reference Production as Search: The Impact of Domain Size on the Production of Distinguishing Descriptions

When producing a description of a target referent in a visual context, speakers need to choose a set of properties that distinguish it from its distractors. Computational models of language production/generation usually model this as a search process and predict that the time taken will increase both with the number of distractors in a scene and with the number of properties required to distinguish the target. These predictions are reminiscent of classic findings in visual search; however, unlike models of reference production, visual search models also predict that search can become very efficient under certain conditions, something that reference production models do not consider. This paper investigates the predictions of these models empirically. In two experiments, we show that the time taken to plan a referring expression-as reflected by speech onset latencies-is influenced by distractor set size and by the number of properties required, but this crucially depends on the discriminability of the properties under consideration. We discuss the implications for current models of reference production and recent work on the role of salience in visual search.