Visual search has memory

Visual Search in the Real World: Evidence for the Formation of Distractor Representations

Visual search in the real world often requires that we search the same environment a number of times for different targets. What is the fate of information about fixated distractor objects during these searches? Here, participants searched the same array of real objects on a tabletop twice for two different targets successively whilst wearing a head-mounted eye-tracker. We found that fixating an object when it was a distractor in the first search facilitated search for that same object when it became the target in the second search. The results suggest that the location and identity of fixated distractor objects are represented to a level that guides subsequent searches, even when this information is not required at the time of fixation.

Location and identity memory of saccade targets

While the memory of objects' identity and of their spatiotopic location may sustain transsaccadic spatial constancy, the memory of their retinotopic location may hamper it. Is it then true that saccades perturb retinotopic but not spatiotopic memory? We address this issue by assessing localization performances of the last and of the penultimate saccade target in a series of 2-6 saccades. Upon fixation, nine letter-pairs, eight black and one white, were displayed at 3° eccentricity around fixation within a 20° × 20° grey frame, and subjects were instructed to saccade to the white letter-pair; the cycle was then repeated. Identical conditions were run with the eyes maintaining fixation throughout the trial but with the grey frame moving so as to mimic its retinal displacement when the eyes moved. At the end of a trial, subjects reported the identity and/or the location of the target in either retinotopic (relative to the current fixation dot) or frame-based(1) (relative to the grey frame) coordinates. Saccades degraded target's retinotopic location memory but not its frame-based location or its identity memory. Results are compatible with the notion that spatiotopic representation takes over retinotopic representation during eye movements thereby contributing to the stability of the visual world as its retinal projection jumps on our retina from saccade to saccade.

Rapid Communication: Finding memory in search: The effect of visual working memory load on visual search

There is now substantial evidence that during visual search, previously searched distractors are stored in memory to prevent them from being reselected. Studies examining which memory resources are involved in this process have indicated that while a concurrent spatial working memory task does affect search slopes, depleting visual working memory (VWM) resources does not. In the present study, we confirm that VWM load indeed has no effect on the search slope; however, there is an increase in overall reaction times that is directly related to the number of items held in VWM. Importantly, this effect on search time increases proportionally with the memory load until the capacity of VWM is reached. Furthermore, the search task interfered with the number of items stored in VWM during the concurrent change-detection task. These findings suggest that VWM plays a role in the inhibition of previously searched distractors.

Overlapping functional anatomy for working memory and visual search

Recent behavioural findings using dual-task paradigms demonstrate the importance of both spatial and non-spatial working memory processes in inefficient visual search (Anderson et al. in Exp Psychol 55:301–312, 2008). Here, using functional magnetic resonance imaging (fMRI), we sought to determine whether brain areas recruited during visual search are also involved in working memory. Using visually matched spatial and non-spatial working memory tasks, we confirmed previous behavioural findings that show significant dual-task interference effects occur when inefficient visual search is performed concurrently with either working memory task. Furthermore, we find considerable overlap in the cortical network activated by inefficient search and both working memory tasks. Our findings suggest that the interference effects observed behaviourally may have arisen from competition for cortical processes subserved by these overlapping regions. Drawing on previous findings (Anderson et al. in Exp Brain Res 180:289–302, 2007), we propose that the most likely anatomical locus for these interference effects is the inferior and middle frontal cortex of the right hemisphere. These areas are associated with attentional selection from memory as well as manipulation of information in memory, and we propose that the visual search and working memory tasks used here compete for common processing resources underlying these mechanisms.

A probabilistic model of overt visual attention for cognitive robots

Visual attention is one of the major requirements for a robot to serve as a cognitive companion for human. The robotic visual attention is mostly concerned with overt attention which accompanies head and eye movements of a robot. In this case, each movement of the camera head triggers a number of events, namely transformation of the camera and the image coordinate systems, change of content of the visual field, and partial appearance of the stimuli. All of these events contribute to the reduction in probability of meaningful identification of the next focus of attention. These events are specific to overt attention with head movement and, therefore, their effects are not addressed in the classical models of covert visual attention. This paper proposes a Bayesian model as a robot-centric solution for the overt visual attention problem. The proposed model, while taking inspiration from the primates visual attention mechanism, guides a robot to direct its camera toward behaviorally relevant and/or visually demanding stimuli. A particle filter implementation of this model addresses the challenges involved in overt attention with head movement. Experimental results demonstrate the performance of the proposed model.

Visual search elicits the electrophysiological marker of visual working memory

BACKGROUND

Although limited in capacity, visual working memory (VWM) plays an important role in many aspects of visually-guided behavior. Recent experiments have demonstrated an electrophysiological marker of VWM encoding and maintenance, the contralateral delay activity (CDA), which has been shown in multiple tasks that have both explicit and implicit memory demands. Here, we investigate whether the CDA is evident during visual search, a thoroughly-researched task that is a hallmark of visual attention but has no explicit memory requirements.

METHODOLOGY/PRINCIPAL FINDINGS

The results demonstrate that the CDA is present during a lateralized search task, and that it is similar in amplitude to the CDA observed in a change-detection task, but peaks slightly later. The changes in CDA amplitude during search were strongly correlated with VWM capacity, as well as with search efficiency. These results were paralleled by behavioral findings showing a strong correlation between VWM capacity and search efficiency.

CONCLUSIONS/SIGNIFICANCE

We conclude that the activity observed during visual search was generated by the same neural resources that subserve VWM, and that this activity reflects the maintenance of previously searched distractors.

The 35th Sir Frederick Bartlett Lecture: Eye movements and attention in reading, scene perception, and visual search

Eye movements are now widely used to investigate cognitive processes during reading, scene perception, and visual search. In this article, research on the following topics is reviewed with respect to reading: (a) the perceptual span (or span of effective vision), (b) preview benefit, (c) eye movement control, and (d) models of eye movements. Related issues with respect to eye movements during scene perception and visual search are also reviewed. It is argued that research on eye movements during reading has been somewhat advanced over research on eye movements in scene perception and visual search and that some of the paradigms developed to study reading should be more widely adopted in the study of scene perception and visual search. Research dealing with “real-world” tasks and research utilizing the visual-world paradigm are also briefly discussed.

The mechanism underlying inhibition of saccadic return

Human observers take longer to re-direct gaze to a previously fixated location. Although there has been some exploration of the characteristics of inhibition of saccadic return (ISR), the exact mechanisms by which ISR operates are currently unknown. In the framework of accumulation models of response times, in which evidence is integrated over time to a response threshold, ISR could reflect a reduction in the rate of accumulation for saccades to return locations or an increase in the effective criterion for response. In two experiments, participants generated sequences of three saccades, in response to a peripheral or a central cue. ISR occurred across these manipulations: saccade latency was consistently increased for movements to the immediately previously fixated location. Latency distributions from individual observers were fit with a Linear Ballistic Accumulator model. ISR was best accounted for as a change in the accumulation rate. We suggest this parameter represents the overall desirability of a particular course of action, the evidence for which may be derived from a variety of sensory and non-sensory sources.

The contents of perceptual hypotheses: evidence from rapid resumption of interrupted visual search

Observers can resume a previously interrupted visual search trial significantly more quickly than they can start a new search trial (Lleras, Rensink, & Enns, 2005). This rapid resumption of search is possible because evidence accumulated during the previous exposure, a perceptual hypothesis, can carry over to a subsequent presentation. We present four interrupted visual search experiments in which the content of the perceptual hypotheses used during visual search trials was characterized. These experiments suggest that prior to explicit target identification, observers have accumulated evidence about the locations, but not the identities, of local, task-relevant distractors, as well as preliminary evidence for the identity of the target. Our results characterize the content of perceptual search hypotheses and highlight the utility of interrupted search for studying online search processing prior to target identification.

The speed of free will

Do voluntary and task-driven shifts of attention have the same time course? In order to measure the time needed to voluntarily shift attention, we devised several novel visual search tasks that elicited multiple sequential attentional shifts. Participants could only respond correctly if they attended to the right place at the right time. In control conditions, search tasks were similar but participants were not required to shift attention in any order. Across five experiments, voluntary shifts of attention required 200-300 ms. Control conditions yielded estimates of 35-100 ms for task-driven shifts. We suggest that the slower speed of voluntary shifts reflects the "clock speed of free will". Wishing to attend to something takes more time than shifting attention in response to sensory input.

Attentional demand influences strategies for encoding into visual working memory

Visual selective attention and visual working memory (WM) share the same capacity-limited resources. We investigated whether and how participants can cope with a task in which these 2 mechanisms interfere. The task required participants to scan an array of 9 objects in order to select the target locations and to encode the items presented at these locations into WM (1 to 5 shapes). Determination of the target locations required either few attentional resources ("popout condition") or an attention-demanding serial search ("non pop-out condition"). Participants were able to achieve high memory performance in all stimulation conditions but, in the non popout conditions, this came at the cost of additional processing time. Both empirical evidence and subjective reports suggest that participants invested the additional time in memorizing the locations of all target objects prior to the encoding of their shapes into WM. Thus, they seemed to be unable to interleave the steps of search with those of encoding. We propose that the memory for target locations substitutes for perceptual pop-out and thus may be the key component that allows for flexible coping with the common processing limitations of visual WM and attention. The findings have implications for understanding how we cope with real-life situations in which the demands on visual attention and WM occur simultaneously.

The construct of attention in schizophrenia

Schizophrenia is widely thought to involve deficits of attention. However, the term "attention" can be defined so broadly that impaired performance on virtually any task could be construed as evidence for a deficit in attention, and this has slowed cumulative progress in understanding attention deficits in schizophrenia. To address this problem, we divide the general concept of attention into two distinct constructs: input selection, the selection of task-relevant inputs for further processing; and rule selection, the selective activation of task-appropriate rules. These constructs are closely tied to working memory, because input selection mechanisms are used to control the transfer of information into working memory and because working memory stores the rules used by rule selection mechanisms. These constructs are also closely tied to executive function, because executive systems are used to guide input selection and because rule selection is itself a key aspect of executive function. Within the domain of input selection, it is important to distinguish between the control of selection--the processes that guide attention to task-relevant inputs--and the implementation of selection--the processes that enhance the processing of the relevant inputs and suppress the irrelevant inputs. Current evidence suggests that schizophrenia involves a significant impairment in the control of selection but little or no impairment in the implementation of selection. Consequently, the CNTRICS (Cognitive Neuroscience Treatment Research to Improve Cognition in Schizophrenia) participants agreed by consensus that attentional control should be a priority target for measurement and treatment research in schizophrenia.

The obligatory effects of memory on eye movements

Previous work has shown that eye movement behaviour is affected by previous experience, such that alterations in viewing patterns can be observed to previously viewed compared to novel displays. The current work addresses the extent to which such effects of memory on eye movement behaviour are obligatory; that is, we examined whether prior experience could alter subsequent eye movement behaviour under a variety of testing conditions, for stimuli that varied on the nature of the prior exposure. While task demands influenced whether viewing was predominantly directed to the novel versus familiar faces, viewing of the familiar faces was distinguished from viewing of the novel faces, regardless of whether the task required incidental encoding or intentional retrieval. Changes in scanning of previously viewed over novel faces emerged early in viewing; in particular, viewing duration of the first fixation to the familiar faces was often significantly different from the duration of the first fixation directed to the novel faces, regardless of whether prior exposure was solely in the context of the experiment or due to real-world exposure. These findings suggest that representations maintained in memory may be retrieved and compared with presented information obligatorily.

Visual search and foraging compared in a large-scale search task

It has been argued that visual search is a valid model for human foraging. However, the two tasks differ greatly in terms of the coding of space and the effort required to search. Here we describe a direct comparison between visually guided searches (as studied in visual search tasks) and foraging that is not based upon a visually distinct target, within the same context. The experiment was conducted in a novel apparatus, where search locations were indicated by an array of lights embedded in the floor. In visually guided conditions participants searched for a target defined by the presence of a feature (red target amongst green distractors) or the absence of a feature (green target amongst red and green distractors). Despite the expanded search scale and the different response requirements, these conditions followed the pattern found in conventional visual search paradigms: feature-present search latencies were not linearly related to display size, whereas feature-absent searches were longer as the number of distractors increased. In a non-visually guided foraging condition, participants searched for a target that was only visible once the switch was activated. This resulted in far longer latencies that rose markedly with display size. Compared to eye-movements in previous visual search studies, there were few revisit errors to previously inspected locations in this condition. This demonstrates the important distinction between visually guided and non-visually guided foraging processes, and shows that the visual search paradigm is an equivocal model for general search in any context. We suggest a comprehensive model of human spatial search behaviour needs to include search at a small and large scale as well as visually guided and non-visually guided search.

A Role for Spatial and Nonspatial Working Memory Processes in Visual Search

Searching a cluttered visual scene for a specific item of interest can take several seconds to perform if the target item is difficult to discriminate from surrounding items. Whether working memory processes are utilized to guide the path of attentional selection during such searches remains under debate. Previous studies have found evidence to support a role for spatial working memory in inefficient search, but the role of nonspatial working memory remains unclear. Here, we directly compared the role of spatial and nonspatial working memory for both an efficient and inefficient search task. In Experiment 1, we used a dual-task paradigm to investigate the effect of performing visual search within the retention interval of a spatial working memory task. Importantly, by incorporating two working memory loads (low and high) we were able to make comparisons between dual-task conditions, rather than between dual-task and single-task conditions. This design allows any interference effects observed to be attributed to changes in memory load, rather than to nonspecific effects related to “dual-task” performance. We found that the efficiency of the inefficient search task declined as spatial memory load increased, but that the efficient search task remained efficient. These results suggest that spatial memory plays an important role in inefficient but not efficient search. In Experiment 2, participants performed the same visual search tasks within the retention interval of visually matched spatial and verbal working memory tasks. Critically, we found comparable dual-task interference between inefficient search and both the spatial and nonspatial working memory tasks, indicating that inefficient search recruits working memory processes common to both domains.

Evidence from a response choice task reveals a selection bias in the attentional cueing paradigm

In a typical attentional cueing paradigm, irrelevant peripheral cues produce early facilitation (fast responses) followed by later inhibition (slow responses) to cued locations. Here we examine whether cues not only influence the speed with which responses are produced, but also impact or bias which location is ultimately selected as requiring a response. Specifically, can cues influence not only the speed with which we respond but also influence the behavior produced? To examine this question, a choice localization task was used in which no targets were presented, and subjects were asked to choose which effector (left hand, right hand) to use in response to a centrally presented tone. Thus, following either a left or right peripheral cue, and then a central tone, subjects were free to respond with either their left or right hand. Early facilitation and later inhibition with this choice procedure were found in both response times and the proportion of responses to the cued and uncued locations. These results suggest that there are processes which initially bias response selection toward cued locations and then subsequently bias response selection away from cued locations.

Visual search is guided by prospective and retrospective memory

Although there has been some controversy as to whether attention is guided by memory during visual search, recent findings have suggested that memory helps to prevent attention from needlessly reinspecting examined items. Until now, it has been assumed that some form of retrospective memory is responsible for keeping track of examined items and preventing revisitations. Alternatively, some form of prospective memory, such as strategic scanpath planning, could be responsible for guiding attention away from examined items. We used a new technique that allowed us to selectively prevent retrospective or prospective memory from contributing to search. We demonstrated that both retrospective and prospective memory guide attention during visual search.

Memory for the search path: evidence for a high-capacity representation of search history

Using a gaze-contingent paradigm, we directly measured observers' memory capacity for fixated distractor locations during search. After approximately half of the search objects had been fixated, they were masked and a spatial probe appeared at either a previously fixated location or a non-fixated location; observers then rated their confidence that the target had appeared at the probed location. Observers were able to differentiate the 12 most recently fixated distractor locations from non-fixated locations, but analyses revealed that these locations were represented fairly coarsely. We conclude that there exists a high-capacity, but low-resolution, memory for a search path.

Saccade trajectory under simultaneous inhibition for two locations

A saccade trajectory often curves away from the location of a non-target stimulus that appears before saccade execution. Spatial inhibition may prevent the saccade from moving toward the non-target stimulus. However, little is known about how simultaneous inhibition for multiple locations affects saccade trajectories. In this study, we examined the effects from two inhibited locations on saccade trajectories. The results show that the saccade trajectories depend on the inhibited locations, and the effect of inhibiting two locations on the trajectory was a summation of the effect of inhibiting each location. A simulation study using the initial interference model also suggests that the effect of each inhibition was summed up to modulate the initial saccade direction.

Saccades and covert shifts of attention during active visual search: spatial distributions, memory, and items per fixation

Target detection during active visual search was examined. The chance corrected spatial distribution of target detection was found to be symmetrically distributed around the point of fixation and, unexpectedly, was independent of the proximity of fixations to the display boundaries. Memory was found to play a very limited role in target detection, but a significant role in the guidance of eye movements. A model of covert shifts was used to estimate the number and spatial distribution of shifts required to explain observed performance. An increase from one to five shifts per fixation across increasing array set size as estimated by two different methods was inconsistent with unchanging fixation durations, suggesting that multiple covert shifts are not occurring during the fixations in active search.

Involvement of prefrontal cortex in visual search

Visual search for target items embedded within a set of distracting items has consistently been shown to engage regions of occipital and parietal cortex, but the contribution of different regions of prefrontal cortex remains unclear. Here, we used fMRI to compare brain activity in 12 healthy participants performing efficient and inefficient search tasks in which target discriminability and the number of distractor items were manipulated. Matched baseline conditions were incorporated to control for visual and motor components of the tasks, allowing cortical activity associated with each type of search to be isolated. Region of interest analysis was applied to critical regions of prefrontal cortex to determine whether their involvement was common to both efficient and inefficient search, or unique to inefficient search alone. We found regions of the inferior and middle frontal cortex were only active during inefficient search, whereas an area in the superior frontal cortex (in the region of FEF) was active for both efficient and inefficient search. Thus, regions of ventral as well as dorsal prefrontal cortex are recruited during inefficient search, and we propose that this activity is related to processes that guide, control and monitor the allocation of selective attention.

Visual memory capacity in transsaccadic integration

How we perceive the visual world as stable and unified suggests the existence of transsaccadic integration that retains and integrates visual information from one eye fixation to another eye fixation across saccadic eye movements. However, the capacity of transsaccadic integration is still a subject of controversy. We tested our subjects' memory capacity of two basic visual features, i.e. luminance (Experiment 1) and orientation (Experiment 2), both within a single fixation (i.e. visual working memory) and between separate fixations (i.e. transsaccadic memory). Experiment 2 was repeated, but attention allocation was manipulated using attentional cues at either the target or distracter (Experiment 3). Subjects were able to retain 3-4 objects in transsaccadic memory for luminance and orientation; errors generally increased as saccade size increased; and, subjects were more accurate when attention was allocated to the same location as the impending target. These results were modelled by inputting a noisy extra-retinal signal into an eye-centered feature map. Our results suggest that transsaccadic memory has a similar capacity for storing simple visual features as basic visual memory, but this capacity is dependent both on the metrics of the saccade and allocation of attention.

Memory for where, but not what, is used during visual search

Although the role of memory in visual search is debatable, most researchers agree with a limited-capacity model of memory in visual search. The authors demonstrate the role of memory by replicating previous findings showing that visual search is biased away from old items (previously examined items) and toward new items (nonexamined items). Furthermore, the authors examined the type of memory representations used to bias search by changing an item's individuating feature or location during search. Changing the individuating feature of an item did not disrupt normal search biases. However, when the location of an item changed, normal search biases were disrupted. These results suggest that memory used in visual search is based on items' locations rather than their identity.

Memory processes in multiple-target visual search

Gibson, Li, Skow, Brown, and Cooke (Psychological Science, 11, 324-327, 2000) had participants carry out a search task in which they were required to detect the presence of one or two targets. In order to successfully perform such a multiple-target visual search task, participants had to remember the location of the first target while searching for the second target. In two experiments we investigated the cost of remembering this target location. In Experiment 1, we compared performance on the Gibson et al. task with performance on a more conventional present-absent search task. The comparison suggests a substantial performance cost as measured by reaction time, number of fixations and slope of the search functions. In Experiment 2, we looked in detail at refixations of distractors, which are a direct measure of attentional deployment. We demonstrated that the cost in this multiple-target visual search task was due to an increased number of refixations on previously visited distractors. Such refixations were present right from the start of the search. This change in search behaviour may be caused by the necessity of having to remember a target-allocating memory for the upcoming target may consume memory capacity that may otherwise be available for the tagging of distractors. These results support the notion of limited capacity memory processes in search.

Effect of previously fixated locations on saccade trajectory during free visual search

Recent studies have shown that the saccade trajectory often curved away from an object that was previously attended but irrelevant to the current saccade goal. We investigated whether such curved saccades occur during serial visual search, which requires sequential saccades possibly controlled by inhibition to multiple locations. The results show that the saccade trajectories were affected by at least three previous fixations. Furthermore, the effect of the previous fixations on saccade trajectories decreased exponentially with time or the number of intervening saccades. The relationship between the curved saccade trajectory and inhibition of return during serial visual search was discussed.

Marking rejected distractors: a gaze-contingent technique for measuring memory during search

There is a debate among search theorists as to whether search exploits a memory for rejected distractors. We addressed this question by monitoring eye movements and explicitly marking objects visited by gaze during search. If search is memoryless, markers might be used to reduce distractor reinspections and improve search efficiency, relative to a no-marking baseline. However, if search already uses distractor memory, there should be no differences between marking and no-marking conditions. In four experiments, with stimuli ranging from Os and Qs to realistic scenes, two consistent data patterns emerged: (1) Marking rejected distractors produced no systematic benefit for search efficiency, as measured by reinspections, reaction times, or errors, and (2) distractor reinspection rates were, overall, extremely low. These results suggest that search uses a memory for rejected distractors, at least in those many real-world search tasks in which gaze is free to move.

Aging, memory and visual search

Potential age-related differences in the memory processes that underlie visual search are examined in the present study. Using a dynamic, gaze-contingent search paradigm developed to assess memory for previously examined distractors, older adults demonstrated no memory deficit. Surprisingly, older adults made fewer refixations compared to their younger counterparts, indicating better memory for previously inspected objects. This improved memory was not the result of a speed-accuracy trade-off or larger Inhibition-of-Return effects for older than for younger adults. Additional analyses suggested that older adults may derive their benefit from finer spatial encoding of search items. These findings suggest that some of the memory processes that support visual search are relatively age invariant.