Automatic and intentional memory processes in visual search

Target selection during "snapshot" foraging

While previous foraging studies have identified key variables that determine attentional selection, they are affected by the global statistics of the tasks. In most studies, targets are selected one at a time without replacement while distractor numbers remain constant, steadily reducing the ratios of targets to distractors with every selection. We designed a foraging task with a sequence of local "snapshots" of foraging displays, with each snapshot requiring a target selection. This enabled tighter control of local target and distractor type ratios while maintaining the flavor of a sequential, multiple-target foraging task. Observers saw only six items for each target selection during a "snapshot" containing varying numbers of two target types and two distractor types. After each selection, a new six-item array (the following snapshot) immediately appeared, centered on the locus of the last selected target. We contrasted feature-based and conjunction-based foraging and analyzed the data by the proportion of different target types in each trial. We found that target type proportion affected selection, with longer response times during conjunction foraging when the number of the alternate target types was greater than the repeated target types. In addition, the choice of target in each snapshot was influenced by the relative positions of selected targets and distractors during preceding snapshots. Importantly, this shows to what degree previous findings on foraging can be attributed to changing global statistics of the foraging array. We propose that "snapshot foraging" can increase experimental control in understanding how people choose targets during continuous attentional orienting.

Ketamine-Induced Alteration of Working Memory Utility during Oculomotor Foraging Task in Monkeys

Impairments of working memory (WM) are commonly observed in a variety of neurodegenerative disorders but they are difficult to quantitatively assess in clinical cases. Recent studies in experimental animals have used low-dose ketamine (an NMDA receptor antagonist) to disrupt WM, partly mimicking the pathophysiology of schizophrenia. Here, we developed a novel behavioral paradigm to assess multiple components of WM and applied it to monkeys with and without ketamine administration. In an oculomotor foraging task, the animals were presented with 15 identical objects on the screen. One of the objects was associated with a liquid reward, and monkeys were trained to search for the target by generating sequential saccades under a time constraint. We assumed that the occurrence of recursive movements to the same object might reflect WM dysfunction. We constructed a "foraging model" that incorporated (1) memory capacity, (2) memory decay, and (3) utility rate; this model was able to explain more than 92% of the variations in behavioral data obtained from three monkeys. Following systemic administration of low dosages of ketamine, the memory capacity and utility rate were dramatically reduced by 15% and 57%, respectively, while memory decay remained largely unchanged. These results suggested that the behavioral deficits during the blockade of NMDA receptors were mostly due to the decreased usage of short-term memory. Our oculomotor paradigm and foraging model appear to be useful for quantifying multiple components of WM and could be applicable to clinical cases in future studies.

Inhibition of return: An information processing theory of its natures and significance

Inhibition of return (IOR) is an inhibitory aftereffect of visuospatial orienting, typically resulting in slower responses to targets presented in an area that has been recently attended. Since its discovery, myriad research has sought to explain the causes and effects underlying this phenomenon. Here, we briefly summarize the history of the phenomenon, and describe the early work supporting the functional significance of IOR as a foraging facilitator. We then shine a light on the discordance in the literature with respect to mechanism-in particular the lack of theoretical constructs that can consistently explain innumerable dissociations. We then describe three diagnostics (central arrow targets, locus of slack logic and the psychological refractory period, and performance in speed-accuracy space) used to support our theory that there are two forms of inhibition of return-the form which is manifest being contingent upon the activation state of the reflexive oculomotor system. The input form, which operates to decrease the salience of inputs, is generated when the reflexive oculomotor system is suppressed; the output form, which operates to bias responding, is generated when the reflexive oculomotor system is not suppressed. Then, we subject a published data set, wherein inhibitory effects had been generated while the reflexive oculomotor system was either active or suppressed, to diffusion modelling. As we hypothesized, based on the aforementioned theory, the effects of the two forms of IOR were best accounted for by different drift diffusion parameters. The paper ends with a variety of suggestions for further research.

Post-search IOR: Searching for inhibition of return after search

Previous research has indicated that Inhibition of return (IOR) supports visual search by discouraging the re-inspection of recently inspected items during search. However, it is not clear whether IOR persists after a search is completed or whether this depends on the presence of a further search in the same display. To investigate this issue, we had participants search consecutively twice in the same display (Experiment 1). Immediately after the end of the first search and after the end of the second search we probed an item which had been recently inspected or not in the previous search. The results showed that IOR as measured by the saccadic latency to the probed items was absent after the end of each of the two successive searches. In Experiment 2, we measured both saccadic latencies and manual responses in a single-search paradigm. We found that IOR during and after the search was present for saccadic responses but absent for manual responses. This suggests that IOR during and after a visual search depends on the modality of the response and the number of required searches.

Revisiting the global effect and inhibition of return

Saccades toward previously cued locations have longer latencies than saccades toward other locations, a phenomenon known as inhibition of return (IOR). Watanabe (Exp Brain Res 138:330–342. doi:10.1007/s002210100709, 2001) combined IOR with the global effect (where saccade landing points fall in between neighboring objects) to investigate whether IOR can also have a spatial component. When one of two neighboring targets was cued, there was a clear bias away from the cued location. In a condition where both targets were cued, it appeared that the global effect magnitude was similar to the condition without any cues. However, as the latencies in the double cue condition were shorter compared to the no cue condition, it is still an open question whether these results are representative for IOR. Considering the double cue condition can provide valuable insight into the interaction of the mechanisms underlying the two phenomena, here, we revisit this condition in an adapted paradigm. Our paradigm does result in longer latencies for the cued locations, and we find that the magnitude of the global effect is reduced significantly. Unexpectedly, this holds even when only including saccades with the same latencies for both conditions. Thus, the increased latencies associated with IOR cannot directly explain the reduction in global effect. The global effect reduction can likely best be seen as either a result of short-term depression of exogenous visual signals or a result of IOR established at the center of gravity of cues.

Spatial inhibition of return as a function of fixation history, task, and spatial references

In oculomotor selection, each saccade is thought to be automatically biased toward uninspected locations, inhibiting the inefficient behavior of repeatedly refixating the same objects. This automatic bias is related to inhibition of return (IOR). Although IOR seems an appealing property that increases efficiency in visual search, such a mechanism would not be efficient in other tasks. Indeed, evidence for additional, more flexible control over refixations has been provided. Here, we investigated whether task demands implicitly affect the rate of refixations. We measured the probability of refixations after series of six binary saccadic decisions under two conditions: visual search and free viewing. The rate of refixations seems influenced by two effects. One effect is related to the rate of intervening fixations, specifically, more refixations were observed with more intervening fixations. In addition, we observed an effect of task set, with fewer refixations in visual search than in free viewing. Importantly, the history-related effect was more pronounced when sufficient spatial references were provided, suggesting that this effect is dependent on spatiotopic encoding of previously fixated locations. This known history-related bias in gaze direction is not the primary influence on the refixation rate. Instead, multiple factors, such as task set and spatial references, assert strong influences as well.

Seek and you shall remember: scene semantics interact with visual search to build better memories

Memorizing critical objects and their locations is an essential part of everyday life. In the present study, incidental encoding of objects in naturalistic scenes during search was compared to explicit memorization of those scenes. To investigate if prior knowledge of scene structure influences these two types of encoding differently, we used meaningless arrays of objects as well as objects in real-world, semantically meaningful images. Surprisingly, when participants were asked to recall scenes, their memory performance was markedly better for searched objects than for objects they had explicitly tried to memorize, even though participants in the search condition were not explicitly asked to memorize objects. This finding held true even when objects were observed for an equal amount of time in both conditions. Critically, the recall benefit for searched over memorized objects in scenes was eliminated when objects were presented on uniform, non-scene backgrounds rather than in a full scene context. Thus, scene semantics not only help us search for objects in naturalistic scenes, but appear to produce a representation that supports our memory for those objects beyond intentional memorization.

Neural limits to representing objects still within view

Visual working memory is an online workspace for temporarily representing visual information from the environment. The two most prevalent empirical characteristics of working memory are that it is supported by sustained neural activity over a delay period and it has a severely limited capacity for representing multiple items simultaneously. Traditionally, such delay activity and capacity limits have been considered to be exclusive for maintaining information about objects that are no longer visible to the observers. Here, by contrast, we provide both neurophysiological and psychophysical evidence that the sustained neural activity and capacity limits for items that are continuously visible to the human observer are indistinguishable from those measured for items that are no longer visible. This holds true even when the observers know that the objects will not disappear from the visual field. These results demonstrate that our explicit representation of objects that are still "in view" is far more limited than previously assumed.

The development of organized visual search

Visual search plays an important role in guiding behavior. Children have more difficulty performing conjunction search tasks than adults. The present research evaluates whether developmental differences in children's ability to organize serial visual search (i.e., search organization skills) contribute to performance limitations in a typical conjunction search task. We evaluated 134 children between the ages of 2 and 17 on separate tasks measuring search for targets defined by a conjunction of features or by distinct features. Our results demonstrated that children organize their visual search better as they get older. As children's skills at organizing visual search improve they become more accurate at locating targets with conjunction of features amongst distractors, but not for targets with distinct features. Developmental limitations in children's abilities to organize their visual search of the environment are an important component of poor conjunction search in young children. In addition, our findings provide preliminary evidence that, like other visuospatial tasks, exposure to reading may influence children's spatial orientation to the visual environment when performing a visual search.

New Evidence for Strategic Differences between Static and Dynamic Search Tasks: An Individual Observer Analysis of Eye Movements

Two experiments are reported that further explore the processes underlying dynamic search. In Experiment 1, observers’ oculomotor behavior was monitored while they searched for a randomly oriented T among oriented L distractors under static and dynamic viewing conditions. Despite similar search slopes, eye movements were less frequent and more spatially constrained under dynamic viewing relative to static, with misses also increasing more with target eccentricity in the dynamic condition. These patterns suggest that dynamic search involves a form of sit-and-wait strategy in which search is restricted to a small group of items surrounding fixation. To evaluate this interpretation, we developed a computational model of a sit-and-wait process hypothesized to underlie dynamic search. In Experiment 2 we tested this model by varying fixation position in the display and found that display positions optimized for a sit-and-wait strategy resulted in higher d′ values relative to a less optimal location. We conclude that different strategies, and therefore underlying processes, are used to search static and dynamic displays.

The interplay of episodic and semantic memory in guiding repeated search in scenes

It seems intuitive to think that previous exposure or interaction with an environment should make it easier to search through it and, no doubt, this is true in many real-world situations. However, in a recent study, we demonstrated that previous exposure to a scene does not necessarily speed search within that scene. For instance, when observers performed as many as 15 searches for different objects in the same, unchanging scene, the speed of search did not decrease much over the course of these multiple searches (Võ & Wolfe, 2012). Only when observers were asked to search for the same object again did search become considerably faster. We argued that our naturalistic scenes provided such strong "semantic" guidance-e.g., knowing that a faucet is usually located near a sink-that guidance by incidental episodic memory-having seen that faucet previously-was rendered less useful. Here, we directly manipulated the availability of semantic information provided by a scene. By monitoring observers' eye movements, we found a tight coupling of semantic and episodic memory guidance: Decreasing the availability of semantic information increases the use of episodic memory to guide search. These findings have broad implications regarding the use of memory during search in general and particularly during search in naturalistic scenes.

Inhibition of return: Twenty years after

When responding to a suddenly appearing stimulus, we are slower and/or less accurate when the stimulus occurs at the same location of a previous event than when it appears in a new location. This phenomenon, often referred to as inhibition of return (IOR), has fostered a huge amount of research in the last 20 years. In this selective review, which introduces a Special Issue of Cognitive Neuropsychology dedicated to IOR, we discuss some of the methods used for eliciting IOR and its boundary conditions. We also address its debated relationships with orienting of attention, succinctly review findings of altered IOR in normal elderly and neuropsychiatric patients, and present results concerning its possible neural bases. We conclude with an outline of the papers collected in this issue, which offer a more in-depth treatment of behavioural, neural, and theoretical issues related to IOR.

Active inhibition and memory promote exploration and search of natural scenes

Active exploration of the visual world depends on sequential shifts of gaze that bring prioritized regions of a scene into central vision. The efficiency of this system is commonly attributed to a mechanism of "inhibition of return" (IOR) that discourages re-examination of previously-visited locations. Such a process is fundamental to computational models of attentional selection and paralleled by neurophysiological observations of inhibition of target-related activity in visuomotor areas. However, studies examining eye movements in naturalistic visual scenes appear to contradict the hypothesis that IOR promotes exploration. Instead, these reports reveal a surprisingly strong tendency to shift gaze back to the previously fixated location, suggesting that refixations might even be facilitated under natural conditions. Here we resolve this apparent contradiction, based on a probabilistic analysis of gaze patterns recorded during both free-viewing and search of naturalistic scenes. By simulating saccadic selection based on instantaneous influences alone, we show that the observed frequency of return saccades is in fact substantially less than predicted for a memoryless system, demonstrating that refixation is actively inhibited under natural viewing conditions. Furthermore, these observations reveal that gaze history significantly influences the way in which natural scenes are explored, contrary to accounts that suggest visual search has no memory.

Do object refixations during scene viewing indicate rehearsal in visual working memory?

Do refixations serve a rehearsal function in visual working memory (VWM)? We analyzed refixations from observers freely viewing multiobject scenes. An eyetracker was used to limit the viewing of a scene to a specified number of objects fixated after the target (intervening objects), followed by a four-alternative forced choice recognition test. Results showed that the probability of target refixation increased with the number of fixated intervening objects, and these refixations produced a 16% accuracy benefit over the first five intervening-object conditions. Additionally, refixations most frequently occurred after fixations on only one to two other objects, regardless of the intervening-object condition. These behaviors could not be explained by random or minimally constrained computational models; a VWM component was required to completely describe these data. We explain these findings in terms of a monitor-refixate rehearsal system: The activations of object representations in VWM are monitored, with refixations occurring when these activations decrease suddenly.

When does repeated search in scenes involve memory? Looking at versus looking for objects in scenes

One might assume that familiarity with a scene or previous encounters with objects embedded in a scene would benefit subsequent search for those items. However, in a series of experiments we show that this is not the case: When participants were asked to subsequently search for multiple objects in the same scene, search performance remained essentially unchanged over the course of searches despite increasing scene familiarity. Similarly, looking at target objects during previews, which included letter search, 30 seconds of free viewing, or even 30 seconds of memorizing a scene, also did not benefit search for the same objects later on. However, when the same object was searched for again memory for the previous search was capable of producing very substantial speeding of search despite many different intervening searches. This was especially the case when the previous search engagement had been active rather than supported by a cue. While these search benefits speak to the strength of memory-guided search when the same search target is repeated, the lack of memory guidance during initial object searches-despite previous encounters with the target objects-demonstrates the dominance of guidance by generic scene knowledge in real-world search.

Rapid Communication: Finding memory in search: The effect of visual working memory load on visual search

There is now substantial evidence that during visual search, previously searched distractors are stored in memory to prevent them from being reselected. Studies examining which memory resources are involved in this process have indicated that while a concurrent spatial working memory task does affect search slopes, depleting visual working memory (VWM) resources does not. In the present study, we confirm that VWM load indeed has no effect on the search slope; however, there is an increase in overall reaction times that is directly related to the number of items held in VWM. Importantly, this effect on search time increases proportionally with the memory load until the capacity of VWM is reached. Furthermore, the search task interfered with the number of items stored in VWM during the concurrent change-detection task. These findings suggest that VWM plays a role in the inhibition of previously searched distractors.

Visual search elicits the electrophysiological marker of visual working memory

BACKGROUND

Although limited in capacity, visual working memory (VWM) plays an important role in many aspects of visually-guided behavior. Recent experiments have demonstrated an electrophysiological marker of VWM encoding and maintenance, the contralateral delay activity (CDA), which has been shown in multiple tasks that have both explicit and implicit memory demands. Here, we investigate whether the CDA is evident during visual search, a thoroughly-researched task that is a hallmark of visual attention but has no explicit memory requirements.

METHODOLOGY/PRINCIPAL FINDINGS

The results demonstrate that the CDA is present during a lateralized search task, and that it is similar in amplitude to the CDA observed in a change-detection task, but peaks slightly later. The changes in CDA amplitude during search were strongly correlated with VWM capacity, as well as with search efficiency. These results were paralleled by behavioral findings showing a strong correlation between VWM capacity and search efficiency.

CONCLUSIONS/SIGNIFICANCE

We conclude that the activity observed during visual search was generated by the same neural resources that subserve VWM, and that this activity reflects the maintenance of previously searched distractors.

A Role for Spatial and Nonspatial Working Memory Processes in Visual Search

Searching a cluttered visual scene for a specific item of interest can take several seconds to perform if the target item is difficult to discriminate from surrounding items. Whether working memory processes are utilized to guide the path of attentional selection during such searches remains under debate. Previous studies have found evidence to support a role for spatial working memory in inefficient search, but the role of nonspatial working memory remains unclear. Here, we directly compared the role of spatial and nonspatial working memory for both an efficient and inefficient search task. In Experiment 1, we used a dual-task paradigm to investigate the effect of performing visual search within the retention interval of a spatial working memory task. Importantly, by incorporating two working memory loads (low and high) we were able to make comparisons between dual-task conditions, rather than between dual-task and single-task conditions. This design allows any interference effects observed to be attributed to changes in memory load, rather than to nonspecific effects related to “dual-task” performance. We found that the efficiency of the inefficient search task declined as spatial memory load increased, but that the efficient search task remained efficient. These results suggest that spatial memory plays an important role in inefficient but not efficient search. In Experiment 2, participants performed the same visual search tasks within the retention interval of visually matched spatial and verbal working memory tasks. Critically, we found comparable dual-task interference between inefficient search and both the spatial and nonspatial working memory tasks, indicating that inefficient search recruits working memory processes common to both domains.

Visual search is guided by prospective and retrospective memory

Although there has been some controversy as to whether attention is guided by memory during visual search, recent findings have suggested that memory helps to prevent attention from needlessly reinspecting examined items. Until now, it has been assumed that some form of retrospective memory is responsible for keeping track of examined items and preventing revisitations. Alternatively, some form of prospective memory, such as strategic scanpath planning, could be responsible for guiding attention away from examined items. We used a new technique that allowed us to selectively prevent retrospective or prospective memory from contributing to search. We demonstrated that both retrospective and prospective memory guide attention during visual search.

Memory for the search path: evidence for a high-capacity representation of search history

Using a gaze-contingent paradigm, we directly measured observers' memory capacity for fixated distractor locations during search. After approximately half of the search objects had been fixated, they were masked and a spatial probe appeared at either a previously fixated location or a non-fixated location; observers then rated their confidence that the target had appeared at the probed location. Observers were able to differentiate the 12 most recently fixated distractor locations from non-fixated locations, but analyses revealed that these locations were represented fairly coarsely. We conclude that there exists a high-capacity, but low-resolution, memory for a search path.

Age-related differences in the use of background layout in visual search

The effect of background layout on visual search performance, and more specifically on the tendency to refixate previously inspected locations and objects, was investigated. Older and younger adults performed a search task in which a background layout or landmark was present or absent in a gaze contingent visual search paradigm. Regardless of age, participants demonstrated fewer refixations when landmarks were present, with older adults showing a larger landmark advantage. This visual search advantage did not come at the cost of saccadic latency. Furthermore, the visual search performance advantage obtained in the presence of a background layout or landmark was observed both for individuals with small and large memory spans.

Memory for where, but not what, is used during visual search

Although the role of memory in visual search is debatable, most researchers agree with a limited-capacity model of memory in visual search. The authors demonstrate the role of memory by replicating previous findings showing that visual search is biased away from old items (previously examined items) and toward new items (nonexamined items). Furthermore, the authors examined the type of memory representations used to bias search by changing an item's individuating feature or location during search. Changing the individuating feature of an item did not disrupt normal search biases. However, when the location of an item changed, normal search biases were disrupted. These results suggest that memory used in visual search is based on items' locations rather than their identity.

Aging, memory and visual search

Potential age-related differences in the memory processes that underlie visual search are examined in the present study. Using a dynamic, gaze-contingent search paradigm developed to assess memory for previously examined distractors, older adults demonstrated no memory deficit. Surprisingly, older adults made fewer refixations compared to their younger counterparts, indicating better memory for previously inspected objects. This improved memory was not the result of a speed-accuracy trade-off or larger Inhibition-of-Return effects for older than for younger adults. Additional analyses suggested that older adults may derive their benefit from finer spatial encoding of search items. These findings suggest that some of the memory processes that support visual search are relatively age invariant.