Visual search has memory

Numerosity estimates for attended and unattended items in visual search

The goal of this research was to examine memories created for the number of items during a visual search task. Participants performed a visual search task for a target defined by a single feature (Experiment 1A), by a conjunction of features (Experiment 1B), or by a specific spatial configuration of features (Experiment 1C). On some trials following the search task, subjects were asked to recall the total number of items in the previous display. In all search types, participants underestimated the total number of items, but the severity of the underestimation varied depending on the efficiency of the search. In three follow-up studies (Experiments 2A, 2B, and 2C) using the same visual stimuli, the participants' only task was to estimate the number of items on each screen. Participants still underestimated the numerosity of the items, although the degree of underestimation was smaller than in the search tasks and did not depend on the type of visual stimuli. In Experiment 3, participants were asked to recall the number of items in a display only once. Subjects still displayed a tendency to underestimate, indicating that the underestimation effects seen in Experiments 1A-1C were not attributable to knowledge of the estimation task. The degree of underestimation depends on the efficiency of the search task, with more severe underestimation in efficient search tasks. This suggests that the lower attentional demands of very efficient searches leads to less encoding of numerosity of the distractor set.

Independence of long-term contextual memory and short-term perceptual hypotheses: Evidence from contextual cueing of interrupted search

Observers are able to resume an interrupted search trial faster relative to responding to a new, unseen display. This finding of rapid resumption is attributed to short-term perceptual hypotheses generated on the current look and confirmed upon subsequent looks at the same display. It has been suggested that the contents of perceptual hypotheses are similar to those of other forms of memory acquired long-term through repeated exposure to the same search displays over the course of several trials, that is, the memory supporting “contextual cueing.” In three experiments, we investigated the relationship between short-term perceptual hypotheses and long-term contextual memory. The results indicated that long-term, contextual memory of repeated displays neither affected the generation nor the confirmation of short-term perceptual hypotheses for these displays. Furthermore, the analysis of eye movements suggests that long-term memory provides an initial benefit in guiding attention to the target, whereas in subsequent looks guidance is entirely based on short-term perceptual hypotheses. Overall, the results reveal a picture of both long- and short-term memory contributing to reliable performance gains in interrupted search, while exerting their effects in an independent manner.

Semantic and Syntactic Associations During Word Search Modulate the Relationship Between Attention and Subsequent Memory

Two experiments were conducted to investigate how linguistic information influences attention allocation in visual search and memory for words. In Experiment 1, participants searched for the synonym of a cue word among five words. The distractors included one antonym and three unrelated words. In Experiment 2, participants were asked to judge whether the five words presented on the screen comprise a valid sentence. The relationships among words were sentential, semantically related or unrelated. A memory recognition task followed. Results in both experiments showed that linguistically related words produced better memory performance. We also found that there were significant interactions between linguistic relation conditions and memorization on eye-movement measures, indicating that good memory for words relied on frequent and long fixations during search in the unrelated condition but to a much lesser extent in linguistically related conditions. We conclude that semantic and syntactic associations attenuate the link between overt attention allocation and subsequent memory performance, suggesting that linguistic relatedness can somewhat compensate for a relative lack of attention during word search.

Cognitive architecture enables comprehensive predictive models of visual search

With a simple demonstration model, Hulleman & Olivers (H&O) effectively argue that theories of visual search need an overhaul. We point to related literature in which visual search is modeled in even more detail through the use of computational cognitive architectures that incorporate fundamental perceptual, cognitive, and motor mechanisms; the result of such work thus far bolsters their arguments considerably.

A novel computational model to probe visual search deficits during motor performance

Successful execution of many motor skills relies on well-organized visual search (voluntary eye movements that actively scan the environment for task-relevant information). Although impairments of visual search that result from brain injuries are linked to diminished motor performance, the neural processes that guide visual search within this context remain largely unknown. The first objective of this study was to examine how visual search in healthy adults and stroke survivors is used to guide hand movements during the Trail Making Test (TMT), a neuropsychological task that is a strong predictor of visuomotor and cognitive deficits. Our second objective was to develop a novel computational model to investigate combinatorial interactions between three underlying processes of visual search (spatial planning, working memory, and peripheral visual processing). We predicted that stroke survivors would exhibit deficits in integrating the three underlying processes, resulting in deteriorated overall task performance. We found that normal TMT performance is associated with patterns of visual search that primarily rely on spatial planning and/or working memory (but not peripheral visual processing). Our computational model suggested that abnormal TMT performance following stroke is associated with impairments of visual search that are characterized by deficits integrating spatial planning and working memory. This innovative methodology provides a novel framework for studying how the neural processes underlying visual search interact combinatorially to guide motor performance.

NEW & NOTEWORTHY

Visual search has traditionally been studied in cognitive and perceptual paradigms, but little is known about how it contributes to visuomotor performance. We have developed a novel computational model to examine how three underlying processes of visual search (spatial planning, working memory, and peripheral visual processing) contribute to visual search during a visuomotor task. We show that deficits integrating spatial planning and working memory underlie abnormal performance in stroke survivors with frontoparietal damage.

Distractor Dwelling, Skipping, and Revisiting Determine Target Absent Performance in Difficult Visual Search

Some targets in visual search are more difficult to find than others. In particular, a target that is similar to the distractors is more difficult to find than a target that is dissimilar to the distractors. Efficiency differences between easy and difficult searches are manifest not only in target-present trials but also in target-absent trials. In fact, even physically identical displays are searched through with different efficiency depending on the searched-for target. Here, we monitored eye movements in search for a target similar to the distractors (difficult search) versus a target dissimilar to the distractors (easy search). We aimed to examine three hypotheses concerning the causes of differential search efficiencies in target-absent trials: (a) distractor dwelling (b) distractor skipping, and (c) distractor revisiting. Reaction times increased with target similarity which is consistent with existing theories and replicates earlier results. Eye movement data indicated guidance in target trials, even though search was very slow. Dwelling, skipping, and revisiting contributed to low search efficiency in difficult search, with dwelling being the strongest factor. It is argued that differences in dwell time account for a large amount of total search time differences.

Modeling Lag-2 Revisits to Understand Trade-Offs in Mixed Control of Fixation Termination During Visual Search

An important question about eye-movement behavior is when the decision is made to terminate a fixation and program the following saccade. Different approaches have found converging evidence in favor of a mixed-control account, in which there is some overlap between processing information at fixation and planning the following saccade. We examined one interesting instance of mixed control in visual search: lag-2 revisits, during which observers fixate a stimulus, move to a different stimulus, and then revisit the first stimulus on the next fixation. Results show that the probability of lag-2 revisits occurring increased with the number of target-similar stimuli, and revisits were preceded by a brief fixation on the intervening distractor stimulus. We developed the Efficient Visual Sampling (EVS) computational model to simulate our findings (fixation durations and fixation locations) and to provide insight into mixed control of fixations and the perceptual, cognitive, and motor processes that produce lag-2 revisits.

Spatial inhibition of return as a function of fixation history, task, and spatial references

In oculomotor selection, each saccade is thought to be automatically biased toward uninspected locations, inhibiting the inefficient behavior of repeatedly refixating the same objects. This automatic bias is related to inhibition of return (IOR). Although IOR seems an appealing property that increases efficiency in visual search, such a mechanism would not be efficient in other tasks. Indeed, evidence for additional, more flexible control over refixations has been provided. Here, we investigated whether task demands implicitly affect the rate of refixations. We measured the probability of refixations after series of six binary saccadic decisions under two conditions: visual search and free viewing. The rate of refixations seems influenced by two effects. One effect is related to the rate of intervening fixations, specifically, more refixations were observed with more intervening fixations. In addition, we observed an effect of task set, with fewer refixations in visual search than in free viewing. Importantly, the history-related effect was more pronounced when sufficient spatial references were provided, suggesting that this effect is dependent on spatiotopic encoding of previously fixated locations. This known history-related bias in gaze direction is not the primary influence on the refixation rate. Instead, multiple factors, such as task set and spatial references, assert strong influences as well.

When High-Capacity Readers Slow Down and Low-Capacity Readers Speed Up: Working Memory and Locality Effects

We examined the effects of argument-head distance in SVO and SOV languages (Spanish and German), while taking into account readers' working memory capacity and controlling for expectation (Levy, 2008) and other factors. We predicted only locality effects, that is, a slowdown produced by increased dependency distance (Gibson, 2000; Lewis and Vasishth, 2005). Furthermore, we expected stronger locality effects for readers with low working memory capacity. Contrary to our predictions, low-capacity readers showed faster reading with increased distance, while high-capacity readers showed locality effects. We suggest that while the locality effects are compatible with memory-based explanations, the speedup of low-capacity readers can be explained by an increased probability of retrieval failure. We present a computational model based on ACT-R built under the previous assumptions, which is able to give a qualitative account for the present data and can be tested in future research. Our results suggest that in some cases, interpreting longer RTs as indexing increased processing difficulty and shorter RTs as facilitation may be too simplistic: The same increase in processing difficulty may lead to slowdowns in high-capacity readers and speedups in low-capacity ones. Ignoring individual level capacity differences when investigating locality effects may lead to misleading conclusions.

Persistence in eye movement during visual search

As any cognitive task, visual search involves a number of underlying processes that cannot be directly observed and measured. In this way, the movement of the eyes certainly represents the most explicit and closest connection we can get to the inner mechanisms governing this cognitive activity. Here we show that the process of eye movement during visual search, consisting of sequences of fixations intercalated by saccades, exhibits distinctive persistent behaviors. Initially, by focusing on saccadic directions and intersaccadic angles, we disclose that the probability distributions of these measures show a clear preference of participants towards a reading-like mechanism (geometrical persistence), whose features and potential advantages for searching/foraging are discussed. We then perform a Multifractal Detrended Fluctuation Analysis (MF-DFA) over the time series of jump magnitudes in the eye trajectory and find that it exhibits a typical multifractal behavior arising from the sequential combination of saccades and fixations. By inspecting the time series composed of only fixational movements, our results reveal instead a monofractal behavior with a Hurst exponent , which indicates the presence of long-range power-law positive correlations (statistical persistence). We expect that our methodological approach can be adopted as a way to understand persistence and strategy-planning during visual search.

A geometric method for computing ocular kinematics and classifying gaze events using monocular remote eye tracking in a robotic environment

BACKGROUND

Robotic and virtual-reality systems offer tremendous potential for improving assessment and rehabilitation of neurological disorders affecting the upper extremity. A key feature of these systems is that visual stimuli are often presented within the same workspace as the hands (i.e., peripersonal space). Integrating video-based remote eye tracking with robotic and virtual-reality systems can provide an additional tool for investigating how cognitive processes influence visuomotor learning and rehabilitation of the upper extremity. However, remote eye tracking systems typically compute ocular kinematics by assuming eye movements are made in a plane with constant depth (e.g. frontal plane). When visual stimuli are presented at variable depths (e.g. transverse plane), eye movements have a vergence component that may influence reliable detection of gaze events (fixations, smooth pursuits and saccades). To our knowledge, there are no available methods to classify gaze events in the transverse plane for monocular remote eye tracking systems. Here we present a geometrical method to compute ocular kinematics from a monocular remote eye tracking system when visual stimuli are presented in the transverse plane. We then use the obtained kinematics to compute velocity-based thresholds that allow us to accurately identify onsets and offsets of fixations, saccades and smooth pursuits. Finally, we validate our algorithm by comparing the gaze events computed by the algorithm with those obtained from the eye-tracking software and manual digitization.

RESULTS

Within the transverse plane, our algorithm reliably differentiates saccades from fixations (static visual stimuli) and smooth pursuits from saccades and fixations when visual stimuli are dynamic.

CONCLUSIONS

The proposed methods provide advancements for examining eye movements in robotic and virtual-reality systems. Our methods can also be used with other video-based or tablet-based systems in which eye movements are performed in a peripersonal plane with variable depth.

Working memory differences in long-distance dependency resolution

There is a wealth of evidence showing that increasing the distance between an argument and its head leads to more processing effort, namely, locality effects; these are usually associated with constraints in working memory (DLT: Gibson, 2000; activation-based model: Lewis and Vasishth, 2005). In SOV languages, however, the opposite effect has been found: antilocality (see discussion in Levy et al., 2013). Antilocality effects can be explained by the expectation-based approach as proposed by Levy (2008) or by the activation-based model of sentence processing as proposed by Lewis and Vasishth (2005). We report an eye-tracking and a self-paced reading study with sentences in Spanish together with measures of individual differences to examine the distinction between expectation- and memory-based accounts, and within memory-based accounts the further distinction between DLT and the activation-based model. The experiments show that (i) antilocality effects as predicted by the expectation account appear only for high-capacity readers; (ii) increasing dependency length by interposing material that modifies the head of the dependency (the verb) produces stronger facilitation than increasing dependency length with material that does not modify the head; this is in agreement with the activation-based model but not with the expectation account; and (iii) a possible outcome of memory load on low-capacity readers is the increase in regressive saccades (locality effects as predicted by memory-based accounts) or, surprisingly, a speedup in the self-paced reading task; the latter consistent with good-enough parsing (Ferreira et al., 2002). In sum, the study suggests that individual differences in working memory capacity play a role in dependency resolution, and that some of the aspects of dependency resolution can be best explained with the activation-based model together with a prediction component.

Comparing target detection errors in visual search and manually-assisted search

Subjects searched for low- or high-prevalence targets among static nonoverlapping items or items piled in heaps that could be moved using a computer mouse. We replicated the classical prevalence effect both in visual search and when unpacking items from heaps, with more target misses under low prevalence. Moreover, we replicated our previous finding that while unpacking, people often move the target item without noticing (the unpacking error) and determined that these errors also increase under low prevalence. On the basis of a comparison of item movements during the manually-assisted search and eye movements during static visual search, we suggest that low prevalence leads to broadly reduced diligence during search but that the locus of this reduced diligence depends on the nature of the task. In particular, while misses during visual search often arise from a failure to inspect all of the items, misses during manually-assisted search more often result from a failure to adequately inspect individual items. Indeed, during manually-assisted search, over 90 % of target misses occurred despite subjects having moved the target item during search.

Fixation duration surpasses pupil size as a measure of memory load in free viewing

Oculomotor behavior reveals, not only the acquisition of visual information at fixation, but also the accumulation of information in memory across subsequent fixations. Two candidate measures were considered as indicators of such dynamic visual memory load: fixation duration and pupil size. While recording these measures, we displayed an arrangement of 3, 4 or 5 targets among distractors. Both occurred in various orientations. Participants searched for targets and reported whether in a subsequent display one of them had changed orientation. We determined to what extent fixation duration and pupil size indicate dynamic memory load, as a function of the number of targets fixated during the search. We found that fixation duration reflects the number of targets, both when this number is within and above the limit of working memory capacity. Pupil size reflects the number of targets only when it exceeds the capacity limit. Moreover, the duration of fixations on successive targets but not on distractors increases whereas pupil size does not. The increase in fixation duration with number of targets both within and above working memory capacity suggests that in free viewing fixation duration is sensitive to actual memory load as well as to processing load, whereas pupil size is indicative of processing load only. Two alternative models relating visual attention and working memory are considered relevant to these results. We discuss the results as supportive of a model which involves a temporary buffer in the interaction of attention and working memory.

Exploration versus exploitation in space, mind, and society

Search is a ubiquitous property of life. Although diverse domains have worked on search problems largely in isolation, recent trends across disciplines indicate that the formal properties of these problems share similar structures and, often, similar solutions. Moreover, internal search (e.g., memory search) shows similar characteristics to external search (e.g., spatial foraging), including shared neural mechanisms consistent with a common evolutionary origin across species. Search problems and their solutions also scale from individuals to societies, underlying and constraining problem solving, memory, information search, and scientific and cultural innovation. In summary, search represents a core feature of cognition, with a vast influence on its evolution and processes across contexts and requiring input from multiple domains to understand its implications and scope.

Searching for the highest number

When viewing a collection of products how does a consumer decide which one to buy? To do this task, the consumer not only needs to evaluate the desirability of the products, taking into account factors such as quality and price, but also needs to search through the products to find the most desirable one. We studied the search process using an abstraction of a common consumer choice task. In our task, observers searched an array of numbers for the highest. Crucially, the observers did not know in advance what this number would be, which made it difficult to know when the search should be terminated. In this way, our search task mimicked a problem often faced by consumers in a supermarket setting where they also may not know in advance what the most desirable product will be. We compared several computational models. We found that our data was best described by a process that assumes that observers terminate their search when they find a number that exceeds an internal threshold. Depending on the observer and the circumstances, this threshold appeared either to be fixed or to decrease over the course of the trial. This threshold can explain why in some situations the observers terminate the search without inspecting all the numbers in the display, whereas in other situations observers act in a seemingly irrational manner, continuing the search even after inspecting all the numbers.

Exploring the breadth of the top-down representations that control attentional disengagement

Recent studies attribute top-down control as the primary determinant of the speed with which attention can be disengaged from an object, and disengagement has received increased interest as a means to distinguish between top-down and bottom-up accounts of attention capture. We present the results of three experiments exploring the breadth of the representations that delay attentional disengagement based on top-down goals. Experiments 1 and 2 examined whether objects similar to an observers' search target, but differing in luminance and hue, delayed the reallocation of attention in a search paradigm designed to isolate disengagement time. Experiment 3 explored whether the representations that delay disengagement are based on absolute similarity with the search target, or are tuned based on target/nontarget relationships. These three studies confirmed the role of top-down goals in automatically contributing to dwell times and revealed that the representations that underlie disengagement effects are broad (automatically delaying disengagement for items similar, but not identical to, the search target). In some cases, attention sets appeared to be graded in nature, but in others target-distractor relationships influenced the degree to which an irrelevant item held attention. Implications for theories of attention capture and potential functional significance of these automatic effects are discussed.

Seek and you shall remember: scene semantics interact with visual search to build better memories

Memorizing critical objects and their locations is an essential part of everyday life. In the present study, incidental encoding of objects in naturalistic scenes during search was compared to explicit memorization of those scenes. To investigate if prior knowledge of scene structure influences these two types of encoding differently, we used meaningless arrays of objects as well as objects in real-world, semantically meaningful images. Surprisingly, when participants were asked to recall scenes, their memory performance was markedly better for searched objects than for objects they had explicitly tried to memorize, even though participants in the search condition were not explicitly asked to memorize objects. This finding held true even when objects were observed for an equal amount of time in both conditions. Critically, the recall benefit for searched over memorized objects in scenes was eliminated when objects were presented on uniform, non-scene backgrounds rather than in a full scene context. Thus, scene semantics not only help us search for objects in naturalistic scenes, but appear to produce a representation that supports our memory for those objects beyond intentional memorization.

Visual neglect: is there a relationship between impaired spatial working memory and re-cancellation?

In visual search tasks, neglect patients tend to explore and repeatedly re-cancel stimuli on the ipsilesional side, as if they did not realize that they had previously examined the rightward locations favoured by their lateral bias. The aim of this study was to explore the hypothesis that a spatial working memory deficit explains these ipsilesional re-cancellation errors in neglect patients. For the first time, we evaluated spatial working memory and re-cancellation through separate and independent tasks in a group of patients with right hemisphere damage and a diagnosis of left neglect. Results showed impaired spatial working memory in neglect patients. Compared to the control group, neglect patients cancelled fewer targets and made more re-cancellations both on the left side and on the right side. The spatial working memory deficit appears to be related to re-cancellations, but only for some neglect patients. Alternative interpretations of re-exploration of space are discussed.

Analysis of microsaccades and pupil dilation reveals a common decisional origin during visual search

During free viewing visual search, observers often refixate the same locations several times before and after target detection is reported with a button press. We analyzed the rate of microsaccades in the sequence of refixations made during visual search and found two important components. One related to the visual content of the region being fixated; fixations on targets generate more microsaccades and more microsaccades are generated for those targets that are more difficult to disambiguate. The other empathizes non-visual decisional processes; fixations containing the button press generate more microsaccades than those made on the same target but without the button press. Pupil dilation during the same refixations reveals a similar modulation. We inferred that generic sympathetic arousal mechanisms are part of the articulated complex of perceptual processes governing fixational eye movements.

Neural limits to representing objects still within view

Visual working memory is an online workspace for temporarily representing visual information from the environment. The two most prevalent empirical characteristics of working memory are that it is supported by sustained neural activity over a delay period and it has a severely limited capacity for representing multiple items simultaneously. Traditionally, such delay activity and capacity limits have been considered to be exclusive for maintaining information about objects that are no longer visible to the observers. Here, by contrast, we provide both neurophysiological and psychophysical evidence that the sustained neural activity and capacity limits for items that are continuously visible to the human observer are indistinguishable from those measured for items that are no longer visible. This holds true even when the observers know that the objects will not disappear from the visual field. These results demonstrate that our explicit representation of objects that are still "in view" is far more limited than previously assumed.

The development of organized visual search

Visual search plays an important role in guiding behavior. Children have more difficulty performing conjunction search tasks than adults. The present research evaluates whether developmental differences in children's ability to organize serial visual search (i.e., search organization skills) contribute to performance limitations in a typical conjunction search task. We evaluated 134 children between the ages of 2 and 17 on separate tasks measuring search for targets defined by a conjunction of features or by distinct features. Our results demonstrated that children organize their visual search better as they get older. As children's skills at organizing visual search improve they become more accurate at locating targets with conjunction of features amongst distractors, but not for targets with distinct features. Developmental limitations in children's abilities to organize their visual search of the environment are an important component of poor conjunction search in young children. In addition, our findings provide preliminary evidence that, like other visuospatial tasks, exposure to reading may influence children's spatial orientation to the visual environment when performing a visual search.

A common discrete resource for visual working memory and visual search

Visual search, a dominant paradigm within attention research, requires observers to rapidly identify targets hidden among distractors. Major models of search presume that working memory (WM) provides the on-line work space for evaluating potential targets. According to this hypothesis, individuals with higher WM capacity should search more efficiently, because they should be able to apprehend a larger number of search elements at a time. Nevertheless, no compelling evidence of such a correlation has emerged, and this null result challenges a growing consensus that there is strong overlap between the neural processes that limit internal storage and those that limit external selection. Here, we provide multiple demonstrations of robust correlations between WM capacity and search efficiency, and we document a key boundary condition for observing this link. Finally, examination of a neural measure of visual selection capacity (the N2pc) demonstrates that visual search and WM storage are constrained by a common discrete resource.

New Evidence for Strategic Differences between Static and Dynamic Search Tasks: An Individual Observer Analysis of Eye Movements

Two experiments are reported that further explore the processes underlying dynamic search. In Experiment 1, observers’ oculomotor behavior was monitored while they searched for a randomly oriented T among oriented L distractors under static and dynamic viewing conditions. Despite similar search slopes, eye movements were less frequent and more spatially constrained under dynamic viewing relative to static, with misses also increasing more with target eccentricity in the dynamic condition. These patterns suggest that dynamic search involves a form of sit-and-wait strategy in which search is restricted to a small group of items surrounding fixation. To evaluate this interpretation, we developed a computational model of a sit-and-wait process hypothesized to underlie dynamic search. In Experiment 2 we tested this model by varying fixation position in the display and found that display positions optimized for a sit-and-wait strategy resulted in higher d′ values relative to a less optimal location. We conclude that different strategies, and therefore underlying processes, are used to search static and dynamic displays.

Is Sternberg’s Memory Scanning Task Really a Short-Term Memory Task?

Sternberg’s paradigm is currently viewed as a typical short-term memory task and is widely used to tap mnemonic capacities in neuroscience studies. However, Sternberg’s original procedure includes an experimental constraint – recalling the sequence of digits in order – which was not reused in the following studies. In previous research ( Corbin & Marquer, 2008 , 2009 ), we showed that the recall constraint has an impact on the quantitative results as well as on the strategies implemented. These findings led us to wonder whether the presence or absence of this simple experimental constraint could also affect the processes implemented in Sternberg’s task. In order to answer this question, we analyzed the relationships between the performance levels of 50 participants on Sternberg’s task on various well-known span tasks and on a classical visual search task. The results showed that, in the recall condition, Sternberg’s paradigm appears to be a verbal working memory task, whereas in the no-recall condition, the task appears to be a recognition task that involves visuospatial memory capacities. In this latter condition, the processes implemented may be more similar to those implemented in visual search tasks.

The interplay of episodic and semantic memory in guiding repeated search in scenes

It seems intuitive to think that previous exposure or interaction with an environment should make it easier to search through it and, no doubt, this is true in many real-world situations. However, in a recent study, we demonstrated that previous exposure to a scene does not necessarily speed search within that scene. For instance, when observers performed as many as 15 searches for different objects in the same, unchanging scene, the speed of search did not decrease much over the course of these multiple searches (Võ & Wolfe, 2012). Only when observers were asked to search for the same object again did search become considerably faster. We argued that our naturalistic scenes provided such strong "semantic" guidance-e.g., knowing that a faucet is usually located near a sink-that guidance by incidental episodic memory-having seen that faucet previously-was rendered less useful. Here, we directly manipulated the availability of semantic information provided by a scene. By monitoring observers' eye movements, we found a tight coupling of semantic and episodic memory guidance: Decreasing the availability of semantic information increases the use of episodic memory to guide search. These findings have broad implications regarding the use of memory during search in general and particularly during search in naturalistic scenes.

A Bayesian Optimal Foraging Model of Human Visual Search

Real-world visual searches often contain a variable and unknown number of targets. Such searches present difficult metacognitive challenges, as searchers must decide when to stop looking for additional targets, which results in high miss rates in multiple-target searches. In the study reported here, we quantified human strategies in multiple-target search via an ecological optimal foraging model and investigated whether searchers adapt their strategies to complex target-distribution statistics. Separate groups of individuals searched displays with the number of targets per trial sampled from different geometric distributions but with the same overall target prevalence. As predicted by optimal foraging theory, results showed that individuals searched longer when they expected more targets to be present and adjusted their expectations on-line during each search by taking into account the higher-order, across-trial target distributions. However, compared with modeled ideal observers, participants systematically responded as if the target distribution were more uniform than it was, which suggests that training could improve multiple-target search performance.

Simultaneous control of attention by multiple working memory representations

Working memory representations play a key role in controlling attention by making it possible to shift attention to task-relevant objects. Visual working memory has a capacity of three to four objects, but recent studies suggest that only one representation can guide attention at a given moment. We directly tested this proposal by monitoring eye movements while observers performed a visual search task in which they attempted to limit attention to objects drawn in two colors. When the observers were motivated to attend to one color at a time, they searched many consecutive items of one color (long run lengths) and exhibited a delay prior to switching gaze from one color to the other (switch cost). In contrast, when they were motivated to attend to both colors simultaneously, observers' gaze switched back and forth between the two colors frequently (short run lengths), with no switch cost. Thus, multiple working memory representations can concurrently guide attention.

When do I quit? The search termination problem in visual search

In visual search tasks, observers look for targets in displays or scenes containing distracting, non-target items. Most of the research on this topic has concerned the finding of those targets. Search termination is a less thoroughly studied topic. When is it time to abandon the current search? The answer is fairly straight forward when the one and only target has been found (There are my keys.). The problem is more vexed if nothing has been found (When is it time to stop looking for a weapon at the airport checkpoint?) or when the number of targets is unknown (Have we found all the tumors?). This chapter reviews the development of ideas about quitting time in visual search and offers an outline of our current theory.

Guidance of visual search by memory and knowledge

To behave intelligently in the world, humans must be able to find objects efficiently within the complex environments they inhabit. A growing proportion of the literature on visual search is devoted to understanding this type of natural search. In the present chapter, I review the literature on visual search through natural scenes, focusing on the role of memory and knowledge in guiding attention to task-relevant objects.

Active inhibition and memory promote exploration and search of natural scenes

Active exploration of the visual world depends on sequential shifts of gaze that bring prioritized regions of a scene into central vision. The efficiency of this system is commonly attributed to a mechanism of "inhibition of return" (IOR) that discourages re-examination of previously-visited locations. Such a process is fundamental to computational models of attentional selection and paralleled by neurophysiological observations of inhibition of target-related activity in visuomotor areas. However, studies examining eye movements in naturalistic visual scenes appear to contradict the hypothesis that IOR promotes exploration. Instead, these reports reveal a surprisingly strong tendency to shift gaze back to the previously fixated location, suggesting that refixations might even be facilitated under natural conditions. Here we resolve this apparent contradiction, based on a probabilistic analysis of gaze patterns recorded during both free-viewing and search of naturalistic scenes. By simulating saccadic selection based on instantaneous influences alone, we show that the observed frequency of return saccades is in fact substantially less than predicted for a memoryless system, demonstrating that refixation is actively inhibited under natural viewing conditions. Furthermore, these observations reveal that gaze history significantly influences the way in which natural scenes are explored, contrary to accounts that suggest visual search has no memory.

Do object refixations during scene viewing indicate rehearsal in visual working memory?

Do refixations serve a rehearsal function in visual working memory (VWM)? We analyzed refixations from observers freely viewing multiobject scenes. An eyetracker was used to limit the viewing of a scene to a specified number of objects fixated after the target (intervening objects), followed by a four-alternative forced choice recognition test. Results showed that the probability of target refixation increased with the number of fixated intervening objects, and these refixations produced a 16% accuracy benefit over the first five intervening-object conditions. Additionally, refixations most frequently occurred after fixations on only one to two other objects, regardless of the intervening-object condition. These behaviors could not be explained by random or minimally constrained computational models; a VWM component was required to completely describe these data. We explain these findings in terms of a monitor-refixate rehearsal system: The activations of object representations in VWM are monitored, with refixations occurring when these activations decrease suddenly.

When does repeated search in scenes involve memory? Looking at versus looking for objects in scenes

One might assume that familiarity with a scene or previous encounters with objects embedded in a scene would benefit subsequent search for those items. However, in a series of experiments we show that this is not the case: When participants were asked to subsequently search for multiple objects in the same scene, search performance remained essentially unchanged over the course of searches despite increasing scene familiarity. Similarly, looking at target objects during previews, which included letter search, 30 seconds of free viewing, or even 30 seconds of memorizing a scene, also did not benefit search for the same objects later on. However, when the same object was searched for again memory for the previous search was capable of producing very substantial speeding of search despite many different intervening searches. This was especially the case when the previous search engagement had been active rather than supported by a cue. While these search benefits speak to the strength of memory-guided search when the same search target is repeated, the lack of memory guidance during initial object searches-despite previous encounters with the target objects-demonstrates the dominance of guidance by generic scene knowledge in real-world search.

Incidental learning speeds visual search by lowering response thresholds, not by improving efficiency: evidence from eye movements

When observers search for a target object, they incidentally learn the identities and locations of "background" objects in the same display. This learning can facilitate search performance, eliciting faster reaction times for repeated displays. Despite these findings, visual search has been successfully modeled using architectures that maintain no history of attentional deployments; they are amnesic (e.g., Guided Search Theory). In the current study, we asked two questions: 1) under what conditions does such incidental learning occur? And 2) what does viewing behavior reveal about the efficiency of attentional deployments over time? In two experiments, we tracked eye movements during repeated visual search, and we tested incidental memory for repeated nontarget objects. Across conditions, the consistency of search sets and spatial layouts were manipulated to assess their respective contributions to learning. Using viewing behavior, we contrasted three potential accounts for faster searching with experience. The results indicate that learning does not result in faster object identification or greater search efficiency. Instead, familiar search arrays appear to allow faster resolution of search decisions, whether targets are present or absent.