Incidental visual memory for targets and distractors in visual search

Visual long-term memory and change blindness: Different effects of pre- and post-change information on one-shot change detection using meaningless geometric objects

To clarify the relationship between visual long-term memory (VLTM) and online visual processing, we investigated whether and how VLTM involuntarily affects the performance of a one-shot change detection task using images consisting of six meaningless geometric objects. In the study phase, participants observed pre-change (Experiment 1), post-change (Experiment 2), or both pre- and post-change (Experiment 3) images appearing in the subsequent change detection phase. In the change detection phase, one object always changed between pre- and post-change images and participants reported which object was changed. Results showed that VLTM of pre-change images enhanced the performance of change detection, while that of post-change images decreased accuracy. Prior exposure to both pre- and post-change images did not influence performance. These results indicate that pre-change information plays an important role in change detection, and that information in VLTM related to the current task does not always have a positive effect on performance.

Seek and you shall remember: scene semantics interact with visual search to build better memories

Memorizing critical objects and their locations is an essential part of everyday life. In the present study, incidental encoding of objects in naturalistic scenes during search was compared to explicit memorization of those scenes. To investigate if prior knowledge of scene structure influences these two types of encoding differently, we used meaningless arrays of objects as well as objects in real-world, semantically meaningful images. Surprisingly, when participants were asked to recall scenes, their memory performance was markedly better for searched objects than for objects they had explicitly tried to memorize, even though participants in the search condition were not explicitly asked to memorize objects. This finding held true even when objects were observed for an equal amount of time in both conditions. Critically, the recall benefit for searched over memorized objects in scenes was eliminated when objects were presented on uniform, non-scene backgrounds rather than in a full scene context. Thus, scene semantics not only help us search for objects in naturalistic scenes, but appear to produce a representation that supports our memory for those objects beyond intentional memorization.

The target effect: visual memory for unnamed search targets

Search targets are typically remembered much better than other objects even when they are viewed for less time. However, targets have two advantages that other objects in search displays do not have: They are identified categorically before the search, and finding them represents the goal of the search task. The current research investigated the contributions of both of these types of information to the long-term visual memory representations of search targets. Participants completed either a predefined search or a unique-object search in which targets were not defined with specific categorical labels before searching. Subsequent memory results indicated that search target memory was better than distractor memory even following ambiguously defined searches and when the distractors were viewed significantly longer. Superior target memory appears to result from a qualitatively different representation from those of distractor objects, indicating that decision processes influence visual memory.

Priorities for selection and representation in natural tasks

Selecting and remembering visual information is an active and competitive process. In natural environments, representations are tightly coupled to task. Objects that are task-relevant are remembered better due to a combination of increased selection for fixation and strategic control of encoding and/or retaining viewed information. However, it is not understood how physically manipulating objects when performing a natural task influences priorities for selection and memory. In this study, we compare priorities for selection and memory when actively engaged in a natural task with first-person observation of the same object manipulations. Results suggest that active manipulation of a task-relevant object results in a specific prioritization for object position information compared with other properties and compared with action observation of the same manipulations. Experiment 2 confirms that this spatial prioritization is likely to arise from manipulation rather than differences in spatial representation in real environments and the movies used for action observation. Thus, our findings imply that physical manipulation of task relevant objects results in a specific prioritization of spatial information about task-relevant objects, possibly coupled with strategic de-prioritization of colour memory for irrelevant objects.

The interplay of episodic and semantic memory in guiding repeated search in scenes

It seems intuitive to think that previous exposure or interaction with an environment should make it easier to search through it and, no doubt, this is true in many real-world situations. However, in a recent study, we demonstrated that previous exposure to a scene does not necessarily speed search within that scene. For instance, when observers performed as many as 15 searches for different objects in the same, unchanging scene, the speed of search did not decrease much over the course of these multiple searches (Võ & Wolfe, 2012). Only when observers were asked to search for the same object again did search become considerably faster. We argued that our naturalistic scenes provided such strong "semantic" guidance-e.g., knowing that a faucet is usually located near a sink-that guidance by incidental episodic memory-having seen that faucet previously-was rendered less useful. Here, we directly manipulated the availability of semantic information provided by a scene. By monitoring observers' eye movements, we found a tight coupling of semantic and episodic memory guidance: Decreasing the availability of semantic information increases the use of episodic memory to guide search. These findings have broad implications regarding the use of memory during search in general and particularly during search in naturalistic scenes.

The importance of being relevant

This review aims at an understanding of the binding process by synthesizing the extant perspectives regarding binding. It begins with a consideration of the biological explanations of binding, viz., conjunctive coding, synchrony, and reentrant mechanisms. Thereafter binding is reviewed as a psychological process guided by top-down signals. The stages and types of binding proposed by various researchers are discussed in this section. The next section introduces Working Memory (WM) as the executive directing the top-down signals. After that it is described how WM works by selecting relevant sensory input, followed by a detailed consideration of the debate regarding objects vs. features with the conclusion that relevance is the key factor determining what is processed. The next section considers other factors affecting the selection of relevant input. Then, we shift focus to describe what happens to irrelevant input - whether it is discarded at the outset or is gradually inhibited, and whether inhibition is a perceptual or post-perceptual process. The concluding section describes the process of binding as currently understood on the basis of the literature included in the review. To summarize, it appears that initially the "object" is conceptualized as an instantaneous bundle of all features. However, only relevant features of stimuli are gradually integrated to form a stable representation of the object. Concomitantly, irrelevant features are removed from the object representations. Empirical evidence suggests that the inhibition of irrelevant features occurs over time and is presumably a process within WM.

Does an instruction to forget enhance memory for other presented items?

In an item-method directed forgetting paradigm, participants were required to attend to one of two colored words presented on opposite sides of a central fixation stimulus; they were instructed to Remember or Forget the attended item. On a subsequent recognition test, the Attended words showed a typical directed forgetting effect with better recognition of Remember words than Forget words. Our interest was in the fate of the Unattended words. When the study display disappeared before the memory instruction, there was no effect of that instruction on unattended words; when the study display remained visible during presentation of the memory instruction, there was a reverse directed forgetting effect with better recognition of unattended words from Forget trials than from Remember trials. Incidental encoding of task-irrelevant stimuli occurs following presentation of a Forget instruction - but only when those task-irrelevant stimuli are still visible in the external environment.

Aging, eye movements, and object-location memory

This study investigated whether “intentional” instructions could improve older adults' object memory and object-location memory about a scene by promoting object-oriented viewing. Eye movements of younger and older adults were recorded while they viewed a photograph depicting 12 household objects in a cubicle with or without the knowledge that memory about these objects and their locations would be tested (intentional vs. incidental encoding). After viewing, participants completed recognition and relocation tasks. Both instructions and age affected viewing behaviors and memory. Relative to incidental instructions, intentional instructions resulted in more accurate memory about object identity and object-location binding, but did not affect memory accuracy about overall positional configuration. More importantly, older adults exhibited more object-oriented viewing in the intentional than incidental condition, supporting the environmental support hypothesis.

Guidance of visual search by memory and knowledge

To behave intelligently in the world, humans must be able to find objects efficiently within the complex environments they inhabit. A growing proportion of the literature on visual search is devoted to understanding this type of natural search. In the present chapter, I review the literature on visual search through natural scenes, focusing on the role of memory and knowledge in guiding attention to task-relevant objects.

Shifting attention among working memory representations: testing cue type, awareness, and strategic control

It is well known that visual working memory (VWM) performance is modulated by attentional cues presented during encoding. Interestingly, retrospective cues presented after encoding, but prior to the test phase also improve performance. This improvement in performance is termed the retro-cue benefit. We investigated whether the retro-cue benefit is sensitive to cue type, whether participants were aware of their improvement in performance due to the retro-cue, and whether the effect was under strategic control. Experiment 1 compared the potential cueing benefits of abrupt onset retro-cues relying on bottom-up attention, number retro-cues relying on top-down attention, and arrow retro-cues, relying on a mixture of both. We found a significant retro-cue effect only for arrow retro-cues. In Experiment 2, we tested participants' awareness of their use of the informative retro-cue and found that they were aware of their improved performance. In Experiment 3, we asked whether participants have strategic control over the retro-cue. The retro-cue was difficult to ignore, suggesting that strategic control is low. The retro-cue effect appears to be within conscious awareness but not under full strategic control.

Incidental learning speeds visual search by lowering response thresholds, not by improving efficiency: evidence from eye movements

When observers search for a target object, they incidentally learn the identities and locations of "background" objects in the same display. This learning can facilitate search performance, eliciting faster reaction times for repeated displays. Despite these findings, visual search has been successfully modeled using architectures that maintain no history of attentional deployments; they are amnesic (e.g., Guided Search Theory). In the current study, we asked two questions: 1) under what conditions does such incidental learning occur? And 2) what does viewing behavior reveal about the efficiency of attentional deployments over time? In two experiments, we tracked eye movements during repeated visual search, and we tested incidental memory for repeated nontarget objects. Across conditions, the consistency of search sets and spatial layouts were manipulated to assess their respective contributions to learning. Using viewing behavior, we contrasted three potential accounts for faster searching with experience. The results indicate that learning does not result in faster object identification or greater search efficiency. Instead, familiar search arrays appear to allow faster resolution of search decisions, whether targets are present or absent.

Learning in repeated visual search

Visual search (e.g., finding a specific object in an array of other objects) is performed most effectively when people are able to ignore distracting nontargets. In repeated search, however, incidental learning of object identities may facilitate performance. In three experiments, with over 1,100 participants, we examined the extent to which search could be facilitated by object memory and by memory for spatial layouts. Participants searched for new targets (real-world, nameable objects) embedded among repeated distractors. To make the task more challenging, some participants performed search for multiple targets, increasing demands on visual working memory (WM). Following search, memory for search distractors was assessed using a surprise two-alternative forced choice recognition memory test with semantically matched foils. Search performance was facilitated by distractor object learning and by spatial memory; it was most robust when object identity was consistently tied to spatial locations and weakest (or absent) when object identities were inconsistent across trials. Incidental memory for distractors was better among participants who searched under high WM load, relative to low WM load. These results were observed when visual search included exhaustive-search trials (Experiment 1) or when all trials were self-terminating (Experiment 2). In Experiment 3, stimulus exposure was equated across WM load groups by presenting objects in a single-object stream; recognition accuracy was similar to that in Experiments 1 and 2. Together, the results suggest that people incidentally generate memory for nontarget objects encountered during search and that such memory can facilitate search performance.

The 35th Sir Frederick Bartlett Lecture: Eye movements and attention in reading, scene perception, and visual search

Eye movements are now widely used to investigate cognitive processes during reading, scene perception, and visual search. In this article, research on the following topics is reviewed with respect to reading: (a) the perceptual span (or span of effective vision), (b) preview benefit, (c) eye movement control, and (d) models of eye movements. Related issues with respect to eye movements during scene perception and visual search are also reviewed. It is argued that research on eye movements during reading has been somewhat advanced over research on eye movements in scene perception and visual search and that some of the paradigms developed to study reading should be more widely adopted in the study of scene perception and visual search. Research dealing with “real-world” tasks and research utilizing the visual-world paradigm are also briefly discussed.

Taking a new look at looking at nothing

A crucial question in cognitive science is how linguistic and visual information are integrated. Previous research has shown that eye movements to objects in the visual environment are locked to linguistic input. More surprisingly, listeners fixate on now-empty regions that had previously been occupied by relevant objects. This 'looking at nothing' phenomenon has been linked to the claim that the visual system constructs sparse representations of the external world and relies on saccades and fixations to extract information in a just-in-time manner. Our model provides a different explanation: based on recent work in visual cognition and memory, it assumes that the visual system creates and stores detailed internal memory representations, and that looking at nothing facilitates retrieval of those representations.

The contents of visual memory are only partly under volitional control

When we look around within a visual scene, is visual information automatically placed in visual memory during each saccade, or can we control which information is retained and which is excluded? We examined this question in five experiments by requiring participants to remember sequentially presented visual shapes or faces-some of which were marked for encoding (targets) and others that were supposed to be ignored (distractors)-over a 1-sec delay. The results show that distractors were retained in visual memory, regardless of stimulus category, suggesting that it is a general phenomenon. Whether or not participants were allowed to prepare for a target or distractor did not modulate distractor intrusion. When attention coupled with eye movements could be used to select targets, distractors were no longer encoded into memory. When eye movements were constrained, distractors once again intruded into memory. These findings suggest that top-down control processes are insufficient to filter the contents of visual memory.

Age differences in what is viewed and remembered in complex conjunction search

Older and younger adults searched arrays of 12 unique real-world photographs for a specified object (e.g., a yellow drill) among distractors (e.g., yellow telephone, red drill, and green door). Eye-tracking data from 24 of 48 participants in each age group showed generally similar search patterns for the younger and older adults but there were some interesting differences. Older adults processed all the items in the arrays more slowly than the younger adults (e.g., they had longer fixation durations, gaze durations, and total times), but this difference was exaggerated for target items. We also found that older and younger adults differed in the sequence in which objects were searched, with younger adults fixating the target objects earlier in the trial than older adults. Despite the relatively longer fixation times on the targets (in comparison to the distractors) for older adults, a surprise visual recognition test revealed a sizeable age deficit for target memory but, importantly, no age differences for distractor memory.

Differential detection of global luminance and contrast changes across saccades and flickers during active scene perception

How sensitive are viewers to changes in global image properties across saccades during active real-world scene perception? This question was investigated by globally increasing and/or decreasing luminance or contrast in photographs of real-world scenes across saccadic eye movements or during matched brief interruptions in a flicker paradigm. The results from two experiments demonstrated very poor sensitivity to global image changes in both the saccade-contingent and flicker paradigms, suggesting that the specific values of basic sensory properties do not contribute to the perception of stability across saccades during complex scene perception. In addition, overall sensitivity was significantly worse in the saccade-contingent change paradigm than the flicker paradigm, suggesting that the flicker paradigm is an imperfect simulation of transsaccadic vision.

How does the purpose of inspection influence the potency of visual salience in scene perception?

Salience-map models have been taken to suggest that the locations of eye fixations are determined by the extent of the low-level discontinuities in an image. While such models have found some support, an increasing emphasis on the task viewers are performing implies that these models must combine with cognitive demands to describe how the eyes are guided efficiently. An experiment is reported in which eye movements to objects in photographs were examined while viewers performed a memory-encoding task or one of two search tasks. The objects depicted in the scenes had known salience ranks according to a popular model. Participants fixated higher-salience objects sooner and more often than lower-salience objects, but only when memorising scenes. This difference shows that salience-map models provide useful predictions even in complex scenes and late in viewing. However, salience had no effects when searching for a target defined by category or exemplar. The results suggest that salience maps are not used to guide the eyes in these tasks, that cognitive override by task demands can be total, and that modelling top-down search is important but may not be easily accomplished within a salience-map framework.

Change detection: evidence for information accumulation in flicker paradigms

Change detection in rapidly alternating pictures separated by a blank frame has been shown to be very difficult (e.g., [Rensink, R. A., O'Regan, J. K., & Clark, J. J. (1997). To see or not to see: The need for attention to perceive changes in scenes. Psychological Science, 8, 368-373]). The three experiments reported here focus on the mechanism behind detection. More specifically, we explored whether information about the stimulus material accumulates in visual memory and thereby improves change detection. For that purpose the first experiment varied the number of repetitions of the original and modified stimulus version. Results showed that detection improved with more repetitions. The second experiment demonstrated that repetition performance improved more when both the original and the modified picture were repeated. Finally, the third experiment strengthened these findings by showing poorer detection performance when the repetition sequence was randomized. Together, our findings suggest accumulation of information in memory over picture presentations and moreover improved performance when both picture types were repeated. These results underline the importance of developing representations for both picture versions in change detection.

Distractor familiarity leads to more efficient visual search for complex stimuli

Previous reports suggest that distractor familiarity plays an important role in determining visual search efficiency. However, the specific tasks used in those studies limit the extension of their findings to real-world situations and everyday images. In the present study, subjects engaged in a prolonged period of search experience as a control of their level of familiarity with a large set of target and distractor images. Reaction times and search slopes decreased dramatically over this period, especially for trials with a large target eccentricity and many distractors. Following extended practice, search among familiar distractors was more efficient than search among unfamiliar distractors. Furthermore, we found that familiar targets were located more efficiently than unfamiliar targets and that subjects were faster at locating targets that they had experienced in the majority of the search trials. These results show that prolonged visual experience facilitates processing of both target and distractor items during search.