Consequences of display changes during interrupted visual search: Rapid resumption is target specific

Interruption in visual search: a systematic review

Visual search, the process of trying to find a target presented among distractors, is a much-studied cognitive task. Less well-studied is the condition in which the search task is interrupted before the target is found. The consequences of such interruptions in visual search have been investigated across various disciplines, which has resulted in diverse and at times contradictory findings. The aim of this systematic review is to provide a more cohesive understanding of the effects of interruptions in visual search. For this purpose, we identified 28 studies that met our inclusion criteria. To facilitate a more organized and comprehensive analysis, we grouped the studies based on three dimensions: the search environment, the interruption aftermath, and the type of the interrupting event. While interruptions in visual search are variable and manifest differently across studies, our review provides a foundational scheme for a more cohesive understanding of the subject. This categorization serves as a starting point for exploring potential future directions, which we delineate in our conclusions.

Using the flicker task to estimate visual working memory storage capacity

Studies of visual working memory (VWM) typically have used a "one-shot" change detection task to arrive at a capacity estimate of three to four objects, with additional limits imposed by the precision of the information needed for each object. Unlike the one-shot task, the flicker change detection task permits measurement of VWM capacity over time and with larger numbers of objects present in the scene, but it has rarely been used to assess the capacity of VWM. We used the flicker task to examine (a) whether capacity is close to the typical three to four items when using subtly different stimuli; (b) which dependent measure provides the most meaningful estimate of the capacity of VWM in the flicker task (response time or number of changes viewed); (c) whether capacity remains fixed at three to four items for displays containing many more objects; and (d) how VWM operates over time, with repeated opportunities to encode, retain, and compare elements in a display. Four experiments using grids of simple items varying only in luminance or color revealed a range for VWM capacity limits that was largely impervious to changes in display duration, interstimulus intervals, and array size. This estimate of VWM capacity was correlated with an estimate from the more typical one-shot task, further validating the flicker task as a tool for measuring the capacity of VWM.

Independence of long-term contextual memory and short-term perceptual hypotheses: Evidence from contextual cueing of interrupted search

Observers are able to resume an interrupted search trial faster relative to responding to a new, unseen display. This finding of rapid resumption is attributed to short-term perceptual hypotheses generated on the current look and confirmed upon subsequent looks at the same display. It has been suggested that the contents of perceptual hypotheses are similar to those of other forms of memory acquired long-term through repeated exposure to the same search displays over the course of several trials, that is, the memory supporting “contextual cueing.” In three experiments, we investigated the relationship between short-term perceptual hypotheses and long-term contextual memory. The results indicated that long-term, contextual memory of repeated displays neither affected the generation nor the confirmation of short-term perceptual hypotheses for these displays. Furthermore, the analysis of eye movements suggests that long-term memory provides an initial benefit in guiding attention to the target, whereas in subsequent looks guidance is entirely based on short-term perceptual hypotheses. Overall, the results reveal a picture of both long- and short-term memory contributing to reliable performance gains in interrupted search, while exerting their effects in an independent manner.

Fixations are not all created equal: An objection to mindless visual search

This call to revolution in theories of visual search does not go far enough. Treating fixations as uniform is an oversimplification that obscures the critical role of the mind. We remind readers that what happens during a fixation depends on mindset, as shown in studies of search strategy and of humans' ability to rapidly resume search following an interruption.

The Neural Mechanisms of Prediction in Visual Search

The speed of visual search depends on bottom-up stimulus features (e.g., we quickly locate a red item among blue distractors), but it is also facilitated by the presence of top-down perceptual predictions about the item. Here, we identify the nature, source, and neuronal substrate of the predictions that speed up resumed visual search. Human subjects were presented with a visual search array that was repeated up to 4 times, while brain activity was recorded using magnetoencephalography (MEG). Behaviorally, we observed a bimodal reaction time distribution for resumed visual search, indicating that subjects were extraordinarily rapid on a proportion of trials. MEG data demonstrated that these rapid-response trials were associated with a prediction of (1) target location, as reflected by alpha-band (8-12 Hz) lateralization; and (2) target identity, as reflected by beta-band (15-30 Hz) lateralization. Moreover, we show that these predictions are likely generated in a network consisting of medial superior frontal cortex and right temporo-parietal junction. These findings underscore the importance and nature of perceptual hypotheses for efficient visual search.

The role of prediction in perception: Evidence from interrupted visual search

Recent studies of rapid resumption-an observer's ability to quickly resume a visual search after an interruption-suggest that predictions underlie visual perception. Previous studies showed that when the search display changes unpredictably after the interruption, rapid resumption disappears. This conclusion is at odds with our everyday experience, where the visual system seems to be quite efficient despite continuous changes of the visual scene; however, in the real world, changes can typically be anticipated based on previous knowledge. The present study aimed to evaluate whether changes to the visual display can be incorporated into the perceptual hypotheses, if observers are allowed to anticipate such changes. Results strongly suggest that an interrupted visual search can be rapidly resumed even when information in the display has changed after the interruption, so long as participants not only can anticipate them, but also are aware that such changes might occur.

Effects of display complexity on location and feature inhibition

Inhibition of return refers to the lengthening of reaction times (RTs) to a target when a preceding stimulus has occupied the same location in space. Recently, we observed a robust inhibitory effect for color and shape in moderately complex displays: It is more difficult to detect a target with a particular nonspatial attribute if a stimulus with the same attribute was recently the focus of attention. Such nonspatial inhibitory effects have not generally been found in simpler displays. In the present study, we test how location-based and nonspatial inhibitory effects vary as a function of display complexity (eight, six, four, and two locations). The results demonstrated that (1) location-based inhibition effects were much stronger in more complex displays, whereas the nonspatial inhibition was only slightly stronger in more complex displays; (2) nonspatial inhibitory effects emerged at longer stimulus onset asynchronies than did location-based effects; and (3) nonspatial inhibition appeared only when cues and targets occurred in the same locations, confirming that pure feature repetition does not produce a cost. Taken together, the results are consistent with perceptual processes based on object files that are organized by spatial location. Using somewhat more complex displays than are most commonly employed provides a more sensitive method for observing the role of inhibitory processes in facilitating visual search. In addition, using a relatively wide set of cue-target timing relationships is necessary in order to clearly see how inhibitory effects operate.

Attention and visual memory in visualization and computer graphics

A fundamental goal of visualization is to produce images of data that support visual analysis, exploration, and discovery of novel insights. An important consideration during visualization design is the role of human visual perception. How we "see" details in an image can directly impact a viewer's efficiency and effectiveness. This paper surveys research on attention and visual perception, with a specific focus on results that have direct relevance to visualization and visual analytics. We discuss theories of low-level visual perception, then show how these findings form a foundation for more recent work on visual memory and visual attention. We conclude with a brief overview of how knowledge of visual attention and visual memory is being applied in visualization and graphics. We also discuss how challenges in visualization are motivating research in psychophysics.

Rapid resumption of interrupted search is independent of age-related improvements in visual search

In this study, 7-19-year-olds performed an interrupted visual search task in two experiments. Our question was whether the tendency to respond within 500ms after a second glimpse of a display (the rapid resumption effect [Psychological Science, 16 (2005) 684-688]) would increase with age in the same way as overall search efficiency. The results indicated no correlation of rapid resumption with search speed either across age groups (7, 9, 11, and 19years) or at the level of individual participants. Moreover, relocating the target randomly between looks reduced the rate of rapid resumption in a very similar way at each age. These results imply that implicit perceptual prediction during search is invariant across this age range and is distinct from other critical processes such as feature integration and control over spatial attention.

Delayed offset of distracters masks a local target

Object substitution masking (OSM) is observed when a brief target surrounded with a mask is presented among distracter stimuli and cannot be identified when it and the distracters disappear but the mask remains in view. We probed whether OSM also occurs without a local mask object when the distracters remain after target offset. We also varied the congruence between the local target and the global search display and the grouping properties of the delayed offset distracters. A target was briefly presented in a global object configuration of distracters that had delayed offset. Results showed that OSM could be observed with delayed offset of distracters grouped into a global mask shape. Congruence of the shapes of the global and local objects did not affect OSM, suggesting that a generalized abstract visual pattern representation of the global object may not be involved in OSM nor did the grouping properties of the delayed offset distracters influence OSM.

The contents of perceptual hypotheses: evidence from rapid resumption of interrupted visual search

Observers can resume a previously interrupted visual search trial significantly more quickly than they can start a new search trial (Lleras, Rensink, & Enns, 2005). This rapid resumption of search is possible because evidence accumulated during the previous exposure, a perceptual hypothesis, can carry over to a subsequent presentation. We present four interrupted visual search experiments in which the content of the perceptual hypotheses used during visual search trials was characterized. These experiments suggest that prior to explicit target identification, observers have accumulated evidence about the locations, but not the identities, of local, task-relevant distractors, as well as preliminary evidence for the identity of the target. Our results characterize the content of perceptual search hypotheses and highlight the utility of interrupted search for studying online search processing prior to target identification.

Event-related potentials reveal rapid verification of predicted visual input

Human information processing depends critically on continuous predictions about upcoming events, but the temporal convergence of expectancy-based top-down and input-driven bottom-up streams is poorly understood. We show that, during reading, event-related potentials differ between exposure to highly predictable and unpredictable words no later than 90 ms after visual input. This result suggests an extremely rapid comparison of expected and incoming visual information and gives an upper temporal bound for theories of top-down and bottom-up interactions in object recognition.

What's next? New evidence for prediction in human vision

Everyday visual experience involves making implicit predictions, as revealed by our surprise when something disturbs our expectations. Many theories of vision have been premised on the central role played by prediction. Yet, implicit prediction in human vision has been difficult to assess in the laboratory, and many results have not distinguished between the indisputably important role of memory and the future-oriented aspect of prediction. Now, a new and unexpected finding - that humans can resume an interrupted visual search much faster than they can start a new search - offers new hope, because the rapid resumption of a search seems to depend on participants forming an implicit prediction of what they will see after the interruption. These findings combined with results of recent neurophysiology studies provide a framework for studying implicit prediction in perception.