Rapid Resumption of Interrupted Visual Search: New Insights on the Interaction Between Vision and Memory

Interruption in visual search: a systematic review

Visual search, the process of trying to find a target presented among distractors, is a much-studied cognitive task. Less well-studied is the condition in which the search task is interrupted before the target is found. The consequences of such interruptions in visual search have been investigated across various disciplines, which has resulted in diverse and at times contradictory findings. The aim of this systematic review is to provide a more cohesive understanding of the effects of interruptions in visual search. For this purpose, we identified 28 studies that met our inclusion criteria. To facilitate a more organized and comprehensive analysis, we grouped the studies based on three dimensions: the search environment, the interruption aftermath, and the type of the interrupting event. While interruptions in visual search are variable and manifest differently across studies, our review provides a foundational scheme for a more cohesive understanding of the subject. This categorization serves as a starting point for exploring potential future directions, which we delineate in our conclusions.

The impact of different distractions on outdoor visual search and object memory

We investigated whether and how different types of search distractions affect visual search behavior and target memory while participants searched in a real-world environment. They searched either undistracted (control condition), listened to a podcast (auditory distraction), counted down aloud at intervals of three while searching (executive working memory load), or were forced to stop the search on half of the trials (time pressure). In line with findings from laboratory settings, participants searched longer but made fewer errors when the target was absent than when it was present, regardless of distraction condition. Furthermore, compared to the auditory distraction condition, the executive working memory load led to higher error rates (but not longer search times). In a surprise memory test after the end of the search tasks, recognition was better for previously present targets than for absent targets. Again, this was regardless of the previous distraction condition, although significantly fewer targets were remembered by the participants in the executive working memory load condition than by those in the control condition. The findings suggest that executive working memory load, but likely not auditory distraction and time pressure affected visual search performance and target memory in a real-world environment.

The performance costs of interruption during visual search are determined by the type of search task

Prior research has shown that interruptions lead to a variety of performance costs. However, these costs are heterogenous and poorly understood. Under some circumstances, interruptions lead to large decreases in accuracy on the primary task, whereas in others task duration increases, but task accuracy is unaffected. Presently, the underlying cause of these costs is unclear. The Memory for Goals model suggests that interruptions interfere with the ability to represent the current goal of the primary task. Here, we test the idea that working memory (WM) may play a critical role in representing the current goal and thus may underlie the observed costs associated with interruption. In two experiments, we utilized laboratory-based visual search tasks, which differed in their WM demands, in order to assess how this difference influenced the observed interruption costs. Interruptions led to more severe performance costs when the target of the search changed on each trial. When the search target was consistent across trials, the cost of interruption was greatly reduced. This suggests that the WM demands associated with the primary task play an important role in determining the performance costs of interruption. Our findings suggest that it is important for research to consider the cognitive processes a task engages in order to predict the nature of the adverse effects of interruption in applied settings such as radiology.

Visual working memory load does not affect the overall stimulus processing time in visual search

The dual-task paradigm is widely used in studying the interaction between visual search and working memory. A number of studies showed that holding items in working memory delays the overall response time (RT) in visual search, but it does not affect the efficiency of search (i.e., the slope of the RT × set size function). Why the memory load merely affects the overall RT? Some researchers proposed that this load-effect on overall RT may be caused by factors that only affect response selection processes, while others argued that it may reflect the effect of visual working memory load on visual search. This study investigated the two competing hypotheses by measuring the threshold stimulus exposure duration (TSED) for successfully fulfilling a search task. Experiment 1 replicated the large overall RT difference with the RT method but only found a small though reliable overall TSED difference with the TSED method. Experiment 2, with better controls, found no TSED difference by manipulating the visual working memory load. Experiment 3 showed that the TSED is not influenced by processes in the response selection stage. The present findings suggest that the overall stimulus processing time in visual search is not affected by visual working memory load and that the effect of memory load on overall RT is largely due to factors affecting response selection alone.

Visual Search May Not Require Target Representation in Working Memory or Long-Term Memory

We spend a lot of time searching for things. If we know what we are looking for in advance, a memory representation of the target will be created to guide search. But if the identity of the search target is revealed simultaneously with the presentation of the search array, is a similar memory representation formed? In the present study, 96 observers determined whether a central target was present in a peripheral search array. The results revealed that as long as the central target remained available for inspection (even if only in iconic memory), observers reinspected it after each distractor was checked, apparently forgoing consolidation of the target into working memory. The present findings challenged the assumption that evaluating items in a search array must involve comparison with a template in working memory.

Using the flicker task to estimate visual working memory storage capacity

Studies of visual working memory (VWM) typically have used a "one-shot" change detection task to arrive at a capacity estimate of three to four objects, with additional limits imposed by the precision of the information needed for each object. Unlike the one-shot task, the flicker change detection task permits measurement of VWM capacity over time and with larger numbers of objects present in the scene, but it has rarely been used to assess the capacity of VWM. We used the flicker task to examine (a) whether capacity is close to the typical three to four items when using subtly different stimuli; (b) which dependent measure provides the most meaningful estimate of the capacity of VWM in the flicker task (response time or number of changes viewed); (c) whether capacity remains fixed at three to four items for displays containing many more objects; and (d) how VWM operates over time, with repeated opportunities to encode, retain, and compare elements in a display. Four experiments using grids of simple items varying only in luminance or color revealed a range for VWM capacity limits that was largely impervious to changes in display duration, interstimulus intervals, and array size. This estimate of VWM capacity was correlated with an estimate from the more typical one-shot task, further validating the flicker task as a tool for measuring the capacity of VWM.

Distraction in diagnostic radiology: How is search through volumetric medical images affected by interruptions?

Observational studies have shown that interruptions are a frequent occurrence in diagnostic radiology. The present study used an experimental design in order to quantify the cost of these interruptions during search through volumetric medical images. Participants searched through chest CT scans for nodules that are indicative of lung cancer. In half of the cases, search was interrupted by a series of true or false math equations. The primary cost of these interruptions was an increase in search time with no corresponding increase in accuracy or lung coverage. This time cost was not modulated by the difficulty of the interruption task or an individual's working memory capacity. Eye-tracking suggests that this time cost was driven by impaired memory for which regions of the lung were searched prior to the interruption. Potential interventions will be discussed in the context of these results.

Independence of long-term contextual memory and short-term perceptual hypotheses: Evidence from contextual cueing of interrupted search

Observers are able to resume an interrupted search trial faster relative to responding to a new, unseen display. This finding of rapid resumption is attributed to short-term perceptual hypotheses generated on the current look and confirmed upon subsequent looks at the same display. It has been suggested that the contents of perceptual hypotheses are similar to those of other forms of memory acquired long-term through repeated exposure to the same search displays over the course of several trials, that is, the memory supporting “contextual cueing.” In three experiments, we investigated the relationship between short-term perceptual hypotheses and long-term contextual memory. The results indicated that long-term, contextual memory of repeated displays neither affected the generation nor the confirmation of short-term perceptual hypotheses for these displays. Furthermore, the analysis of eye movements suggests that long-term memory provides an initial benefit in guiding attention to the target, whereas in subsequent looks guidance is entirely based on short-term perceptual hypotheses. Overall, the results reveal a picture of both long- and short-term memory contributing to reliable performance gains in interrupted search, while exerting their effects in an independent manner.

Fixations are not all created equal: An objection to mindless visual search

This call to revolution in theories of visual search does not go far enough. Treating fixations as uniform is an oversimplification that obscures the critical role of the mind. We remind readers that what happens during a fixation depends on mindset, as shown in studies of search strategy and of humans' ability to rapidly resume search following an interruption.

The effects of brief visual interruption tasks on drivers' ability to resume their visual search for a pre-cued hazard

Driver visual distraction is known to increase the likelihood of being involved in a crash, especially for long glances inside the vehicle. The detrimental impact of these in-vehicle glances may carry over and disrupt the ongoing processing of information after the driver glances back up on the road. This study explored the effect of different types of visual tasks inside the vehicle on the top-down processes that guide the detection and monitoring of road hazards after the driver glances back towards the road. Using a driving simulator, 56 participants were monitored with an eye tracking system while they navigated various hazardous scenarios in one of four experimental conditions. In all conditions, a potential hazard was visible 4-5s before the driver could strike the potential hazard were it to materialize. All interruptions were exactly two seconds in length. After the interruption the potential hazard again became visible for about a half-second after which the driver passed by the hazard. The nature of the in-vehicle visual interruption presented to the participants was varied across conditions: (1) Visual interruptions comprised of spatial, driving unrelated, tasks; (2) visual interruptions comprised of non-spatial, driving unrelated, tasks; (3) visual interruptions with no tasks added; and (4) no visual interruptions. In the first three conditions drivers glancing on the forward roadway was momentarily interrupted (either with or without a task) just after the potential hazard first became visible by the occurrence of an in-vehicle task lasting two seconds. In the last condition (no interruptions) the driver could not see the potential hazard after it just became visible because of obstructions in the built or natural environment. The obstruction (like the interruption) lasted for two seconds. In other words, across all conditions the hazard was visible, then became invisible, and finally became visible again. Importantly, the results show that the presence of an interruption (as opposed to an obstruction) negatively impacted drivers' ability to anticipate the potential hazard. Moreover, the various types of interruptions had differential effects on hazard detection. The implications of this study for the design of in-vehicle displays are discussed.

The Neural Mechanisms of Prediction in Visual Search

The speed of visual search depends on bottom-up stimulus features (e.g., we quickly locate a red item among blue distractors), but it is also facilitated by the presence of top-down perceptual predictions about the item. Here, we identify the nature, source, and neuronal substrate of the predictions that speed up resumed visual search. Human subjects were presented with a visual search array that was repeated up to 4 times, while brain activity was recorded using magnetoencephalography (MEG). Behaviorally, we observed a bimodal reaction time distribution for resumed visual search, indicating that subjects were extraordinarily rapid on a proportion of trials. MEG data demonstrated that these rapid-response trials were associated with a prediction of (1) target location, as reflected by alpha-band (8-12 Hz) lateralization; and (2) target identity, as reflected by beta-band (15-30 Hz) lateralization. Moreover, we show that these predictions are likely generated in a network consisting of medial superior frontal cortex and right temporo-parietal junction. These findings underscore the importance and nature of perceptual hypotheses for efficient visual search.

Memory for a Hazard is Interrupted by Performance of a Secondary In-Vehicle Task

Driver visual distraction is known to increase the likelihood of being involved in a crash, especially for long glances. Recent evidence further suggests that the detrimental impact of these glances carries over and disrupting the ongoing processing of information after the eyes return to the road. This study aimed at exploring the effect of different types of visual disruptions on the top-down processes that guide the detection and monitoring of road hazards. Using a driving simulator, 56 participants were monitored with an eye tracking system while they navigated various hazardous scenarios in one of four experimental conditions: (1) Visual interruptions comprised of spatial, driving unrelated, tasks; (2) visual interruptions comprised of non-spatial, driving unrelated, tasks; (3) visual interruptions with no tasks added; and (4) no visual interruptions. In the first three conditions drivers were momentarily interrupted (either with or without a task) prior to the hazard occurrence. The visual interruption was aimed to simulate a glance inside the vehicle either with or without the need to process driving irrelevant information. Results show that the various types of tasks had differential effects on hazard detection. Implications of this study are discussed.

The effects of momentary visual disruption on hazard anticipation and awareness in driving

OBJECTIVE

Driver distraction is known to increase crash risk, especially when a driver glances inside the vehicle for especially long periods of time. Though it is clear that such glances increase the risk for the driver when looking inside the vehicle, it is less clear how these glances disrupt the ongoing processing of information outside the vehicle once the driver's eyes return to the road. The present study was aimed at exploring the effect of in-vehicle glances on the top-down processes that guide the detection and monitoring of hazards on the forward roadway.

METHOD

Using a driving simulator, 12 participants were monitored with an eye-tracking system while they navigated various hazardous scenarios. Six participants were momentarily interrupted by a visual secondary task (simulating a glance inside the vehicle) prior to the occurrence of a potential hazard and 6 were not.

RESULTS

Eye movement analyses showed that interrupted drivers often failed to continue scanning for a potential hazard when their forward view reappeared, especially when the potential threat could not easily be localized. Additionally, drivers' self-appraisal of workload and performance of the driving task indicated that, contrary to what one might expect, drivers in the interruption condition reported workload levels lower than and performance equal to drivers in the no interruption condition.

CONCLUSIONS

Drivers who are momentarily disrupted even for a brief duration are at risk of missing important information when they return their gaze to the forward roadway. In addition, because they are not aware of missing this information they are likely to continue engaging in in-vehicle tasks even though they are demonstrably unsafe. The implications for safety, calibration, and targeted remediation are discussed.

The role of prediction in perception: Evidence from interrupted visual search

Recent studies of rapid resumption-an observer's ability to quickly resume a visual search after an interruption-suggest that predictions underlie visual perception. Previous studies showed that when the search display changes unpredictably after the interruption, rapid resumption disappears. This conclusion is at odds with our everyday experience, where the visual system seems to be quite efficient despite continuous changes of the visual scene; however, in the real world, changes can typically be anticipated based on previous knowledge. The present study aimed to evaluate whether changes to the visual display can be incorporated into the perceptual hypotheses, if observers are allowed to anticipate such changes. Results strongly suggest that an interrupted visual search can be rapidly resumed even when information in the display has changed after the interruption, so long as participants not only can anticipate them, but also are aware that such changes might occur.

Effects of display complexity on location and feature inhibition

Inhibition of return refers to the lengthening of reaction times (RTs) to a target when a preceding stimulus has occupied the same location in space. Recently, we observed a robust inhibitory effect for color and shape in moderately complex displays: It is more difficult to detect a target with a particular nonspatial attribute if a stimulus with the same attribute was recently the focus of attention. Such nonspatial inhibitory effects have not generally been found in simpler displays. In the present study, we test how location-based and nonspatial inhibitory effects vary as a function of display complexity (eight, six, four, and two locations). The results demonstrated that (1) location-based inhibition effects were much stronger in more complex displays, whereas the nonspatial inhibition was only slightly stronger in more complex displays; (2) nonspatial inhibitory effects emerged at longer stimulus onset asynchronies than did location-based effects; and (3) nonspatial inhibition appeared only when cues and targets occurred in the same locations, confirming that pure feature repetition does not produce a cost. Taken together, the results are consistent with perceptual processes based on object files that are organized by spatial location. Using somewhat more complex displays than are most commonly employed provides a more sensitive method for observing the role of inhibitory processes in facilitating visual search. In addition, using a relatively wide set of cue-target timing relationships is necessary in order to clearly see how inhibitory effects operate.

Attention and visual memory in visualization and computer graphics

A fundamental goal of visualization is to produce images of data that support visual analysis, exploration, and discovery of novel insights. An important consideration during visualization design is the role of human visual perception. How we "see" details in an image can directly impact a viewer's efficiency and effectiveness. This paper surveys research on attention and visual perception, with a specific focus on results that have direct relevance to visualization and visual analytics. We discuss theories of low-level visual perception, then show how these findings form a foundation for more recent work on visual memory and visual attention. We conclude with a brief overview of how knowledge of visual attention and visual memory is being applied in visualization and graphics. We also discuss how challenges in visualization are motivating research in psychophysics.

Effects of menu structure and touch screen scrolling style on the variability of glance durations during in-vehicle visual search tasks

The effects of alternative navigation device display features on drivers' visual sampling efficiency while searching forpoints of interest were studied in two driving simulation experiments with 40 participants. Given that the number of display items was sufficient, display features that facilitate resumption of visual search following interruptions were expected to lead to more consistent in-vehicle glance durations. As predicted, compared with a grid-style menu, searching information in a list-style menu while driving led to smaller variance in durations of in-vehicle glances, in particular with nine item displays. Kinetic touch screen scrolling induced a greater number of very short in-vehicle glances than scrolling with arrow buttons. The touch screen functionality did not significantly diminish the negative effects of the grid-menu compared with physical controls with list-style menus. The findings suggest that resumability of self-paced, in-vehicle visual search tasks could be assessed with the measures of variance of in-vehicle glance duration distributions. Statement of Relevance: The reported research reveals display design factors affecting safety-relevant variability of in-vehicle glance durations and provides a theoretical framework for explaining the effects. The research can have a significant methodical value for driver distraction research and practical value for the design and testing of in-vehicle user interfaces.

Rapid resumption of interrupted search is independent of age-related improvements in visual search

In this study, 7-19-year-olds performed an interrupted visual search task in two experiments. Our question was whether the tendency to respond within 500ms after a second glimpse of a display (the rapid resumption effect [Psychological Science, 16 (2005) 684-688]) would increase with age in the same way as overall search efficiency. The results indicated no correlation of rapid resumption with search speed either across age groups (7, 9, 11, and 19years) or at the level of individual participants. Moreover, relocating the target randomly between looks reduced the rate of rapid resumption in a very similar way at each age. These results imply that implicit perceptual prediction during search is invariant across this age range and is distinct from other critical processes such as feature integration and control over spatial attention.

A taxonomy of external and internal attention

Attention is a core property of all perceptual and cognitive operations. Given limited capacity to process competing options, attentional mechanisms select, modulate, and sustain focus on information most relevant for behavior. A significant problem, however, is that attention is so ubiquitous that it is unwieldy to study. We propose a taxonomy based on the types of information that attention operates over--the targets of attention. At the broadest level, the taxonomy distinguishes between external attention and internal attention. External attention refers to the selection and modulation of sensory information. External attention selects locations in space, points in time, or modality-specific input. Such perceptual attention can also select features defined across any of these dimensions, or object representations that integrate over space, time, and modality. Internal attention refers to the selection, modulation, and maintenance of internally generated information, such as task rules, responses, long-term memory, or working memory. Working memory, in particular, lies closest to the intersection between external and internal attention. The taxonomy provides an organizing framework that recasts classic debates, raises new issues, and frames understanding of neural mechanisms.

The contents of perceptual hypotheses: evidence from rapid resumption of interrupted visual search

Observers can resume a previously interrupted visual search trial significantly more quickly than they can start a new search trial (Lleras, Rensink, & Enns, 2005). This rapid resumption of search is possible because evidence accumulated during the previous exposure, a perceptual hypothesis, can carry over to a subsequent presentation. We present four interrupted visual search experiments in which the content of the perceptual hypotheses used during visual search trials was characterized. These experiments suggest that prior to explicit target identification, observers have accumulated evidence about the locations, but not the identities, of local, task-relevant distractors, as well as preliminary evidence for the identity of the target. Our results characterize the content of perceptual search hypotheses and highlight the utility of interrupted search for studying online search processing prior to target identification.

Event-related potentials reveal rapid verification of predicted visual input

Human information processing depends critically on continuous predictions about upcoming events, but the temporal convergence of expectancy-based top-down and input-driven bottom-up streams is poorly understood. We show that, during reading, event-related potentials differ between exposure to highly predictable and unpredictable words no later than 90 ms after visual input. This result suggests an extremely rapid comparison of expected and incoming visual information and gives an upper temporal bound for theories of top-down and bottom-up interactions in object recognition.

What's next? New evidence for prediction in human vision

Everyday visual experience involves making implicit predictions, as revealed by our surprise when something disturbs our expectations. Many theories of vision have been premised on the central role played by prediction. Yet, implicit prediction in human vision has been difficult to assess in the laboratory, and many results have not distinguished between the indisputably important role of memory and the future-oriented aspect of prediction. Now, a new and unexpected finding - that humans can resume an interrupted visual search much faster than they can start a new search - offers new hope, because the rapid resumption of a search seems to depend on participants forming an implicit prediction of what they will see after the interruption. These findings combined with results of recent neurophysiology studies provide a framework for studying implicit prediction in perception.

The mechanisms of feature inheritance as predicted by a systems-level model of visual attention and decision making

Feature inheritance provides evidence that properties of an invisible target stimulus can be attached to a following mask. We apply a systemslevel model of attention and decision making to explore the influence of memory and feedback connections in feature inheritance. We find that the presence of feedback loops alone is sufficient to account for feature inheritance. Although our simulations do not cover all experimental variations and focus only on the general principle, our result appears of specific interest since the model was designed for a completely different purpose than to explain feature inheritance. We suggest that feedback is an important property in visual perception and provide a description of its mechanism and its role in perception.

Visual similarity in masking and priming: The critical role of task relevance

Cognitive scientists use rapid image sequences to study both the emergence of conscious perception (visual masking) and the unconscious processes involved in response preparation (masked priming). The present study asked two questions: (1) Does image similarity influence masking and priming in the same way? (2) Are similarity effects in both tasks governed by the extent of feature overlap in the images or only by task-relevant features? Participants in Experiment 1 classified human faces using a single dimension even though the faces varied in three dimensions (emotion, race, sex). Abstract geometric shapes and colors were tested in the same way in Experiment 2. Results showed that similarity reduced the visibility of the target in the masking task and increased response speed in the priming task, pointing to a double-dissociation between the two tasks. Results also showed that only task-relevant (not objective) similarity influenced masking and priming, implying that both tasks are influenced from the beginning by intentions of the participant. These findings are interpreted within the framework of a reentrant theory of visual perception. They imply that intentions can influence object formation prior to the separation of vision for perception and vision for action.

Change detection: evidence for information accumulation in flicker paradigms

Change detection in rapidly alternating pictures separated by a blank frame has been shown to be very difficult (e.g., [Rensink, R. A., O'Regan, J. K., & Clark, J. J. (1997). To see or not to see: The need for attention to perceive changes in scenes. Psychological Science, 8, 368-373]). The three experiments reported here focus on the mechanism behind detection. More specifically, we explored whether information about the stimulus material accumulates in visual memory and thereby improves change detection. For that purpose the first experiment varied the number of repetitions of the original and modified stimulus version. Results showed that detection improved with more repetitions. The second experiment demonstrated that repetition performance improved more when both the original and the modified picture were repeated. Finally, the third experiment strengthened these findings by showing poorer detection performance when the repetition sequence was randomized. Together, our findings suggest accumulation of information in memory over picture presentations and moreover improved performance when both picture types were repeated. These results underline the importance of developing representations for both picture versions in change detection.

Crossmodal change blindness between vision and touch

Change blindness is the name given to people's inability to detect changes introduced between two consecutively-presented scenes when they are separated by a distractor that masks the transients that are typically associated with change. Change blindness has been reported within vision, audition, and touch, but has never before been investigated when successive patterns are presented to different sensory modalities. In the study reported here, we investigated change detection performance when the two to-be-compared stimulus patterns were presented in the same sensory modality (i.e., both visual or both tactile) and when one stimulus pattern was tactile while the other was presented visually or vice versa. The two to-be-compared patterns were presented consecutively, separated by an empty interval, or else separated by a masked interval. In the latter case, the masked interval could either be tactile or visual. The first experiment investigated visual-tactile and tactile-visual change detection performance. The results showed that in the absence of masking, participants detected changes in position accurately, despite the fact that the two to-be-compared displays were presented in different sensory modalities. Furthermore, when a mask was presented between the two to-be-compared displays, crossmodal change blindness was elicited no matter whether the mask was visual or tactile. The results of two further experiments showed that performance was better overall in the unimodal (visual or tactile) conditions than in the crossmodal conditions. These results suggest that certain of the processes underlying change blindness are multisensory in nature. We discuss these findings in relation to recent claims regarding the crossmodal nature of spatial attention.

How do we track invisible objects?

We previously demonstrated that observers in multiple object tracking experiments can successfully track targets when all the objects simultaneously vanish for periods lasting several hundred milliseconds (Alvarez, Horowitz, Arsenio, Dimase, and Wolfe, 2005). How do observers do this? Since observers can track objects that move behind occluders (e.g., Scholl and Pylyshyn, 1999), they may treat a temporal gap as a case of complete occlusion. If so, performance should improve if occlusion cues (deletion and accretion) are provided and items disappear and reappear one by one (asynchronously), rather than simultaneously. However, we found better performance with simultaneous than with asynchronous disappearance (Experiment 1), whereas occlusion cues were detrimental (Experiment 2). We propose that observers tolerate a gap in tracking by storing the current task state when objects vanish and resuming tracking on the basis of that memory when the objects reappear (a task-switching account).

Interrupting infants' persisting object representations: an object-based limit?

Making sense of the visual world requires keeping track of objects as the same persisting individuals over time and occlusion. Here we implement a new paradigm using 10-month-old infants to explore the processes and representations that support this ability in two ways. First, we demonstrate that persisting object representations can be maintained over brief interruptions from additional independent events - just as a memory of a traffic scene may be maintained through a brief glance in the rearview mirror. Second, we demonstrate that this ability is nevertheless subject to an object-based limit: if an interrupting event involves enough objects (carefully controlling for overall salience), then it will impair the maintenance of other persisting object representations even though it is an independent event. These experiments demonstrate how object representations can be studied via their 'interruptibility', and the results are consistent with the idea that infants' persisting object representations are constructed and maintained by capacity-limited mid-level 'object-files'.