Measuring and modeling attentional dwell time

A cortical zoom-in operation underlies covert shifts of visual spatial attention

Shifting the focus of attention without moving the eyes poses challenges for signal coding in visual cortex in terms of spatial resolution, signal routing, and cross-talk. Little is known how these problems are solved during focus shifts. Here, we analyze the spatiotemporal dynamic of neuromagnetic activity in human visual cortex as a function of the size and number of focus shifts in visual search. We find that large shifts elicit activity modulations progressing from highest (IT) through mid-level (V4) to lowest hierarchical levels (V1). Smaller shifts cause those modulations to start at lower levels in the hierarchy. Successive shifts involve repeated backward progressions through the hierarchy. We conclude that covert focus shifts arise from a cortical coarse-to-fine process progressing from retinotopic areas with larger toward areas with smaller receptive fields. This process localizes the target and increases the spatial resolution of selection, which resolves the above issues of cortical coding.

Advances in the application of a computational Theory of Visual Attention (TVA): Moving towards more naturalistic stimuli and game-like tasks

The theory of visual attention, “TVA”, is an influential and formal theory of attentional selection. It is widely applied in clinical assessment of attention and fundamental attention research. However, most TVA-based research is based on accuracy data from letter report experiments performed in controlled laboratory environments. While such basic approaches to questions regarding attentional selection are undoubtedly useful, recent technological advances have enabled the use of increasingly sophisticated experimental paradigms involving more realistic scenarios. Notably, these studies have in many cases resulted in different estimates of capacity limits than those found in studies using traditional TVA-based assessment. Here we review recent developments in TVA-based assessment of attention that goes beyond the use of letter report experiments and experiments performed in controlled laboratory environments. We show that TVA can be used with other tasks and new stimuli, that TVA-based parameter estimation can be embedded into complex scenarios, such as games that can be used to investigate particular problems regarding visual attention, and how TVA-based simulations of “visual foraging” can elucidate attentional control in more naturalistic tasks. We also discuss how these developments may inform future advances of TVA.

The time course of salience: not entirely caused by salience

Visual salience is a key component of attentional selection, the process that guards the scarce resources needed for conscious recognition and perception. In previous works, we proposed a measure of visual salience based on a formal theory of visual selection. However, the strength of visual salience depends on the time course as well as local physical contrasts. Evidence from multiple experimental designs in the literature suggests that the strength of salience rises initially and declines after approximately 150 ms. The present article amends the theory-based salience measure beyond local physical contrasts to the time course of salience. It does so through a first experiment which reveals that-contrary to expectations-salience is not reduced during the first 150 ms after onset. Instead, the overall visual processing capacity is severely reduced, which corresponds to a reduced processing speed of all stimuli in the visual field. A second experiment confirms this conclusion by replicating the result. We argue that the slower stimulus processing may have been overlooked previously because the attentional selection mechanism had not yet been modeled in studies on the time course of salience.

A novel driving assessment combining hazard perception, hazard prediction and theory questions

A new hazard test was created using high-fidelity computer animation containing ten hazards. Sixty learner drivers and sixty experienced drivers sat either a hazard-perception version of this test (requiring timed responses to materialized hazards) or a hazard-prediction variant of the test (where the screen is occluded as the hazard begins to appear and drivers are asked 'What happens next?'). Recent studies have demonstrated that the prediction test format outperforms the hazard perception format using naturalistic video, but there has not yet been a study replicating this effect with computer-animated materials similar to the quality of those used in the official UK hazard perception test. The new test also included eleven theory questions designed to probe drivers' knowledge of the rules of the road. The results demonstrated that both test variants differentiated between driver groups with considerable effect sizes. Theory-question scores were comparable across learner and experienced driver groups, reflecting learners' preparation for the test and possible issues with memory decay and overwriting in the experienced group. As an interesting aside, driving-related video game play negatively correlated with hazard perception performance, but not with hazard prediction scores. Some individual hazards better suited the prediction or perception test format, raising the possibility of a future hybrid test that combines the two approaches.

Endogenous cueing effects for detection can be accounted for by a decision model of selective attention

Spatial cues help participants detect a visual target when it appears at the cued location. One hypothesis for this cueing effect, called selective perception, is that cueing a location enhances perceptual encoding at that location. Another hypothesis, called selective decision, is that the cue has no effect on perception, but instead provides prior information that facilitates decision-making. We distinguished these hypotheses by comparing a simultaneous display with two spatial locations to sequential displays with two temporal intervals. The simultaneous condition had a partially valid spatial cue, and the sequential condition had a partially valid temporal cue. Selective perception predicts no cueing effect for sequential displays given there is enough time to switch attention. In contrast, selective decision predicts cueing effects for sequential displays regardless of time. We used endogenous cueing of a detection-like coarse orientation discrimination task with clear displays (no external noise or postmasks). Results showed cueing effects for the sequential condition, supporting a decision account of selective attention for endogenous cueing of detection-like tasks.

Attentional competition across saccadic eye movements

Human behavior is guided by visual object recognition. For being recognized, objects compete for limited attentional processing resources. The more objects compete, the lower is performance in recognizing each individual object. Here, we ask whether this competition is confined to eye fixations, periods of relatively stable gaze, or whether it extends from one fixation to the next, across saccadic eye movements. Participants made saccades to a peripheral saccade target. After the saccade, a letter was briefly presented within the saccade target and terminated by a mask. Object recognition of the letter was assessed as participants' report. Critically, either no, two, or four additional non-target objects appeared before the saccade. In Experiment 1, presaccadic non-targets were task-irrelevant and had no effects on postsaccadic object recognition. In Experiment 2, presaccadic non-targets were task-relevant and, here, postsaccadic object recognition deteriorated with increasing number of presaccadic non-targets. As suggested by Experiment 3 and a mathematical model, this effect was due to a slowing down but also a delayed start of visual processing after the saccade. Together, our findings show that objects compete for recognition across saccades, but only if they are task-relevant. This reveals an attentional mechanism of task-driven object recognition that is interlaced with active saccade-mediated vision.

Poking Left To Be Right? A Model-Based Analysis of Temporal Order Judged by Mice

The theory of visual attention (TVTVA) provides a formal framework for the assessment of visual attention and related processes. Its center is a mathematical model of visual encoding processes and discretely defined components of attention. Building on this model, TVTVA offers quantitative and process-related explanations for a variety of phenomena in the domain of visual attention. Because the theory relies on very general assumptions which might hold true for other domains of sensory processing, we tested its possible explanatory value for tactile processing in mice. Reanalyzing published data of temporal-order judgments by mice, we show how a TVTVA-based analysis identifies the processes which drive observable behavior and that it comes to conclusions quite different from those of conventional analyses of temporal-order judgments. According to this analysis, despite the same overall capacity dedicated to the task, some mice assume attentional biases toward one side, possibly to optimize their overall performance. We suggest that TVTVA's concepts provide a powerful point of vantage to find explanations for observable behavior where conventional analysis easily leads to dead ends.

Behavioral and Brain Measures of Phasic Alerting Effects on Visual Attention

In the present study, we investigated effects of phasic alerting on visual attention in a partial report task, in which half of the displays were preceded by an auditory warning cue. Based on the computational Theory of Visual Attention (TVA), we estimated parameters of spatial and non-spatial aspects of visual attention and measured event-related lateralizations (ERLs) over visual processing areas. We found that the TVA parameter sensory effectiveness a, which is thought to reflect visual processing capacity, significantly increased with phasic alerting. By contrast, the distribution of visual processing resources according to task relevance and spatial position, as quantified in parameters top-down control α and spatial bias w_index, was not modulated by phasic alerting. On the electrophysiological level, the latencies of ERLs in response to the task displays were reduced following the warning cue. These results suggest that phasic alerting facilitates visual processing in a general, unselective manner and that this effect originates in early stages of visual information processing.

Measuring Attention and Visual Processing Speed by Model-based Analysis of Temporal-order Judgments

This protocol describes how to conduct temporal-order experiments to measure visual processing speed and the attentional resource distribution. The proposed method is based on a new and synergistic combination of three components: the temporal-order judgments (TOJ) paradigm, Bundesen's Theory of Visual Attention (TVA), and a hierarchical Bayesian estimation framework. The method provides readily interpretable parameters, which are supported by the theoretical and neurophysiological underpinnings of TVA. Using TOJs, TVA-based estimates can be obtained for a broad range of stimuli, whereas traditional paradigms used with TVA are mainly limited to letters and digits. Finally, the meaningful parameters of the proposed model allow for the establishment of a hierarchical Bayesian model. Such a statistical model allows assessing results in one coherent analysis both on the subject and the group level. To demonstrate the feasibility and versatility of this new approach, three experiments are reported with attention manipulations in synthetic pop-out displays, natural images, and a cued letter-report paradigm.

Out with the old? The role of selective attention in retaining targets in partial report

In the partial-report task, subjects are asked to report only a portion of the items presented. Selective attention chooses which objects to represent in short-term memory (STM) on the basis of their relevance. Because STM is limited in capacity, one must sometimes choose which objects are removed from memory in light of new relevant information. We tested the hypothesis that the choices among newly presented information and old information in STM involve the same process-that both are acts of selective attention. We tested this hypothesis using a two-display partial-report procedure. In this procedure, subjects had to select and retain relevant letters (targets) from two sequentially presented displays. If selection in perception and retention in STM are the same process, then irrelevant letters (distractors) in the second display, which demanded attention because of their similarity to the targets, should have decreased target report from the first display. This effect was not obtained in any of four experiments. Thus, choosing objects to keep in STM is not the same process as choosing new objects to bring into STM.

Peripheral Visual Cues: Their Fate in Processing and Effects on Attention and Temporal-Order Perception

Peripheral visual cues lead to large shifts in psychometric distributions of temporal-order judgments. In one view, such shifts are attributed to attention speeding up processing of the cued stimulus, so-called prior entry. However, sometimes these shifts are so large that it is unlikely that they are caused by attention alone. Here we tested the prevalent alternative explanation that the cue is sometimes confused with the target on a perceptual level, bolstering the shift of the psychometric function. We applied a novel model of cued temporal-order judgments, derived from Bundesen's Theory of Visual Attention. We found that cue-target confusions indeed contribute to shifting psychometric functions. However, cue-induced changes in the processing rates of the target stimuli play an important role, too. At smaller cueing intervals, the cue increased the processing speed of the target. At larger intervals, inhibition of return was predominant. Earlier studies of cued TOJs were insensitive to these effects because in psychometric distributions they are concealed by the conjoint effects of cue-target confusions and processing rate changes.

Episodic Short-Term Recognition Requires Encoding into Visual Working Memory: Evidence from Probe Recognition after Letter Report

Human vision is organized in discrete processing episodes (e.g., eye fixations or task-steps). Object information must be transmitted across episodes to enable episodic short-term recognition: recognizing whether a current object has been seen in a previous episode. We ask whether episodic short-term recognition presupposes that objects have been encoded into capacity-limited visual working memory (VWM), which retains visual information for report. Alternatively, it could rely on the activation of visual features or categories that occurs before encoding into VWM. We assessed the dependence of episodic short-term recognition on VWM by a new paradigm combining letter report and probe recognition. Participants viewed displays of 10 letters and reported as many as possible after a retention interval (whole report). Next, participants viewed a probe letter and indicated whether it had been one of the 10 letters (probe recognition). In Experiment 1, probe recognition was more accurate for letters that had been encoded into VWM (reported letters) compared with non-encoded letters (non-reported letters). Interestingly, those letters that participants reported in their whole report had been near to one another within the letter displays. This suggests that the encoding into VWM proceeded in a spatially clustered manner. In Experiment 2, participants reported only one of 10 letters (partial report) and probes either referred to this letter, to letters that had been near to it, or far from it. Probe recognition was more accurate for near than for far letters, although none of these letters had to be reported. These findings indicate that episodic short-term recognition is constrained to a small number of simultaneously presented objects that have been encoded into VWM.

The attention-weighted sample-size model of visual short-term memory: Attention capture predicts resource allocation and memory load

We investigated the capacity of visual short-term memory (VSTM) in a phase discrimination task that required judgments about the configural relations between pairs of black and white features. Sewell et al. (2014) previously showed that VSTM capacity in an orientation discrimination task was well described by a sample-size model, which views VSTM as a resource comprised of a finite number of noisy stimulus samples. The model predicts the invariance of [Formula: see text] , the sum of squared sensitivities across items, for displays of different sizes. For phase discrimination, the set-size effect significantly exceeded that predicted by the sample-size model for both simultaneously and sequentially presented stimuli. Instead, the set-size effect and the serial position curves with sequential presentation were predicted by an attention-weighted version of the sample-size model, which assumes that one of the items in the display captures attention and receives a disproportionate share of resources. The choice probabilities and response time distributions from the task were well described by a diffusion decision model in which the drift rates embodied the assumptions of the attention-weighted sample-size model.

Breaking Object Correspondence Across Saccadic Eye Movements Deteriorates Object Recognition

Visual perception is based on information processing during periods of eye fixations that are interrupted by fast saccadic eye movements. The ability to sample and relate information on task-relevant objects across fixations implies that correspondence between presaccadic and postsaccadic objects is established. Postsaccadic object information usually updates and overwrites information on the corresponding presaccadic object. The presaccadic object representation is then lost. In contrast, the presaccadic object is conserved when object correspondence is broken. This helps transsaccadic memory but it may impose attentional costs on object recognition. Therefore, we investigated how breaking object correspondence across the saccade affects postsaccadic object recognition. In Experiment 1, object correspondence was broken by a brief postsaccadic blank screen. Observers made a saccade to a peripheral object which was displaced during the saccade. This object reappeared either immediately after the saccade or after the blank screen. Within the postsaccadic object, a letter was briefly presented (terminated by a mask). Observers reported displacement direction and letter identity in different blocks. Breaking object correspondence by blanking improved displacement identification but deteriorated postsaccadic letter recognition. In Experiment 2, object correspondence was broken by changing the object's contrast-polarity. There were no object displacements and observers only reported letter identity. Again, breaking object correspondence deteriorated postsaccadic letter recognition. These findings identify transsaccadic object correspondence as a key determinant of object recognition across the saccade. This is in line with the recent hypothesis that breaking object correspondence results in separate representations of presaccadic and postsaccadic objects which then compete for limited attentional processing resources (Schneider, 2013). Postsaccadic object recognition is then deteriorated because less resources are available for processing postsaccadic objects.

Effects of monitoring for visual events on distinct components of attention

Monitoring the environment for visual events while performing a concurrent task requires adjustment of visual processing priorities. By use of Bundesen's (1990) Theory of Visual Attention, we investigated how monitoring for an object-based brief event affected distinct components of visual attention in a concurrent task. The perceptual salience of the event was varied. Monitoring reduced the processing speed in the concurrent task, and the reduction was stronger when the event was less salient. The monitoring task neither affected the temporal threshold of conscious perception nor the storage capacity of visual short-term memory nor the efficiency of top-down controlled attentional selection.

Emotion based attentional priority for storage in visual short-term memory

A plethora of research demonstrates that the processing of emotional faces is prioritised over non-emotive stimuli when cognitive resources are limited (this is known as ‘emotional superiority’). However, there is debate as to whether competition for processing resources results in emotional superiority per se, or more specifically, threat superiority. Therefore, to investigate prioritisation of emotional stimuli for storage in visual short-term memory (VSTM), we devised an original VSTM report procedure using schematic (angry, happy, neutral) faces in which processing competition was manipulated. In Experiment 1, display exposure time was manipulated to create competition between stimuli. Participants (n = 20) had to recall a probed stimulus from a set size of four under high (150 ms array exposure duration) and low (400 ms array exposure duration) perceptual processing competition. For the high competition condition (i.e. 150 ms exposure), results revealed an emotional superiority effect per se. In Experiment 2 (n = 20), we increased competition by manipulating set size (three versus five stimuli), whilst maintaining a constrained array exposure duration of 150 ms. Here, for the five-stimulus set size (i.e. maximal competition) only threat superiority emerged. These findings demonstrate attentional prioritisation for storage in VSTM for emotional faces. We argue that task demands modulated the availability of processing resources and consequently the relative magnitude of the emotional/threat superiority effect, with only threatening stimuli prioritised for storage in VSTM under more demanding processing conditions. Our results are discussed in light of models and theories of visual selection, and not only combine the two strands of research (i.e. visual selection and emotion), but highlight a critical factor in the processing of emotional stimuli is availability of processing resources, which is further constrained by task demands.

Selective visual processing across competition episodes: a theory of task-driven visual attention and working memory

The goal of this review is to introduce a theory of task-driven visual attention and working memory (TRAM). Based on a specific biased competition model, the ‘theory of visual attention’ (TVA) and its neural interpretation (NTVA), TRAM introduces the following assumption. First, selective visual processing over time is structured in competition episodes. Within an episode, that is, during its first two phases, a limited number of proto-objects are competitively encoded—modulated by the current task—in activation-based visual working memory (VWM). In processing phase 3, relevant VWM objects are transferred via a short-term consolidation into passive VWM. Second, each time attentional priorities change (e.g. after an eye movement), a new competition episode is initiated. Third, if a phase 3 VWM process (e.g. short-term consolidation) is not finished, whereas a new episode is called, a protective maintenance process allows its completion. After a VWM object change, its protective maintenance process is followed by an encapsulation of the VWM object causing attentional resource costs in trailing competition episodes. Viewed from this perspective, a new explanation of key findings of the attentional blink will be offered. Finally, a new suggestion will be made as to how VWM items might interact with visual search processes.

Temporal expectancy in the context of a theory of visual attention

Temporal expectation is expectation with respect to the timing of an event such as the appearance of a certain stimulus. In this paper, temporal expectancy is investigated in the context of the theory of visual attention (TVA), and we begin by summarizing the foundations of this theoretical framework. Next, we present a parametric experiment exploring the effects of temporal expectation on perceptual processing speed in cued single-stimulus letter recognition with unspeeded motor responses. The length of the cue-stimulus foreperiod was exponentially distributed with one of six hazard rates varying between blocks. We hypothesized that this manipulation would result in a distinct temporal expectation in each hazard rate condition. Stimulus exposures were varied such that both the temporal threshold of conscious perception (t0 ms) and the perceptual processing speed (v letters s(-1)) could be estimated using TVA. We found that the temporal threshold t0 was unaffected by temporal expectation, but the perceptual processing speed v was a strikingly linear function of the logarithm of the hazard rate of the stimulus presentation. We argue that the effects on the v values were generated by changes in perceptual biases, suggesting that our perceptual biases are directly related to our temporal expectations.