A dynamic model of how feature cues guide spatial attention

Top-down modulation of DLPFC in visual search: a study based on fMRI and TMS

Effective visual search is essential for daily life, and attention orientation as well as inhibition of return play a significant role in visual search. Researches have established the involvement of dorsolateral prefrontal cortex in cognitive control during selective attention. However, neural evidence regarding dorsolateral prefrontal cortex modulates inhibition of return in visual search is still insufficient. In this study, we employed event-related functional magnetic resonance imaging and dynamic causal modeling to develop modulation models for two types of visual search tasks. In the region of interest analyses, we found that the right dorsolateral prefrontal cortex and temporoparietal junction were selectively activated in the main effect of search type. Dynamic causal modeling results indicated that temporoparietal junction received sensory inputs and only dorsolateral prefrontal cortex →temporoparietal junction connection was modulated in serial search. Such neural modulation presents a significant positive correlation with behavioral reaction time. Furthermore, theta burst stimulation via transcranial magnetic stimulation was utilized to modulate the dorsolateral prefrontal cortex region, resulting in the disappearance of the inhibition of return effect during serial search after receiving continuous theta burst stimulation. Our findings provide a new line of causal evidence that the top-down modulation by dorsolateral prefrontal cortex influences the inhibition of return effect during serial search possibly through the retention of inhibitory tagging via working memory storage.

Contingent capture by color is sensitive to categorical color perception

Contingent capture (CC) theory postulates that attention can only be captured by top-down matching stimuli. Although the contingent capture of attention is a well-known and thoroughly studied phenomenon, there is still no consensus on the characteristics of the top-down template which guides the search for colors. We tried to replicate the classical contingent capture effect on color (Experiment 1) and then added linguistic processing to this perceptual effect (Experiment 2). In Experiment 1, attention was indeed captured by the cues of the same color as the target, while the cues of different colors were successfully ignored. In Experiment 2, the cue color was never identical to the target color but would either belong to the same linguistic category or not (i.e., linguistic matching and linguistic nonmatching cues). In both cases, cues were made to be equally perceptually distant from the target. Although, attention was captured by both cue types, the degree of capture was significantly higher for linguistic matching cues. Our research replicated the classic contingent capture effect but on color, and also demonstrated the effect of color categories in the search task. In short, we demonstrated the effect of color categories in the search task. Results show that the template for color search contains physical characteristics of color, as well as information about color category names.

Motor-related signals support localization invariance for stable visual perception

Our ability to perceive a stable visual world in the presence of continuous movements of the body, head, and eyes has puzzled researchers in the neuroscience field for a long time. We reformulated this problem in the context of hierarchical convolutional neural networks (CNNs)-whose architectures have been inspired by the hierarchical signal processing of the mammalian visual system-and examined perceptual stability as an optimization process that identifies image-defining features for accurate image classification in the presence of movements. Movement signals, multiplexed with visual inputs along overlapping convolutional layers, aided classification invariance of shifted images by making the classification faster to learn and more robust relative to input noise. Classification invariance was reflected in activity manifolds associated with image categories emerging in late CNN layers and with network units acquiring movement-associated activity modulations as observed experimentally during saccadic eye movements. Our findings provide a computational framework that unifies a multitude of biological observations on perceptual stability under optimality principles for image classification in artificial neural networks.

Spatial Attention Enhances Crowded Stimulus Encoding Across Modeled Receptive Fields by Increasing Redundancy of Feature Representations

Any visual system, biological or artificial, must make a trade-off between the number of units used to represent the visual environment and the spatial resolution of the sampling array. Humans and some other animals are able to allocate attention to spatial locations to reconfigure the sampling array of receptive fields (RFs), thereby enhancing the spatial resolution of representations without changing the overall number of sampling units. Here, we examine how representations of visual features in a fully convolutional neural network interact and interfere with each other in an eccentricity-dependent RF pooling array and how these interactions are influenced by dynamic changes in spatial resolution across the array. We study these feature interactions within the framework of visual crowding, a well-characterized perceptual phenomenon in which target objects in the visual periphery that are easily identified in isolation are much more difficult to identify when flanked by similar nearby objects. By separately simulating effects of spatial attention on RF size and on the density of the pooling array, we demonstrate that the increase in RF density due to attention is more beneficial than changes in RF size for enhancing target classification for crowded stimuli. Furthermore, by varying target/flanker spacing, as well as the spatial extent of attention, we find that feature redundancy across RFs has more influence on target classification than the fidelity of the feature representations themselves. Based on these findings, we propose a candidate mechanism by which spatial attention relieves visual crowding through enhanced feature redundancy that is mostly due to increased RF density.

A neuro-computational model of visual attention with multiple attentional control sets

In numerous activities, humans need to attend to multiple sources of visual information at the same time. Although several recent studies support the evidence of this ability, the mechanism of multi-item attentional processing is still a matter of debate and has not been investigated much by previous computational models. Here, we present a neuro-computational model aiming to address specifically the question of how subjects attend to two items that deviate defined by feature and location. We simulate the experiment of Adamo et al. (2010) which required subjects to use two different attentional control sets, each a combination of color and location. The structure of our model is composed of two components "attention" and "decision-making". The important aspect of our model is its dynamic equations that allow us to simulate the time course of processes at a neural level that occur during different stages until a decision is made. We analyze in detail the conditions under which our model matches the behavioral and EEG data from human subjects. Consistent with experimental findings, our model supports the hypothesis of attending to two control settings concurrently. In particular, our model proposes that initially, feature-based attention operates in parallel across the scene, and only in ongoing processing, a selection by the location takes place.

Mechanisms of feature binding in visual working memory are stable over long delays

The ability to accurately retain the binding between the features of different objects is a critical element of visual working memory. The underlying mechanism can be elucidated by analyzing correlations of response errors in dual-report experiments, in which participants have to report two features of a single item from a previously viewed stimulus array. Results from separate previous studies using different cueing conditions have indicated that location takes a privileged role in mediating binding between other features, in that largely independent response errors have been observed when location was used as a cue, but errors were highly correlated when location was one of the reported features. Earlier results from change detection tasks likewise support such a special role of location, but they also suggest that this role is substantially reduced for longer retention intervals in favor of object-based representation. In the present study, we replicated the findings of previous dual-report tasks with different cueing conditions, using matched stimuli and procedures. Moreover, we show that the observed patterns of error correlations remain qualitatively unchanged with longer retention intervals. Fits with neural population models demonstrate that the behavioral results at long, as well as short, delays are best explained by memory representations in independent feature maps, in which an item's features are bound to each other only via their shared location.

Optimal attention tuning in a neuro-computational model of the visual cortex-basal ganglia-prefrontal cortex loop

Visual attention is widely considered a vital factor in the perception and analysis of a visual scene. Several studies explored the effects and mechanisms of top-down attention, but the mechanisms that determine the attentional signal are less explored. By developing a neuro-computational model of visual attention including the visual cortex-basal ganglia loop, we demonstrate how attentional alignment can evolve based on dopaminergic reward during a visual search task. Unlike most previous modeling studies of feature-based attention, we do not implement a manually predefined attention template. Dopamine-modulated covariance learning enable the basal ganglia to learn rewarded associations between the visual input and the attentional gain represented in the PFC of the model. Hence, the model shows human-like performance on a visual search task by optimally tuning the attention signal. In particular, similar as in humans, this reward-based tuning in the model leads to an attentional template that is not centered on the target feature, but a relevant feature deviating away from the target due to the presence of highly similar distractors. Further analyses of the model shows, attention is mainly guided by the signal-to-noise ratio between target and distractors.

Flexible weighting of target features based on distractor context

Models of attention posit that attentional priority is established by summing the saliency and relevancy signals from feature-selective maps. The dimension-weighting account further hypothesizes that information from each feature-selective map is weighted based on expectations of how informative each dimension will be. In the current studies, we investigated the question of whether attentional biases to the features of a conjunction target (color and orientation) differ when one dimension is expected to be more diagnostic of the target. In a series of color-orientation conjunction search tasks, observers saw an exact cue for the upcoming target, while the probability of distractors sharing a target feature in each dimension was manipulated. In one context, distractors were more likely to share the target color, and in another, distractors were more likely to share the target orientation. The results indicated that despite an overall bias toward color, attentional priority to each target feature was flexibly adjusted according to distractor context: RT and accuracy performance was better when the diagnostic feature was expected than unexpected. This occurred both when the distractor context was learned implicitly and explicitly. These results suggest that feature-based enhancement can occur selectively for the dimension expected to be most informative in distinguishing the target from distractors.

Visual Search: How Do We Find What We Are Looking For?

In visual search tasks, observers look for targets among distractors. In the lab, this often takes the form of multiple searches for a simple shape that may or may not be present among other items scattered at random on a computer screen (e.g., Find a red T among other letters that are either black or red.). In the real world, observers may search for multiple classes of target in complex scenes that occur only once (e.g., As I emerge from the subway, can I find lunch, my friend, and a street sign in the scene before me?). This article reviews work on how search is guided intelligently. I ask how serial and parallel processes collaborate in visual search, describe the distinction between search templates in working memory and target templates in long-term memory, and consider how searches are terminated.

Limited interactions between space- and feature-based attention in visually sparse displays

Top-down visual attention selectively filters sensory input so relevant information receives preferential processing. Feature-based attention (FBA) enhances the representation of relevant low-level features, whereas space-based attention (SBA) enhances information at relevant location(s). The present study investigates whether the unique influences of SBA and FBA combine to facilitate behavior in a perceptually demanding discrimination task. We first demonstrated that, independently, both color and location pre-cues could effectively direct attention to facilitate perceptual decision making of a target. We then examined the combined effects of SBA and FBA in the same design by deploying a predictive color arrow pre-cue. Only SBA effects were observed in performance accuracy and reaction time. However, we detected a reaction time cost when a valid spatial cue was paired with a feature cue. A computational perceptual decision-making model largely provided converging evidence that contributions from FBA were restricted to facilitating the speed with which the relevant item was identified. Our results suggest that both selection mechanisms can be used in isolation to resolve a perceptually challenging target in a sparse display, but with little additive perceptual benefit when cued simultaneously. We conclude that there is at least some higher order interdependence between space-based and feature-based selection during decision making under specific conditions.

Humans can efficiently look for but not select multiple visual objects

The human brain recurrently prioritizes task-relevant over task-irrelevant visual information. A central question is whether multiple objects can be prioritized simultaneously. To answer this, we let observers search for two colored targets among distractors. Crucially, we independently varied the number of target colors that observers anticipated, and the number of target colors actually used to distinguish the targets in the display. This enabled us to dissociate the preparation of selection mechanisms from the actual engagement of such mechanisms. Multivariate classification of electroencephalographic activity allowed us to track selection of each target separately across time. The results revealed only small neural and behavioral costs associated with preparing for selecting two objects, but substantial costs when engaging in selection. Further analyses suggest this cost is the consequence of neural competition resulting in limited parallel processing, rather than a serial bottleneck. The findings bridge diverging theoretical perspectives on capacity limitations of feature-based attention.

Investigating the role of verbal templates in contingent capture by color

To investigate if top-down contingent capture by color cues relies on verbal or semantic templates, we combined different stimuli representing colors physically or semantically in six contingent-capture experiments. In contingent capture, only cues that match the top-down search templates lead to validity effects (shorter search times and fewer errors for validly than for invalidly cued targets) resulting from attentional capture by the cue. We compared validity effects of color cues and color-word cues in top-down search for color targets (Experiment 1a) and color-word targets (Experiment 2). We also compared validity effects of color cues and color-associated symbolic cues during search for color targets (Experiment 1b) and of color-word cues during search for both color and color-word targets (Experiment 3). Only cues of the same stimulus category as the target (either color or color-word cues) captured attention. This makes it unlikely that color search is based on verbal or semantic search templates. Additionally, the validity effect of matching color-word cues during search for color-word targets was neither changed by cue-target graphic (font) similarity versus dissimilarity (Experiment 4) nor by articulatory suppression (Experiment 5). These results suggested either a phonological long-term memory template or an orthographically mediated effect of the color-word cues during search for color-words. Altogether, our findings are in line with a pronounced role of color-based templates during contingent capture by color and do not support semantic or verbal influences in this situation.

Corollary Discharge Contributions to Perceptual Continuity Across Saccades

Our vision depends upon shifting our high-resolution fovea to objects of interest in the visual field. Each saccade displaces the image on the retina, which should produce a chaotic scene with jerks occurring several times per second. It does not. This review examines how an internal signal in the primate brain (a corollary discharge) contributes to visual continuity across saccades. The article begins with a review of evidence for a corollary discharge in the monkey and evidence from inactivation experiments that it contributes to perception. The next section examines a specific neuronal mechanism for visual continuity, based on corollary discharge that is referred to as visual remapping. Both the basic characteristics of this anticipatory remapping and the factors that control it are enumerated. The last section considers hypotheses relating remapping to the perceived visual continuity across saccades, including remapping's contribution to perceived visual stability across saccades.

Dynamics of Feature-based Attentional Selection during Color–Shape Conjunction Search

Feature-based attentional selection is accomplished by increasing the gain of sensory neurons encoding target-relevant features while decreasing that of other features. But how do these mechanisms work when targets and distractors share features? We investigated this in a simplified color–shape conjunction search task using ERP components (N2pc, PD, and SPCN) that index lateralized attentional processing. In Experiment 1, we manipulated the presence and frequency of color distractors while holding shape distractors constant. We tested the hypothesis that the color distractor would capture attention, requiring active suppression such that processing of the target can continue. Consistent with this hypothesis, we found that color distractors consistently captured attention, as indexed by a significant N2pc, but were reactively suppressed (indexed by PD). Interestingly, when the color distractor was present, target processing was sustained (indexed by SPCN), suggesting that the dynamics of attentional competition involved distractor suppression interlinked with sustained target processing. In Experiment 2, we examined the contribution of shape to the dynamics of attentional competition under similar conditions. In contrast to color distractors, shape distractors did not reliably capture attention, even when the color distractor was very frequent and attending to target shape would be beneficial. Together, these results suggest that target-colored objects are prioritized during color–shape conjunction search, and the ability to select the target is delayed while target-colored distractors are actively suppressed.

First unitary, then divided: the temporal dynamics of dividing attention

Whether focused visual attention can be divided has been the topic of much investigation, and there is a compelling body of evidence showing that, at least under certain conditions, attention can be divided and deployed as two independent foci. Three experiments were conducted to examine whether attention can be deployed in divided form from the outset, or whether it is first deployed as a unitary focus before being divided. To test this, we adapted the methodology of Jefferies, Enns, and Di Lollo (Journal of Experimental Psychology: Human Perception and Performance 40: 465, 2014), who used a dual-stream Attentional Blink paradigm and two letter-pair targets. One aspect of the AB, Lag-1 sparing, has been shown to occur only if the second target pair appears within the focus of attention. By presenting the second target pair at various spatial locations and assessing the magnitude of Lag-1 sparing, we probed the spatial distribution of attention. By systematically manipulating the stimulus-onset-asynchrony between the targets, we also tracked changes to the spatial distribution of attention over time. The results showed that even under conditions which encourage the division of attention, the attentional focus is first deployed in unitary form before being divided. It is then maintained in divided form only briefly before settling on a single location.

A Neurodynamic Model of Feature-Based Spatial Selection

Huang and Pashler (2007) suggested that feature-based attention creates a special form of spatial representation, which is termed a Boolean map. It partitions the visual scene into two distinct and complementary regions: selected and not selected. Here, we developed a model of a recurrent competitive network that is capable of state-dependent computation. It selects multiple winning locations based on a joint top-down cue. We augmented a model of the WTA circuit that is based on linear-threshold units with two computational elements: dendritic non-linearity that acts on the excitatory units and activity-dependent modulation of synaptic transmission between excitatory and inhibitory units. Computer simulations showed that the proposed model could create a Boolean map in response to a featured cue and elaborate it using the logical operations of intersection and union. In addition, it was shown that in the absence of top-down guidance, the model is sensitive to bottom-up cues such as saliency and abrupt visual onset.

The architecture of interaction between visual working memory and visual attention

In five experiments, we examined whether a task-irrelevant item in visual working memory (VWM) interacts with perceptual selection when VWM must also be used to maintain a template representation of a search target. This question is critical to distinguishing between competing theories specifying the architecture of interaction between VWM and attention. The single-item template hypothesis (SIT) posits that only a single item in VWM can be maintained in a state that interacts with attention. Thus, the secondary item should be inert with respect to attentional guidance. The multiple-item template hypothesis (MIT) posits that multiple items can be maintained in a state that interacts with attention; thus, both the target representation and the secondary item should be capable of guiding selection. This question has been addressed previously in attention capture studies, but the results have been ambiguous. Here, we modified these earlier paradigms to optimize sensitivity to capture. Capture by a distractor matching the secondary item in VWM was observed consistently across multiple types of search task (abstract arrays and natural scenes), multiple dependent measures (search reaction time (RT) and oculomotor capture), multiple memory dimensions (color and shape), and multiple search stimulus dimensions (color, shape, common objects), providing strong support for the MIT. (PsycINFO Database Record

Whatever you do, don't look at the...: Evaluating guidance by an exclusionary attentional template

People can use a target template consisting of one or more features to guide attention and gaze to matching objects in a search array. But can we also use feature information to guide attention away from known irrelevant items? Some studies found a benefit from foreknowledge of a distractor feature, whereas others found a cost. Importantly, previous work has largely relied on end-of-trial manual responses; it is unclear how feature-guided avoidance might unfold as candidate objects are inspected. In the current experiments, participants were cued with a distractor feature to avoid, then performed a visual search task while eye movements were recorded. Participants initially fixated a to-be-avoided object more frequently than predicted by chance, but they also demonstrated avoidance of cue-matching objects later in the trial. When provided more time between cue stimulus and search array, participants continued to be initially captured by a cued-color item. Furthermore, avoidance of cue-matching objects later in the trial was not contingent on initial capture by a cue-matching object. These results suggest that the conflicting findings in previous negative-cue experiments may be explained by a mixture of two independent processes: initial attentional capture by memory-matching items and later avoidance of known irrelevant items. (PsycINFO Database Record

The Role of the Oculomotor System in Updating Visual-Spatial Working Memory across Saccades

Visual-spatial working memory (VSWM) helps us to maintain and manipulate visual information in the absence of sensory input. It has been proposed that VSWM is an emergent property of the oculomotor system. In the present study we investigated the role of the oculomotor system in updating of spatial working memory representations across saccades. Participants had to maintain a location in memory while making a saccade to a different location. During the saccade the target was displaced, which went unnoticed by the participants. After executing the saccade, participants had to indicate the memorized location. If memory updating fully relies on cancellation driven by extraretinal oculomotor signals, the displacement should have no effect on the perceived location of the memorized stimulus. However, if postsaccadic retinal information about the location of the saccade target is used, the perceived location will be shifted according to the target displacement. As it has been suggested that maintenance of accurate spatial representations across saccades is especially important for action control, we used different ways of reporting the location held in memory; a match-to-sample task, a mouse click or by making another saccade. The results showed a small systematic target displacement bias in all response modalities. Parametric manipulation of the distance between the to-be-memorized stimulus and saccade target revealed that target displacement bias increased over time and changed its spatial profile from being initially centered on locations around the saccade target to becoming spatially global. Taken together results suggest that we neither rely exclusively on extraretinal nor on retinal information in updating working memory representations across saccades. The relative contribution of retinal signals is not fixed but depends on both the time available to integrate these signals as well as the distance between the saccade target and the remembered location.

Localized Attentional Interference Affects Object Individuation, Not Feature Detection

Modern theorists conceptualize visual selective attention as a competition between object representations for the control of extrastriate receptive fields, an account supported by the finding that attentional selection of one stimulus can degrade processing of nearby stimuli. In the present study the conditions that produce reciprocal interference between attended stimuli are examined. Each display contained either no, one, or two feature-defined target items among an array of homogeneous distractors. Observers performed two tasks, feature detection and object individuation. The feature-detection task required observers to determine if any targets were present within the display. The object-individuation task required observers to determine if the number of targets was exactly two. Spatially mediated interference between target pairs occurred in the object-individuation task, but had no effect on feature detection. Results suggest that localized interference between attended stimuli occurs only when observers are required to resolve the features of individual objects, consistent with the competitive interaction models of attention.

Modeling the Effects of Perceptual Load: Saliency, Competitive Interactions, and Top-Down Biases

A computational model of visual selective attention has been implemented to account for experimental findings on the Perceptual Load Theory (PLT) of attention. The model was designed based on existing neurophysiological findings on attentional processes with the objective to offer an explicit and biologically plausible formulation of PLT. Simulation results verified that the proposed model is capable of capturing the basic pattern of results that support the PLT as well as findings that are considered contradictory to the theory. Importantly, the model is able to reproduce the behavioral results from a dilution experiment, providing thus a way to reconcile PLT with the competing Dilution account. Overall, the model presents a novel account for explaining PLT effects on the basis of the low-level competitive interactions among neurons that represent visual input and the top-down signals that modulate neural activity. The implications of the model concerning the debate on the locus of selective attention as well as the origins of distractor interference in visual displays of varying load are discussed.

The spatial profile of mask-induced compression for perception and action

Stimuli briefly flashed just before a saccade are perceived closer to the saccade target, a phenomenon known as saccadic compression of space. We have recently demonstrated that similar mislocalizations of flashed stimuli can be observed in the absence of saccades: brief probes were attracted towards a visual reference when followed by a mask. To examine the spatial profile of this new phenomenon of masked-induced compression, here we used a pair of references that draw the probe into the gap between them. Strong compression was found when we masked the probe and presented it following a reference pair, whereas little or no compression occurred for the probe without the reference pair or without the mask. When the two references were arranged vertically, horizontal mislocalizations prevailed. That is, probes presented to the left or right of the vertically arranged references were "drawn in" to be seen aligned with the references. In contrast, when we arranged the two references horizontally, we found vertical compression for stimuli presented above or below the references. Finally, when participants were to indicate the perceived probe location by making an eye movement towards it, saccade landing positions were compressed in a similar fashion as perceptual judgments, confirming the robustness of mask-induced compression. Our findings challenge pure oculomotor accounts of saccadic compression of space that assume a vital role for saccade-specific signals such as corollary discharge or the updating of eye position. Instead, we suggest that saccade- and mask-induced compression both reflect how the visual system deals with disruptions.

Theta oscillations modulate attentional search performance periodically

Visual search--finding a target element among similar-looking distractors--is one of the prevailing experimental methods to study attention. Current theories of visual search postulate an early stage of feature extraction interacting with an attentional process that selects candidate targets for further analysis; in difficult search situations, this selection is iterated until the target is found. Although such theories predict an intrinsic periodicity in the neuronal substrates of attentional search, this prediction has not been extensively tested in human electrophysiology. Here, using EEG and TMS, we study attentional periodicities in visual search. EEG measurements indicated that successful and unsuccessful search trials were associated with different amounts of poststimulus oscillatory amplitude and phase-locking at ∼6 Hz and opposite prestimulus oscillatory phase at ∼6 Hz. A trial-by-trial comparison of pre- and poststimulus ∼6 Hz EEG phases revealed that the functional interplay between prestimulus brain states, poststimulus oscillations, and successful search performance was mediated by a partial phase reset of ongoing oscillations. Independently, TMS applied over occipital cortex at various intervals after search onset demonstrated a periodic pattern of interference at ∼6 Hz. The converging evidence from independent TMS and EEG measurements demonstrates that attentional search is modulated periodically by brain oscillations. This periodicity is naturally compatible with a sequential exploration by attention, although a parallel but rhythmically modulated attention spotlight cannot be entirely ruled out.

Computing with a canonical neural circuits model with pool normalization and modulating feedback

Evidence suggests that the brain uses an operational set of canonical computations like normalization, input filtering, and response gain enhancement via reentrant feedback. Here, we propose a three-stage columnar architecture of cascaded model neurons to describe a core circuit combining signal pathways of feedforward and feedback processing and the inhibitory pooling of neurons to normalize the activity. We present an analytical investigation of such a circuit by first reducing its detail through the lumping of initial feedforward response filtering and reentrant modulating signal amplification. The resulting excitatory-inhibitory pair of neurons is analyzed in a 2D phase-space. The inhibitory pool activation is treated as a separate mechanism exhibiting different effects. We analyze subtractive as well as divisive (shunting) interaction to implement center-surround mechanisms that include normalization effects in the characteristics of real neurons. Different variants of a core model architecture are derived and analyzed--in particular, individual excitatory neurons (without pool inhibition), the interaction with an inhibitory subtractive or divisive (i.e., shunting) pool, and the dynamics of recurrent self-excitation combined with divisive inhibition. The stability and existence properties of these model instances are characterized, which serve as guidelines to adjust these properties through proper model parameterization. The significance of the derived results is demonstrated by theoretical predictions of response behaviors in the case of multiple interacting hypercolumns in a single and in multiple feature dimensions. In numerical simulations, we confirm these predictions and provide some explanations for different neural computational properties. Among those, we consider orientation contrast-dependent response behavior, different forms of attentional modulation, contrast element grouping, and the dynamic adaptation of the silent surround in extraclassical receptive field configurations, using only slight variations of the same core reference model.

Global selection of saccadic target features by neurons in area v4

Psychophysical and neurophysiological studies indicate that during the preparation of saccades, visual processing at the target location is facilitated automatically by the deployment of attention. It has been assumed that the neural mechanisms involved in presaccadic shifts of attention are purely spatial in nature. Saccade preparation modulates the visual responses of neurons within extrastriate area V4, where the responses to targets are enhanced and responses to nontargets are suppressed. We tested whether this effect also engages a nonspatial form of modulation. We measured the responses of area V4 neurons to oriented gratings in two monkeys (Macaca mulatta) making delayed saccades to targets distant from the neuronal receptive field (RF). We varied the orientation of both the RF stimulus and the saccadic target. We found that, in addition to the spatial modulation, saccade preparation involves a feature-dependent modulation of V4 neuronal responses. Specifically, we found that the suppression of area V4 responses to nontarget stimuli during the preparation of saccades depends on the features of the saccadic target. Presaccadic suppression was absent when the features of the saccadic target matched the features preferred by individual V4 neurons. This feature-dependent modulation occurred in the absence of any feature-attention task. We show that our observations are consistent with a computational framework in which feature-based effects automatically emerge from saccade-related feedback signals that are spatial in nature.

A saliency-based bottom-up visual attention model for dynamic scenes analysis

This work proposes a model of visual bottom-up attention for dynamic scene analysis. Our work adds motion saliency calculations to a neural network model with realistic temporal dynamics [(e.g., building motion salience on top of De Brecht and Saiki Neural Networks 19:1467-1474, (2006)]. The resulting network elicits strong transient responses to moving objects and reaches stability within a biologically plausible time interval. The responses are statistically different comparing between earlier and later motion neural activity; and between moving and non-moving objects. We demonstrate the network on a number of synthetic and real dynamical movie examples. We show that the model captures the motion saliency asymmetry phenomenon. In addition, the motion salience computation enables sudden-onset moving objects that are less salient in the static scene to rise above others. Finally, we include strong consideration for the neural latencies, the Lyapunov stability, and the neural properties being reproduced by the model.

Where do we store the memory representations that guide attention?

During the last decade one of the most contentious and heavily studied topics in the attention literature has been the role that working memory representations play in controlling perceptual selection. The hypothesis has been advanced that to have attention select a certain perceptual input from the environment, we only need to represent that item in working memory. Here we summarize the work indicating that the relationship between what representations are maintained in working memory and what perceptual inputs are selected is not so simple. First, it appears that attentional selection is also determined by high-level task goals that mediate the relationship between working memory storage and attentional selection. Second, much of the recent work from our laboratory has focused on the role of long-term memory in controlling attentional selection. We review recent evidence supporting the proposal that working memory representations are critical during the initial configuration of attentional control settings, but that after those settings are established long-term memory representations play an important role in controlling which perceptual inputs are selected by mechanisms of attention.

A mixture distribution of spatial attention

Although it may seem paradoxical, the unified-focus and multiple-foci theories of spatial selective attention are both well supported by experimental evidence. However, the apparent contradiction is illusory and the two competing views may be reconciled by a closer examination of the spatial mechanisms involved. We propose that the deployment of attention may be modeled as a mixture of individual distributions of attention and we tested this hypothesis in two experiments. Participants had to identify targets among distractors, with the targets presented at various distances from the cued locations. Experiment 1 confirmed that the distribution of attention may be described by a mixture of individual distributions, each centered at a cued location. Experiment 2 showed that cue separation is an important determinant of whether spatial attention is divided or not.

Mechanisms and representations of language-mediated visual attention

The experimental investigation of language-mediated visual attention is a promising way to study the interaction of the cognitive systems involved in language, vision, attention, and memory. Here we highlight four challenges for a mechanistic account of this oculomotor behavior: the levels of representation at which language-derived and vision-derived representations are integrated; attentional mechanisms; types of memory; and the degree of individual and group differences. Central points in our discussion are (a) the possibility that local microcircuitries involving feedforward and feedback loops instantiate a common representational substrate of linguistic and non-linguistic information and attention; and (b) that an explicit working memory may be central to explaining interactions between language and visual attention. We conclude that a synthesis of further experimental evidence from a variety of fields of inquiry and the testing of distinct, non-student, participant populations will prove to be critical.

AN EMBEDDED SALIENCY MAP ESTIMATOR SCHEME: APPLICATION TO VIDEO ENCODING

In this paper we propose a novel saliency-based computational model for visual attention. This model processes both top-down (goal directed) and bottom-up information. Processing in the top-down channel creates the so called skin conspicuity map and emulates the visual search for human faces performed by humans. This is clearly a goal directed task but is generic enough to be context independent. Processing in the bottom-up information channel follows the principles set by Itti et al. but it deviates from them by computing the orientation, intensity and color conspicuity maps within a unified multi-resolution framework based on wavelet subband analysis. In particular, we apply a wavelet based approach for efficient computation of the topographic feature maps. Given that wavelets and multiresolution theory are naturally connected the usage of wavelet decomposition for mimicking the center surround process in humans is an obvious choice. However, our implementation goes further. We utilize the wavelet decomposition for inline computation of the features (such as orientation angles) that are used to create the topographic feature maps. The bottom-up topographic feature maps and the top-down skin conspicuity map are then combined through a sigmoid function to produce the final saliency map. A prototype of the proposed model was realized through the TMDSDMK642-0E DSP platform as an embedded system allowing real-time operation. For evaluation purposes, in terms of perceived visual quality and video compression improvement, a ROI-based video compression setup was followed. Extended experiments concerning both MPEG-1 as well as low bit-rate MPEG-4 video encoding were conducted showing significant improvement in video compression efficiency without perceived deterioration in visual quality.

Neural mechanisms of saccade target selection: gated accumulator model of the visual-motor cascade

We review a new computational model developed to understand how evidence about stimulus salience in visual search is translated into a saccade command. The model uses the activity of visually responsive neurons in the frontal eye field as evidence for stimulus salience that is accumulated in a network of stochastic accumulators to produce accurate and timely saccades. We discovered that only when the input to the accumulation process was gated could the model account for the variability in search performance and predict the dynamics of movement neuron discharge rates. This union of cognitive modeling and neurophysiology indicates how the visual-motor transformation can occur, and provides a concrete mapping between neuron function and specific cognitive processes.

Split of spatial attention as predicted by a systems-level model of visual attention

Can we attend to multiple distinct spatial locations at the same time? According to a recent psychophysical study [J. Dubois et al. (2009)Journal of Vision, 9, 3.1-11] such a split of spatial attention might be limited to short periods of time. Following N. P. Bichot et al. [(1999)Perception & Psychophysics, 61, 403-423] subjects had to report the identity of multiple letters that were briefly presented at different locations, while two of these locations (targets) were relevant for a concurrent shape comparison task. In addition to the design used by Bichot et al. stimulus onset asynchrony between shape onset and letters was systematically varied. In general, the performance of subjects was superior at target locations. Furthermore, for short stimulus onset asynchronies, performance was simultaneously increasing at both target locations. For longer stimulus onset asynchronies, however, performance deteriorated at one of the target locations while increasing at the other target location. It was hypothesized that this dynamic deployment of attention might be caused by competitive processes in saccade-related structures such as the frontal eye field. Here we simulated the task of Dubois et al. using a systems-level model of attention. Our results are consistent with recent findings in the frontal eye field obtained during covert visual search, and they support the view of a transient deployment of spatial attention to multiple stimuli in the early epoch of target selection.

Computational models of spatial updating in peri-saccadic perception

Perceptual phenomena that occur around the time of a saccade, such as peri-saccadic mislocalization or saccadic suppression of displacement, have often been linked to mechanisms of spatial stability. These phenomena are usually regarded as errors in processes of trans-saccadic spatial transformations and they provide important tools to study these processes. However, a true understanding of the underlying brain processes that participate in the preparation for a saccade and in the transfer of information across it requires a closer, more quantitative approach that links different perceptual phenomena with each other and with the functional requirements of ensuring spatial stability. We review a number of computational models of peri-saccadic spatial perception that provide steps in that direction. Although most models are concerned with only specific phenomena, some generalization and interconnection between them can be obtained from a comparison. Our analysis shows how different perceptual effects can coherently be brought together and linked back to neuronal mechanisms on the way to explaining vision across saccades.

Biased Competition in Visual Processing Hierarchies: A Learning Approach Using Multiple Cues

In this contribution, we present a large-scale hierarchical system for object detection fusing bottom-up (signal-driven) processing results with top-down (model or task-driven) attentional modulation. Specifically, we focus on the question of how the autonomous learning of invariant models can be embedded into a performing system and how such models can be used to define object-specific attentional modulation signals. Our system implements bi-directional data flow in a processing hierarchy. The bottom-up data flow proceeds from a preprocessing level to the hypothesis level where object hypotheses created by exhaustive object detection algorithms are represented in a roughly retinotopic way. A competitive selection mechanism is used to determine the most confident hypotheses, which are used on the system level to train multimodal models that link object identity to invariant hypothesis properties. The top-down data flow originates at the system level, where the trained multimodal models are used to obtain space- and feature-based attentional modulation signals, providing biases for the competitive selection process at the hypothesis level. This results in object-specific hypothesis facilitation/suppression in certain image regions which we show to be applicable to different object detection mechanisms. In order to demonstrate the benefits of this approach, we apply the system to the detection of cars in a variety of challenging traffic videos. Evaluating our approach on a publicly available dataset containing approximately 3,500 annotated video images from more than 1 h of driving, we can show strong increases in performance and generalization when compared to object detection in isolation. Furthermore, we compare our results to a late hypothesis rejection approach, showing that early coupling of top-down and bottom-up information is a favorable approach especially when processing resources are constrained.

A nonlinear dynamics model for simulating long range correlations of cognitive bistability

Simulation results of bistable perception due to ambiguous visual stimuli are presented which are obtained with a behavioral nonlinear dynamics model using perception-attention-memory coupling. This model provides an explanation of recent experimental results of Gao et al. (Cogn Process 7:105-112, 2006a) and it supports their speculation that the fractal character of perceptual dominance time series may be understood in terms of nonlinear and reentrant dynamics of brain processing. Percept reversals are induced by attention fatigue and noise, with an attention bias which balances the relative percept duration. Dynamical coupling of the attention bias to the perception state introduces memory effects leading to significant long range correlations of perceptual duration times as quantified by the Hurst parameter H > 0.5 (Mandelbrot, The fractal geometry of nature, 1991), in agreement with Gao et al. (Cogn Process 7:105-112, 2006a).

Reaction time distributions constrain models of visual search

Many experiments have investigated visual search for simple stimuli like colored bars or alphanumeric characters. When eye movements are not a limiting factor, these tasks tend to produce roughly linear functions relating reaction time (RT) to the number of items in the display (set size). The slopes of the RTxset size functions for different searches fall on a continuum from highly efficient (slopes near zero) to inefficient (slopes>25-30ms/item). Many theories of search can produce the correct pattern of mean RTs. Producing the correct RT distributions is more difficult. In order to guide future modeling, we have collected a very large data set (about 112,000 trials) on three tasks: an efficient color feature search, an inefficient search for a 2 among 5s, and an intermediate colorxorientation conjunction search. The RT distributions have interesting properties. For example, target absent distributions overlap target present more than would be expected if the decision to end search were based on a simple elapsed time threshold. Other qualitative properties of the RT distributions falsify some classes of model. For example, normalized RT distributions do not change shape as set size changes as a standard self-terminating model predicts that they should.

A computational model of Basal Ganglia and its role in memory retrieval in rewarded visual memory tasks

Visual working memory (WM) tasks involve a network of cortical areas such as inferotemporal, medial temporal and prefrontal cortices. We suggest here to investigate the role of the basal ganglia (BG) in the learning of delayed rewarded tasks through the selective gating of thalamocortical loops. We designed a computational model of the visual loop linking the perirhinal cortex, the BG and the thalamus, biased by sustained representations in prefrontal cortex. This model learns concurrently different delayed rewarded tasks that require to maintain a visual cue and to associate it to itself or to another visual object to obtain reward. The retrieval of visual information is achieved through thalamic stimulation of the perirhinal cortex. The input structure of the BG, the striatum, learns to represent visual information based on its association to reward, while the output structure, the substantia nigra pars reticulata, learns to link striatal representations to the disinhibition of the correct thalamocortical loop. In parallel, a dopaminergic cell learns to associate striatal representations to reward and modulates learning of connections within the BG. The model provides testable predictions about the behavior of several areas during such tasks, while providing a new functional organization of learning within the BG, putting emphasis on the learning of the striatonigral connections as well as the lateral connections within the substantia nigra pars reticulata. It suggests that the learning of visual WM tasks is achieved rapidly in the BG and used as a teacher for feedback connections from prefrontal cortex to posterior cortices.

Influence of low-level stimulus features, task dependent factors, and spatial biases on overt visual attention

Visual attention is thought to be driven by the interplay between low-level visual features and task dependent information content of local image regions, as well as by spatial viewing biases. Though dependent on experimental paradigms and model assumptions, this idea has given rise to varying claims that either bottom-up or top-down mechanisms dominate visual attention. To contribute toward a resolution of this discussion, here we quantify the influence of these factors and their relative importance in a set of classification tasks. Our stimuli consist of individual image patches (bubbles). For each bubble we derive three measures: a measure of salience based on low-level stimulus features, a measure of salience based on the task dependent information content derived from our subjects' classification responses and a measure of salience based on spatial viewing biases. Furthermore, we measure the empirical salience of each bubble based on our subjects' measured eye gazes thus characterizing the overt visual attention each bubble receives. A multivariate linear model relates the three salience measures to overt visual attention. It reveals that all three salience measures contribute significantly. The effect of spatial viewing biases is highest and rather constant in different tasks. The contribution of task dependent information is a close runner-up. Specifically, in a standardized task of judging facial expressions it scores highly. The contribution of low-level features is, on average, somewhat lower. However, in a prototypical search task, without an available template, it makes a strong contribution on par with the two other measures. Finally, the contributions of the three factors are only slightly redundant, and the semi-partial correlation coefficients are only slightly lower than the coefficients for full correlations. These data provide evidence that all three measures make significant and independent contributions and that none can be neglected in a model of human overt visual attention.

Visual stability based on remapping of attention pointers

When we move our eyes, we easily keep track of where relevant things are in the world. Recent proposals link this stability to the shifting of receptive fields of neurons in eye movement and attention control areas. Reports of 'spatiotopic' visual aftereffects have also been claimed to support this shifting connectivity even at an early level, but these results have been challenged. Here, the process of updating visual location is described as predictive shifts of location 'pointers' to attended targets, analogous to predictive activation seen cross-modally. We argue that these location pointers, the core operators of spatial attention, are linked to identity information and that such a link is necessary to establish a workable visual architecture and to explain frequently reported positive spatiotopic biases.

Visual and motor connectivity and the distribution of calcium-binding proteins in macaque frontal eye field: implications for saccade target selection

The frontal eye field (FEF) contributes to directing visual attention and saccadic eye movement through intrinsic processing, interactions with extrastriate visual cortical areas (e.g., V4), and projections to subcortical structures (e.g., superior colliculus, SC). Several models have been proposed to describe the relationship between the allocation of visual attention and the production of saccades. We obtained anatomical information that might provide useful constraints on these models by evaluating two characteristics of FEF. First, we investigated the laminar distribution of efferent connections from FEF to visual areas V4 + TEO and to SC. Second, we examined the laminar distribution of different populations of GABAergic neurons in FEF. We found that the neurons in FEF that project to V4 + TEO are located predominantly in the supragranular layers, colocalized with the highest density of calbindin- and calretinin-immunoreactive inhibitory interneurons. In contrast, the cell bodies of neurons that project to SC are found only in layer 5 of FEF, colocalized primarily with parvalbumin inhibitory interneurons. None of the neurons in layer 5 that project to V4 + TEO also project to SC. These results provide useful constraints for cognitive models of visual attention and saccade production by indicating that different populations of neurons project to extrastriate visual cortical areas and to SC. This finding also suggests that FEF neurons projecting to visual cortex and SC are embedded in different patterns of intracortical circuitry.

Attentional guidance relies on a winner-take-all mechanism

The finding that attention can encompass several non-contiguous items at once challenges the current models of visual search based on a winner-take-all mechanism assuming the selection of a single object. It has been proposed instead that attentional guidance involves mechanisms selecting all relevant items simultaneously. In order to test this hypothesis, we studied attentional allocation during various visual search tasks. We confirmed that attention can indeed select several items concurrently but on the basis of their spatial relation, not relevance. This finding corroborates the view that during visual search, attentional guidance is based on a winner-take-all mechanism.

Efficient coding correlates with spatial frequency tuning in a model of V1 receptive field organization

Efficient coding has been proposed to play an essential role in early visual processing. While several approaches used an objective function to optimize a particular aspect of efficient coding, such as the minimization of mutual information or the maximization of sparseness, we here explore how different estimates of efficient coding in a model with nonlinear dynamics and Hebbian learning determine the similarity of model receptive fields to V1 data with respect to spatial tuning. Our simulation results indicate that most measures of efficient coding correlate with the similarity of model receptive field data to V1 data, that is, optimizing the estimate of efficient coding increases the similarity of the model data to experimental data. However, the degree of the correlation varies with the different estimates of efficient coding, and in particular, the variance in the firing pattern of each cell does not predict a similarity of model and experimental data.

Statistical decision theory to relate neurons to behavior in the study of covert visual attention

Scrutiny of the numerous physiology and imaging studies of visual attention reveal that integration of results from neuroscience with the classic theories of visual attention based on behavioral work is not simple. The different subfields have pursued different questions, used distinct experimental paradigms and developed diverse models. The purpose of this review is to use statistical decision theory and computational modeling to relate classic theories of attention in psychological research to neural observables such as mean firing rate or functional imaging BOLD response, tuning functions, Fano factor, neuronal index of detectability and area under the receiver operating characteristic (ROC). We focus on cueing experiments and attempt to distinguish two major leading theories in the study of attention: limited resources model/increased sensitivity vs. selection/differential weighting. We use Bayesian ideal observer (BIO) modeling, in which predictive cues or prior knowledge change the differential weighting (prior) of sensory information to generate predictions of behavioral and neural observables based on Gaussian response variables and Poisson process neural based models. The ideal observer model can be modified to represent a number of classic psychological theories of visual attention by including hypothesized human attentional limited resources in the same way sequential ideal observer analysis has been used to include physiological processing components of human spatial vision (Geisler, W. S. (1989). Sequential ideal-observer analysis of visual discrimination. Psychological Review 96, 267-314.). In particular we compare new biologically plausible implementations of the BIO and variant models with limited resources. We find a close relationship between the behavioral effects of cues predicted by the models developed in the field of human psychophysics and their neuron-based analogs. Critically, we show that cue effects on experimental observables such as mean neural activity, variance, Fano factor and neuronal index of detectability can be consistent with the two major theoretical models of attention depending on whether the neuron is assumed to be computing likelihoods, log-likelihoods or a simple model operating directly on the Poisson variable. Change in neuronal tuning functions can also be consistent with both theories depending on whether the change in tuning is along the dimension being experimentally cued or a different dimension. We show that a neuron's sensitivity appropriately measured using the area under the Receive Operating Characteristic curve can be used to distinguish across both theories and is robust to the many transformations of the decision variable. We provide a summary table with the hope that it might provide some guidance in interpreting past results as well as planning future studies.