From salience to saccades: multiple-alternative gated stochastic accumulator model of visual search

Separate evaluation of target facilitation and distractor suppression in the activity of macaque lateral intraparietal neurons during visual search

During visual search, neurons in the lateral intraparietal area (LIP) discriminate the target from distractors by exhibiting stronger activation when the target appears within the receptive field than when it appears outside the receptive field. It is generally thought that such target-discriminative activity is produced by the combination of target-related facilitation and distractor-related suppression. However, little is known about how the target-discriminative activity is constituted by these two types of neural modulation. To address this issue, we recorded activity from LIP of monkeys performing a visual search task that consisted of target-present and target-absent trials. Monkeys had to make a saccade to a target in the target-present trials, whereas they had to maintain fixation in the target-absent trials, in which only distractors were presented. By introducing the activity from the latter trials as neutral activity, we were able to separate the target-discriminative activity into target-related elevation and distractor-related reduction components. We found that the target-discriminative activity of most LIP neurons consisted of the combination of target-related elevation and distractor-related reduction or only target-related elevation. In contrast, target-discriminative activity composed of only distractor-related reduction was observed for very few neurons. We also found that, on average, target-related elevation was stronger and occurred earlier compared with distractor-related reduction. Finally, we consider possible underlying mechanisms, including lateral inhibitory interactions, responsible for target-discriminative activity in visual search. The present findings provide insight into how neuronal modulations shape target-discriminative activity during visual search.

Simultaneous modeling of visual saliency and value computation improves predictions of economic choice

Many decisions we make require visually identifying and evaluating numerous alternatives quickly. These usually vary in reward, or value, and in low-level visual properties, such as saliency. Both saliency and value influence the final decision. In particular, saliency affects fixation locations and durations, which are predictive of choices. However, it is unknown how saliency propagates to the final decision. Moreover, the relative influence of saliency and value is unclear. Here we address these questions with an integrated model that combines a perceptual decision process about where and when to look with an economic decision process about what to choose. The perceptual decision process is modeled as a drift-diffusion model (DDM) process for each alternative. Using psychophysical data from a multiple-alternative, forced-choice task, in which subjects have to pick one food item from a crowded display via eye movements, we test four models where each DDM process is driven by (i) saliency or (ii) value alone or (iii) an additive or (iv) a multiplicative combination of both. We find that models including both saliency and value weighted in a one-third to two-thirds ratio (saliency-to-value) significantly outperform models based on either quantity alone. These eye fixation patterns modulate an economic decision process, also described as a DDM process driven by value. Our combined model quantitatively explains fixation patterns and choices with similar or better accuracy than previous models, suggesting that visual saliency has a smaller, but significant, influence than value and that saliency affects choices indirectly through perceptual decisions that modulate economic decisions.

Attentional resource allocation during a cued saccade task

Attentional selection of sensory information and motor output is critical for successful interaction with one's surroundings. However, organization of attentional processes involved in selection of salient visual information, decision making, and movement planning has not yet been fully elucidated. We hypothesized that attentional processes involved in these tasks can function independently and draw from separate resources. If true, challenging the capacity limit of one attentional process would not affect performance of others. Healthy participants performed a cued saccade task in which target cues were embedded in a central stream of letters in a Rapid Serial Visual Presentation (RSVP). Participants performed saccades as quickly and as accurately as possible to a peripheral target location based on cue presentation within the central letter stream. To challenge visual attention, we parametrically varied the duration at which each letter of the RSVP was presented (50-200ms). In a separate experiment we challenged motor attention by increasing the number of possible saccade trajectories (1-6 peripheral targets). As expected, increasing attentional load in one domain of the task negatively affected performance in that domain, while performance in other domains was unaffected. We interpret our results as support for the independent allocation of attentional resources, at least in the early stages of processing, required across components of a cued saccade task. Deciphering the contributions of attention during visuomotor tasks is a critical step to understanding how humans process information necessary to successfully interact with the environment.

Persistence of internal representations of alternative voluntary actions

We have investigated a situation in which externally available response alternatives and their internal representations could be dissociated, by suddenly removing some action alternatives from the response space during the interval between the free selection and the execution of a voluntary action. Choice reaction times in this situation were related to the number of initially available response alternatives, rather than to the number of alternatives available effectively available after the change in the external environment. The internal representations of response alternatives appeared to persist after external changes actually made the corresponding action unavailable. This suggests a surprising dynamics of voluntary action representations: counterfactual response alternatives persist, and may even be actively maintained, even when they are not available in reality. Our results highlight a representational basis for the counterfactual course of action. Such representations may play a key role in feelings of regret, disappointment, or frustration. These feelings all involve persistent representation of counterfactual response alternatives that may not actually be available in the environment.

Macrocircuits: decision networks

Decision-making requires stimulus categorization and localization to guide accurate responses that can be produced through multiple effectors. The success of actions is monitored so that performance can be adjusted to achieve goals. This review will survey recent empirical and theoretical developments very selectively with an emphasis on neurophysiological data from nonhuman primates that provide the clearest information about neural mechanisms.

Salience-based selection: attentional capture by distractors less salient than the target

Current accounts of attentional capture predict the most salient stimulus to be invariably selected first. However, existing salience and visual search models assume noise in the map computation or selection process. Consequently, they predict the first selection to be stochastically dependent on salience, implying that attention could even be captured first by the second most salient (instead of the most salient) stimulus in the field. Yet, capture by less salient distractors has not been reported and salience-based selection accounts claim that the distractor has to be more salient in order to capture attention. We tested this prediction using an empirical and modeling approach of the visual search distractor paradigm. For the empirical part, we manipulated salience of target and distractor parametrically and measured reaction time interference when a distractor was present compared to absent. Reaction time interference was strongly correlated with distractor salience relative to the target. Moreover, even distractors less salient than the target captured attention, as measured by reaction time interference and oculomotor capture. In the modeling part, we simulated first selection in the distractor paradigm using behavioral measures of salience and considering the time course of selection including noise. We were able to replicate the result pattern we obtained in the empirical part. We conclude that each salience value follows a specific selection time distribution and attentional capture occurs when the selection time distributions of target and distractor overlap. Hence, selection is stochastic in nature and attentional capture occurs with a certain probability depending on relative salience.

Neural mechanisms of speed-accuracy tradeoff

Intelligent agents balance speed of responding with accuracy of deciding. Stochastic accumulator models commonly explain this speed-accuracy tradeoff by strategic adjustment of response threshold. Several laboratories identify specific neurons in prefrontal and parietal cortex with this accumulation process, yet no neurophysiological correlates of speed-accuracy tradeoff have been described. We trained macaque monkeys to trade speed for accuracy on cue during visual search and recorded the activity of neurons in the frontal eye field. Unpredicted by any model, we discovered that speed-accuracy tradeoff is accomplished through several distinct adjustments. Visually responsive neurons modulated baseline firing rate, sensory gain, and the duration of perceptual processing. Movement neurons triggered responses with activity modulated in a direction opposite of model predictions. Thus, current stochastic accumulator models provide an incomplete description of the neural processes accomplishing speed-accuracy tradeoffs. The diversity of neural mechanisms was reconciled with the accumulator framework through an integrated accumulator model constrained by requirements of the motor system.

Distinct neural mechanisms of distractor suppression in the frontal and parietal lobe

The posterior parietal cortex and the prefrontal cortex are associated with eye movements and visual attention, but their specific contributions are poorly understood. We compared the dorsolateral prefrontal cortex (dlPFC) and the lateral intraparietal area (LIP) in monkeys using a memory saccade task in which a salient distractor flashed at a variable timing and location during the memory delay. We found that the two areas had similar responses to target selection, but made distinct contributions to distractor suppression. Distractor responses were more strongly suppressed and more closely correlated with performance in the dlPFC relative to LIP. Moreover, reversible inactivation of the dlPFC produced much larger increases in distractibility than inactivation of LIP. These findings suggest that LIP and dlPFC mediate different aspects of selective attention. Although both areas can contribute to the perceptual selection of salient information, the dlPFC has a decisive influence on whether and how attended stimulus is linked with actions.

Impact of correlated neural activity on decision-making performance

Stimulus from the environment that guides behavior and informs decisions is encoded in the firing rates of neural populations. Neurons in the populations, however, do not spike independently: spike events are correlated from cell to cell. To what degree does this apparent redundancy have an impact on the accuracy with which decisions can be made and the computations required to optimally decide? We explore these questions for two illustrative models of correlation among cells. Each model is statistically identical at the level of pairwise correlations but differs in higher-order statistics that describe the simultaneous activity of larger cell groups. We find that the presence of correlations can diminish the performance attained by an ideal decision maker to either a small or large extent, depending on the nature of the higher-order correlations. Moreover, although this optimal performance can in some cases be obtained using the standard integration-to-bound operation, in others it requires a nonlinear computation on incoming spikes. Overall, we conclude that a given level of pairwise correlations, even when restricted to identical neural populations, may not always indicate redundancies that diminish decision-making performance.

Supplementary eye field during visual search: salience, cognitive control, and performance monitoring

How supplementary eye field (SEF) contributes to visual search is unknown. Inputs from cortical and subcortical structures known to represent visual salience suggest that SEF may serve as an additional node in this network. This hypothesis was tested by recording action potentials and local field potentials (LFPs) in two monkeys performing an efficient pop-out visual search task. Target selection modulation, tuning width, and response magnitude of spikes and LFP in SEF were compared with those in frontal eye field. Surprisingly, only ∼2% of SEF neurons and ∼8% of SEF LFP sites selected the location of the search target. The absence of salience in SEF may be due to an absence of appropriate visual afferents, which suggests that these inputs are a necessary anatomical feature of areas representing salience. We also tested whether SEF contributes to overcoming the automatic tendency to respond to a primed color when the target identity switches during priming of pop-out. Very few SEF neurons or LFP sites modulated in association with performance deficits following target switches. However, a subset of SEF neurons and LFPs exhibited strong modulation following erroneous saccades to a distractor. Altogether, these results suggest that SEF plays a limited role in controlling ongoing visual search behavior, but may play a larger role in monitoring search performance.

Attention, learning, and the value of information

Despite many studies on selective attention, fundamental questions remain about its nature and neural mechanisms. Here I draw from the animal and machine learning fields that describe attention as a mechanism for active learning and uncertainty reduction and explore the implications of this view for understanding visual attention and eye movement control. I propose that a closer integration of these different views has the potential greatly to expand our understanding of oculomotor control and our ability to use this system as a window into high level but poorly understood cognitive functions, including the capacity for curiosity and exploration and for inferring internal models of the external world.

Response variability of frontal eye field neurons modulates with sensory input and saccade preparation but not visual search salience

Discharge rate modulation of frontal eye field (FEF) neurons has been identified with a representation of visual search salience (physical conspicuity and behavioral relevance) and saccade preparation. We tested whether salience or saccade preparation are evident in the trial-to-trial variability of discharge rate. We quantified response variability via the Fano factor in FEF neurons recorded in monkeys performing efficient and inefficient visual search tasks. Response variability declined following stimulus presentation in most neurons, but despite clear discharge rate modulation, variability did not change with target salience. Instead, we found that response variability was modulated by stimulus luminance and the number of items in the visual field independently of attentional demands. Response variability declined to a minimum before saccade initiation, and presaccadic response variability was directionally tuned. In addition, response variability was correlated with the response time of memory-guided saccades. These results indicate that the trial-by-trial response variability of FEF neurons reflects saccade preparation and the strength of sensory input, but not visual search salience or attentional allocation.

Perceptions as hypotheses: saccades as experiments

If perception corresponds to hypothesis testing (Gregory, 1980); then visual searches might be construed as experiments that generate sensory data. In this work, we explore the idea that saccadic eye movements are optimal experiments, in which data are gathered to test hypotheses or beliefs about how those data are caused. This provides a plausible model of visual search that can be motivated from the basic principles of self-organized behavior: namely, the imperative to minimize the entropy of hidden states of the world and their sensory consequences. This imperative is met if agents sample hidden states of the world efficiently. This efficient sampling of salient information can be derived in a fairly straightforward way, using approximate Bayesian inference and variational free-energy minimization. Simulations of the resulting active inference scheme reproduce sequential eye movements that are reminiscent of empirically observed saccades and provide some counterintuitive insights into the way that sensory evidence is accumulated or assimilated into beliefs about the world.