Parallel and Serial Neural Mechanisms for Visual Search in Macaque Area V4

Dynamic target match signals in perirhinal cortex can be explained by instantaneous computations that act on dynamic input from inferotemporal cortex

Finding sought objects requires the brain to combine visual and target signals to determine when a target is in view. To investigate how the brain implements these computations, we recorded neural responses in inferotemporal cortex (IT) and perirhinal cortex (PRH) as macaque monkeys performed a delayed-match-to-sample target search task. Our data suggest that visual and target signals were combined within or before IT in the ventral visual pathway and then passed onto PRH, where they were reformatted into a more explicit target match signal over ∼10-15 ms. Accounting for these dynamics in PRH did not require proposing dynamic computations within PRH itself but, rather, could be attributed to instantaneous PRH computations performed upon an input representation from IT that changed with time. We found that the dynamics of the IT representation arose from two commonly observed features: individual IT neurons whose response preferences were not simply rescaled with time and variable response latencies across the population. Our results demonstrate that these types of time-varying responses have important consequences for downstream computation and suggest that dynamic representations can arise within a feedforward framework as a consequence of instantaneous computations performed upon time-varying inputs.

Feature attention for binocular disparity in primate area MT depends on tuning strength

Attending to a stimulus modulates the responses of sensory neurons that represent features of that stimulus, a phenomenon named "feature attention." For example, attending to a stimulus containing upward motion enhances the responses of upward-preferring direction-selective neurons in the middle temporal area (MT) and suppresses the responses of downward-preferring neurons, even when the attended stimulus is outside of the spatial receptive fields of the recorded neurons (Treue S, Martinez-Trujillo JC. Nature 399: 575-579, 1999). This modulation renders the representation of sensory information across a neuronal population more selective for the features present in the attended stimulus (Martinez-Trujillo JC, Treue S. Curr Biol 14: 744-751, 2004). We hypothesized that if feature attention modulates neurons according to their tuning preferences, it should also be sensitive to their tuning strength, which is the magnitude of the difference in responses to preferred and null stimuli. We measured how the effects of feature attention on MT neurons in rhesus monkeys (Macaca mulatta) depended on the relationship between features-in our case, direction of motion and binocular disparity-of the attended stimulus and a neuron's tuning for those features. We found that, as for direction, attention to stimuli containing binocular disparity cues modulated the responses of MT neurons and that the magnitude of the modulation depended on both a neuron's tuning preferences and its tuning strength. Our results suggest that modulation by feature attention may depend not just on which features a neuron represents but also on how well the neuron represents those features.

Saccades and shifting receptive fields: anticipating consequences or selecting targets?

Saccadic eye movements cause frequent and substantial displacements of the retinal image, but those displacements go unnoticed. It has been widely assumed that this perceived stability emerges from the shifting of visual receptive fields from their current, presaccadic locations to their future, postsaccadic locations in anticipation of the retinal consequences of saccades. Although evidence consistent with this anticipatory remapping has accumulated over the years, more recent work suggests an alternative view. In this opinion article, we examine the evidence of presaccadic receptive field shifts and their relationship to the perceptual changes that accompany saccades. We argue that both reflect the selection of targets for saccades rather than the anticipation of a displaced retinal image.

Deep learning in neural networks: an overview

In recent years, deep artificial neural networks (including recurrent ones) have won numerous contests in pattern recognition and machine learning. This historical survey compactly summarizes relevant work, much of it from the previous millennium. Shallow and Deep Learners are distinguished by the depth of their credit assignment paths, which are chains of possibly learnable, causal links between actions and effects. I review deep supervised learning (also recapitulating the history of backpropagation), unsupervised learning, reinforcement learning & evolutionary computation, and indirect search for short programs encoding deep and large networks.

Interactions between space-based and feature-based attention

Although early research suggested that attention to nonspatial features (i.e., red) was confined to stimuli appearing at an attended spatial location, more recent research has emphasized the global nature of feature-based attention. For example, a distractor sharing a target feature may capture attention even if it occurs at a task-irrelevant location. Such findings have been used to argue that feature-based attention operates independently of spatial attention. However, feature-based attention may nonetheless interact with spatial attention, yielding larger feature-based effects at attended locations than at unattended locations. The present study tested this possibility. In 2 experiments, participants viewed a rapid serial visual presentation (RSVP) stream and identified a target letter defined by its color. Target-colored distractors were presented at various task-irrelevant locations during the RSVP stream. We found that feature-driven attentional capture effects were largest when the target-colored distractor was closer to the attended location. These results demonstrate that spatial attention modulates the strength of feature-based attention capture, calling into question the prior evidence that feature-based attention operates in a global manner that is independent of spatial attention.

Dynamic integration of task-relevant visual features in posterior parietal cortex

The primate visual system consists of multiple hierarchically organized cortical areas, each specialized for processing distinct aspects of the visual scene. For example, color and form are encoded in ventral pathway areas such as V4 and inferior temporal cortex, while motion is preferentially processed in dorsal pathway areas such as the middle temporal area. Such representations often need to be integrated perceptually to solve tasks that depend on multiple features. We tested the hypothesis that the lateral intraparietal area (LIP) integrates disparate task-relevant visual features by recording from LIP neurons in monkeys trained to identify target stimuli composed of conjunctions of color and motion features. We show that LIP neurons exhibit integrative representations of both color and motion features when they are task relevant and task-dependent shifts of both direction and color tuning. This suggests that LIP plays a role in flexibly integrating task-relevant sensory signals.

Rejection positivity predicts trial-to-trial reaction times in an auditory selective attention task: a computational analysis of inhibitory control

A series of computer simulations using variants of a formal model of attention (Melara and Algom, 2003) probed the role of rejection positivity (RP), a slow-wave electroencephalographic (EEG) component, in the inhibitory control of distraction. Behavioral and EEG data were recorded as participants performed auditory selective attention tasks. Simulations that modulated processes of distractor inhibition accounted well for reaction-time (RT) performance, whereas those that modulated target excitation did not. A model that incorporated RP from actual EEG recordings in estimating distractor inhibition was superior in predicting changes in RT as a function of distractor salience across conditions. A model that additionally incorporated momentary fluctuations in EEG as the source of trial-to-trial variation in performance precisely predicted individual RTs within each condition. The results lend support to the linking proposition that RP controls the speed of responding to targets through the inhibitory control of distractors.

The role of visual area V4 in the discrimination of partially occluded shapes

The primate brain successfully recognizes objects, even when they are partially occluded. To begin to elucidate the neural substrates of this perceptual capacity, we measured the responses of shape-selective neurons in visual area V4 while monkeys discriminated pairs of shapes under varying degrees of occlusion. We found that neuronal shape selectivity always decreased with increasing occlusion level, with some neurons being notably more robust to occlusion than others. The responses of neurons that maintained their selectivity across a wider range of occlusion levels were often sufficiently sensitive to support behavioral performance. Many of these same neurons were distinctively selective for the curvature of local boundary features and their shape tuning was well fit by a model of boundary curvature (curvature-tuned neurons). A significant subset of V4 neurons also signaled the animal's upcoming behavioral choices; these decision signals had short onset latencies that emerged progressively later for higher occlusion levels. The time course of the decision signals in V4 paralleled that of shape selectivity in curvature-tuned neurons: shape selectivity in curvature-tuned neurons, but not others, emerged earlier than the decision signals. These findings provide evidence for the involvement of contour-based mechanisms in the segmentation and recognition of partially occluded objects, consistent with psychophysical theory. Furthermore, they suggest that area V4 participates in the representation of the relevant sensory signals and the generation of decision signals underlying discrimination.

How do we select multiple features? Transient costs for selecting two colors rather than one, persistent costs for color-location conjunctions

In a previous study Lo, Howard, & Holcombe (Vision Research 63:20-33, 2012), selecting two colors did not induce a performance cost, relative to selecting one color. For example, requiring possible report of both a green and a red target did not yield a worse performance than when both targets were green. Yet a cost of selecting multiple colors was observed when selection needed be contingent on both color and location. When selecting a red target to the left and a green target to the right, superimposing a green distractor to the left and a red distractor to the right impeded performance. Possibly, participants cannot confine attention to a color at a particular location. As a result, distractors that share the target colors disrupt attentional selection of the targets. The attempt to select the targets must then be repeated, which increases the likelihood that the trial terminates when selection is not effective, even for long trials. Consistent with this, here we find a persistent cost of selecting two colors when the conjunction of color and location is needed, but the cost is confined to short exposure durations when the observer just has to monitor red and green stimuli without the need to use the location information. These results suggest that selecting two colors is time-consuming but effective, whereas selection of simultaneous conjunctions is never entirely successful.

Effects of spatial and feature attention on disparity-rendered structure-from-motion stimuli in the human visual cortex

An important advance in the study of visual attention has been the identification of a non-spatial component of attention that enhances the response to similar features or objects across the visual field. Here we test whether this non-spatial component can co-select individual features that are perceptually bound into a coherent object. We combined human psychophysics and functional magnetic resonance imaging (fMRI) to demonstrate the ability to co-select individual features from perceptually coherent objects. Our study used binocular disparity and visual motion to define disparity structure-from-motion (dSFM) stimuli. Although the spatial attention system induced strong modulations of the fMRI response in visual regions, the non-spatial system’s ability to co-select features of the dSFM stimulus was less pronounced and variable across subjects. Our results demonstrate that feature and global feature attention effects are variable across participants, suggesting that the feature attention system may be limited in its ability to automatically select features within the attended object. Careful comparison of the task design suggests that even minor differences in the perceptual task may be critical in revealing the presence of global feature attention.

Global selection of saccadic target features by neurons in area v4

Psychophysical and neurophysiological studies indicate that during the preparation of saccades, visual processing at the target location is facilitated automatically by the deployment of attention. It has been assumed that the neural mechanisms involved in presaccadic shifts of attention are purely spatial in nature. Saccade preparation modulates the visual responses of neurons within extrastriate area V4, where the responses to targets are enhanced and responses to nontargets are suppressed. We tested whether this effect also engages a nonspatial form of modulation. We measured the responses of area V4 neurons to oriented gratings in two monkeys (Macaca mulatta) making delayed saccades to targets distant from the neuronal receptive field (RF). We varied the orientation of both the RF stimulus and the saccadic target. We found that, in addition to the spatial modulation, saccade preparation involves a feature-dependent modulation of V4 neuronal responses. Specifically, we found that the suppression of area V4 responses to nontarget stimuli during the preparation of saccades depends on the features of the saccadic target. Presaccadic suppression was absent when the features of the saccadic target matched the features preferred by individual V4 neurons. This feature-dependent modulation occurred in the absence of any feature-attention task. We show that our observations are consistent with a computational framework in which feature-based effects automatically emerge from saccade-related feedback signals that are spatial in nature.

Lesions of prefrontal cortex reduce attentional modulation of neuronal responses and synchrony in V4

It is widely held that the frontal eye field (FEF) in prefrontal cortex (PFC) modulates processing in visual cortex with attention, although the evidence for a necessary role is equivocal. To help identify critical sources of attentional feedback to area V4, we surgically removed the entire lateral PFC, including the FEF, in one hemisphere and transected the corpus callosum and anterior commisure in two macaques. This deprived V4 of PFC input in one hemisphere while keeping the other hemisphere intact. In the absence of PFC, attentional effects on neuronal responses and synchrony in V4 were significantly reduced and the remaining effects of attention were delayed in time indicating a critical role of PFC. Conversely, distracters captured attention and influenced V4 responses. However, because the effects of attention in V4 were not eliminated by PFC lesions, other sources of top-down attentional control signals to visual cortex must exist outside of PFC.

Predictive saccade target selection in superior colliculus during visual search

Searching for a visual object naturally involves sequences of gaze fixations, during which the current foveal image is analyzed and the next object to inspect is selected as a saccade target. Fixation durations during such sequences are short, suggesting that saccades may be concurrently processed. Therefore, the selection of the next saccade target may occur before the current saccade target is acquired. To test this hypothesis, we trained four female rhesus monkeys (Macaca mulatta) to perform a multiple-fixation visual conjunction search task. We simultaneously recorded the activity of sensorimotor neurons in the midbrain superior colliculus (SC) in two monkeys. In this task, monkeys made multiple fixations before foveating the target. Fixation durations were significantly shorter than the latency of the initial responses to the search display, with approximately one-quarter being shorter than the shortest response latencies. The time at which SC sensorimotor activity discriminated the target from distracters occurred significantly earlier for the selection of subsequent fixations than for the selection of the first fixation. Target selection during subsequent fixations occurred even before the visual afferent delay in more than half of the neuronal sample, suggesting that the process of selection can encompass at least two future saccade targets. This predictive selection was present even when differences in saccade latencies were taken into account. Altogether, these findings demonstrate how neural representations on the visual salience map are processed in parallel, thus facilitating visual search.

Spatial ranking strategy and enhanced peripheral vision discrimination optimize performance and efficiency of visual sequential search

Visual sequential search might use a peripheral spatial ranking of the scene to put the next target of the sequence in the correct order. This strategy, indeed, might enhance the discriminative capacity of the human peripheral vision and spare neural resources associated with foveation. However, it is not known how exactly the peripheral vision sustains sequential search and whether the sparing of neural resources has a cost in terms of performance. To elucidate these issues, we compared strategy and performance during an alpha-numeric sequential task where peripheral vision was modulated in three different conditions: normal, blurred, or obscured. If spatial ranking is applied to increase the peripheral discrimination, its use as a strategy in visual sequencing should differ according to the degree of discriminative information that can be obtained from the periphery. Moreover, if this strategy spares neural resources without impairing the performance, its use should be associated with better performance. We found that spatial ranking was applied when peripheral vision was fully available, reducing the number and time of explorative fixations. When the periphery was obscured, explorative fixations were numerous and sparse; when the periphery was blurred, explorative fixations were longer and often located close to the items. Performance was significantly improved by this strategy. Our results demonstrated that spatial ranking is an efficient strategy adopted by the brain in visual sequencing to highlight peripheral detection and discrimination; it reduces the neural cost by avoiding unnecessary foveations, and promotes sequential search by facilitating the onset of a new saccade.

Functions of gamma-band synchronization in cognition: from single circuits to functional diversity across cortical and subcortical systems

Gamma-band activity (30-90 Hz) and the synchronization of neural activity in the gamma-frequency range have been observed in different cortical and subcortical structures and have been associated with different cognitive functions. However, it is still unknown whether gamma-band synchronization subserves a single universal function or a diversity of functions across the full spectrum of cognitive processes. Here, we address this question reviewing the mechanisms of gamma-band oscillation generation and the functions associated with gamma-band activity across several cortical and subcortical structures. Additionally, we raise a plausible explanation of why gamma rhythms are found so ubiquitously across brain structures. Gamma band activity originates from the interplay between inhibition and excitation. We stress that gamma oscillations, associated with this interplay, originate from basic functional motifs that conferred advantages for low-level system processing and multiple cognitive functions throughout evolution. We illustrate the multifunctionality of gamma-band activity by considering its role in neural systems for perception, selective attention, memory, motivation and behavioral control. We conclude that gamma-band oscillations support multiple cognitive processes, rather than a single one, which, however, can be traced back to a limited set of circuit motifs which are found universally across species and brain structures.

Intellectual disabilities and power spectra analysis during sleep: a new perspective on borderline intellectual functioning

BACKGROUND

The role of sleep in cognitive processes has been confirmed by a growing number of reports for all ages of life. Analysing sleep electroencephalogram (EEG) spectra may be useful to study cortical organisation in individuals with Borderline Intellectual Functioning (BIF), as seen in other disturbances even if it is not considered a disease. The aim of this study was to determine if the sleep EEG power spectra in children with BIF could be different from typically developing children.

METHODS

Eighteen BIF (12 males) (mean age 11.04; SD ± 1.07) and 24 typical developing children (14 men) (mean age 10.98; SD ± 1.76; P = 0.899) underwent an overnight polysomnography (PSG) recording in the Sleep Laboratory of the Clinic of Child and Adolescent Neuropsychiatry, after one adaptation night. Sleep was subdivided into 30-s epochs and sleep stages were scored according to the standard criteria and the power spectra were calculated for the Cz-A2 channel using the sleep analysis software Hypnolab 1.2 (SWS Soft, Italy) by means of the Fast Fourier Transform and the power spectrum was calculated for frequencies between 0.5 and 60 Hz with a frequency step of 1 Hz and then averaged across the following bands delta (0.5-4 Hz), theta (5-7 Hz), alpha (8-11 Hz), sigma (11-15 Hz), and beta (16-30 Hz), gamma (30-60 Hz) for S2, SWS and REM (Rapid Eye Movement) sleep stages.

RESULTS

BIF have a reduced sleep duration (total sleep time; P < 0.001), and an increased rate of stage shifts (P < 0.001) and awakenings (P < 0.001) and WASO (wakefulness after sleep onset) % (P < 0.001); the stage 2% (P < 0.001), and REM% (P < 0.001) were lower and slow-wave sleep percentage was slightly higher (P < 0.001). All children with BIF had an AHI (apnoea-hypopnea index) less than 1 (mean AHI = 0.691 ± 0.236) with a mean oxygen saturation of 97.6% and a periodic leg movement index (PLMI) less than 5 (mean PLMI = 2.94 ± 1.56). All sleep stages had a significant reduction in gamma frequency (30-60 Hz) (P < 0.001) and an increased delta frequency (0.5-4.0 Hz) (P < 0.001) power in BIF subjects compared with typically developing children.

CONCLUSION

Our findings shed light on the importance of sleep for cognition processes particularly in cognitive borderline dysfunction and the role of EEG spectral power analysis to recognize sleep characteristics in BIF children.

Decision-related activity in sensory neurons may depend on the columnar architecture of cerebral cortex

Many studies have reported correlations between the activity of sensory neurons and animals' judgments in discrimination tasks. Here, we suggest that such neuron-behavior correlations may require a cortical map for the task relevant features. This would explain why studies using discrimination tasks based on disparity in area V1 have not found these correlations: V1 contains no map for disparity. This scheme predicts that activity of V1 neurons correlates with decisions in an orientation-discrimination task. To test this prediction, we trained two macaque monkeys in a coarse orientation discrimination task using band-pass-filtered dynamic noise. The two orientations were always 90° apart and task difficulty was controlled by varying the orientation bandwidth of the filter. While the trained animals performed this task, we recorded from orientation-selective V1 neurons (n = 82, n = 31 for Monkey 1, n = 51 for Monkey 2). For both monkeys, we observed significant correlation (quantified as "choice probabilities") of the V1 activity with the monkeys' perceptual judgments (mean choice probability 0.54, p = 10(-5)). In one of these animals, we had previously measured choice probabilities in a disparity discrimination task in V1, which had been at chance (0.49, not significantly different from 0.5). The choice probabilities in this monkey for the orientation discrimination task were significantly larger than those for the disparity discrimination task (p = 0.032). These results are predicted by our suggestion that choice probabilities are only observed for cortical sensory neurons that are organized in maps for the task-relevant feature.

Determinants of Global Color-Based Selection in Human Visual Cortex

Feature attention operates in a spatially global way, with attended feature values being prioritized for selection outside the focus of attention. Accounts of global feature attention have emphasized feature competition as a determining factor. Here, we use magnetoencephalographic recordings in humans to test whether competition is critical for global feature selection to arise. Subjects performed a color/shape discrimination task in one visual field (VF), while irrelevant color probes were presented in the other unattended VF. Global effects of color attention were assessed by analyzing the response to the probe as a function of whether or not the probe's color was a target-defining color. We find that global color selection involves a sequence of modulations in extrastriate cortex, with an initial phase in higher tier areas (lateral occipital complex) followed by a later phase in lower tier retinotopic areas (V3/V4). Importantly, these modulations appeared with and without color competition in the focus of attention. Moreover, early parts of the modulation emerged for a task-relevant color not even present in the focus of attention. All modulations, however, were eliminated during simple onset-detection of the colored target. These results indicate that global color-based attention depends on target discrimination independent of feature competition in the focus of attention.

Neural responses to target features outside a search array are enhanced during conjunction but not unique-feature search

The visual world is typically too complex to permit full apprehension of its content from a single fixation. Humans therefore use visual search to direct attention and eye movements to locations or objects of interest in cluttered scenes. Psychophysical investigations have revealed that observers can select target elements from within an array of distractors on the basis of their spatial location or simple features, such as color. It remains unclear, however, how stimuli that lie outside the current search array are represented in the visual system. To investigate this, we recorded continuous neural activity using EEG while participants searched a foveal array of colored targets and distractors, and ignored irrelevant objects in the periphery. Search targets were defined either by a unique feature within the array or by a conjunction of features. Objects outside the array could match the target or distractor color within the array, or otherwise possessed a baseline (neutral) color present only in the periphery. The search array and irrelevant peripheral objects flickered at unique rates and thus evoked distinct frequency-tagged neural oscillations. During conjunction but not unique-feature search, target-colored objects outside the array evoked enhanced activity relative to distractor-colored and neutral objects. The results suggest that feature-based selection applies to stimuli at ignored peripheral locations, but only when central targets compete with distractors within the array. Distractor-colored and neutral objects evoked equivalent oscillatory responses, suggesting that feature-based selection at ignored locations during visual search arises exclusively from enhancement rather than suppression of neural activity.

Visual space is compressed in prefrontal cortex before eye movements

We experience the visual world through a series of saccadic eye movements, each one shifting our gaze to bring objects of interest to the fovea for further processing. Although such movements lead to frequent and substantial displacements of the retinal image, these displacements go unnoticed. It is widely assumed that a primary mechanism underlying this apparent stability is an anticipatory shifting of visual receptive fields (RFs) from their presaccadic to their postsaccadic locations before movement onset. Evidence of this predictive 'remapping' of RFs has been particularly apparent within brain structures involved in gaze control. However, critically absent among that evidence are detailed measurements of visual RFs before movement onset. Here we show that during saccade preparation, rather than remap, RFs of neurons in a prefrontal gaze control area massively converge towards the saccadic target. We mapped the visual RFs of prefrontal neurons during stable fixation and immediately before the onset of eye movements, using multi-electrode recordings in monkeys. Following movements from an initial fixation point to a target, RFs remained stationary in retinocentric space. However, in the period immediately before movement onset, RFs shifted by as much as 18 degrees of visual angle, and converged towards the target location. This convergence resulted in a threefold increase in the proportion of RFs responding to stimuli near the target region. In addition, like in human observers, the population of prefrontal neurons grossly mislocalized presaccadic stimuli as being closer to the target. Our results show that RF shifts do not predict the retinal displacements due to saccades, but instead reflect the overriding perception of target space during eye movements.

Optimal attentional modulation of a neural population

Top-down attention has often been separately studied in the contexts of either optimal population coding or biasing of visual search. Yet, both are intimately linked, as they entail optimally modulating sensory variables in neural populations according to top-down goals. Designing experiments to probe top-down attentional modulation is difficult because non-linear population dynamics are hard to predict in the absence of a concise theoretical framework. Here, we describe a unified framework that encompasses both contexts. Our work sheds light onto the ongoing debate on whether attention modulates neural response gain, tuning width, and/or preferred feature. We evaluate the framework by conducting simulations for two tasks: (1) classification (discrimination) of two stimuli s a and s b and (2) searching for a target T among distractors D. Results demonstrate that all of gain, tuning, and preferred feature modulation happen to different extents, depending on stimulus conditions and task demands. The theoretical analysis shows that task difficulty (linked to difference Δ between s a and s b , or T, and D) is a crucial factor in optimal modulation, with different effects in discrimination vs. search. Further, our framework allows us to quantify the relative utility of neural parameters. In easy tasks (when Δ is large compared to the density of the neural population), modulating gains and preferred features is sufficient to yield nearly optimal performance; however, in difficult tasks (smaller Δ), modulating tuning width becomes necessary to improve performance. This suggests that the conflicting reports from different experimental studies may be due to differences in tasks and in their difficulties. We further propose future electrophysiology experiments to observe different types of attentional modulation in a same neuron.

Selective attention in the honeybee optic lobes precedes behavioral choices

Attention allows animals to respond selectively to competing stimuli, enabling some stimuli to evoke a behavioral response while others are ignored. How the brain does this remains mysterious, although it is increasingly evident that even animals with the smallest brains display this capacity. For example, insects respond selectively to salient visual stimuli, but it is unknown where such selectivity occurs in the insect brain, or whether neural correlates of attention might predict the visual choices made by an insect. Here, we investigate neural correlates of visual attention in behaving honeybees (Apis mellifera). Using a closed-loop paradigm that allows tethered, walking bees to actively control visual objects in a virtual reality arena, we show that behavioral fixation increases neuronal responses to flickering, frequency-tagged stimuli. Attention-like effects were reduced in the optic lobes during replay of the same visual sequences, when bees were not able to control the visual displays. When bees were presented with competing frequency-tagged visual stimuli, selectivity in the medulla (an optic ganglion) preceded behavioral selection of a stimulus, suggesting that modulation of early visual processing centers precedes eventual behavioral choices made by these insects.

Attention in the real world: toward understanding its neural basis

The efficient selection of behaviorally relevant objects from cluttered environments supports our everyday goals. Attentional selection has typically been studied in search tasks involving artificial and simplified displays. Although these studies have revealed important basic principles of attention, they do not explain how the brain efficiently selects familiar objects in complex and meaningful real-world scenes. Findings from recent neuroimaging studies indicate that real-world search is mediated by 'what' and 'where' attentional templates that are implemented in high-level visual cortex. These templates represent target-diagnostic properties and likely target locations, respectively, and are shaped by object familiarity, scene context, and memory. We propose a framework for real-world search that incorporates these recent findings and specifies directions for future study.

Perceptual asynchrony for motion

Psychophysical experiments show that two different visual attributes, color and motion, processed in different areas of the visual brain, are perceived at different times relative to each other (Moutoussis and Zeki, 1997a). Here we demonstrate psychophysically that two variants of the same attribute, motion, which have the same temporal structure and are processed in the same visual areas, are also processed asynchronously. When subjects were asked to pair up–down motion of dots in one half of their hemifield with up-right motion in the other, they perceived the two directions of motion asynchronously, with the advantage in favor of up-right motion; when they were asked to pair the motion of white dots moving against a black background with that of red dots moving against an equiluminant green background, they perceived the luminant motion first, thus demonstrating a perceptual advantage of luminant over equiluminant motion. These results were not affected by motion speed or perceived motion “streaks.” We thus interpret these results to reflect the different processing times produced by luminant and equiluminant motion stimuli or by different degrees of motion direction change, thus adding to the evidence that processing time within the visual system is a major determinant of perceptual time.

Neurosystems: brain rhythms and cognitive processing

Neuronal rhythms are ubiquitous features of brain dynamics, and are highly correlated with cognitive processing. However, the relationship between the physiological mechanisms producing these rhythms and the functions associated with the rhythms remains mysterious. This article investigates the contributions of rhythms to basic cognitive computations (such as filtering signals by coherence and/or frequency) and to major cognitive functions (such as attention and multi-modal coordination). We offer support to the premise that the physiology underlying brain rhythms plays an essential role in how these rhythms facilitate some cognitive operations.

The interaction of feature and space based orienting within the attention set

The processing of sensory information relies on interacting mechanisms of sustained attention and attentional capture, both of which operate in space and on object features. While evidence indicates that exogenous attentional capture, a mechanism previously understood to be automatic, can be eliminated while concurrently performing a demanding task, we reframe this phenomenon within the theoretical framework of the “attention set” (Most et al., 2005). Consequently, the specific prediction that cuing effects should reappear when feature dimensions of the cue overlap with those in the attention set (i.e., elements of the demanding task) was empirically tested and confirmed using a dual-task paradigm involving both sustained attention and attentional capture, adapted from Santangelo et al. (2007). Participants were required to either detect a centrally presented target presented in a stream of distractors (the primary task), or respond to a spatially cued target (the secondary task). Importantly, the spatial cue could either share features with the target in the centrally presented primary task, or not share any features. Overall, the findings supported the attention set hypothesis showing that a spatial cuing effect was only observed when the peripheral cue shared a feature with objects that were already in the attention set (i.e., the primary task). However, this finding was accompanied by differential attentional orienting dependent on the different types of objects within the attention set, with feature-based orienting occurring for target-related objects, and additional spatial-based orienting for distractor-related objects.

Theta band activity in response to emotional expressions and its relationship with gamma band activity as revealed by MEG and advanced beamformer source imaging

Neuronal oscillations in the theta and gamma bands have been shown to be important for cognition. Here we examined the temporal and spatial relationship between the two frequency bands in emotional processing using magnetoencephalography and an advanced dynamic beamformer source imaging method called synthetic aperture magnetometry. We found that areas including the amygdala, visual and frontal cortex showed significant event-related synchronization in both bands, suggesting a functional association of neuronal oscillations in the same areas in the two bands. However, while the temporal profile in both bands was similar in the amygdala, the peak in gamma band power was much earlier within both visual and frontal areas. Our results do not support a traditional view that the localizations of lower and higher frequencies are spatially distinct. Instead, they suggest that in emotional processing, neuronal oscillations in the gamma and theta bands may reflect, at least in visual and frontal cortex either different but related functional processes or, perhaps more probably, different computational components of the same functional process.

Extrafoveal preview benefit during free-viewing visual search in the monkey

Previous studies have shown that subjects require less time to process a stimulus at the fovea after a saccade if they have viewed the same stimulus in the periphery immediately prior to the saccade. This extrafoveal preview benefit indicates that information about the visual form of an extrafoveally viewed stimulus can be transferred across a saccade. Here, we extend these findings by demonstrating and characterizing a similar extrafoveal preview benefit in monkeys during a free-viewing visual search task. We trained two monkeys to report the orientation of a target among distractors by releasing one of two bars with their hand; monkeys were free to move their eyes during the task. Both monkeys took less time to indicate the orientation of the target after foveating it, when the target lay closer to the fovea during the previous fixation. An extrafoveal preview benefit emerged even if there was more than one intervening saccade between the preview and the target fixation, indicating that information about target identity could be transferred across more than one saccade and could be obtained even if the search target was not the goal of the next saccade. An extrafoveal preview benefit was also found for distractor stimuli. These results aid future physiological investigations of the extrafoveal preview benefit.

Bottom-up and top-down attention: different processes and overlapping neural systems

The brain is limited in its capacity to process all sensory stimuli present in the physical world at any point in time and relies instead on the cognitive process of attention to focus neural resources according to the contingencies of the moment. Attention can be categorized into two distinct functions: bottom-up attention, referring to attentional guidance purely by externally driven factors to stimuli that are salient because of their inherent properties relative to the background; and top-down attention, referring to internal guidance of attention based on prior knowledge, willful plans, and current goals. Over the past few years, insights on the neural circuits and mechanisms of bottom-up and top-down attention have been gained through neurophysiological experiments. Attention affects the mean neuronal firing rate as well as its variability and correlation across neurons. Although distinct processes mediate the guidance of attention based on bottom-up and top-down factors, a common neural apparatus, the frontoparietal network, is essential in both types of attentional processes.

Visual working memory modulates low-level saccade target selection: evidence from rapidly generated saccades in the global effect paradigm

In three experiments, we examined the influence of visual working memory (VWM) on the metrics of saccade landing position in a global effect paradigm. Participants executed a saccade to the more eccentric object in an object pair appearing on the horizontal midline, to the left or right of central fixation. While completing the saccade task, participants maintained a color in VWM for an unrelated memory task. Either the color of the saccade target matched the memory color (target match), the color of the distractor matched the memory color (distractor match), or the colors of neither object matched the memory color (no match). In the no-match condition, saccades tended to land at the midpoint between the two objects: the global, or averaging, effect. However, when one of the two objects matched VWM, the distribution of landing position shifted toward the matching object, both for target match and for distractor match. VWM modulation of landing position was observed even for the fastest quartile of saccades, with a mean latency as low as 112 ms. Effects of VWM on such rapidly generated saccades, with latencies in the express-saccade range, indicate that VWM interacts with the initial sweep of visual sensory processing, modulating perceptual input to oculomotor systems and thereby biasing oculomotor selection. As a result, differences in memory match produce effects on landing position similar to the effects generated by differences in physical salience.

Production, control, and visual guidance of saccadic eye movements

Primate vision is served by rapid shifts of gaze called saccades. This review will survey current knowledge and particular problems concerning the neural control and guidance of gaze shifts.

Neural chronometry and coherency across speed-accuracy demands reveal lack of homomorphism between computational and neural mechanisms of evidence accumulation

The stochastic accumulation framework provides a mechanistic, quantitative account of perceptual decision-making and how task performance changes with experimental manipulations. Importantly, it provides an elegant account of the speed-accuracy trade-off (SAT), which has long been the litmus test for decision models, and also mimics the activity of single neurons in several key respects. Recently, we developed a paradigm whereby macaque monkeys trade speed for accuracy on cue during visual search task. Single-unit activity in frontal eye field (FEF) was not homomorphic with the architecture of models, demonstrating that stochastic accumulators are an incomplete description of neural activity under SAT. This paper summarizes and extends this work, further demonstrating that the SAT leads to extensive, widespread changes in brain activity never before predicted. We will begin by reviewing our recently published work that establishes how spiking activity in FEF accomplishes SAT. Next, we provide two important extensions of this work. First, we report a new chronometric analysis suggesting that increases in perceptual gain with speed stress are evident in FEF synaptic input, implicating afferent sensory-processing sources. Second, we report a new analysis demonstrating selective influence of SAT on frequency coupling between FEF neurons and local field potentials. None of these observations correspond to the mechanics of current accumulator models.

Visual cortical gamma-band activity during free viewing of natural images

Gamma-band activity in visual cortex has been implicated in several cognitive operations, like perceptual grouping and attentional selection. So far, it has been studied primarily under well-controlled visual fixation conditions and using well-controlled stimuli, like isolated bars or patches of grating. If gamma-band activity is to subserve its purported functions outside of the laboratory, it should be present during natural viewing conditions. We recorded neuronal activity with a 252-channel electrocorticographic (ECoG) grid covering large parts of the left hemisphere of 2 macaque monkeys, while they freely viewed natural images. We found that natural viewing led to pronounced gamma-band activity in the visual cortex. In area V1, gamma-band activity during natural viewing showed a clear spectral peak indicative of oscillatory activity between 50 and 80 Hz and was highly significant for each of 65 natural images. Across the ECoG grid, gamma-band activity during natural viewing was present over most of the recorded visual cortex and absent over most remaining cortex. After saccades, the gamma peak frequency slid down to 30–40 Hz at around 80 ms postsaccade, after which the sustained 50- to 80-Hz gamma-band activity resumed. We propose that gamma-band activity plays an important role during natural viewing.

Dissociating oculomotor contributions to spatial and feature-based selection

Saccades not only deliver the high-resolution retinal image requisite for visual perception, but processing stages associated with saccade target selection affect visual perception even before the eye movement starts. These presaccadic effects are thought to arise from two visual selection mechanisms: spatial selection that enhances processing of the saccade target location and feature-based selection that enhances processing of the saccade target features. By measuring oculomotor performance and perceptual discrimination, we determined which selection mechanisms are associated with saccade preparation. We observed both feature-based and space-based selection during saccade preparation but found that feature-based selection was neither related to saccade initiation nor was it affected by simultaneously observed redistribution of spatial selection. We conclude that oculomotor selection biases visual selection only in a spatial, feature-unspecific manner.

Evidence for differential top-down and bottom-up suppression in posterior parietal cortex

When searching for an object, we usually avoid items that are visually different from the target and objects or places that have been searched already. Previous studies have shown that neural activity in the lateral intraparietal area (LIP) can be used to guide this behaviour; responses to task irrelevant stimuli or to stimuli that have been fixated previously in the trial are reduced compared with responses to potential targets. Here, we test the hypothesis that these reduced responses have a different genesis. Two animals were trained on a visual foraging task, in which they had to find a target among a number of physically identical potential targets (T) and task irrelevant distractors. We recorded neural activity and local field potentials (LFPs) in LIP while the animals performed the task. We found that LFP power was similar for potential targets and distractors but was greater in the alpha and low beta bands when a previously fixated T was in the response field. We interpret these data to suggest that the reduced single-unit response to distractors is a bottom-up feed-forward result of processing in earlier areas and the reduced response to previously fixated Ts is a result of active top-down suppression.

Predictability of spatial and non-spatial target properties improves perception in the pre-saccadic interval

In a dual-task paradigm with a perceptual discrimination task and a concurrent saccade task, we examined participants' ability to make use of prior knowledge of a critical property of the perceptual target to improve discrimination. Previous research suggests that during a short time window before a saccade, covert attention is imperatively directed towards the saccade target location. Consequently, discrimination of perceptual targets at the saccade target location is better than at other locations. We asked whether the obligatory pre-saccadic attention shift prevents perceptual benefits arising for perceptual target stimuli with predictable as opposed to non-predictable properties. We compared conditions in which the color or location of the perceptual target was constant to conditions in which those properties varied randomly across trials. In addition to the expected improvements of perception at the saccade target location, we found perception to be better with constant than with random properties of the perceptual target. Thus, color or location information about an upcoming perceptual target facilitates perception even while spatial attention is shifted to the saccade target. The improvement occurred irrespective of the saccade target location, which suggests that the underlying mechanism is independent of the pre-saccadic attention shift, but alternative interpretations are discussed as well.

Top-down beta rhythms support selective attention via interlaminar interaction: a model

Cortical rhythms have been thought to play crucial roles in our cognitive abilities. Rhythmic activity in the beta frequency band, around 20 Hz, has been reported in recent studies that focused on neural correlates of attention, indicating that top-down beta rhythms, generated in higher cognitive areas and delivered to earlier sensory areas, can support attentional gain modulation. To elucidate functional roles of beta rhythms and underlying mechanisms, we built a computational model of sensory cortical areas. Our simulation results show that top-down beta rhythms can activate ascending synaptic projections from L5 to L4 and L2/3, responsible for biased competition in superficial layers. In the simulation, slow-inhibitory interneurons are shown to resonate to the 20 Hz input and modulate the activity in superficial layers in an attention-related manner. The predicted critical roles of these cells in attentional gain provide a potential mechanism by which cholinergic drive can support selective attention.

Top-down signals originate from higher cognitive areas such as parietal and prefrontal cortex and propagate to earlier stages of the brain. They have been thought to be associated with selective attention, and recent physiological studies suggest that top-down signals in the beta frequency band can support selective attention. In this study, we employ a computational model to investigate potential mechanisms by which top-down beta rhythms can influence neural responses induced by presentation of stimuli. The model includes several cell types, reportedly crucial for generating cortical rhythmic activity in the gamma and beta frequency bands, and the simulation results show that top-down beta rhythms are capable of reproducing experimentally observed attentional effects on neural responses to visual stimuli. These modulatory effects of top-down beta rhythms are mainly induced via activation of ascending inhibition originating from deep layer slow inhibitory interneurons. Since the excitability of slow interneurons can be increased by cholinergic neuromodulators, these interneurons may mediate the effects of cholinergic tone on attention.

Dynamic oscillatory processes governing cued orienting and allocation of auditory attention

In everyday listening situations, we need to constantly switch between alternative sound sources and engage attention according to cues that match our goals and expectations. The exact neuronal bases of these processes are poorly understood. We investigated oscillatory brain networks controlling auditory attention using cortically constrained fMRI-weighted magnetoencephalography/EEG source estimates. During consecutive trials, participants were instructed to shift attention based on a cue, presented in the ear where a target was likely to follow. To promote audiospatial attention effects, the targets were embedded in streams of dichotically presented standard tones. Occasionally, an unexpected novel sound occurred opposite to the cued ear to trigger involuntary orienting. According to our cortical power correlation analyses, increased frontoparietal/temporal 30-100 Hz gamma activity at 200-1400 msec after cued orienting predicted fast and accurate discrimination of subsequent targets. This sustained correlation effect, possibly reflecting voluntary engagement of attention after the initial cue-driven orienting, spread from the TPJ, anterior insula, and inferior frontal cortices to the right FEFs. Engagement of attention to one ear resulted in a significantly stronger increase of 7.5-15 Hz alpha in the ipsilateral than contralateral parieto-occipital cortices 200-600 msec after the cue onset, possibly reflecting cross-modal modulation of the dorsal visual pathway during audiospatial attention. Comparisons of cortical power patterns also revealed significant increases of sustained right medial frontal cortex theta power, right dorsolateral pFC and anterior insula/inferior frontal cortex beta power, and medial parietal cortex and posterior cingulate cortex gamma activity after cued versus novelty-triggered orienting (600-1400 msec). Our results reveal sustained oscillatory patterns associated with voluntary engagement of auditory spatial attention, with the frontoparietal and temporal gamma increases being best predictors of subsequent behavioral performance.

Signals in inferotemporal and perirhinal cortex suggest an untangling of visual target information

Finding sought visual targets requires our brains to flexibly combine working memory information about what we are looking for with visual information about what we are looking at. To investigate the neural computations involved in finding visual targets, we recorded neural responses in inferotemporal (IT) and perirhinal (PRH) cortex as macaque monkeys performed a task that required them to find targets within sequences of distractors. We found similar amounts of total task-specific information in both areas, however, information about whether a target was in view was more accessible using a linear read-out (i.e. was more “untangled”) in PRH. Consistent with the flow of information from IT to PRH, we also found that task-relevant information arrived earlier in IT. PRH responses were well-described by a functional model in which “untangling” computations in PRH reformat input from IT by combining neurons with asymmetric tuning correlations for target matches and distractors.

The development of organized visual search

Visual search plays an important role in guiding behavior. Children have more difficulty performing conjunction search tasks than adults. The present research evaluates whether developmental differences in children's ability to organize serial visual search (i.e., search organization skills) contribute to performance limitations in a typical conjunction search task. We evaluated 134 children between the ages of 2 and 17 on separate tasks measuring search for targets defined by a conjunction of features or by distinct features. Our results demonstrated that children organize their visual search better as they get older. As children's skills at organizing visual search improve they become more accurate at locating targets with conjunction of features amongst distractors, but not for targets with distinct features. Developmental limitations in children's abilities to organize their visual search of the environment are an important component of poor conjunction search in young children. In addition, our findings provide preliminary evidence that, like other visuospatial tasks, exposure to reading may influence children's spatial orientation to the visual environment when performing a visual search.

Spatial and temporal scales of neuronal correlation in visual area V4

The spiking activity of nearby cortical neurons is correlated on both short and long time scales. Understanding this shared variability in firing patterns is critical for appreciating the representation of sensory stimuli in ensembles of neurons, the coincident influences of neurons on common targets, and the functional implications of microcircuitry. Our knowledge about neuronal correlations, however, derives largely from experiments that used different recording methods, analysis techniques, and cortical regions. Here we studied the structure of neuronal correlation in area V4 of alert macaques using recording and analysis procedures designed to match those used previously in primary visual cortex (V1), the major input to V4. We found that the spatial and temporal properties of correlations in V4 were remarkably similar to those of V1, with two notable differences: correlated variability in V4 was approximately one-third the magnitude of that in V1 and synchrony in V4 was less temporally precise than in V1. In both areas, spontaneous activity (measured during fixation while viewing a blank screen) was approximately twice as correlated as visual-evoked activity. The results provide a foundation for understanding how the structure of neuronal correlation differs among brain regions and stages in cortical processing and suggest that it is likely governed by features of neuronal circuits that are shared across the visual cortex.

Individual differences in attention strategies during detection, fine discrimination, and coarse discrimination

Interacting with the environment requires the ability to flexibly direct attention to relevant features. We examined the degree to which individuals attend to visual features within and across Detection, Fine Discrimination, and Coarse Discrimination tasks. Electroencephalographic (EEG) responses were measured to an unattended peripheral flickering (4 or 6 Hz) grating while individuals (n = 33) attended to orientations that were offset by 0°, 10°, 20°, 30°, 40°, and 90° from the orientation of the unattended flicker. These unattended responses may be sensitive to attentional gain at the attended spatial location, since attention to features enhances early visual responses throughout the visual field. We found no significant differences in tuning curves across the three tasks in part due to individual differences in strategies. We sought to characterize individual attention strategies using hierarchical Bayesian modeling, which grouped individuals into families of curves that reflect attention to the physical target orientation ("on-channel") or away from the target orientation ("off-channel") or a uniform distribution of attention. The different curves were related to behavioral performance; individuals with "on-channel" curves had lower thresholds than individuals with uniform curves. Individuals with "off-channel" curves during Fine Discrimination additionally had lower thresholds than those assigned to uniform curves, highlighting the perceptual benefits of attending away from the physical target orientation during fine discriminations. Finally, we showed that a subset of individuals with optimal curves ("on-channel") during Detection also demonstrated optimal curves ("off-channel") during Fine Discrimination, indicating that a subset of individuals can modulate tuning optimally for detection and discrimination.

Top-down modulation of lateral interactions in visual cortex

The primary visual cortex (V1) changes its computation according to the perceptual task being performed. We propose that this cognitive modulation results from gating of V1 intrinsic connections. To test this idea, using behavioral paradigms that engage top-down modulation of V1 contextual interactions, we recorded from chronically implanted electrode arrays in macaques. We observed task-dependent changes in interactions between V1 sites measured both by correlation between spike trains and by coherence between local field potentials (LFP-LFP coherence). The direction of the changes in aggregate activity, as measured by LFPs, depended on perceptual strategy: perceptual grouping increased LFP coherence between sites crucial for the task, whereas perceptual segregation lowered the LFP coherence. Using spiking activity as a measure, we found that the behaviorally driven changes in correlation structure between neurons dramatically increased the stimulus-related information that they convey; this additional increase in encoded information at the level of neuronal ensembles equals that obtained from task-driven reconfigurations of neural tuning curves. The improvements in information encoding were strongest for stimuli with greatest discrimination difficulty.