Analysis of spiking synchrony in visual cortex reveals distinct types of top-down modulation signals for spatial and object-based attention

The activity of a border ownership selective (BOS) neuron indicates where a foreground object is located relative to its (classical) receptive field (RF). A population of BOS neurons thus provides an important component of perceptual grouping, the organization of the visual scene into objects. In previous theoretical work, it has been suggested that this grouping mechanism is implemented by a population of dedicated grouping (“G”) cells that integrate the activity of the distributed feature cells representing an object and, by feedback, modulate the same cells, thus making them border ownership selective. The feedback modulation by G cells is thought to also provide the mechanism for object-based attention. A recent modeling study showed that modulatory common feedback, implemented by synapses with N-methyl-D-aspartate (NMDA)-type glutamate receptors, accounts for the experimentally observed synchrony in spike trains of BOS neurons and the shape of cross-correlations between them, including its dependence on the attentional state. However, that study was limited to pairs of BOS neurons with consistent border ownership preferences, defined as two neurons tuned to respond to the same visual object, in which attention decreases synchrony. But attention has also been shown to increase synchrony in neurons with inconsistent border ownership selectivity. Here we extend the computational model from the previous study to fully understand these effects of attention. We postulate the existence of a second type of G-cell that represents spatial attention by modulating the activity of all BOS cells in a spatially defined area. Simulations of this model show that a combination of spatial and object-based mechanisms fully accounts for the observed pattern of synchrony between BOS neurons. Our results suggest that modulatory feedback from G-cells may underlie both spatial and object-based attention.

Vision allows us to make sense out of a very complex signal, the patterns of light rays reaching our eyes. Two mechanisms are essential for this: perceptual organization which structures the input into meaningful visual objects, and attention which selects only the most important parts in the input. Prior work suggests that both of these mechanisms are implemented by neurons called grouping cells. These organize the object features into coherent entities (perceptual grouping) and access them as needed (selective attention). For technical reasons it is difficult to observe grouping cells but their effect can be seen in the influence they have on responses of other classes of cells. These responses have been measured experimentally and it was found that they depend in unexpected ways on where the subject was attending. Using a computational model, we here demonstrate that the responses can be understood in terms of the interaction between two kinds of selective attention, both of which are known to occur in primate perception. One is attention to a specific area in the environment, the other is to specific objects. A model including both of these attentional mechanisms generates neuronal responses in agreement with the observed patterns of neural activity.

Searching for object pointers in the visual cortex

We perceive objects as permanent and stable despite frequent occlusions and eye movements, but their representation in the visual cortex is neither permanent nor stable. Feature selective cells respond only as long as objects are visible, and their responses depend on eye position. We explored the hypothesis that the system maintains object pointers that provide permanence and stability. Pointers should send facilitatory signals to the feature cells of an object, and these signals should persist across temporary occlusions and remap to compensate for image displacements caused by saccades. Here, we searched for such signals in monkey areas V2 and V4 (Macaca mulatta). We developed a new paradigm in which a monkey freely inspects an array of objects in search for reward while some of the objects are being occluded temporarily by opaque drifting strips. Two types of objects were used to manipulate attention. The results were as follows. 1) Eye movements indicated a robust representation of location and type of the occluded objects; 2) in neurons of V4, but not V2, occluded objects produced elevated activity relative to blank condition; 3) the elevation of activity was reduced for objects that had been fixated immediately before the current fixation ('inhibition of return'); and 4) when attended, or when the target of a saccade, visible objects produced enhanced responses in V4, but occluded objects produced no modulation. Although results 1-3 confirm the hypothesis, the absence of modulation under occlusion is not consistent. Further experiments are needed to resolve this discrepancy.NEW & NOTEWORTHY The way we perceive objects as permanent contrasts with the short-lived responses of visual cortical neurons. A theory postulates pointers that give objects continuity, predicting a class of neurons that respond not only to visual objects but also when an occluded object moves into their receptive field. Here, we tested this theory with a novel paradigm in which a monkey freely scans an array of objects while some of them are transiently occluded.

Figure-Ground Organization in Natural Scenes: Performance of a Recurrent Neural Model Compared with Neurons of Area V2

A crucial step in understanding visual input is its organization into meaningful components, in particular object contours and partially occluded background structures. This requires that all contours are assigned to either the foreground or the background (border ownership assignment). While earlier studies showed that neurons in primate extrastriate cortex signal border ownership for simple geometric shapes, recent studies show consistent border ownership coding also for complex natural scenes. In order to understand how the brain performs this task, we developed a biologically plausible recurrent neural network that is fully image computable. Our model uses local edge detector ( B ) cells and grouping ( G ) cells whose activity represents proto-objects based on the integration of local feature information. G cells send modulatory feedback connections to those B cells that caused their activation, making the B cells border ownership selective. We found close agreement between our model and neurophysiological results in terms of the timing of border ownership signals (BOSs) as well as the consistency of BOSs across scenes. We also benchmarked our model on the Berkeley Segmentation Dataset and achieved performance comparable to recent state-of-the-art computer vision approaches. Our proposed model provides insight into the cortical mechanisms of figure-ground organization.

Figure and ground: how the visual cortex integrates local cues for global organization

Inferring figure-ground organization in two-dimensional images may require different complementary strategies. For isolated objects, it has been shown that mechanisms in visual cortex exploit the overall distribution of contours, but in images of cluttered scenes where the grouping of contours is not obvious, that strategy would fail. However, natural scenes contain local features, specifically contour junctions, that may contribute to the definition of object regions. To study the role of local features in the assignment of border ownership, we recorded single-cell activity from visual cortex in awake behaving Macaca mulatta. We tested configurations perceived as two overlapping figures in which T- and L-junctions depend on the direction of overlap, whereas the overall distribution of contours provides no valid information. While recording responses to the occluding contour, we varied direction of overlap and variably masked some of the critical contour features to determine their influences and their interactions. On average, most features influenced the responses consistently, producing either enhancement or suppression depending on border ownership. Different feature types could have opposite effects even at the same location. Features far from the receptive field produced effects as strong as near features and with the same short latency. Summation was highly nonlinear: any single feature produced more than two-thirds of the effect of all features together. These findings reveal fast and highly specific organization mechanisms, supporting a previously proposed model in which "grouping cells" integrate widely distributed edge signals with specific end-stopped signals to modulate the original edge signals by feedback. NEW & NOTEWORTHY Seeing objects seems effortless, but defining objects in a scene requires sophisticated neural mechanisms. For isolated objects, the visual cortex groups contours based on overall distribution, but this strategy does not work for cluttered scenes. Here, we demonstrate mechanisms that integrate local contour features like T- and L-junctions to resolve clutter. The process is fast, evaluates widely distributed features, and gives any single feature a decisive influence on figure-ground representation.

Short-term depression and transient memory in sensory cortex

Persistent neuronal activity is usually studied in the context of short-term memory localized in central cortical areas. Recent studies show that early sensory areas also can have persistent representations of stimuli which emerge quickly (over tens of milliseconds) and decay slowly (over seconds). Traditional positive feedback models cannot explain sensory persistence for at least two reasons: (i) They show attractor dynamics, with transient perturbations resulting in a quasi-permanent change of system state, whereas sensory systems return to the original state after a transient. (ii) As we show, those positive feedback models which decay to baseline lose their persistence when their recurrent connections are subject to short-term depression, a common property of excitatory connections in early sensory areas. Dual time constant network behavior has also been implemented by nonlinear afferents producing a large transient input followed by much smaller steady state input. We show that such networks require unphysiologically large onset transients to produce the rise and decay observed in sensory areas. Our study explores how memory and persistence can be implemented in another model class, derivative feedback networks. We show that these networks can operate with two vastly different time courses, changing their state quickly when new information is coming in but retaining it for a long time, and that these capabilities are robust to short-term depression. Specifically, derivative feedback networks with short-term depression that acts differentially on positive and negative feedback projections are capable of dynamically changing their time constant, thus allowing fast onset and slow decay of responses without requiring unrealistically large input transients.

Figure-ground organization in the visual cortex: does meaning matter?

Figure-ground organization in the visual cortex is generally assumed to be based partly on general rules and partly on specific influences of object recognition in higher centers as found in the temporal lobe. To see if shape familiarity influences figure-ground organization, we tested border ownership-selective neurons in monkey V1/V2 with silhouettes of human and monkey face profiles and "nonsense" silhouettes constructed by mirror-reversing the front part of the profile. We found no superiority of face silhouettes compared with nonsense shapes in eliciting border-ownership signals overall. However, in some neurons, border-ownership signals differed strongly between the two categories consistently across many different profile shapes. Surprisingly, this category selectivity appeared as early as 70 ms after stimulus onset, which is earlier than the typical latency of shape-selective responses but compatible with the earliest face-selective responses in the inferior temporal lobe. Although our results provide no evidence for a delayed top-down influence from object recognition centers, they indicate sophisticated shape categorization mechanisms that are much faster than generally assumed. NEW & NOTEWORTHY A long-standing question is whether low-level sensory representations in cortex are influenced by cognitive "top-down" signals. We studied figure-ground organization in the visual cortex by comparing border-ownership signals for face profiles and matched nonsense shapes. We found no sign of "face superiority" in the population border-ownership signal. However, some neurons consistently differentiated between the face and nonsense categories early on, indicating the presence of shape classification mechanisms that are much faster than previously assumed.

Figure-Ground Organization in Visual Cortex for Natural Scenes

Figure-ground organization and border-ownership assignment are essential for understanding natural scenes. It has been shown that many neurons in the macaque visual cortex signal border-ownership in displays of simple geometric shapes such as squares, but how well these neurons resolve border-ownership in natural scenes is not known. We studied area V2 neurons in behaving macaques with static images of complex natural scenes. We found that about half of the neurons were border-ownership selective for contours in natural scenes, and this selectivity originated from the image context. The border-ownership signals emerged within 70 ms after stimulus onset, only ∼30 ms after response onset. A substantial fraction of neurons were highly consistent across scenes. Thus, the cortical mechanisms of figure-ground organization are fast and efficient even in images of complex natural scenes. Understanding how the brain performs this task so fast remains a challenge.

Spike synchrony generated by modulatory common input through NMDA-type synapses

Common excitatory input to neurons increases their firing rates and the strength of the spike correlation (synchrony) between them. Little is known, however, about the synchronizing effects of modulatory common input. Here, we show that modulatory common input with the slow synaptic kinetics of N-methyl-d-aspartate (NMDA) receptors enhances firing rates and also produces synchrony. Tight synchrony (correlations on the order of milliseconds) always increases with modulatory strength. Unexpectedly, the relationship between strength of modulation and strength of loose synchrony (tens of milliseconds) is not monotonic: The strongest loose synchrony is obtained for intermediate modulatory amplitudes. This finding explains recent neurophysiological results showing that in cortical areas V1 and V2, presumed modulatory top-down input due to contour grouping increases (loose and tight) synchrony but that additional modulatory input due to top-down attention does not change tight synchrony and actually decreases loose synchrony. These neurophysiological findings are understood from our model of integrate-and-fire neurons under the assumption that contour grouping as well as attention lead to additive modulatory common input through NMDA-type synapses. In contrast, circuits with common projections through model α-amino-3-hydroxy-5-methyl-4-isoxazolepropionic acid receptors did not exhibit the paradoxical decrease of synchrony with increased input. Our results suggest that NMDA receptors play a critical role in top-down response modulation in the visual cortex.

Figure-ground organization and the emergence of proto-objects in the visual cortex

A long history of studies of perception has shown that the visual system organizes the incoming information early on, interpreting the 2D image in terms of a 3D world and producing a structure that provides perceptual continuity and enables object-based attention. Recordings from monkey visual cortex show that many neurons, especially in area V2, are selective for border ownership. These neurons are edge selective and have ordinary classical receptive fields (CRF), but in addition their responses are modulated (enhanced or suppressed) depending on the location of a 'figure' relative to the edge in their receptive field. Each neuron has a fixed preference for location on one side or the other. This selectivity is derived from the image context far beyond the CRF. This paper reviews evidence indicating that border ownership selectivity reflects the formation of early object representations ('proto-objects'). The evidence includes experiments showing (1) reversal of border ownership signals with change of perceived object structure, (2) border ownership specific enhancement of responses in object-based selective attention, (3) persistence of border ownership signals in accordance with continuity of object perception, and (4) remapping of border ownership signals across saccades and object movements. Findings 1 and 2 can be explained by hypothetical grouping circuits that sum contour feature signals in search of objectness, and, via recurrent projections, enhance the corresponding low-level feature signals. Findings 3 and 4 might be explained by assuming that the activity of grouping circuits persists and can be remapped. Grouping, persistence, and remapping are fundamental operations of vision. Finding these operations manifest in low-level visual areas challenges traditional views of visual processing. New computational models need to be developed for a comprehensive understanding of the function of the visual cortex.

Neurophysiological constraints on models of illusory contours

Illusory contours can appear as interpolation between edges of the stimulus, as in the Kanizsa triangle, or run orthogonal to the inducing elements, as in the Ehrenstein illusion. Single-cell recordings from monkey visual cortex suggest that both are produced by the same mechanism. Neural border ownership coding, on the other hand, which shows a much larger range of context integration, might involve a different mechanism.

Remapping of border ownership in the visual cortex

We see objects as having continuity although the retinal image changes frequently. How such continuity is achieved is hard to understand, because neurons in the visual cortex have small receptive fields that are fixed on the retina, which means that a different set of neurons is activated every time the eyes move. Neurons in areas V1 and V2 of the visual cortex signal the local features that are currently in their receptive fields and do not show "remapping" when the image moves. However, subsets of neurons in these areas also carry information about global aspects, such as figure-ground organization. Here we performed experiments to find out whether figure-ground organization is remapped. We recorded single neurons in macaque V1 and V2 in which figure-ground organization is represented by assignment of contours to regions (border ownership). We found previously that border-ownership signals persist when a figure edge is switched to an ambiguous edge by removing the context. We now used this paradigm to see whether border ownership transfers when the ambiguous edge is moved across the retina. In the new position, the edge activated a different set of neurons at a different location in cortex. We found that border ownership was transferred to the newly activated neurons. The transfer occurred whether the edge was moved by a saccade or by moving the visual display. Thus, although the contours are coded in retinal coordinates, their assignment to objects is maintained across movements of the retinal image.

A century of Gestalt psychology in visual perception: I. Perceptual grouping and figure-ground organization

In 1912, Max Wertheimer published his paper on phi motion, widely recognized as the start of Gestalt psychology. Because of its continued relevance in modern psychology, this centennial anniversary is an excellent opportunity to take stock of what Gestalt psychology has offered and how it has changed since its inception. We first introduce the key findings and ideas in the Berlin school of Gestalt psychology, and then briefly sketch its development, rise, and fall. Next, we discuss its empirical and conceptual problems, and indicate how they are addressed in contemporary research on perceptual grouping and figure-ground organization. In particular, we review the principles of grouping, both classical (e.g., proximity, similarity, common fate, good continuation, closure, symmetry, parallelism) and new (e.g., synchrony, common region, element and uniform connectedness), and their role in contour integration and completion. We then review classic and new image-based principles of figure-ground organization, how it is influenced by past experience and attention, and how it relates to shape and depth perception. After an integrated review of the neural mechanisms involved in contour grouping, border ownership, and figure-ground perception, we conclude by evaluating what modern vision science has offered compared to traditional Gestalt psychology, whether we can speak of a Gestalt revival, and where the remaining limitations and challenges lie. A better integration of this research tradition with the rest of vision science requires further progress regarding the conceptual and theoretical foundations of the Gestalt approach, which is the focus of a second review article.

The speed of context integration in the visual cortex

The observation of figure-ground selectivity in neurons of the visual cortex shows that these neurons can be influenced by the image context far beyond the classical receptive field. To clarify the nature of the context integration mechanism, we studied the latencies of neural edge signals, comparing the emergence of context-dependent definition of border ownership with the onset of local edge definition (contrast polarity; stereoscopic depth order). Single-neuron activity was recorded in areas V1 and V2 of Macaca mulatta under behaviorally induced fixation. Whereas local edge definition emerged immediately (<13 ms) after the edge onset response, the context-dependent signal was delayed by about 30 ms. To see if the context influence was mediated by horizontal fibers within cortex, we measured the latencies of border ownership signals for two conditions in which the relevant context information was located at different distances from the receptive field and compared the latency difference with the difference predicted from horizontal signal propagation. The prediction was based on the increase in cortical distance, computed from the mapping of the test stimuli in the cortex, and the known conduction velocities of horizontal fibers. The measured latencies increased with cortical distance, but much less than predicted by the horizontal propagation hypothesis. Probability calculations showed that an explanation of the context influence by horizontal signal propagation alone is highly unlikely, whereas mechanisms involving back projections from other extrastriate areas are plausible.

Representation of object continuity in the visual cortex

An amazing feature of our visual system is the ability to detect and track objects in the stream of continually changing retinal images. Theories have proposed that the system creates temporary internal representations that persist across changing images, providing continuity. However, how such representations are formed in the brain is not known. Here we examined the time course of the responses of border-ownership-selective neurons in the visual cortex to displays that portray object continuity. We found that the neurons signal border ownership immediately when new objects appear, but when a border that has been assigned to one object is reassigned to another object while the first remains in the display, the initial responses persist. The neurons continue to signal the initial assignment despite the presence of contradicting figure--ground cues. We propose that border ownership selectivity reflects mechanisms that create object continuity.

Short-term memory for figure-ground organization in the visual cortex

Whether the visual system uses a buffer to store image information and the duration of that storage have been debated intensely in recent psychophysical studies. The long phases of stable perception of reversible figures suggest a memory that persists for seconds. But persistence of similar duration has not been found in signals of the visual cortex. Here, we show that figure-ground signals in the visual cortex can persist for a second or more after the removal of the figure-ground cues. When new figure-ground information is presented, the signals adjust rapidly, but when a figure display is changed to an ambiguous edge display, the signals decay slowly--a behavior that is characteristic of memory devices. Figure-ground signals represent the layout of objects in a scene, and we propose that a short-term memory for object layout is important in providing continuity of perception in the rapid stream of images flooding our eyes.

Synchrony and the binding problem in macaque visual cortex

We tested the binding-by-synchrony hypothesis which proposes that object representations are formed by synchronizing spike activity between neurons that code features of the same object. We studied responses of 32 pairs of neurons recorded with microelectrodes 3 mm apart in the visual cortex of macaques performing a fixation task. Upon mapping the receptive fields of the neurons, a quadrilateral was generated so that two of its sides were centered in the receptive fields at the optimal orientations. This one-figure condition was compared with a two-figure condition in which the neurons were stimulated by two separate figures, keeping the local edges in the receptive fields identical. For each neuron, we also determined its border ownership selectivity (H. Zhou, H. S. Friedman, & R. von der Heydt, 2000). We examined both synchronization and correlation at nonzero time lag. After correcting for effects of the firing rate, we found that synchrony did not depend on the binding condition. However, finding synchrony in a pair of neurons was correlated with finding border-ownership selectivity in both members of the pair. This suggests that the synchrony reflected the connectivity in the network that generates border ownership assignment. Thus, we have not found evidence to support the binding-by-synchrony hypothesis.

Figure-ground mechanisms provide structure for selective attention

Attention depends on figure-ground organization: figures draw attention, whereas shapes of the ground tend to be ignored. Recent research has revealed mechanisms for figure-ground organization in the visual cortex, but how these mechanisms relate to the attention process remains unclear. Here we show that the influences of figure-ground organization and volitional (top-down) attention converge in single neurons of area V2 in Macaca mulatta. Although we found assignment of border ownership for attended and for ignored figures, attentional modulation was stronger when the attended figure was located on the neuron's preferred side of border ownership. When the border between two overlapping figures was placed in the receptive field, responses depended on the side of attention, and enhancement was generally found on the neuron's preferred side of border ownership. This correlation suggests that the neural network that creates figure-ground organization also provides the interface for the top-down selection process.

A neural model of figure-ground organization

Psychophysical studies suggest that figure-ground organization is a largely autonomous process that guides--and thus precedes--allocation of attention and object recognition. The discovery of border-ownership representation in single neurons of early visual cortex has confirmed this view. Recent theoretical studies have demonstrated that border-ownership assignment can be modeled as a process of self-organization by lateral interactions within V2 cortex. However, the mechanism proposed relies on propagation of signals through horizontal fibers, which would result in increasing delays of the border-ownership signal with increasing size of the visual stimulus, in contradiction with experimental findings. It also remains unclear how the resulting border-ownership representation would interact with attention mechanisms to guide further processing. Here we present a model of border-ownership coding based on dedicated neural circuits for contour grouping that produce border-ownership assignment and also provide handles for mechanisms of selective attention. The results are consistent with neurophysiological and psychophysical findings. The model makes predictions about the hypothetical grouping circuits and the role of feedback between cortical areas.

Neural representation of transparent overlay

Perceptual transparency is a surprising phenomenon in which a number of regions of different shades organize into overlaying transparent objects. We recorded single neuron responses from Macaca mulatta area V2 to a display of two bright and two dark squares that appeared as two overlaying bars. We found that neurons assign border ownership according to the transparent interpretation, representing the shapes of the bars rather than the squares.

Border-ownership-dependent tilt aftereffect

Single-cell recordings from macaque visual cortex have shown orientation-selective neurons in area in V2 code for border ownership [J. Neurosci. 20, 6594 (2000)]: Each piece of contrast border is represented by two pools of neurons whose relative firing rate indicates the side of border ownership. Here we show that the human visual cortex uses a similar coding scheme by demonstrating a border-ownership-contingent tilt aftereffect. The aftereffect was specific for the adapted location, indicating that the adapted neurons have small receptive fields. We conclude that figure-ground organization is represented by border-ownership-selective neurons at early stages in the human visual cortex.

Figure and ground in the visual cortex: v2 combines stereoscopic cues with gestalt rules

Figure-ground organization is a process by which the visual system identifies some image regions as foreground and others as background, inferring 3D layout from 2D displays. A recent study reported that edge responses of neurons in area V2 are selective for side-of-figure, suggesting that figure-ground organization is encoded in the contour signals (border ownership coding). Here, we show that area V2 combines two strategies of computation, one that exploits binocular stereoscopic information for the definition of local depth order, and another that exploits the global configuration of contours (Gestalt factors). These are combined in single neurons so that the "near" side of the preferred 3D edge generally coincides with the preferred side-of-figure in 2D displays. Thus, area V2 represents the borders of 2D figures as edges of surfaces, as if the figures were objects in 3D space. Even in 3D displays, Gestalt factors influence the responses and can enhance or null the stereoscopic depth information.

The coding of uniform colour figures in monkey visual cortex

Psychophysical studies indicate that perception of the colour and brightness of a surface depends on neural signals evoked by the borders of the surface rather than its interior. The visual cortex emphasizes contrast borders, but it is unclear whether colour surface signals also exist, whether colour border signals are orientation selective or mainly non-oriented, and whether cortical processing tends to separate colour and form information. To address these questions we examined the representation of uniform colour figures by recording single neuron activity from areas V1 and V2 in alert macaque monkeys during behaviourally induced fixation. Three aspects of coding were quantified: colour, orientation and edge selectivity. The occurrence of colour selectivity was not correlated with orientation or edge selectivity. The fraction of colour-selective cells was the same (64 % in layers 2 and 3 of V1, 45 % in V2) for oriented and non-oriented cells, and for edge-selective and surface-responsive cells. Oriented cells were often highly selective in colour space, and about 40 % of them were selective for edge polarity or border ownership. Thus, contrary to the idea of feature maps, colour, orientation and edge polarity are multiplexed in cortical signals. The results from V2 were similar to those from upper-layer V1, indicating that cortical processing does not strive to separate form and colour information. Oriented cells were five times more frequent than non-oriented cells. Thus, the vast majority of colour-coded cells are orientation tuned. Based on response profiles across a 4 deg square figure, and the relative frequency of oriented and non-oriented cells, we estimate that the cortical colour signal is 5-6 times stronger for the edges than for the surface of the figure. The frequency of oriented colour cells and their ability to code edge polarity indicate that these cells play a major role in the representation of surface colour.