Visual curve tracing properties

Recurrent neural networks that learn multi-step visual routines with reinforcement learning

Many cognitive problems can be decomposed into series of subproblems that are solved sequentially by the brain. When subproblems are solved, relevant intermediate results need to be stored by neurons and propagated to the next subproblem, until the overarching goal has been completed. We will here consider visual tasks, which can be decomposed into sequences of elemental visual operations. Experimental evidence suggests that intermediate results of the elemental operations are stored in working memory as an enhancement of neural activity in the visual cortex. The focus of enhanced activity is then available for subsequent operations to act upon. The main question at stake is how the elemental operations and their sequencing can emerge in neural networks that are trained with only rewards, in a reinforcement learning setting. We here propose a new recurrent neural network architecture that can learn composite visual tasks that require the application of successive elemental operations. Specifically, we selected three tasks for which electrophysiological recordings of monkeys' visual cortex are available. To train the networks, we used RELEARNN, a biologically plausible four-factor Hebbian learning rule, which is local both in time and space. We report that networks learn elemental operations, such as contour grouping and visual search, and execute sequences of operations, solely based on the characteristics of the visual stimuli and the reward structure of a task. After training was completed, the activity of the units of the neural network elicited by behaviorally relevant image items was stronger than that elicited by irrelevant ones, just as has been observed in the visual cortex of monkeys solving the same tasks. Relevant information that needed to be exchanged between subroutines was maintained as a focus of enhanced activity and passed on to the subsequent subroutines. Our results demonstrate how a biologically plausible learning rule can train a recurrent neural network on multistep visual tasks.

A region complexity effect masquerading as object-based attention

A large portion of the evidence for object-based attention comes from experiments using the two-rectangle paradigm introduced by Egly, Driver, and Rafal (1994), in which response times are longer when the two stimulus locations relevant to the task are on separate objects. In the new experiments presented here, response times are longer when the two locations are part of the same object but are separated by a concavity in the object, so that the region directly between the two locations is crossed by the object's boundaries. Response times when the two locations are separated by the concavity are not statistically different from when they are on two separate objects. The results are similar for a two-letter comparison task and for a spatial cuing task. Thus, in these experiments, the response time increase does not reflect the cost of shifting attention from object to object, because it appears when the two locations are on the same object, and it is not increased when they are on different objects. Instead, it seems to reflect the complexity of the region between the two stimulus locations. This finding raises questions about whether data from previous two-rectangle experiments should be attributed to object-based attention.

Testing between the gunpowder fuse and the filling-hose analogy for mental curve tracing using electroencephalography: boom!

Curve tracing occurs when a line is followed covertly to accomplish a task, for example, to determine whether two landmarks are on the same line or not (could Highway 61 take Robert Johnson from New Orleans to St Louis?). Previous work suggests that attention either moves along the curve, momentarily activating local representations of the curve during this process, leaving little or no trace of this activation once attention has passed, or attention spreads along the curve, resulting in an activated state along the entire portion of the curve that was traced. We re-examined this issue using event-related potentials. Curves to be traced were presented briefly to encourage a rapid deployment of attention. The curves started on the vertical midline and passed into the left or right visual field and terminated either on the vertical midline or at a lateral position. We measured a posterior contralateral negativity (relative to the visual field of the traced curve) that offset more rapidly when the curve was traced back to the midline than when it remained lateral. The results suggest that attention travels along the curve like fire on a fuse, with activation returning to baseline once the flame has passed.

Object-based attention is mediated by collinearity of targets

Same-object bias occurs when tasks associated with processing a single object are faster than tasks associated with two objects. Over five experiments we assessed whether same-object bias is mediated by the collinearity of the targets. Participants decided whether two targets, presented either within a single object or across two objects, were the same or different. Results showed that same-object bias only occurred when targets appeared on the same straight line within the same object. When targets appeared in the same object but were separated by an angle or corner, within-object facilitation was eliminated or greatly reduced. In a final experiment, response times to two targets that were collinear but on separate objects were responded to faster than were noncollinear targets on the same object. This suggests that collinearity between targets mediates the effect found in this paradigm, at least to a greater extent than colour grouping.

Layer-specificity in the effects of attention and working memory on activity in primary visual cortex

Neuronal activity in early visual cortex depends on attention shifts but the contribution to working memory has remained unclear. Here, we examine neuronal activity in the different layers of the primary visual cortex (V1) in an attention-demanding and a working memory task. A current-source density analysis reveales top-down inputs in the superficial layers and layer 5, and an increase in neuronal firing rates most pronounced in the superficial and deep layers and weaker in input layer 4. This increased activity is strongest in the attention task but it is also highly reliable during working memory delays. A visual mask erases the V1 memory activity, but it reappeares at a later point in time. These results provide new insights in the laminar circuits involved in the top-down modulation of activity in early visual cortex in the presence and absence of visual stimuli.

Serial grouping of 2D-image regions with object-based attention in humans

After an initial stage of local analysis within the retina and early visual pathways, the human visual system creates a structured representation of the visual scene by co-selecting image elements that are part of behaviorally relevant objects. The mechanisms underlying this perceptual organization process are only partially understood. We here investigate the time-course of perceptual grouping of two-dimensional image-regions by measuring the reaction times of human participants and report that it is associated with the gradual spread of object-based attention. Attention spreads fastest over large and homogeneous areas and is slowed down at locations that require small-scale processing. We find that the time-course of the object-based selection process is well explained by a 'growth-cone' model, which selects surface elements in an incremental, scale-dependent manner. We discuss how the visual cortical hierarchy can implement this scale-dependent spread of object-based attention, leveraging the different receptive field sizes in distinct cortical areas.

DOI:http://dx.doi.org/10.7554/eLife.14320.001

When we look at an object, we perceive it as a whole. However, this is not how the brain processes objects. Instead, cells at early stages of the visual system respond selectively to single features of the object, such as edges. Moreover, each cell responds to its target feature in only a small region of space known as its receptive field. At higher levels of the visual system, cells respond to more complex features: angles rather than edges, for example. The receptive fields of the cells are also larger. For us to see an object, the brain must therefore 'stitch' together diverse features into a unified impression.

This process is termed perceptual grouping. But how does it work? Jeurissen et al. hypothesized that this process depends on the visual system’s attention spreading over a region in the image occupied by an object, and that the speed of the process will depend on the size of the receptive fields involved. If an image region is narrow, the visual system must recruit cells with small receptive fields to process the individual features. Grouping will therefore be slow. By contrast, if the object consists of large uniform areas lacking in detail, grouping should be fast. These assumptions give rise to a model called the “growth-conemodel”, which makes a number of specific predictions about reaction times during perceptual grouping.

Jeurissen et al. tested the growth-cone model’s predictions by measuring the speed of perceptual grouping in 160 human volunteers. These volunteers looked at an image made up of two simple shapes, and reported whether two dots fell on the same or different shapes. The results supported the growth-cone model. People were able to group large and uniform areas quickly, but were slower for narrow areas. Grouping also took more time when the distance between the dots increased. Hence, perceptual grouping of everyday objects calls on a step-by-step process that resembles solving a small maze.

The results also revealed that perceptual grouping of simple shapes relies on the spreading of visual attention over the relevant object. Furthermore, the data support the hypothesis that perceptual grouping makes use of the different sizes of receptive fields at various levels of the visual system. Further research will be needed to translate these findings to the more complex natural scenes we encounter in our daily lives.

DOI:http://dx.doi.org/10.7554/eLife.14320.002

A growth-cone model for the spread of object-based attention during contour grouping

BACKGROUND

Object-based attention can group image elements of spatially extended objects into coherent representations, but its mechanisms have remained unclear. The mechanisms for object-based attention may include shape-selective neurons in higher visual cortical areas that feed back to lower areas to simultaneously enhance the representation of all image elements of a relevant shape. Another, nonexclusive mechanism is the spread of attention in early visual cortex according to Gestalt rules, which could successively add new elements to a growing object representation.

RESULTS

We investigated the dynamics of object-based attention in the primary visual cortex of monkeys trained to perform a contour-grouping task. The animals mentally traced a target curve through the visual field and ignored a distracting curve. Neuronal responses elicited by the target curve were enhanced relative to those elicited by distracting curve. Remarkably, the response enhancement was delayed for neurons with receptive fields farther from the start of the tracing process. We could therefore measure propagation speed and found that it was low if curves were nearby and that it increased if curves were far apart. The results are well explained by an "attentional growth-cone" model, which holds that the response enhancement can spread in multiple visual cortical areas with different receptive field sizes at a speed of approximately 50 ms per receptive field.

CONCLUSIONS

Our findings support an active role for early visual areas in object-based attention because neurons in these areas gradually spread enhanced activity over the representation of relevant objects.

The time course of perceptual grouping in natural scenes

Visual perception starts with localized filters that subdivide the image into fragments that undergo separate analyses. The visual system has to reconstruct objects by grouping image fragments that belong to the same object. A widely held view is that perceptual grouping occurs in parallel across the visual scene and without attention. To test this idea, we measured the speed of grouping in pictures of animals and vehicles. In a classification task, these pictures were categorized efficiently. In an image-parsing task, participants reported whether two cues fell on the same or different objects, and we measured reaction times. Despite the participants' fast object classification, perceptual grouping required more time if the distance between cues was larger, and we observed an additional delay when the cues fell on different parts of a single object. Parsing was also slower for inverted than for upright objects. These results imply that perception starts with rapid object classification and that rapid classification is followed by a serial perceptual grouping phase, which is more efficient for objects in a familiar orientation than for objects in an unfamiliar orientation.

Incremental grouping of image elements in vision

One important task for the visual system is to group image elements that belong to an object and to segregate them from other objects and the background. We here present an incremental grouping theory (IGT) that addresses the role of object-based attention in perceptual grouping at a psychological level and, at the same time, outlines the mechanisms for grouping at the neurophysiological level. The IGT proposes that there are two processes for perceptual grouping. The first process is base grouping and relies on neurons that are tuned to feature conjunctions. Base grouping is fast and occurs in parallel across the visual scene, but not all possible feature conjunctions can be coded as base groupings. If there are no neurons tuned to the relevant feature conjunctions, a second process called incremental grouping comes into play. Incremental grouping is a time-consuming and capacity-limited process that requires the gradual spread of enhanced neuronal activity across the representation of an object in the visual cortex. The spread of enhanced neuronal activity corresponds to the labeling of image elements with object-based attention.

The impact of map orientation and generalisation on congestion decisions: a comparison of schematic-egocentric and topographic-allocentric maps

Map information for drivers is usually presented in an allocentric-topographic form (as with printed maps) or in an egocentric-schematic form (as with road signs). The advent of new variable message boards on UK motorways raises the possibility of presenting road maps to reflect congestion ahead. Should these maps be allocentric-topographic or egocentric-schematic? This was assessed in an eye tracking study, with participants viewing maps of a motorway network in order to identify whether any congestion was relevant to their intended route. The schematic-egocentric maps were responded to most accurately with shorter fixation durations suggesting easier processing. In particular, the driver's entrance and intended exit from the map were attended to more in the allocentric maps. Individual differences in mental rotation ability also seem to contribute to poor performance on allocentric maps. The results favour schematic-egocentric maps for roadside congestion information, but also provide theoretical insights into map-rotation and individual differences. Statement of Relevance: This study informs designers and policy makers about optimum representations of traffic congestion on roadside variable message signs and, furthermore, demonstrates that individual differences contribute to problems with processing certain sign types. Schematic-egocentric representations of a motorway network produced the best results, as noted in behavioural and eye movement measures.

Visual cognition

Visual cognition, high-level vision, mid-level vision and top-down processing all refer to decision-based scene analyses that combine prior knowledge with retinal input to generate representations. The label "visual cognition" is little used at present, but research and experiments on mid- and high-level, inference-based vision have flourished, becoming in the 21st century a significant, if often understated part, of current vision research. How does visual cognition work? What are its moving parts? This paper reviews the origins and architecture of visual cognition and briefly describes some work in the areas of routines, attention, surfaces, objects, and events (motion, causality, and agency). Most vision scientists avoid being too explicit when presenting concepts about visual cognition, having learned that explicit models invite easy criticism. What we see in the literature is ample evidence for visual cognition, but few or only cautious attempts to detail how it might work. This is the great unfinished business of vision research: at some point we will be done with characterizing how the visual system measures the world and we will have to return to the question of how vision constructs models of objects, surfaces, scenes, and events.

Formation of visual "objects" in the early computation of spatial relations

Perceptual grouping is the process by which elements in the visual image are aggregated into larger and more complex structures, i.e., "objects." This paper reports a study of the spatial factors and time-course of the development of objects over the course of the first few hundred milliseconds of visual processing. The methodology uses the now well-established idea of an "object benefit" for certain kinds of tasks (here, faster within-object than between-objects probe comparisons) to test what the visual system in fact treats as an object at each point during processing. The study tested line segment pairs in a wide variety of spatial configurations at a range of exposure times, in each case measuring the strength of perceptual grouping as reflected in the magnitude of the object benefit. Factors tested included nonaccidental properties such as collinearity, cotermination, and parallelism; contour relatability; Gestalt factors such as symmetry and skew symmetry, and several others, all tested at fine (25 msec) time-slices over the course of processing. The data provide detailed information about the comparative strength of these factors in inducing grouping at each point in processing. The result is a vivid picture of the chronology of object formation, as objects progressively coalesce, with fully bound visual objects completed by about 200 msec of processing.

Object and spatial representations in the corner enhancement effect

Cole, Gellatly, and Blurton have shown that targets presented adjacent to geometric corners are detected more efficiently than targets presented adjacent to straight edges. In six experiments, we examined how this corner enhancement effect is modulated by corner-of-object representations (i.e., corners that define an object's shape) and local base-level corners that occur as a result of, for instance, overlapping the straight edges of two objects. The results show that the corner phenomenon is greater for corners of object representations than for corners that do not define an object's shape. We also examined whether the corner effect persists within the contour boundaries of an object, as well as on the outside. The results showed that a spatial gradient of attention accompanies the corner effect outside the contour boundaries of an object but that processing within an object is uniform, with no corner effect occurring. We discuss these findings in relation to space-based and object-based theories of attention.

A gradual spread of attention

The visual system has to segregate objects that are relevant to behavior from other objects and the background, if they are embedded in a visual scene. This segregation process can be time consuming, especially if the relevant object is spatially extended and overlaps with other image components, but the cause of the delays is presently not well understood. In the present study, we used a curve-tracing task to investigate processing delays during the grouping of contour segments into elongated curves. Our results indicate that contour segments that need to be grouped together are labeled with visual attention. Attention gradually spreads from contour segments that were labeled previously to other contours that are colinear and connected to them. The contour-grouping task is completed as soon as attention is directed to the entire curve. We conclude that processing delays during contour grouping are caused by a time-consuming spread of visual attention.

Effects of plane rotation, task, and complexity on recognition of familiar and chimeric objects

We examined the effects of plane rotation, task, and visual complexity on the recognition of familiar and chimeric objects. The effects of rotation, with response times increasing linearly and monotonically with rotation from the upright, were equivalent for tasks requiring different degrees of visual differentiation of the target from contrasting stimuli--namely, (1) deciding whether the stimulus was living or nonliving (semantic classification), (2) deciding whether the stimulus was an object or a nonobject (object decision), and (3) naming. The effects of complexity, with shorter response times to more complex stimuli, were most apparent in semantic classification and object decision and were additive with the effects of rotation. We discuss the implications of these results for theories of the relationship between the process of normalization and the determining of object identity.

Attention! V1 neurons lining up for inspection

In this issue of Neuron, Roelfsema and Spekreijse report that macaque V1 neuron responses are correlated with target choice in a task requiring monkeys to attentively trace a line to plan a saccade. These results provide evidence that V1 is actively involved in the interpretation of visual stimuli.

Mental maze solving

We sought to determine how a visual maze is mentally solved. Human subjects (N = 13) viewed mazes with orthogonal, unbranched paths; each subject solved 200-600 mazes in any specific experiment below. There were four to six openings at the perimeter of the maze, of which four were labeled: one was the entry point and the remainder were potential exits marked by Arabic numerals. Starting at the entry point, in some mazes the path exited, whereas in others it terminated within the maze. Subjects were required to type the number corresponding to the true exit (if the path exited) or type zero (if the path did not exit). In all cases, the only required hand movement was a key press, and thus the hand never physically traveled through the maze. Response times (RT) were recorded and analyzed using a multiple linear regression model. RT increased as a function of key parameters of the maze, namely the length of the main path, the number of turns in the path, the direct distance from entry to termination, and the presence of an exit. The dependence of RT on the number of turns was present even when the path length was fixed in a separate experiment (N = 10 subjects). In a different experiment, subjects solved large and small mazes (N = 3 subjects). The former was the same as the latter but was scaled up by 1.77 times. Thus both kinds of mazes contained the same number of squares but each square subtended 1.77 degrees of visual angle (DVA) in the large maze, as compared to 1 DVA in the small one. We found that the average RT was practically the same in both cases. A multiple regression analysis revealed that the processing coefficients related to maze distance (i.e., path length and direct distance) were reduced by approximately one-half when solving large mazes, as compared to solving small mazes. This means that the efficiency in processing distance-related information almost doubled for scaled-up mazes. In contrast, the processing coefficients for number of turns and exit status were practically the same in the two cases. Finally, the eye movements of three subjects were recorded during maze solution. They consisted of sequences of saccades and fixations. The number of fixations in a trial increased as a linear function of the path length and number of turns. With respect to the fixations themselves, eyes tended to fixate on the main path and to follow it along its course, such that fixations occurring later in time were positioned at progressively longer distances from the entry point. Furthermore, the time the eyes spent at each fixation point increased as a linear function of the length and number of turns in the path segment between the current and the upcoming fixation points. These findings suggest that the maze segment from the current fixation spot to the next is being processed during the fixation time (FT), and that a significant aspect of this processing relates to the length and turns in that segment. We interpreted these relations to mean that the maze was mentally traversed. We then estimated the distance and endpoint of the path mentally traversed within a specific FT; we also hypothesized that the next portion of the main path would be traversed during the ensuing FT, and so on for the whole path. A prediction of this hypothesis is that the upcoming saccade would land the eyes at or near the locus on the path where the mental traversing ended, so that "the eyes would pick up where the mental traversal left off." In this way, a portion of the path would be traversed during a fixation and successive such portions would be strung together closely along the main path to complete the processing of the whole path. We tested this prediction by analyzing the relations between the path distance of mental traverse and the distance along the path between the current and the next fixation spot. (ABSTRACT TRUNCATED)

Objects of attention, objects of perception

Four experiments were conducted, to explore the notion of objects in perception. Taking as a starting point the effects of display content on rapid attention transfer and manipulating curvature, closure, and processing time, a link between objects of attention and objects of perception is proposed. In Experiment 1, a number of parallel, equally spaced, straight lines facilitated attention transfer along the lines, relative to transfer across the lines. In Experiment 2, with curved, closed-contour shapes, no "same-object" facilitation was observed. However, when a longer time interval was provided, in Experiment 3, a same-object advantage started to emerge. In Experiment 4, using the same curved shapes but in a non-speeded distance estimation task, a strong effect of objects was observed. It is argued that attention transfer is facilitated by line tracing but that line tracing is encouraged by objects.

The computation of contour information in complex objects

Perceptual organisation, and especially the computation of contour information, has been the object of considerable interest in the last few years. In the first part of the paper we review recent accounts on the mechanisms involved in the processing of contour. In the second part we report an experiment designed to examine (1) how physical parameters such as spatial proximity and collinearity of elements affect the integration of global contour in objects and (2) whether the activation of stored representations of objects facilitates the computation of contour. Incomplete forms varying in the spacing and the alignment of line segments on their contour were used as stimuli in a matching task. Subjects were asked to decide which of two laterally displayed figures matched a reference form presented previously. The matching target and the distractor were physically identical but differed in their orientation. In one condition the reference object was always an outline drawing of an object. In a second condition the reference object was either a complete object or a more or less identifiable incomplete form. Little variation in performance was found for forms having continuous and discontinuous contour up to a spacing of 5 pixels (10.8 min) between elements. Response times and errors increased abruptly beyond this limit. This effect occurred in the two conditions of reference stimulus, suggesting that the computation of contour information is more affected by physical constraints at early processes than by high-level processes involving activation of stored structural representations of objects.