Tolerances of responses to visual patterns in neurons of the posterior inferotemporal cortex in the macaque against changing stimulus size and orientation, and deleting patterns

Single-cell selectivity and functional architecture of human lateral occipital complex

The human lateral occipital complex (LOC) is more strongly activated by images of objects compared to scrambled controls, but detailed information at the neuronal level is currently lacking. We recorded with microelectrode arrays in the LOC of 2 patients and obtained highly selective single-unit, multi-unit, and high-gamma responses to images of objects. Contrary to predictions derived from functional imaging studies, all neuronal properties indicated that the posterior subsector of LOC we recorded from occupies an unexpectedly high position in the hierarchy of visual areas. Notably, the response latencies of LOC neurons were long, the shape selectivity was spatially clustered, LOC receptive fields (RFs) were large and bilateral, and a number of LOC neurons exhibited three-dimensional (3D)-structure selectivity (a preference for convex or concave stimuli), which are all properties typical of end-stage ventral stream areas. Thus, our results challenge prevailing ideas about the position of the more posterior subsector of LOC in the hierarchy of visual areas.

Single-cell responses to three-dimensional structure in a functionally defined patch in macaque area TEO

Both dorsal and ventral visual pathways harbor several areas sensitive to gradients of binocular disparity (i.e., higher-order disparity). Although a wealth of information exists about disparity processing in early visual (V1, V2, and V3) and end-stage areas, TE in the ventral stream, and the anterior intraparietal area (AIP) in the dorsal stream, little is known about midlevel area TEO in the ventral pathway. We recorded single-unit responses to disparity-defined curved stimuli in a functional magnetic resonance imaging (fMRI) activation elicited by curved surfaces compared with flat surfaces in the macaque area TEO. This fMRI activation contained a small proportion of disparity-selective neurons, with very few of them second-order disparity selective. Overall, this population of TEO neurons did not preserve its three-dimensional structure selectivity across positions in depth, indicating a lack of higher-order disparity selectivity, but showed stronger responses to flat surfaces than to curved surfaces, as predicted by the fMRI experiment. The receptive fields of the responsive TEO cells were relatively small and generally foveal. A linear support vector machine classifier showed that this population of disparity-selective TEO neurons contains reliable information about the sign of curvature and the position in depth of the stimulus. NEW & NOTEWORTHY We recorded in a part of the macaque area TEO that is activated more by curved surfaces than by flat surfaces at different disparities using the same stimuli. In contrast to previous studies, this functional magnetic resonance imaging-defined patch did not contain a large number of higher-order disparity-selective neurons. However, a linear support vector machine could reliably classify both the sign of the disparity gradient and the position in depth of the stimuli.

Very small faces are easily discriminated under long and short exposure times

Acuity measures related to overall face size that can be perceived have not been studied quantitatively. Consequently, experimenters use a wide range of sizes (usually large) without always providing a rationale for their choices. I studied thresholds for face discrimination by presenting both long (500 ms)- and short (17, 33, 50 ms)-duration stimuli. Face width threshold for the long presentation was ~0.2°, and thresholds for the flashed stimuli ranged from ~0.3° for the 17-ms flash to ~0.23° for the 33- and 50-ms flashes. Such thresholds indicate that face stimuli used in physiological or psychophysical experiments are often too large to tap human fine spatial capabilities, and thus interpretations of such experiments should take into account face discrimination acuity. The 0.2° threshold found in this study is incompatible with the prevalent view that faces are represented by a population of specialized “face cells” because those cells do not respond to <1° stimuli and are optimally tuned to >4° faces. Also, the ability to discriminate small, high-spatial frequency flashed face stimuli is inconsistent with models suggesting that fixational drift transforms retinal spatial patterns into a temporal code. It seems therefore that the small image motions occurring during fixation do not disrupt our perception, because all relevant processing is over with before those motions can have significant effects.

NEW & NOTEWORTHY Although face perception is central to human behavior, the minimally perceived face size is not known. This study shows that humans can discriminate very small (~0.2°) faces. Furthermore, even when flashed for tens of milliseconds, ~0.25° faces can be discriminated. Such fine acuity should impact modeling of physiological mechanisms of face perception. The ability to discriminate flashed faces where there is almost no eye movement indicates that eye drift is not essential for visibility.

Seeing a straight line on a curved surface: decoupling of patterns from surfaces by single IT neurons

We have no difficulty seeing a straight line on a curved piece of paper, but in fact, doing so requires decoupling the shape of the surface from the pattern itself. Here we report a novel form of invariance in the visual cortex: single neurons in monkey inferior temporal cortex respond similarly to congruent transformations of patterns and surfaces, in effect decoupling patterns from the surface on which they are overlaid.

We have no difficulty seeing a straight line drawn on a paper even when the paper is bent, but this inference is in fact nontrivial. Doing so requires either matching local features or representing the pattern after factoring out the surface shape. Here we show that single neurons in the monkey inferior temporal (IT) cortex show invariant responses to patterns across rigid and nonrigid changes of surfaces. We recorded neuronal responses to stimuli in which the pattern and the surrounding surface were varied independently. In a subset of neurons, we found pattern-surface interactions that produced similar responses to stimuli across congruent pattern and surface transformations. These interactions produced systematic shifts in curvature tuning of patterns when overlaid on convex and flat surfaces. Our results show that surfaces are factored out of patterns by single neurons, thereby enabling complex perceptual inferences.

NEW & NOTEWORTHY We have no difficulty seeing a straight line on a curved piece of paper, but in fact, doing so requires decoupling the shape of the surface from the pattern itself. Here we report a novel form of invariance in the visual cortex: single neurons in monkey inferior temporal cortex respond similarly to congruent transformations of patterns and surfaces, in effect decoupling patterns from the surface on which they are overlaid.

Contour Curvature As an Invariant Code for Objects in Visual Area V4

Size-invariant object recognition-the ability to recognize objects across transformations of scale-is a fundamental feature of biological and artificial vision. To investigate its basis in the primate cerebral cortex, we measured single neuron responses to stimuli of varying size in visual area V4, a cornerstone of the object-processing pathway, in rhesus monkeys (Macaca mulatta). Leveraging two competing models for how neuronal selectivity for the bounding contours of objects may depend on stimulus size, we show that most V4 neurons (∼70%) encode objects in a size-invariant manner, consistent with selectivity for a size-independent parameter of boundary form: for these neurons, "normalized" curvature, rather than "absolute" curvature, provided a better account of responses. Our results demonstrate the suitability of contour curvature as a basis for size-invariant object representation in the visual cortex, and posit V4 as a foundation for behaviorally relevant object codes.

SIGNIFICANCE STATEMENT

Size-invariant object recognition is a bedrock for many perceptual and cognitive functions. Despite growing neurophysiological evidence for invariant object representations in the primate cortex, we still lack a basic understanding of the encoding rules that govern them. Classic work in the field of visual shape theory has long postulated that a representation of objects based on information about their bounding contours is well suited to mediate such an invariant code. In this study, we provide the first empirical support for this hypothesis, and its instantiation in single neurons of visual area V4.

Coding of relative size in monkey inferotemporal cortex

We seldom mistake a closer object as being larger, even though its retinal image is bigger. One underlying mechanism could be to calculate the size of the retinal image relative to that of another nearby object. Here we set out to investigate whether single neurons in the monkey inferotemporal cortex (IT) are sensitive to the relative size of parts in a display. Each neuron was tested on shapes containing two parts that could be conjoined or spatially separated. Each shape was presented in four versions created by combining the two parts at each of two possible sizes. In this design, neurons sensitive to the absolute size of parts would show the greatest response modulation when both parts are scaled up, whereas neurons encoding relative size would show similar responses. Our main findings are that 1) IT neurons responded similarly to all four versions of a shape, but tuning tended to be more consistent between versions with proportionately scaled parts; 2) in a subpopulation of cells, we observed interactions that resulted in similar responses to proportionately scaled parts; 3) these interactions developed together with sensitivity to absolute size for objects with conjoined parts but developed slightly later for objects with spatially separate parts. Taken together, our results demonstrate for the first time that there is a subpopulation of neurons in IT that encodes the relative size of parts in a display, forming a potential neural substrate for size constancy.

Perceptual learning of simple stimuli modifies stimulus representations in posterior inferior temporal cortex

Practicing simple visual detection and discrimination tasks improves performance, a signature of adult brain plasticity. The neural mechanisms that underlie these changes in performance are still unclear. Previously, we reported that practice in discriminating the orientation of noisy gratings (coarse orientation discrimination) increased the ability of single neurons in the early visual area V4 to discriminate the trained stimuli. Here, we ask whether practice in this task also changes the stimulus tuning properties of later visual cortical areas, despite the use of simple grating stimuli. To identify candidate areas, we used fMRI to map activations to noisy gratings in trained rhesus monkeys, revealing a region in the posterior inferior temporal (PIT) cortex. Subsequent single unit recordings in PIT showed that the degree of orientation selectivity was similar to that of area V4 and that the PIT neurons discriminated the trained orientations better than the untrained orientations. Unlike in previous single unit studies of perceptual learning in early visual cortex, more PIT neurons preferred trained compared with untrained orientations. The effects of training on the responses to the grating stimuli were also present when the animals were performing a difficult orthogonal task in which the grating stimuli were task-irrelevant, suggesting that the training effect does not need attention to be expressed. The PIT neurons could support orientation discrimination at low signal-to-noise levels. These findings suggest that extensive practice in discriminating simple grating stimuli not only affects early visual cortex but also changes the stimulus tuning of a late visual cortical area.

Simultaneous shape repulsion and global assimilation in the perception of aspect ratio

Although local interactions involving orientation and spatial frequency are well understood, less is known about spatial interactions involving higher level pattern features. We examined interactive coding of aspect ratio, a prevalent two-dimensional feature. We measured perception of two simultaneously flashed ellipses by randomly post-cueing one of them and having observers indicate its aspect ratio. Aspect ratios interacted in two ways. One manifested as an aspect-ratio-repulsion effect. For example, when a slightly tall ellipse and a taller ellipse were simultaneously flashed, the less tall ellipse appeared flatter and the taller ellipse appeared even taller. This repulsive interaction was long range, occurring even when the ellipses were presented in different visual hemifields. The other interaction manifested as a global assimilation effect. An ellipse appeared taller when it was a part of a global vertical organization than when it was a part of a global horizontal organization. The repulsion and assimilation effects temporally dissociated as the former slightly strengthened, and the latter disappeared when the ellipse-to-mask stimulus onset asynchrony was increased from 40 to 140 ms. These results are consistent with the idea that shape perception emerges from rapid lateral and hierarchical neural interactions.

Representing the forest before the trees: a global advantage effect in monkey inferotemporal cortex

Hierarchical stimuli (large shapes composed of small shapes) have long been used to study how humans perceive the global and the local content of a scene--the forest and the trees. Studies using these stimuli have revealed a global advantage effect: humans consistently report global shape faster than local shape. The neuronal underpinnings of this effect remain unclear. Here we demonstrate a correlate and possible mechanism in monkey inferotemporal cortex (IT). Inferotemporal neurons signal the global content of a hierarchical display approximately 30 ms before they signal its local content. This is a specific expression of a general principle, related to spatial scale or spatial frequency rather than to hierarchical level, whereby the representation of a large shape develops in IT before that of a small shape. These findings provide support for a coarse-to-fine model of visual scene representation.

Long-term speeding in perceptual switches mediated by attention-dependent plasticity in cortical visual processing

Binocular rivalry has been extensively studied to understand the mechanisms that control switches in visual awareness and much has been revealed about the contributions of stimulus and cognitive factors. Because visual processes are fundamentally adaptive, however, it is also important to understand how experience alters the dynamics of perceptual switches. When observers viewed binocular rivalry repeatedly over many days, the rate of perceptual switches increased as much as 3-fold. This long-term rivalry speeding exhibited a pattern of image-feature specificity that ruled out primary contributions from strategic and nonsensory factors and implicated neural plasticity occurring in both low- and high-level visual processes in the ventral stream. Furthermore, the speeding occurred only when the rivaling patterns were voluntarily attended, suggesting that the underlying neural plasticity selectively engages when stimuli are behaviorally relevant. Long-term rivalry speeding may thus reflect broader mechanisms that facilitate quick assessments of signals that contain multiple behaviorally relevant interpretations.

Adaptive modulation of color salience contingent upon global form coding and task relevance

Extensive research on local color aftereffects has revealed perceptual consequences of opponent color coding in the retina and the LGN, and of orientation-and/or spatial-frequency-contingent color coding in early cortical visual areas (e.g., V1 and V2). Here, we report a color aftereffect that depends crucially on global-form-contingent color processing. Brief viewing of colored items (passively viewed, ignored, or attended) reduced the salience of the previewed color in a subsequent task of color-based visual search. This color-salience aftereffect was relatively insensitive to variations (between color preview and search) in local image features, but was substantially affected by changes in global configuration (e.g. the presence or absence of perceptual unitization); the global-form dependence of the aftereffect was also modulated by task demands. The overall results suggest that (1) color salience is adaptively modulated (from fixation to fixation), drawing attention to a new color in visual-search contexts, and (2) these modulations seem to be mediated by global-form-and-color-selective neural processing in mid to late stages of the ventral visual pathway (e.g., V4 and IT), in combination with task-dependent feedback from higher cortical areas (e.g., prefrontal cortex).

Long- and short-range reward expectancy in the primate orbitofrontal cortex

The orbitofrontal cortex (OFC) is important in motivation and emotion. We previously reported reward expectancy-related delay activities during a delayed reaction time task in primate OFC neurons. To further investigate the significance of the OFC in motivational operations, we examined pre-instruction, baseline activities of OFC neurons in relation to reward expectancy during the delayed reaction time task. In this task, an instruction cue indicated whether reward would be present or absent in the trial. Each set of four consecutive trials constituted one block within which three different kinds of rewards and one trial with no reward were given in a fixed order that differed from the monkey's reward preference. We identified two types of OFC neurons with reward expectancy-related pre-instruction activities: Step-type neurons showed stepwise changes (increase or decrease) in pre-instruction activity toward the trial with a particular outcome, which usually was the most or least attractive within a block; Pref-type neurons showed pre-instruction activity changes according to the monkey's preference for each trial's outcome. We propose that Step-type and Pref-type neurons are related to long-range and short-range reward expectancies of a particular outcome, respectively. The OFC is considered to play important roles in goal-directed behaviour by adjusting the motivational level toward a certain (current or future) outcome of a particular motivational significance based on the two kinds of reward expectancy processes. Impairments in goal-directed behaviour by OFC patients may be caused by a lack of long-range expectancy or by a deficit in compromising between short-range and long-range expectancies.

Attentional selection of overlapped shapes: a study using brief shape aftereffects

Prior studies using brief stimulus sequences revealed "opponent shape aftereffects", indicative of direct opponent coding of global shape attributes such as aspect ratio, skew, taper, curvature, and convexity (perhaps in IT). Further, aftereffects from overlapped opponent pairs of adaptor shapes (e.g., concave and convex shapes) were substantially modulated by attention [Vision Res. 41 (2001) 3883]. Hypothetically, (1) attention might weight the attended and ignored contours at early stages of processing, or (2) it might sway opposing neural activity (e.g., of convex- vs. concave-tuned units) at the stage of opponent shape coding. Attentional modulation was equivalent for opponent pairs (producing opposite aftereffects) and non-opponent pairs (producing orthogonal aftereffects) of overlapped adaptor shapes, whether convexity or aspect-ratio aftereffects were measured. Further, the degree of attentional modulation obtained for these aftereffects (approximately 60%) was comparable to that obtained for V4 cells [J. Neurosci. 19 (1999) 1736]. Taken together, differential contour weighting appears to be the primary mechanism of attentional modulation of brief shape aftereffects.

Evidence for perceptual "trapping" and adaptation in multistable binocular rivalry

When a different pattern is presented to each eye, the perceived image spontaneously alternates between the two patterns (binocular rivalry); the dynamics of these bistable alternations are known to be stochastic. Examining multistable binocular rivalry (involving four dominant percepts), we demonstrated path dependence and on-line adaptation, which were equivalent whether perceived patterns were formed by single-eye dominance or by mixed-eye dominance. The spontaneous perceptual transitions tended to get trapped within a pair of related global patterns (e.g., opponent shapes and symmetric patterns), and during such trapping, the probability of returning to the repeatedly experienced patterns gradually decreased (postselection pattern adaptation). These results suggest that the structure of global shape coding and its adaptation play a critical role in directing spontaneous alternations of visual awareness in perceptual multistability.

Attention-dependent brief adaptation to contour orientation: a high-level aftereffect for convexity?

In contrast to the abundant literature investigating how orientation coding depends on edges defined by various image features, relatively little is known about how coding of orientation might also depend on the two distinct functional roles that oriented edges commonly play. Oriented lines can delineate outline contours of a figure or they can form texture. The results of five experiments using orientation aftereffects measured with brief tests (27 ms, backward masked; adapt-to-test interval=201 ms) provided evidence that brief stimuli (<135 ms) selectively adapt coding of contour-line orientation rather than coding of line-texture orientation. Furthermore, parametric results revealed that the rapidly adapting aftereffects for contour orientation are characterized by (1) broad orientation tuning (peaking at +/-30 degrees to +/-50 degrees from test orientation), (2) indifference as to how the contours are defined (e.g. bright lines, high-pass-filtered lines, faint lines generated by the spatial inhomogeneity of visual sensitivity), (3) rapid saturation at low contrast energy, (4) strong modulation by selective attention, and (5) relative size tolerance. These characteristics appear to parallel those of cells in the high end of the visual form processing pathway (such as inferotemporal cortex). It is thus suggested that the rapidly adapting contour orientation aftereffects reported here may be mediated by high-level neural units that encode global configurations of orientation (e.g. convexity and concavity).