The integration of multiple stimulus features by V1 neurons

Gamma Responses to Colored Natural Stimuli Can Be Predicted from Local Low-Level Stimulus Features

The role of gamma rhythm (30-80 Hz) in visual processing is debated; stimuli like gratings and hue patches generate strong gamma, but many natural images do not. Could image gamma responses be predicted by approximating images as gratings or hue patches? Surprisingly, this question remains unanswered, since the joint dependence of gamma on multiple features is poorly understood. We recorded local field potentials and electrocorticogram from two female monkeys while presenting natural images and parametric stimuli varying along several feature dimensions. Gamma responses to different grating/hue features were separable, allowing for a multiplicative model based on individual features. By fitting a hue patch to the image around the receptive field, this simple model could predict gamma responses to chromatic images across scales with reasonably high accuracy. Our results provide a simple "baseline" model to predict gamma from local image properties, against which more complex models of natural vision can be tested.

High-precision mapping reveals the structure of odor coding in the human brain

Odor perception is inherently subjective. Previous work has shown that odorous molecules evoke distributed activity patterns in olfactory cortices, but how these patterns map on to subjective odor percepts remains unclear. In the present study, we collected neuroimaging responses to 160 odors from 3 individual subjects (18 h per subject) to probe the neural coding scheme underlying idiosyncratic odor perception. We found that activity in the orbitofrontal cortex (OFC) represents the fine-grained perceptual identity of odors over and above coarsely defined percepts, whereas this difference is less pronounced in the piriform cortex (PirC) and amygdala. Furthermore, the implementation of perceptual encoding models enabled us to predict olfactory functional magnetic resonance imaging responses to new odors, revealing that the dimensionality of the encoded perceptual spaces increases from the PirC to the OFC. Whereas encoding of lower-order dimensions generalizes across subjects, encoding of higher-order dimensions is idiosyncratic. These results provide new insights into cortical mechanisms of odor coding and suggest that subjective olfactory percepts reside in the OFC.

Trailblazers in Neuroscience: Using compositionality to understand how parts combine in whole objects

A fundamental question for any visual system is whether its image representation can be understood in terms of its components. Decomposing any image into components is challenging because there are many possible decompositions with no common dictionary, and enumerating them leads to a combinatorial explosion. Even in perception, many objects are readily seen as containing parts, but there are many exceptions. These exceptions include objects that are not perceived as containing parts, properties like symmetry that cannot be localized to any single part, and also special categories like words and faces whose perception is widely believed to be holistic. Here, I describe a novel approach we have used to address these issues and evaluate compositionality at the behavioral and neural levels. The key design principle is to create a large number of objects by combining a small number of pre-defined components in all possible ways. This allows for building component-based models that explain whole objects using a combination of these components. Importantly, any systematic error in model fits can be used to detect the presence of emergent or holistic properties. Using this approach, we have found that whole object representations are surprisingly predictable from their components, that some components are preferred to others in perception, and that emergent properties can be discovered or explained using compositional models. Thus, compositionality is a powerful approach for understanding how whole objects relate to their parts.

Symmetry-Based Representations for Artificial and Biological General Intelligence

Biological intelligence is remarkable in its ability to produce complex behavior in many diverse situations through data efficient, generalizable, and transferable skill acquisition. It is believed that learning "good" sensory representations is important for enabling this, however there is little agreement as to what a good representation should look like. In this review article we are going to argue that symmetry transformations are a fundamental principle that can guide our search for what makes a good representation. The idea that there exist transformations (symmetries) that affect some aspects of the system but not others, and their relationship to conserved quantities has become central in modern physics, resulting in a more unified theoretical framework and even ability to predict the existence of new particles. Recently, symmetries have started to gain prominence in machine learning too, resulting in more data efficient and generalizable algorithms that can mimic some of the complex behaviors produced by biological intelligence. Finally, first demonstrations of the importance of symmetry transformations for representation learning in the brain are starting to arise in neuroscience. Taken together, the overwhelming positive effect that symmetries bring to these disciplines suggest that they may be an important general framework that determines the structure of the universe, constrains the nature of natural tasks and consequently shapes both biological and artificial intelligence.

Joint encoding of facial identity, orientation, gaze, and expression in the middle dorsal face area

The last two decades have established that a network of face-selective areas in the temporal lobe of macaque monkeys supports the visual processing of faces. Each area within the network contains a large fraction of face-selective cells. And each area encodes facial identity and head orientation differently. A recent brain-imaging study discovered an area outside of this network selective for naturalistic facial motion, the middle dorsal (MD) face area. This finding offers the opportunity to determine whether coding principles revealed inside the core network would generalize to face areas outside the core network. We investigated the encoding of static faces and objects, facial identity, and head orientation, dimensions which had been studied in multiple areas of the core face-processing network before, as well as facial expressions and gaze. We found that MD populations form a face-selective cluster with a degree of selectivity comparable to that of areas in the core face-processing network. MD encodes facial identity robustly across changes in head orientation and expression, it encodes head orientation robustly against changes in identity and expression, and it encodes expression robustly across changes in identity and head orientation. These three dimensions are encoded in a separable manner. Furthermore, MD also encodes the direction of gaze in addition to head orientation. Thus, MD encodes both structural properties (identity) and changeable ones (expression and gaze) and thus provides information about another animal's direction of attention (head orientation and gaze). MD contains a heterogeneous population of cells that establish a multidimensional code for faces.

Closed loop motor-sensory dynamics in human vision

Vision is obtained with a continuous motion of the eyes. The kinematic analysis of eye motion, during any visual or ocular task, typically reveals two (kinematic) components: saccades, which quickly replace the visual content in the retinal fovea, and drifts, which slowly scan the image after each saccade. While the saccadic exchange of regions of interest (ROIs) is commonly considered to be included in motor-sensory closed-loops, it is commonly assumed that drifts function in an open-loop manner, that is, independent of the concurrent visual input. Accordingly, visual perception is assumed to be based on a sequence of open-loop processes, each initiated by a saccade-triggered retinal snapshot. Here we directly challenged this assumption by testing the dependency of drift kinematics on concurrent visual inputs using real-time gaze-contingent-display. Our results demonstrate a dependency of the trajectory on the concurrent visual input, convergence of speed to condition-specific values and maintenance of selected drift-related motor-sensory controlled variables, all strongly indicative of drifts being included in a closed-loop brain-world process, and thus suggesting that vision is inherently a closed-loop process.

Representation of shape, space, and attention in monkey cortex

Attentional deficits are core to numerous developmental, neurological, and psychiatric disorders. At the single-cell level, much knowledge has been garnered from studies of shape and spatial properties, as well as from numerous demonstrations of attentional modulation of those properties. Despite this wealth of knowledge of single-cell responses across many brain regions, little is known about how these cellular characteristics relate to population level representations and how such representations relate to behavior; in particular, how these cellular responses relate to the representation of shape, space, and attention, and how these representations differ across cortical areas and streams. Here we will emphasize the role of population coding as a missing link for connecting single-cell properties with behavior. Using a data-driven intrinsic approach to population decoding, we show that both 'what' and 'where' cortical visual streams encode shape, space, and attention, yet demonstrate striking differences in these representations. We suggest that both pathways fully process shape and space, but that differences in representation may arise due to their differing functions and input and output constraints. Moreover, differences in the effects of attention on shape and spatial population representations in the two visual streams suggest two distinct strategies: in a ventral area, attention or task demands modulate the population representations themselves (perhaps to expand or enhance one part at the expense of other parts) while in a dorsal area, at a population representation level, attention effects are weak and nearly non-existent, perhaps in order to maintain veridical representations needed for visuomotor control. We show that an intrinsic approach, as opposed to theory-driven and labeled approaches, is useful for understanding how representations develop and differ across brain regions. Most importantly, these approaches help link cellular properties more tightly with behavior, a much-needed step to better understand and interpret cellular findings and key to providing insights to improve interventions in human disorders.

Multiplicative mixing of object identity and image attributes in single inferior temporal neurons

Vision is a challenging problem because the same object can produce a variety of images on the retina, mixing signals related to its identity with signals related to its viewing attributes, such as size, position, rotation, etc. Precisely how the brain separates these signals to form an efficient representation is unknown. Here, we show that single neurons in high-level visual cortex encode object identity and attribute multiplicatively and that doing so allows for better decoding of each signal.

Object recognition is challenging because the same object can produce vastly different images, mixing signals related to its identity with signals due to its image attributes, such as size, position, rotation, etc. Previous studies have shown that both signals are present in high-level visual areas, but precisely how they are combined has remained unclear. One possibility is that neurons might encode identity and attribute signals multiplicatively so that each can be efficiently decoded without interference from the other. Here, we show that, in high-level visual cortex, responses of single neurons can be explained better as a product rather than a sum of tuning for object identity and tuning for image attributes. This subtle effect in single neurons produced substantially better population decoding of object identity and image attributes in the neural population as a whole. This property was absent both in low-level vision models and in deep neural networks. It was also unique to invariances: when tested with two-part objects, neural responses were explained better as a sum than as a product of part tuning. Taken together, our results indicate that signals requiring separate decoding, such as object identity and image attributes, are combined multiplicatively in IT neurons, whereas signals that require integration (such as parts in an object) are combined additively.

Spatiotemporal Filter for Visual Motion Integration from Pursuit Eye Movements in Humans and Monkeys

Despite the enduring interest in motion integration, a direct measure of the space-time filter that the brain imposes on a visual scene has been elusive. This is perhaps because of the challenge of estimating a 3D function from perceptual reports in psychophysical tasks. We take a different approach. We exploit the close connection between visual motion estimates and smooth pursuit eye movements to measure stimulus-response correlations across space and time, computing the linear space-time filter for global motion direction in humans and monkeys. Although derived from eye movements, we find that the filter predicts perceptual motion estimates quite well. To distinguish visual from motor contributions to the temporal duration of the pursuit motion filter, we recorded single-unit responses in the monkey middle temporal cortical area (MT). We find that pursuit response delays are consistent with the distribution of cortical neuron latencies and that temporal motion integration for pursuit is consistent with a short integration MT subpopulation. Remarkably, the visual system appears to preferentially weight motion signals across a narrow range of foveal eccentricities rather than uniformly over the whole visual field, with a transiently enhanced contribution from locations along the direction of motion. We find that the visual system is most sensitive to motion falling at approximately one-third the radius of the stimulus aperture. Hypothesizing that the visual drive for pursuit is related to the filtered motion energy in a motion stimulus, we compare measured and predicted eye acceleration across several other target forms.SIGNIFICANCE STATEMENT A compact model of the spatial and temporal processing underlying global motion perception has been elusive. We used visually driven smooth eye movements to find the 3D space-time function that best predicts both eye movements and perception of translating dot patterns. We found that the visual system does not appear to use all available motion signals uniformly, but rather weights motion preferentially in a narrow band at approximately one-third the radius of the stimulus. Although not universal, the filter predicts responses to other types of stimuli, demonstrating a remarkable degree of generalization that may lead to a deeper understanding of visual motion processing.

Decoding Target Distance and Saccade Amplitude from Population Activity in the Macaque Lateral Intraparietal Area (LIP)

Primates perform saccadic eye movements in order to bring the image of an interesting target onto the fovea. Compared to stationary targets, saccades toward moving targets are computationally more demanding since the oculomotor system must use speed and direction information about the target as well as knowledge about its own processing latency to program an adequate, predictive saccade vector. In monkeys, different brain regions have been implicated in the control of voluntary saccades, among them the lateral intraparietal area (LIP). Here we asked, if activity in area LIP reflects the distance between fovea and saccade target, or the amplitude of an upcoming saccade, or both. We recorded single unit activity in area LIP of two macaque monkeys. First, we determined for each neuron its preferred saccade direction. Then, monkeys performed visually guided saccades along the preferred direction toward either stationary or moving targets in pseudo-randomized order. LIP population activity allowed to decode both, the distance between fovea and saccade target as well as the size of an upcoming saccade. Previous work has shown comparable results for saccade direction (Graf and Andersen, 2014a,b). Hence, LIP population activity allows to predict any two-dimensional saccade vector. Functional equivalents of macaque area LIP have been identified in humans. Accordingly, our results provide further support for the concept of activity from area LIP as neural basis for the control of an oculomotor brain-machine interface.

V1 neurons respond to luminance changes faster than contrast changes

Luminance and contrast are two major attributes of objects in the visual scene. Luminance and contrast information received by visual neurons are often updated simultaneously. We examined the temporal response properties of neurons in the primary visual cortex (V1) to stimuli whose luminance and contrast were simultaneously changed by 50 Hz. We found that response tuning to luminance changes precedes tuning to contrast changes in V1. For most V1 neurons, the onset time of response tuning to luminance changes was shorter than that to contrast changes. Most neurons carried luminance information in the early response stage, while all neurons carried both contrast and luminance information in the late response stage. The early luminance response suggests that cortical processing for luminance is not as slow as previously thought.

1-D Vision: Encoding of Eye Movements by Simple Receptive Fields

Eye movements (eyeM) are an essential component of visual perception. They allow the sampling and scanning of stationary scenes at various spatial scales, primarily at the scene level, via saccades, and at the local level, via fixational eyeM. Given the constant motion of visual images on the retina, a crucial factor in resolving spatial ambiguities related to the external scene is the exact trajectory of eyeM. We show here that the trajectory of eyeM can be encoded at high resolution by simple retinal receptive fields of the symmetrical type. We also show that such encoding can account for motion illusions such as the Ouchi illusion. In addition, encoding of motion projections along horizontal and vertical symmetrical simple retinal receptive fields entails a kind of Cartesian decomposition of the 2-D image into two 1-D projections.

Distributed processing of color and form in the visual cortex

To what extent does the visual system process color and form separately? Proponents of the segregation view claim that distinct regions of the cortex are dedicated to each of these two dimensions separately. However, evidence is accumulating that color and form processing may, at least to some extent, be intertwined in the brain. In this perspective, we review psychophysical and neurophysiological studies on color and form perception and evaluate their results in light of recent developments in population coding.

Joint tuning for direction of motion and binocular disparity in macaque MT is largely separable

Neurons in sensory cortical areas are tuned to multiple dimensions, or features, of their sensory space. Understanding how single neurons represent multiple features is of great interest for determining the informative dimensions of the neurons' response, the decoding algorithms appropriate for extracting this information from the neuronal population, and for determining where specific transformations occur along the visual hierarchy. Despite the established role of cortical area MT in judgments of motion and depth, it is not known how individual neurons jointly encode the two dimensions. We investigated the joint tuning of individual MT neurons for two visual features: direction of motion and binocular disparity, an important depth cue. We found that a separable, multiplicative combination of tuning for the two features can account for more than 90% of the variance in the joint tuning function for over 91% of MT neurons. These results suggest 1) that each feature can be read out independently from MT by simply averaging across the population without regard to the other feature and 2) that the inseparable representations seen in subsequent areas, such as MST, must be computed beyond MT. Intriguingly, we found that the remaining nonseparable component of the joint tuning function often manifested as small but systematic changes in the neurons' preferences for one feature as the other one was varied. We believe this reflects the local columnar organization of tuning for direction and binocular disparity in MT, indicating that joint tuning may provide a new tool with which to probe functional architecture.

Seeing via Miniature Eye Movements: A Dynamic Hypothesis for Vision

During natural viewing, the eyes are never still. Even during fixation, miniature movements of the eyes move the retinal image across tens of foveal photoreceptors. Most theories of vision implicitly assume that the visual system ignores these movements and somehow overcomes the resulting smearing. However, evidence has accumulated to indicate that fixational eye movements cannot be ignored by the visual system if fine spatial details are to be resolved. We argue that the only way the visual system can achieve its high resolution given its fixational movements is by seeing via these movements. Seeing via eye movements also eliminates the instability of the image, which would be induced by them otherwise. Here we present a hypothesis for vision, in which coarse details are spatially encoded in gaze-related coordinates, and fine spatial details are temporally encoded in relative retinal coordinates. The temporal encoding presented here achieves its highest resolution by encoding along the elongated axes of simple-cell receptive fields and not across these axes as suggested by spatial models of vision. According to our hypothesis, fine details of shape are encoded by inter-receptor temporal phases, texture by instantaneous intra-burst rates of individual receptors, and motion by inter-burst temporal frequencies. We further describe the ability of the visual system to readout the encoded information and recode it internally. We show how reading out of retinal signals can be facilitated by neuronal phase-locked loops (NPLLs), which lock to the retinal jitter; this locking enables recoding of motion information and temporal framing of shape and texture processing. A possible implementation of this locking-and-recoding process by specific thalamocortical loops is suggested. Overall it is suggested that high-acuity vision is based primarily on temporal mechanisms of the sort presented here and low-acuity vision is based primarily on spatial mechanisms.

Neural selectivity and representation of gloss in the monkey inferior temporal cortex

When we view an object, its appearance depends in large part on specific surface reflectance properties; among these is surface gloss, which provides important information about the material composition of the object and the fine structure of its surface. To study how gloss is represented in the visual cortical areas related to object recognition, we examined the responses of neurons in the inferior temporal (IT) cortex of the macaque monkey to a set of object images exhibiting various combinations of specular reflection, diffuse reflection, and roughness, which are important physical parameters of surface gloss. We found that there are neurons in the lower bank of the superior temporal sulcus that selectively respond to specific gloss. This neuronal selectivity was largely maintained when the shape or illumination of the object was modified and perceived glossiness was unchanged. By contrast, neural responses were significantly altered when the pixels of the images were randomly rearranged, and perceived glossiness was dramatically changed. The stimulus preference of these neurons differed from cell to cell, and, as a population, they systematically represented a variety of surface glosses. We conclude that, within the visual cortex, there are mechanisms operating to integrate local image features and extract information about surface gloss and that this information is systematically represented in the IT cortex, an area playing an important role in object recognition.

Conjunctions between motion and disparity are encoded with the same spatial resolution as disparity alone

Neurons in cortical area MT respond well to transparent streaming motion in distinct depth planes, such as caused by observer self-motion, but do not contain subregions excited by opposite directions of motion. We therefore predicted that spatial resolution for transparent motion/disparity conjunctions would be limited by the size of MT receptive fields, just as spatial resolution for disparity is limited by the much smaller receptive fields found in primary visual cortex, V1. We measured this using a novel "joint motion/disparity grating," on which human observers detected motion/disparity conjunctions in transparent random-dot patterns containing dots streaming in opposite directions on two depth planes. Surprisingly, observers showed the same spatial resolution for these as for pure disparity gratings. We estimate the limiting receptive field diameter at 11 arcmin, similar to V1 and much smaller than MT. Higher internal noise for detecting joint motion/disparity produces a slightly lower high-frequency cutoff of 2.5 cycles per degree (cpd) versus 3.3 cpd for disparity. This suggests that information on motion/disparity conjunctions is available in the population activity of V1 and that this information can be decoded for perception even when it is invisible to neurons in MT.

Decoding conjunctions of direction-of-motion and binocular disparity from human visual cortex

Motion and binocular disparity are two features in our environment that share a common correspondence problem. Decades of psychophysical research dedicated to understanding stereopsis suggest that these features interact early in human visual processing to disambiguate depth. Single-unit recordings in the monkey also provide evidence for the joint encoding of motion and disparity across much of the dorsal visual stream. Here, we used functional MRI and multivariate pattern analysis to examine where in the human brain conjunctions of motion and disparity are encoded. Subjects sequentially viewed two stimuli that could be distinguished only by their conjunctions of motion and disparity. Specifically, each stimulus contained the same feature information (leftward and rightward motion and crossed and uncrossed disparity) but differed exclusively in the way these features were paired. Our results revealed that a linear classifier could accurately decode which stimulus a subject was viewing based on voxel activation patterns throughout the dorsal visual areas and as early as V2. This decoding success was conditional on some voxels being individually sensitive to the unique conjunctions comprising each stimulus, thus a classifier could not rely on independent information about motion and binocular disparity to distinguish these conjunctions. This study expands on evidence that disparity and motion interact at many levels of human visual processing, particularly within the dorsal stream. It also lends support to the idea that stereopsis is subserved by early mechanisms also tuned to direction of motion.

The Pulfrich effect in the clinic

First described in 1922 by Carl Pulfrich, the Pulfrich effect is a stereo-illusion thought to be caused by an inter-ocular signal latency difference stimulating neurons jointly tuned to disparity and motion. Clinically, this can be a spontaneous manifestation due to various ocular and central visual pathway pathologies, and cause symptoms independent of a range of routine visual parameters which may seem bizarre to both the patient and the clinician. Eliciting such symptoms of difficulties with motion and depth perception in a clinical history should direct the clinician to the possibility of the presence of the spontaneous Pulfrich effect, and to proceed to test for it. A spontaneous Pulfrich effect may be the first presentation of a previously undiagnosed condition, and warrants further investigation to identify the underlying cause. Alternatively, a spontaneous Pulfrich effect may be the unrecognized complication of a previously diagnosed condition. The symptoms associated with a spontaneous Pulfrich effect can be managed by treating the underlying cause in reversible conditions, or by prescribing appropriate monocular tinted lenses in front of the unaffected or less affected eye in persistent conditions.

A face feature space in the macaque temporal lobe

The ability of primates to effortlessly recognize faces has been attributed to the existence of specialized face areas. One such area, the macaque middle face patch, consists almost entirely of cells that are selective for faces, but the principles by which these cells analyze faces are unknown. We found that middle face patch neurons detect and differentiate faces using a strategy that is both part based and holistic. Cells detected distinct constellations of face parts. Furthermore, cells were tuned to the geometry of facial features. Tuning was most often ramp-shaped, with a one-to-one mapping of feature magnitude to firing rate. Tuning amplitude depended on the presence of a whole, upright face and features were interpreted according to their position in a whole, upright face. Thus, cells in the middle face patch encode axes of a face space specialized for whole, upright faces.

Pulfrich phenomena are coded effectively by a joint motion-disparity process

Pulfrich phenomena are a class of depth illusions generated by an interocular time delay. This may be demonstrated with continuously moving stimuli, stroboscopic displays undergoing apparent motion, or dynamic noise patterns. Previous studies suggest that neurons jointly tuned to motion and disparity may be responsible for the phenomena. Model cells with such joint coding can explain all Pulfrich phenomena in a unified way (N. Qian & R. A. Andersen, 1997). However, the joint-coding idea has been challenged by recent models (J. C. Read & B. G. Cumming, 2005a, 2005c) that focus on the S shaped functions of perceived disparity in stroboscopic Pulfrich effect (M. J. Morgan, 1979). Here we demonstrate fundamental problems with the recent models in terms of causality, physiological plausibility, and definitions for joint and separate coding, and we compare the two coding schemes under physiologically plausible assumptions. We show that joint coding of disparity and either unidirectional or bidirectional motion selectivity can account for the S curves, but unidirectional selectivity is required to explain direction-depth contingency in Pulfrich effects. In contrast, separate coding can explain neither the S curves nor the direction-depth contingency. We conclude that Pulfrich phenomena are logically accounted for by joint encoding of unidirectional-motion and disparity.

Evidence for joint encoding of motion and disparity in human visual perception

Electrophysiological recordings have established that motion and disparity signals are jointly encoded by subpopulations of neurons in visual cortex. However, the question of whether these neurons play a perceptual role has proven challenging and remains open. To answer this question we combined two powerful psychophysical techniques: perceptual adaptation and reverse correlation. Our results provide a detailed picture of how visual information about motion and disparity is processed by human observers, and how this processing is modified by prolonged sensory stimulation. We were able to isolate two perceptual components: a separable component, supported by separate motion and disparity signals, and an inseparable joint component, supported by motion and disparity signals that are concurrently represented at the level of the same neural mechanism. Both components are involved in the perception of stimuli containing motion and disparity information in line with the known existence of corresponding neuronal subpopulations in visual cortex.

Stimulus motion propels traveling waves in binocular rivalry

State transitions in the nervous system often take shape as traveling waves, whereby one neural state is replaced by another across space in a wave-like manner. In visual perception, transitions between the two mutually exclusive percepts that alternate when the two eyes view conflicting stimuli (binocular rivalry) may also take shape as traveling waves. The properties of these waves point to a neural substrate of binocular rivalry alternations that have the hallmark signs of lower cortical areas. In a series of experiments, we show a potent interaction between traveling waves in binocular rivalry and stimulus motion. The course of the traveling wave is biased in the motion direction of the suppressed stimulus that gains dominance by means of the wave-like transition. Thus, stimulus motion may propel the traveling wave across the stimulus to the extent that the stimulus motion dictates the traveling wave's direction completely. Using a computational model, we show that a speed-dependent asymmetry in lateral inhibitory connections between retinotopically organized and motion-sensitive neurons can explain our results. We argue that such a change in suppressive connections may play a vital role in the resolution of dynamic occlusion situations.

Acoustic Features of Rhesus Vocalizations and Their Representation in the Ventrolateral Prefrontal Cortex

Communication is one of the fundamental components of both human and nonhuman animal behavior. Auditory communication signals (i.e., vocalizations) are especially important in the socioecology of several species of nonhuman primates such as rhesus monkeys. In rhesus, the ventrolateral prefrontal cortex (vPFC) is thought to be part of a circuit involved in representing vocalizations and other auditory objects. To further our understanding of the role of the vPFC in processing vocalizations, we characterized the spectrotemporal features of rhesus vocalizations, compared these features with other classes of natural stimuli, and then related the rhesus-vocalization acoustic features to neural activity. We found that the range of these spectrotemporal features was similar to that found in other ensembles of natural stimuli, including human speech, and identified the subspace of these features that would be particularly informative to discriminate between different vocalizations. In a first neural study, however, we found that the tuning properties of vPFC neurons did not emphasize these particularly informative spectrotemporal features. In a second neural study, we found that a first-order linear model (the spectrotemporal receptive field) is not a good predictor of vPFC activity. The results of these two neural studies are consistent with the hypothesis that the vPFC is not involved in coding the first-order acoustic properties of a stimulus but is involved in processing the higher-order information needed to form representations of auditory objects.

Receptive versus perceptive fields from the reverse-correlation viewpoint

This brief review article brings together a series of related experiments in psychophysics and physiology that show striking similarities between measurements in human observers and in single neurons. We consider seven pairs of primary research articles, each pair consisting of one paper in physiology and one in psychophysics, and we highlight common features between receptive and perceptive fields obtained using reverse correlation. We conclude by discussing how to assess the validity of perceptive fields as predictors of human responses, and by deriving a novel expression for the maximum trial-by-trial predictability attainable by any model for any psychophysical task.

Spatial resolution for feature binding is impaired in peripheral and amblyopic vision

We measured spatial resolution for discriminating targets that differed from nearby distractors in either color or orientation or their conjunction. In the fovea of normal human observers, whenever both attributes are big enough to be individually visible, their conjunction is also visible. In the periphery, the two attributes may be visible, but their conjunction may be invisible. We found a similar impairment in resolving conjunctions for the fovea of deprived eyes of humans with abnormal visual development (amblyopia). These results are quantitatively explained by a model of primary visual cortex (V1) in which orientation and color maps are imperfectly co-registered topographically. Our results in persons with amblyopia indicate that the ability of the fovea to compensate for this poor co-registration is consolidated by visual experience during postnatal development.

Spatiotemporal structure of nonlinear subunits in macaque visual cortex

The primate visual system is arranged hierarchically, starting from the retina and continuing through a series of extrastriate visual areas. Selectivity for motion is first found in individual neurons in the primate visual cortex (V1), in which many simple cells respond selectively to the direction and speed of moving stimuli. Beyond simple cells, most studies of direction selectivity have focused on either V1 complex cells or neurons in the middle temporal area (MT/V5). To understand how visual information is transferred along this pathway, we have studied all three types of neurons, using a reverse correlation procedure to obtain high spatial and temporal resolution maps of activity for different motion stimuli. Most complex and MT cells showed strong second-order interactions, indicating that they were tuned for particular displacements of an apparent motion stimulus. The spatiotemporal structure of these interactions showed a high degree of similarity between the populations of V1 complex cells and MT cells, in terms of the spatiotemporal limits and preferences for motion and their two-dimensional spatial structure. Much of the structure in the V1 and MT second-order kernels could be accounted for on the basis of the first-order responses of V1 simple cells, under the assumption of a Reichardt or motion-energy type of computation.

Tuning for shape dimensions in macaque inferior temporal cortex

It is widely assumed that distributed bell-shaped tuning (e.g. Radial Basis functions) characterizes the shape selectivity of macaque inferior temporal (IT) neurons, analogous to the orientation or spatial frequency tuning found in early visual cortex. Demonstrating such tuning properties requires testing the responses of neurons for different values along dimensions of shape. We recorded the responses of single macaque IT neurons to variations of a rectangle and a triangle along simple shape dimensions, such as taper and axis curvature. The neurons showed systematic response modulation along these dimensions, with the greatest response, on average, to the highest values on the dimensions, e.g. to the most curved shapes. Within the range of values tested, the response functions were monotonic rather than bell-shaped. Multi-dimensional scaling of the neural responses showed that these simple shape dimensions were coded orthogonally by IT neurons: the degree and direction of responses modulation (i.e. the increase or decrease of responses along a dimension) was independent for the different dimensions. Furthermore, for combinations of curvature-related and other simple shape dimensions, the joint tuning was separable, that is well predicted by the product of the tuning for each of the dimensions. The independence of dimensional tuning may provide the neural basis for the independence of psychophysical judgements of multidimensional stimuli.