Image correlates of crowding in natural scenes

Visual crowding is the inability to identify visible features when they are surrounded by other structure in the peripheral field. Since natural environments are replete with structure and most of our visual field is peripheral, crowding represents the primary limit on vision in the real world. However, little is known about the characteristics of crowding under natural conditions. Here we examine where crowding occurs in natural images. Observers were required to identify which of four locations contained a patch of "dead leaves'' (synthetic, naturalistic contour structure) embedded into natural images. Threshold size for the dead leaves patch scaled with eccentricity in a manner consistent with crowding. Reverse correlation at multiple scales was used to determine local image statistics that correlated with task performance. Stepwise model selection revealed that local RMS contrast and edge density at the site of the dead leaves patch were of primary importance in predicting the occurrence of crowding once patch size and eccentricity had been considered. The absolute magnitudes of the regression weights for RMS contrast at different spatial scales varied in a manner consistent with receptive field sizes measured in striate cortex of primate brains. Our results are consistent with crowding models that are based on spatial averaging of features in the early stages of the visual system, and allow the prediction of where crowding is likely to occur in natural images.

Inferring single neuron properties in conductance based balanced networks

Balanced states in large networks are a usual hypothesis for explaining the variability of neural activity in cortical systems. In this regime the statistics of the inputs is characterized by static and dynamic fluctuations. The dynamic fluctuations have a Gaussian distribution. Such statistics allows to use reverse correlation methods, by recording synaptic inputs and the spike trains of ongoing spontaneous activity without any additional input. By using this method, properties of the single neuron dynamics that are masked by the balanced state can be quantified. To show the feasibility of this approach we apply it to large networks of conductance based neurons. The networks are classified as Type I or Type II according to the bifurcations which neurons of the different populations undergo near the firing onset. We also analyze mixed networks, in which each population has a mixture of different neuronal types. We determine under which conditions the intrinsic noise generated by the network can be used to apply reverse correlation methods. We find that under realistic conditions we can ascertain with low error the types of neurons present in the network. We also find that data from neurons with similar firing rates can be combined to perform covariance analysis. We compare the results of these methods (that do not requite any external input) to the standard procedure (that requires the injection of Gaussian noise into a single neuron). We find a good agreement between the two procedures.

Faces in the mist: illusory face and letter detection

We report three behavioral experiments on the spatial characteristics evoking illusory face and letter detection. False detections made to pure noise images were analyzed using a modified reverse correlation method in which hundreds of observers rated a modest number of noise images (480) during a single session. This method was originally developed for brain imaging research, and has been used in a number of fMRI publications, but this is the first report of the behavioral classification images. In Experiment 1 illusory face detection occurred in response to scattered dark patches throughout the images, with a bias to the left visual field. This occurred despite the use of a fixation cross and expectations that faces would be centered. In contrast, illusory letter detection (Experiment 2) occurred in response to centrally positioned dark patches. Experiment 3 included an oval in all displays to spatially constrain illusory face detection. With the addition of this oval the classification image revealed an eyes/nose/mouth pattern. These results suggest that face detection is triggered by a minimal face-like pattern even when these features are not centered in visual focus.

Global properties of natural scenes shape local properties of human edge detectors

Visual cortex analyzes images by first extracting relevant details (e.g., edges) via a large array of specialized detectors. The resulting edge map is then relayed to a processing pipeline, the final goal of which is to attribute meaning to the scene. As this process unfolds, does the global interpretation of the image affect how local feature detectors operate? We characterized the local properties of human edge detectors while we manipulated the extent to which the statistical properties of the surrounding image conformed to those encountered in natural vision. Although some aspects of local processing were unaffected by contextual manipulations, we observed significant alterations in the operating characteristics of the detector which were solely attributable to a higher-level semantic interpretation of the scene, unrelated to lower-level aspects of image statistics. Our results suggest that it may be inaccurate to regard early feature detectors as operating outside the domain of higher-level vision; although there is validity in this approach, a full understanding of their properties requires the inclusion of knowledge-based effects specific to the statistical regularities found in the natural environment.

Visual detection under uncertainty operates via an early static, not late dynamic, non-linearity

SIGNALS IN THE ENVIRONMENT ARE RARELY SPECIFIED EXACTLY: our visual system may know what to look for (e.g., a specific face), but not its exact configuration (e.g., where in the room, or in what orientation). Uncertainty, and the ability to deal with it, is a fundamental aspect of visual processing. The MAX model is the current gold standard for describing how human vision handles uncertainty: of all possible configurations for the signal, the observer chooses the one corresponding to the template associated with the largest response. We propose an alternative model in which the MAX operation, which is a dynamic non-linearity (depends on multiple inputs from several stimulus locations) and happens after the input stimulus has been matched to the possible templates, is replaced by an early static non-linearity (depends only on one input corresponding to one stimulus location) which is applied before template matching. By exploiting an integrated set of analytical and experimental tools, we show that this model is able to account for a number of empirical observations otherwise unaccounted for by the MAX model, and is more robust with respect to the realistic limitations imposed by the available neural hardware. We then discuss how these results, currently restricted to a simple visual detection task, may extend to a wider range of problems in sensory processing.

Psychophysical reverse correlation with multiple response alternatives

Psychophysical reverse-correlation methods such as the "classification image" technique provide a unique tool to uncover the internal representations and decision strategies of individual participants in perceptual tasks. Over the past 30 years, these techniques have gained increasing popularity among both visual and auditory psychophysicists. However, thus far, principled applications of the psychophysical reverse-correlation approach have been almost exclusively limited to two-alternative decision (detection or discrimination) tasks. Whether and how reverse-correlation methods can be applied to uncover perceptual templates and decision strategies in situations involving more than just two response alternatives remain largely unclear. Here, the authors consider the problem of estimating perceptual templates and decision strategies in stimulus identification tasks with multiple response alternatives. They describe a modified correlational approach, which can be used to solve this problem. The approach is evaluated under a variety of simulated conditions, including different ratios of internal-to-external noise, different degrees of correlations between the sensory observations, and various statistical distributions of stimulus perturbations. The results indicate that the proposed approach is reasonably robust, suggesting that it could be used in future empirical studies.

Subpopulations of neurons in visual area v2 perform differentiation and integration operations in space and time

The interconnected areas of the visual system work together to find object boundaries in visual scenes. Primary visual cortex (V1) mainly extracts oriented luminance boundaries, while secondary visual cortex (V2) also detects boundaries defined by differences in texture. How the outputs of V1 neurons are combined to allow for the extraction of these more complex boundaries in V2 is as of yet unclear. To address this question, we probed the processing of orientation signals in single neurons in V1 and V2, focusing on response dynamics of neurons to patches of oriented gratings and to combinations of gratings in neighboring patches and sequential time frames. We found two kinds of response dynamics in V2, both of which were different from those of V1 neurons. While V1 neurons in general preferred one orientation, one subpopulation of V2 neurons (“transient”) showed a temporally dynamic preference, resulting in a preference for changes in orientation. The second subpopulation of V2 neurons (“sustained”) responded similarly to V1 neurons, but with a delay. The dynamics of nonlinear responses to combinations of gratings reinforced these distinctions: the dynamics enhanced the preference of V1 neurons for continuous orientations and the preference of V2 transient neurons for discontinuous ones. We propose that transient neurons in V2 perform a differentiation operation on the V1 input, both spatially and temporally, while the sustained neurons perform an integration operation. We show that a simple feedforward network with delayed inhibition can account for the temporal but not for the spatial differentiation operation.

Natural stimulus statistics alter the receptive field structure of v1 neurons

Studies of the primary visual cortex (V1) have produced models that account for neuronal responses to synthetic stimuli such as sinusoidal gratings. Little is known about how these models generalize to activity during natural vision. We recorded neural responses in area V1 of awake macaques to a stimulus with natural spatiotemporal statistics and to a dynamic grating sequence stimulus. We fit nonlinear receptive field models using each of these data sets and compared how well they predicted time-varying responses to a novel natural visual stimulus. On average, the model fit using the natural stimulus predicted natural visual responses more than twice as accurately as the model fit to the synthetic stimulus. The natural vision model produced better predictions in >75% of the neurons studied. This large difference in predictive power suggests that natural spatiotemporal stimulus statistics activate nonlinear response properties in a different manner than the grating stimulus. To characterize this modulation, we compared the temporal and spatial response properties of the model fits. During natural stimulation, temporal responses often showed a stronger late inhibitory component, indicating an effect of nonlinear temporal summation during natural vision. In addition, spatial tuning underwent complex shifts, primarily in the inhibitory, rather than excitatory, elements of the response profile. These differences in late and spatially tuned inhibition accounted fully for the difference in predictive power between the two models. Both the spatial and temporal statistics of the natural stimulus contributed to the modulatory effects.

Naturalistic auditory contrast improves spectrotemporal coding in the cat inferior colliculus

Statistical analysis of natural sounds and speech reveals logarithmically distributed spectrotemporal modulations that can cover several orders of magnitude. By contrast, most artificial stimuli used to probe auditory function, including pure tones and white noise, have linearly distributed amplitude fluctuations with a limited average dynamic range. Here we explore whether the operating range of the auditory system is physically matched to the statistical structure of natural sounds. We recorded single-unit and multi-unit neuronal activity from the central nucleus of the cat inferior colliculus (ICC) in response to dynamic spectrotemporal sound sequences to determine whether ICC neurons respond preferentially to linear or logarithmic spectrotemporal amplitudes. We varied the intensity, dynamic range, and contrast statistics of these sounds to mimic those of natural and artificial stimuli. ICC neurons exhibited monotonic and nonmonotonic contrast dependencies with increasing dynamic range that were independent of the stimulus intensity. Midbrain neurons had higher firing rates and higher receptive field energies and showed a net improvement in spectrotemporal encoding ability for logarithmic stimuli, with an increase in the mutual information rate of approximately 50% over linear amplitude sounds. This efficient use of logarithmic spectrotemporal modulations by auditory midbrain neurons reflects a neural adaptation to structural regularities in natural sounds and likely underlies human perceptual abilities.

The receptive-field organization of simple cells in primary visual cortex of ferrets under natural scene stimulation

The responses of simple cells in primary visual cortex to sinusoidal gratings can primarily be predicted from their spatial receptive fields, as mapped using spots or bars. Although this quasilinearity is well documented, it is not clear whether it holds for complex natural stimuli. We recorded from simple cells in the primary visual cortex of anesthetized ferrets while stimulating with flashed digitized photographs of natural scenes. We applied standard reverse-correlation methods to quantify the average natural stimulus that invokes a neuronal response. Although these maps cannot be the receptive fields, we find that they still predict the preferred orientation of grating for each cell very well (r = 0.91); they do not predict the spatial-frequency tuning. Using a novel application of the linear reconstruction method called regularized pseudoinverse, we were able to recover high-resolution receptive-field maps from the responses to a relatively small number of natural scenes. These receptive-field maps not only predict the optimum orientation of each cell (r = 0.96) but also the spatial-frequency optimum (r = 0.89); the maps also predict the tuning bandwidths of many cells. Therefore, our first conclusion is that the tuning preferences of the cells are primarily linear and constant across stimulus type. However, when we used these maps to predict the actual responses of the cells to natural scenes, we did find evidence of expansive output nonlinearity and nonlinear influences from outside the classical receptive fields, orientation tuning, and spatial-frequency tuning.

Frequency-specific interaural level difference tuning predicts spatial response patterns of space-specific neurons in the barn owl inferior colliculus

Space-specific neurons in the barn owl's inferior colliculus have spatial receptive fields (RFs) because of sensitivity to interaural time difference and frequency-specific interaural level difference (ILD). These neurons are assumed to be tuned to the frequency-specific ILDs occurring at their spatial RFs, but attempts to assess this tuning with traditional narrowband stimuli have had limited success. Indeed, tuning assessed in this manner, when processed via a linear model of spectral integration, typically explains only approximately half the variance in spatial response patterns. Here we report our findings that frequency-specific ILD tuning of space-specific neurons, when assessed from responses to broadband stimuli, predicted nearly 75% of the variance in spatial responses, using a linear model of spectral integration (p < 0.0001; n = 97 neurons). Furthermore, when we tested neurons using only those frequencies we found to be spatially relevant, we saw that their responses were similar to those elicited by broadband stimuli. When we used frequencies not identified as spatially relevant, such similarity was lacking. Furthermore, spectral components that elicited high firing rates when presented as narrowband stimuli were found in several cases to be irrelevant for or detrimental to the definition of spatial RFs. Thus, neurons achieved sharp spatial tuning by selecting for ILDs of a subset of spectral components in noise, some of which were not identified using narrowband stimuli.

Spectral-temporal receptive fields of nonlinear auditory neurons obtained using natural sounds

The stimulus-response function of many visual and auditory neurons has been described by a spatial-temporal receptive field (STRF), a linear model that for mathematical reasons has until recently been estimated with the reverse correlation method, using simple stimulus ensembles such as white noise. Such stimuli, however, often do not effectively activate high-level sensory neurons, which may be optimized to analyze natural sounds and images. We show that it is possible to overcome the simple-stimulus limitation and then use this approach to calculate the STRFs of avian auditory forebrain neurons from an ensemble of birdsongs. We find that in many cases the STRFs derived using natural sounds are strikingly different from the STRFs that we obtained using an ensemble of random tone pips. When we compare these two models by assessing their predictions of neural response to the actual data, we find that the STRFs obtained from natural sounds are superior. Our results show that the STRF model is an incomplete description of response properties of nonlinear auditory neurons, but that linear receptive fields are still useful models for understanding higher level sensory processing, as long as the STRFs are estimated from the responses to relevant complex stimuli.

Spatial and temporal structure of receptive fields in primate somatosensory area 3b: effects of stimulus scanning direction and orientation

This is the third in a series of studies of the neural representation of tactile spatial form in somatosensory cortical area 3b of the alert monkey. We previously studied the spatial structure of >350 fingerpad receptive fields (RFs) with random-dot patterns scanned in one direction () and at varying velocities (). Those studies showed that area 3b RFs have a wide range of spatial structures that are virtually unaffected by changes in scanning velocity. In this study, 62 area 3b neurons were studied with three to eight scanning directions (58 with four or more directions). The data from all three studies are described accurately by an RF model with three components: (1) a single, central excitatory region of short duration, (2) one or more inhibitory regions, also of short duration, that are adjacent to and nearly synchronous with the excitation, and (3) a region of inhibition that overlaps the excitation partially or totally and is temporally delayed with respect to the first two components. The mean correlation between the observed RFs and the RFs predicted by this three-component model was 0.81. The three-component RFs also predicted orientation sensitivity and preferred orientation to a scanned bar accurately. The orientation sensitivity was determined most strongly by the intensity of the coincident RF inhibition in relation to the excitation. Both orientation sensitivity and this ratio were stronger in the supragranular and infragranular layers than in layer IV.

Spatial summation in the receptive fields of MT neurons

Receptive fields (RFs) of cells in the middle temporal area (MT or V5) of monkeys will often encompass multiple objects under normal image viewing. We therefore have studied how multiple moving stimuli interact when presented within and near the RF of single MT cells. We used moving Gabor function stimuli, <1 degrees in spatial extent and approximately 100 msec in duration, presented on a grid of possible locations over the RF of the cell. Responses to these stimuli were typically robust, and their small spatial and temporal extent allowed detailed mapping of RFs and of interactions between stimuli. The responses to pairs of such stimuli were compared against the responses to the same stimuli presented singly. The responses were substantially less than the sum of the responses to the component stimuli and were well described by a power-law summation model with divisive inhibition. Such divisive inhibition is a key component of recently proposed "normalization" models of cortical physiology and is presumed to arise from lateral interconnections within a region. One open question is whether the normalization occurs only once in primary visual cortex or multiple times in different cortical areas. We addressed this question by exploring the spatial extent over which one stimulus would divide the response to another and found effective normalization from stimuli quite far removed from the RF center. This supports models under which normalization occurs both in MT and in earlier stages.

Functional micro-organization of primary visual cortex: receptive field analysis of nearby neurons

It is well established that multiple stimulus dimensions (e.g., orientation and spatial frequency) are mapped onto the surface of striate cortex. However, the detailed organization of neurons within a local region of striate cortex remains unclear. Within a vertical column, do all neurons have the same response selectivities? And if not, how do they most commonly differ and why? To address these questions, we recorded from nearby pairs of simple cells and made detailed spatiotemporal maps of their receptive fields. From these maps, we extracted and analyzed a variety of response metrics. Our results provide new insights into the local organization of striate cortex. First, we show that nearby neurons seldom have very similar receptive fields, when these fields are characterized in space and time. Thus, there may be less redundancy within a column than previously thought. Moreover, we show that correlated discharge increases with receptive field similarity; thus, the local dissimilarity between neurons may allow for noise reduction by response pooling. Second, we show that several response variables are clustered within striate cortex, including some that have not received much attention such as response latency and temporal frequency. We also demonstrate that other parameters are not clustered, including the spatial phase (or symmetry) of the receptive field. Third, we show that spatial phase is the single parameter that accounts for most of the difference between receptive fields of nearby neurons. We consider the implications of this local diversity of spatial phase for population coding and construction of higher-order receptive fields.

Velocity invariance of receptive field structure in somatosensory cortical area 3b of the alert monkey

This is the second in a series of studies of the neural representation of tactile spatial form in cortical area 3b of the alert monkey. We previously studied the spatial structure of 330 area 3b neuronal receptive fields (RFs) on the fingerpad with random dot patterns scanned at one velocity (40 mm/sec; ). Here, we analyze the temporal structure of 84 neuronal RFs by studying their spatial structure at three scanning velocities (20, 40, and 80 mm/sec). As in the previous study, most RFs contained a single, central, excitatory region and one or more surrounding or flanking inhibitory regions. The mean time delay between skin stimulation and its excitatory effect was 15.5 msec. Except for differences in mean rate, each neuron's response and the spatial structure of its RF were essentially unaffected by scanning velocity. This is the expected outcome when excitatory and inhibitory effects are brief and synchronous. However, that interpretation is consistent neither with the reported timing of excitation and inhibition in somatosensory cortex nor with the third study in this series, which investigates the effect of scanning direction and shows that one component of inhibition lags behind excitation. We reconcile these observations by showing that overlapping (in-field) inhibition delayed relative to excitation can produce RF spatial structure that is unaffected by changes in scanning velocity. Regardless of the mechanisms, the velocity invariance of area 3b RF structure is consistent with the velocity invariance of tactile spatial perception (e.g., roughness estimation and form recognition).

Structure of receptive fields in area 3b of primary somatosensory cortex in the alert monkey

We investigated the two-dimensional structure of area 3b neuronal receptive fields (RFs) in three alert monkeys. Three hundred thirty neurons with RFs on the distal fingerpads were studied with scanned, random dot stimuli. Each neuron was stimulated continuously for 14 min, yielding 20,000 response data points. Excitatory and inhibitory components of each RF were determined with a modified linear regression algorithm. Analyses assessing goodness-of-fit, repeatability, and generality of the RFs were developed. Two hundred forty-seven neurons yielded highly repeatable RF estimates, and most RFs accounted for a large fraction of the explainable response of each neuron. Although the area 3b RF structures appeared to be continuously distributed, certain structural generalities were apparent. Most RFs (94%) contained a single, central region of excitation and one or more regions of inhibition located on one, two, three, or all four sides of the excitatory center. The shape, area, and strength of excitatory and inhibitory RF regions ranged widely. Half the RFs contained almost evenly balanced excitation and inhibition. The findings indicate that area 3b neurons act as local spatiotemporal filters that are maximally excited by the presence of particular stimulus features. We believe that form and texture perception are based on high-level representations and that area 3b is an intermediate stage in the processes leading to these representations. Two possibilities are considered: (1) that these high-level representations are basically somatotopic and that area 3b neurons amplify some features and suppress others, or (2) that these representations are highly transformed and that area 3b effects a step in the transformation.