Mapping of contextual modulation in the population response of primary visual cortex

Divergences in color perception between deep neural networks and humans

Deep neural networks (DNNs) are increasingly proposed as models of human vision, bolstered by their impressive performance on image classification and object recognition tasks. Yet, the extent to which DNNs capture fundamental aspects of human vision such as color perception remains unclear. Here, we develop novel experiments for evaluating the perceptual coherence of color embeddings in DNNs, and we assess how well these algorithms predict human color similarity judgments collected via an online survey. We find that state-of-the-art DNN architectures - including convolutional neural networks and vision transformers - provide color similarity judgments that strikingly diverge from human color judgments of (i) images with controlled color properties, (ii) images generated from online searches, and (iii) real-world images from the canonical CIFAR-10 dataset. We compare DNN performance against an interpretable and cognitively plausible model of color perception based on wavelet decomposition, inspired by foundational theories in computational neuroscience. While one deep learning model - a convolutional DNN trained on a style transfer task - captures some aspects of human color perception, our wavelet algorithm provides more coherent color embeddings that better predict human color judgments compared to all DNNs we examine. These results hold when altering the high-level visual task used to train similar DNN architectures (e.g., image classification versus image segmentation), as well as when examining the color embeddings of different layers in a given DNN architecture. These findings break new ground in the effort to analyze the perceptual representations of machine learning algorithms and to improve their ability to serve as cognitively plausible models of human vision. Implications for machine learning, human perception, and embodied cognition are discussed.

Deep problems with neural network models of human vision

Deep neural networks (DNNs) have had extraordinary successes in classifying photographic images of objects and are often described as the best models of biological vision. This conclusion is largely based on three sets of findings: (1) DNNs are more accurate than any other model in classifying images taken from various datasets, (2) DNNs do the best job in predicting the pattern of human errors in classifying objects taken from various behavioral datasets, and (3) DNNs do the best job in predicting brain signals in response to images taken from various brain datasets (e.g., single cell responses or fMRI data). However, these behavioral and brain datasets do not test hypotheses regarding what features are contributing to good predictions and we show that the predictions may be mediated by DNNs that share little overlap with biological vision. More problematically, we show that DNNs account for almost no results from psychological research. This contradicts the common claim that DNNs are good, let alone the best, models of human object recognition. We argue that theorists interested in developing biologically plausible models of human vision need to direct their attention to explaining psychological findings. More generally, theorists need to build models that explain the results of experiments that manipulate independent variables designed to test hypotheses rather than compete on making the best predictions. We conclude by briefly summarizing various promising modeling approaches that focus on psychological data.

Illusory Essences: A Bias Holding Back Theorizing in Psychological Science

The reliance in psychology on verbal definitions means that psychological research is unusually moored to how humans think and communicate about categories. Psychological concepts (e.g., intelligence, attention) are easily assumed to represent objective, definable categories with an underlying essence. Like the “vital forces” previously thought to animate life, these assumed essences can create an illusion of understanding. By synthesizing a wide range of research lines from cognitive, clinical, and biological psychology and neuroscience, we describe a pervasive tendency across psychological science to assume that essences explain phenomena. Labeling a complex phenomenon can appear as theoretical progress before there is sufficient evidence that the described category has a definable essence or known boundary conditions. Category labels can further undermine progress by masking contingent and contextual relationships and obscuring the need to specify mechanisms. Finally, we highlight examples of promising methods that circumvent the lure of essences and suggest four concrete strategies for identifying and avoiding essentialist intuitions in theory development.

A Disinhibitory Circuit for Contextual Modulation in Primary Visual Cortex

Context guides perception by influencing stimulus saliency. Accordingly, in visual cortex, responses to a stimulus are modulated by context, the visual scene surrounding the stimulus. Responses are suppressed when stimulus and surround are similar but not when they differ. The underlying mechanisms remain unclear. Here, we use optical recordings, manipulations, and computational modeling to show that disinhibitory circuits consisting of vasoactive intestinal peptide (VIP)-expressing and somatostatin (SOM)-expressing inhibitory neurons modulate responses in mouse visual cortex depending on similarity between stimulus and surround, primarily by modulating recurrent excitation. When stimulus and surround are similar, VIP neurons are inactive, and activity of SOM neurons leads to suppression of excitatory neurons. However, when stimulus and surround differ, VIP neurons are active, inhibiting SOM neurons, which leads to relief of excitatory neurons from suppression. We have identified a canonical cortical disinhibitory circuit that contributes to contextual modulation and may regulate perceptual saliency.

Modulating the global orientation bias of the visual system changes population receptive field elongations

The topographical structure of the visual system in individual subjects can be visualized using fMRI. Recently, a radial bias for the long axis of population receptive fields (pRF) has been shown using fMRI. It has been theorized that the elongation of receptive fields pointing toward the fovea results from horizontal local connections bundling orientation selective units mostly parallel to their polar position within the visual field. In order to investigate whether there is a causal relationship between orientation selectivity and pRF elongation the current study employed a global orientation adapter to modulate the orientation bias for the visual system while measuring spatial pRF characteristics. The hypothesis was that the orientation tuning change of neural populations would alter pRF elongations toward the fovea particularly at axial positions parallel and orthogonal to the affected orientation. The results indeed show a different amount of elongation of pRF units and their orientation at parallel and orthogonal axial positions relative to the adapter orientation. Within the lower left hemifield, pRF radial bias and elongation showed an increase during adaptation to a 135° grating while both parameters decreased during the presentation of a 45° adapter stimulus. The lower right visual field showed the reverse pattern. No modulation of the pRF topographies were observed in the upper visual field probably due to a vertical visual field asymmetry of sensitivity toward the low contrast spatial frequency pattern of the adapter stimulus. These data suggest a direct relationship between orientation selectivity and elongation of population units within the visual cortex.

Spatial elongation of population receptive field profiles revealed by model-free fMRI back-projection

Estimates of visual field topographies in human visual cortex obtained through fMRI traveling wave techniques usually provide the parameters of population receptive field (pRF) location (polar angle, eccentricity) and receptive field size. These parameters are obtained by fitting the recorded data to a standard model population receptive field. In this work, pRF profiles are measured directly by back-projecting preprocessed fMRI time-series to sweeps of a bar across the visual field in different angles. The current data suggest that the model-free pRF profiles contain information not only about receptive field location and size but also about the pRF shape characteristics. The elongation (ellipticity) of pRFs decreases along the early visual hierarchy to a different degree for the ventral and the dorsal stream. Furthermore, ellipticity changes as a function of eccentricity. pRF orientation shows a high degree of collinearity with its angular position within the visual field. Using model-free pRF measurements, the traveling wave technique provides additional characteristics of pRF topographies that are not restricted to size and provide robust measures within the single subject.

Is neuroimaging measuring information in the brain?

Psychology moved beyond the stimulus response mapping of behaviorism by adopting an information processing framework. This shift from behavioral to cognitive science was partly inspired by work demonstrating that the concept of information could be defined and quantified (Shannon, 1948). This transition developed further from cognitive science into cognitive neuroscience, in an attempt to measure information in the brain. In the cognitive neurosciences, however, the term information is often used without a clear definition. This paper will argue that, if the formulation proposed by Shannon is applied to modern neuroimaging, then numerous results would be interpreted differently. More specifically, we argue that much modern cognitive neuroscience implicitly focuses on the question of how we can interpret the activations we record in the brain (experimenter-as-receiver), rather than on the core question of how the rest of the brain can interpret those activations (cortex-as-receiver). A clearer focus on whether activations recorded via neuroimaging can actually act as information in the brain would not only change how findings are interpreted but should also change the direction of empirical research in cognitive neuroscience.

Contrast and response gain control depend on cortical map architecture

Visual cortical neurons are sensitive to visual stimulus contrast and most cells adapt their sensitivity to the prevailing visual environment. Specifically, they match the steepest region of their contrast response function to the prevailing contrast (contrast gain control), and reduce spike rates to limit saturation (response gain control). Most neurons are also tuned for stimulus orientation, and neurons with similar orientation preference are clustered together into iso-orientation zones arranged around pinwheels, i.e. points where all orientations are represented. Here we investigated the relationship between the contrast adaptation properties of neurons and their location relative to pinwheels in the orientation preference map. We measured orientation preference maps in cat cortex using optical intrinsic signal imaging. We then characterized the contrast adaptation properties of single neurons located close to pinwheels, in iso-orientation zones, and at regions in between. We found little evidence of differential contrast sensitivity of neurons adapted to zero contrast. However, after adaptation to their preferred orientation at high contrast, changes in both contrast and response gain were greater for neurons near pinwheels compared with other map regions. Therefore, in the adapted state, which is probably typical during natural viewing, there is a spatial map of contrast sensitivity that is associated with the orientation preference map. This differential adaptation revealed a new dimension of cortical functional organization, linking the contrast adaptation of cells with the orientation preference of their nearest neighbours.

Long-range cortical connections give rise to a robust velocity map of V1

This paper proposes a two-dimensional velocity model (2DVM) of the primary visual cortex (V1). The model's novel aspect is that it specifies a particular pattern of long-range cortical temporal connections, via the Connection Algorithm, and shows how the addition of these connections to well-known spatial properties of V1 transforms V1 into a velocity map. The map implies a number of organizational properties of V1: (1) the singularity of each orientation pinwheel contributes to the detection of slow-moving spots across the visual field; (2) the speed component of neuronal velocity selectivity decreases monotonically across each joint orientation contour line for parallel motion and increases monotonically for orthogonal motion; (3) the cells that are direction selective to slow-moving objects are situated in the periphery of V1; and (4) neurons in distinct pinwheels tend to be connected to neurons with similar tuning preferences in other pinwheels. The model accounts for various types of known illusionary perceptions of human vision: perceptual filling-in, illusionary orientation and visual crowding. The three distinguishing features of 2DVM are: (1) it unifies the functional properties of the conventional energy model of V1; (2) it directly relates the functional properties to the known structure of the upper layers of V1; and (3) it implies that the spatial selectivity features of V1 are side effects of its more important role as a velocity map of the visual field.

Distributed processing of color and form in the visual cortex

To what extent does the visual system process color and form separately? Proponents of the segregation view claim that distinct regions of the cortex are dedicated to each of these two dimensions separately. However, evidence is accumulating that color and form processing may, at least to some extent, be intertwined in the brain. In this perspective, we review psychophysical and neurophysiological studies on color and form perception and evaluate their results in light of recent developments in population coding.

A computational neural model of orientation detection based on multiple guesses: comparison of geometrical and algebraic models

The implementation of Hubel-Wiesel hypothesis that orientation selectivity of a simple cell is based on ordered arrangement of its afferent cells has some difficulties. It requires the receptive fields (RFs) of those ganglion cells (GCs) and LGN cells to be similar in size and sub-structure and highly arranged in a perfect order. It also requires an adequate number of regularly distributed simple cells to match ubiquitous edges. However, the anatomical and electrophysiological evidence is not strong enough to support this geometry-based model. These strict regularities also make the model very uneconomical in both evolution and neural computation. We propose a new neural model based on an algebraic method to estimate orientations. This approach synthesizes the guesses made by multiple GCs or LGN cells and calculates local orientation information subject to a group of constraints. This algebraic model need not obey the constraints of Hubel-Wiesel hypothesis, and is easily implemented with a neural network. By using the idea of a satisfiability problem with constraints, we also prove that the precision and efficiency of this model are mathematically practicable. The proposed model makes clear several major questions which Hubel-Wiesel model does not account for. Image-rebuilding experiments are conducted to check whether this model misses any important boundary in the visual field because of the estimation strategy. This study is significant in terms of explaining the neural mechanism of orientation detection, and finding the circuit structure and computational route in neural networks. For engineering applications, our model can be used in orientation detection and as a simulation platform for cell-to-cell communications to develop bio-inspired eye chips.

Interactions between motion and form processing in the human visual system

The predominant view of motion and form processing in the human visual system assumes that these two attributes are handled by separate and independent modules. Motion processing involves filtering by direction-selective sensors, followed by integration to solve the aperture problem. Form processing involves filtering by orientation-selective and size-selective receptive fields, followed by integration to encode object shape. It has long been known that motion signals can influence form processing in the well-known Gestalt principle of common fate; texture elements which share a common motion property are grouped into a single contour or texture region. However, recent research in psychophysics and neuroscience indicates that the influence of form signals on motion processing is more extensive than previously thought. First, the salience and apparent direction of moving lines depends on how the local orientation and direction of motion combine to match the receptive field properties of motion-selective neurons. Second, orientation signals generated by "motion-streaks" influence motion processing; motion sensitivity, apparent direction and adaptation are affected by simultaneously present orientation signals. Third, form signals generated by human body shape influence biological motion processing, as revealed by studies using point-light motion stimuli. Thus, form-motion integration seems to occur at several different levels of cortical processing, from V1 to STS.

A century of Gestalt psychology in visual perception: II. Conceptual and theoretical foundations

Our first review article (Wagemans et al., 2012) on the occasion of the centennial anniversary of Gestalt psychology focused on perceptual grouping and figure-ground organization. It concluded that further progress requires a reconsideration of the conceptual and theoretical foundations of the Gestalt approach, which is provided here. In particular, we review contemporary formulations of holism within an information-processing framework, allowing for operational definitions (e.g., integral dimensions, emergent features, configural superiority, global precedence, primacy of holistic/configural properties) and a refined understanding of its psychological implications (e.g., at the level of attention, perception, and decision). We also review 4 lines of theoretical progress regarding the law of Prägnanz-the brain's tendency of being attracted towards states corresponding to the simplest possible organization, given the available stimulation. The first considers the brain as a complex adaptive system and explains how self-organization solves the conundrum of trading between robustness and flexibility of perceptual states. The second specifies the economy principle in terms of optimization of neural resources, showing that elementary sensors working independently to minimize uncertainty can respond optimally at the system level. The third considers how Gestalt percepts (e.g., groups, objects) are optimal given the available stimulation, with optimality specified in Bayesian terms. Fourth, structural information theory explains how a Gestaltist visual system that focuses on internal coding efficiency yields external veridicality as a side effect. To answer the fundamental question of why things look as they do, a further synthesis of these complementary perspectives is required.

A century of Gestalt psychology in visual perception: I. Perceptual grouping and figure-ground organization

In 1912, Max Wertheimer published his paper on phi motion, widely recognized as the start of Gestalt psychology. Because of its continued relevance in modern psychology, this centennial anniversary is an excellent opportunity to take stock of what Gestalt psychology has offered and how it has changed since its inception. We first introduce the key findings and ideas in the Berlin school of Gestalt psychology, and then briefly sketch its development, rise, and fall. Next, we discuss its empirical and conceptual problems, and indicate how they are addressed in contemporary research on perceptual grouping and figure-ground organization. In particular, we review the principles of grouping, both classical (e.g., proximity, similarity, common fate, good continuation, closure, symmetry, parallelism) and new (e.g., synchrony, common region, element and uniform connectedness), and their role in contour integration and completion. We then review classic and new image-based principles of figure-ground organization, how it is influenced by past experience and attention, and how it relates to shape and depth perception. After an integrated review of the neural mechanisms involved in contour grouping, border ownership, and figure-ground perception, we conclude by evaluating what modern vision science has offered compared to traditional Gestalt psychology, whether we can speak of a Gestalt revival, and where the remaining limitations and challenges lie. A better integration of this research tradition with the rest of vision science requires further progress regarding the conceptual and theoretical foundations of the Gestalt approach, which is the focus of a second review article.

Predicted contextual modulation varies with distance from pinwheel centers in the orientation preference map

In the primary visual cortex (V1) of some mammals, columns of neurons with the full range of orientation preferences converge at the center of a pinwheel-like arrangement, the ‘pinwheel center' (PWC). Because a neuron receives abundant inputs from nearby neurons, the neuron's position on the cortical map likely has a significant impact on its responses to the layout of orientations inside and outside its classical receptive field (CRF). To understand the positional specificity of responses, we constructed a computational model based on orientation preference maps in monkey V1 and hypothetical neuronal connections. The model simulations showed that neurons near PWCs displayed weaker but detectable orientation selectivity within their CRFs, and strongly reduced contextual modulation from extra-CRF stimuli, than neurons distant from PWCs. We suggest that neurons near PWCs robustly extract local orientation within their CRF embedded in visual scenes, and that contextual information is processed in regions distant from PWCs.

Generalization of learning by synchronous waves: from perceptual organization to invariant organization

From a few presentations of an object, perceptual systems are able to extract invariant properties such that novel presentations are immediately recognized. This may be enabled by inferring the set of all representations equivalent under certain transformations. We implemented this principle in a neurodynamic model that stores activity patterns representing transformed versions of the same object in a distributed fashion within maps, such that translation across the map corresponds to the relevant transformation. When a pattern on the map is activated, this causes activity to spread out as a wave across the map, activating all the transformed versions represented. Computational studies illustrate the efficacy of the proposed mechanism. The model rapidly learns and successfully recognizes rotated and scaled versions of a visual representation from a few prior presentations. For topographical maps such as primary visual cortex, the mechanism simultaneously represents identity and variation of visual percepts whose features change through time.