Coding and binding of color and form in visual cortex

Isolating the impact of a visual search template's color and form information on search guidance and verification times

Visual search can be guided by biasing one's attention towards features associated with a target. Prior work has shown that high-fidelity, picture-based cues are more beneficial to search than text-based cues. However, typically picture cues provide both detailed form information and color information that is absent from text-based cues. Given that visual resolution deteriorates with eccentricity, it is not clear that high-fidelity form information would benefit guidance to peripheral objects - much of the picture benefit could be due to color information alone. To address this, we conducted a search task with eye-tracking that had four types of cues that comprised a 2 (text/pictorial cue) × 2 (no color/color) design. We hypothesized that color information would be important for efficient search guidance while high-fidelity form information would be important for efficient verification times. In Experiment 1 cues were a colored picture of the target, a gray-scaled picture of the target, a text-based cue that included color (e.g., "blue shoe"), or a text-based cue without color (e.g., "shoe"). Experiment 2 was a replication of Experiment 1, except that the color word in the text-based cue was presented in the precise color that was the dominant color in the target. Our results show that high-fidelity form information is important for efficient verifications times (with color playing less of a role) and color is important for efficient guidance, but form information also benefits guidance. These results suggest that different features of the cue independently contribute to different aspects of the search process.

Orienting of attention and spatial cognition

Humans orient to their sensory world through foveation of target location or through covert shifts of attention. Orienting provides primacy to the selected location and in humans improves the precision of discrimination. Covert orienting appears to arise separately from the mechanisms involved in saccadic eye movements. Covert orienting can serve to prioritize processing the target even increasing its subjective intensity and its acuity. However, this network does not appear to be involved in the operations related to binding and segmentation. Cells exist in the early visual cortex that are activated by both color and form features without attention, however, color and form appear to remain independent even when oriented to the target that is required to be reported. An understanding of the pathways that connect attention networks to memory networks may allow us to understand more complex aspects of spatial cognition and enhance orienting and thus improve spatial cognition.

Tomatoes Are Red: The Perception of Achromatic Objects Elicits Retrieval of Associated Color Knowledge

When preparing to name an object, semantic knowledge about the object and its attributes is activated, including perceptual properties. It is unclear, however, whether semantic attribute activation contributes to lexical access or is a consequence of activating a concept irrespective of whether that concept is to be named or not. In this study, we measured neural responses using fMRI while participants named objects that are typically green or red, presented in black line drawings. Furthermore, participants underwent two other tasks with the same objects, color naming and semantic judgment, to see if the activation pattern we observe during picture naming is (a) similar to that of a task that requires accessing the color attribute and (b) distinct from that of a task that requires accessing the concept but not its name or color. We used representational similarity analysis to detect brain areas that show similar patterns within the same color category, but show different patterns across the two color categories. In all three tasks, activation in the bilateral fusiform gyri ("Human V4") correlated with a representational model encoding the red-green distinction weighted by the importance of color feature for the different objects. This result suggests that when seeing objects whose color attribute is highly diagnostic, color knowledge about the objects is retrieved irrespective of whether the color or the object itself have to be named.

Categorical consistency facilitates implicit learning of color-number associations

In making sense of the environment, we implicitly learn to associate stimulus attributes that frequently occur together. Is such learning favored for categories over individual items? Here, we introduce a novel paradigm for directly comparing category- to item-level learning. In a category-level experiment, even numbers (2,4,6,8) had a high-probability of appearing in blue, and odd numbers (3,5,7,9) in yellow. Associative learning was measured by the relative performance on trials with low-probability (p = .09) to high-probability (p = .91) number colors. There was strong evidence for associative learning: low-probability performance was impaired (40ms RT increase and 8.3% accuracy decrease relative to high-probability). This was not the case in an item-level experiment with a different group of participants, in which high-probability colors were non-categorically assigned (blue: 2,3,6,7; yellow: 4,5,8,9; 9ms RT increase and 1.5% accuracy increase). The categorical advantage was upheld in an explicit color association report (83% accuracy vs. 43% at the item-level). These results support a conceptual view of perception and suggest empirical bases of categorical, not item-level, color labeling of learning materials.

Comparing the Dominance of Color and Form Information across the Human Ventral Visual Pathway and Convolutional Neural Networks

Color and form information can be decoded in every region of the human ventral visual hierarchy, and at every layer of many convolutional neural networks (CNNs) trained to recognize objects, but how does the coding strength of these features vary over processing? Here, we characterize for these features both their absolute coding strength-how strongly each feature is represented independent of the other feature-and their relative coding strength-how strongly each feature is encoded relative to the other, which could constrain how well a feature can be read out by downstream regions across variation in the other feature. To quantify relative coding strength, we define a measure called the form dominance index that compares the relative influence of color and form on the representational geometry at each processing stage. We analyze brain and CNN responses to stimuli varying based on color and either a simple form feature, orientation, or a more complex form feature, curvature. We find that while the brain and CNNs largely differ in how the absolute coding strength of color and form vary over processing, comparing them in terms of their relative emphasis of these features reveals a striking similarity: For both the brain and for CNNs trained for object recognition (but not for untrained CNNs), orientation information is increasingly de-emphasized, and curvature information is increasingly emphasized, relative to color information over processing, with corresponding processing stages showing largely similar values of the form dominance index.

What multiplexing means for the interpretation of functional MRI data

Despite technology advances that have enabled routine acquisition of functional MRI data with sub-millimeter resolution, the inferences that cognitive neuroscientists must make to link fMRI data to behavior are complicated. Thus, a single dataset subjected to different analyses can be interpreted in different ways. This article presents two optical analogies that can be useful for framing fMRI analyses in a way that allows for multiple interpretations of fMRI data to be valid simultaneously without undermining each other. The first is reflection: when an object is reflected in a mirrored surface, it appears as if the reflected object is sharing space with the mirrored object, but of course it is not. This analogy can be a good guide for interpreting the fMRI signal, since even at sub-millimeter resolutions the signal is determined by a mixture of local and long-range neural computations. The second is refraction. If we view an object through a multi-faceted prism or gemstone, our view will change-sometimes dramatically-depending on our viewing angle. In the same way, interpretation of fMRI data (inference of underlying neuronal activity) can and should be different depending on the analysis approach. Rather than representing a weakness of the methodology, or the superiority of one approach over the other (for example, simple regression analysis versus multi-voxel pattern analysis), this is an expected consequence of how information is multiplexed in the neural networks of the brain: multiple streams of information are simultaneously present in each location. The fact that any one analysis typically shows only one view of the data also puts some parentheses around fMRI practitioners' constant search for ground truth against which to compare their data. By holding our interpretations lightly and understanding that many interpretations of the data can all be true at the same time, we do a better job of preparing ourselves to appreciate, and eventually understand, the complexity of the brain and the behavior it produces.

Functional characterization of correct and incorrect feature integration

Our sensory system constantly receives information from the environment and our own body. Despite our impression to the contrary, we remain largely unaware of this information and often cannot report it correctly. Although perceptual processing does not require conscious effort on the part of the observer, it is often complex, giving rise to errors such as incorrect integration of features (illusory conjunctions). In the present study, we use functional magnetic resonance imaging to study the neural bases of feature integration in a dual task that produced ~30% illusions. A distributed set of regions demonstrated increased activity for correct compared to incorrect (illusory) feature integration, with increased functional coupling between occipital and parietal regions. In contrast, incorrect feature integration (illusions) was associated with increased occipital (V1-V2) responses at early stages, reduced functional connectivity between right occipital regions and the frontal eye field at later stages, and an overall decrease in coactivation between occipital and parietal regions. These results underscore the role of parietal regions in feature integration and highlight the relevance of functional occipito-frontal interactions in perceptual processing.

Mid-range visual functions in relation to higher-order visual functions after stroke

INTRODUCTION

We aimed to investigate whether associations between deficits in "mid-range" visual functions and deficits in higher-order visual cognitive functions in stroke patients are more in line with a hierarchical, two-pathway model of the visual brain, or with a patchwork model, which assumes a parallel organization with many processing routes and cross-talk.

METHODS

A group of 182 ischemic stroke patients was assessed with a new diagnostic set-up for the investigation of a comprehensive range of visuosensory mid-range functions: color, shape, location, orientation, correlated motion, contrast and texture. With logistic regression analyses we investigated the predictive value of these mid-range functions for deficits in visuoconstruction (Copy of the Rey-Complex Figure Test), visual emotion recognition (Ekman 60 Faces Test of the FEEST) and visual memory (computerized Doors-test).

RESULTS

Results showed that performance on most mid-range visual tasks could not predict performance on higher-order visual cognitive tasks. Correlations were low to weak. Impaired visuoconstruction and visual memory were only modestly predicted by a worse location perception. Impaired emotion perception was modestly predicted by a worse orientation perception. In addition, double dissociations were found: there were patients with selective deficits in mid-range visual functions without higher-order visual deficits and vice versa.

CONCLUSIONS

Our findings are not in line with the hierarchical, two-pathway model. Instead, the findings are more in line with alternative "patchwork" models, arguing for a parallel organization with many processing routes and cross-talk. However, future studies are needed to test these alternative models.

Visual Working Memory Adapts to the Nature of Anticipated Interference

Visual working memory has been proven to be relatively robust against interference. However, little is known on whether such robust coding is obligatory, or can be flexibly recruited depending on its expected usefulness. To address this, participants remembered both the color and orientation of a grating. During the maintenance, we inserted a secondary color/orientation memory task, interfering with the primary task. Crucially, we varied the expectations of the type of interference by varying the probability of the two types of intervening task. Behavioral data indicate that to-be-remembered features for which interference is expected are bolstered, whereas to-be-remembered features for which no interference is expected are left vulnerable. This was further supported by fMRI data obtained from visual cortex. In conclusion, the flexibility of visual working memory allows it to strengthen memories for which it anticipates the highest risk of interference.

Temporal dynamics of the neural representation of hue and luminance polarity

Hue and luminance contrast are basic visual features. Here we use multivariate analyses of magnetoencephalography data to investigate the timing of the neural computations that extract them, and whether they depend on common neural circuits. We show that hue and luminance-contrast polarity can be decoded from MEG data and, with lower accuracy, both features can be decoded across changes in the other feature. These results are consistent with the existence of both common and separable neural mechanisms. The decoding time course is earlier and more temporally precise for luminance polarity than hue, a result that does not depend on task, suggesting that luminance contrast is an updating signal that separates visual events. Meanwhile, cross-temporal generalization is slightly greater for representations of hue compared to luminance polarity, providing a neural correlate of the preeminence of hue in perceptual grouping and memory. Finally, decoding of luminance polarity varies depending on the hues used to obtain training and testing data. The pattern of results is consistent with observations that luminance contrast is mediated by both L-M and S cone sub-cortical mechanisms.

Representation of Color, Form, and their Conjunction across the Human Ventral Visual Pathway

Despite decades of research, our understanding of the relationship between color and form processing in the primate ventral visual pathway remains incomplete. Using fMRI multivoxel pattern analysis, we examined coding of color and form, using a simple form feature (orientation) and a mid-level form feature (curvature), in human ventral visual processing regions. We found that both color and form could be decoded from activity in early visual areas V1 to V4, as well as in the posterior color-selective region and shape-selective regions in ventral and lateral occipitotemporal cortex defined based on their univariate selectivity to color or shape, respectively (the central color region only showed color but not form decoding). Meanwhile, decoding biases towards one feature or the other existed in the color- and shape-selective regions, consistent with their univariate feature selectivity reported in past studies. Additional extensive analyses show that while all these regions contain independent (linearly additive) coding for both features, several early visual regions also encode the conjunction of color and the simple, but not the complex, form feature in a nonlinear, interactive manner. Taken together, the results show that color and form are encoded in a biased distributed and largely independent manner across ventral visual regions in the human brain.

Joint representation of color and form in convolutional neural networks: A stimulus-rich network perspective

To interact with real-world objects, any effective visual system must jointly code the unique features defining each object. Despite decades of neuroscience research, we still lack a firm grasp on how the primate brain binds visual features. Here we apply a novel network-based stimulus-rich representational similarity approach to study color and form binding in five convolutional neural networks (CNNs) with varying architecture, depth, and presence/absence of recurrent processing. All CNNs showed near-orthogonal color and form processing in early layers, but increasingly interactive feature coding in higher layers, with this effect being much stronger for networks trained for object classification than untrained networks. These results characterize for the first time how multiple basic visual features are coded together in CNNs. The approach developed here can be easily implemented to characterize whether a similar coding scheme may serve as a viable solution to the binding problem in the primate brain.

Image motion with color contrast suffices to elicit an optokinetic reflex in Xenopus laevis tadpoles

The optokinetic reflex is a closed-loop gaze-stabilizing ocular motor reaction that minimizes residual retinal image slip during vestibulo-ocular reflexes. In experimental isolation, the reflex is usually activated by motion of an achromatic large-field visual background with strong influence of radiance contrast on visual motion estimation and behavioral performance. The presence of color in natural environments, however, suggests that chromatic cues of visual scenes provide additional parameters for image motion detection. Here, we employed Xenopus laevis tadpoles to study the influence of color cues on the performance of the optokinetic reflex and multi-unit optic nerve discharge during motion of a large-field visual scene. Even though the amplitude of the optokinetic reflex decreases with smaller radiance contrast, considerable residual eye movements persist at the ‘point of equiluminance’ of the colored stimuli. Given the color motion preferences of individual optic nerve fibers, the underlying computation potentially originates in retinal circuits. Differential retinal ganglion cell projections and associated ocular motor signal transformation might further reinforce the color dependency in conceptual correspondence with head/body optomotor signaling. Optokinetic reflex performance under natural light conditions is accordingly influenced by radiance contrast as well as by the color composition of the moving visual scene.

Visual Cortex Transcranial Direct Current Stimulation for Proliferative Diabetic Retinopathy Patients: A Double-Blinded Randomized Exploratory Trial

Proliferative diabetic retinopathy (PDR) is a severe complication of diabetes. PDR-related retinal hemorrhages often lead to severe vision loss. The main goals of management are to prevent visual impairment progression and improve residual vision. We explored the potential of transcranial direct current stimulation (tDCS) to enhance residual vision. tDCS applied to the primary visual cortex (V1) may improve visual input processing from PDR patients’ retinas. Eleven PDR patients received cathodal tDCS stimulation of V1 (1 mA for 10 min), and another eleven patients received sham stimulation (1 mA for 30 s). Visual acuity (logarithm of the minimum angle of resolution (LogMAR) scores) and number acuity (reaction times (RTs) and accuracy rates (ARs)) were measured before and immediately after stimulation. The LogMAR scores and the RTs of patients who received cathodal tDCS decreased significantly after stimulation. Cathodal tDCS has no significant effect on ARs. There were no significant changes in the LogMAR scores, RTs, and ARs of PDR patients who received sham stimulation. The results are compatible with our proposal that neuronal noise aggravates impaired visual function in PDR. The therapeutic effect indicates the potential of tDCS as a safe and effective vision rehabilitation tool for PDR patients.

Color Space Geometry Uncovered with Magnetoencephalography

The geometry that describes the relationship among colors, and the neural mechanisms that support color vision, are unsettled. Here, we use multivariate analyses of measurements of brain activity obtained with magnetoencephalography to reverse-engineer a geometry of the neural representation of color space. The analyses depend upon determining similarity relationships among the spatial patterns of neural responses to different colors and assessing how these relationships change in time. We evaluate the approach by relating the results to universal patterns in color naming. Two prominent patterns of color naming could be accounted for by the decoding results: the greater precision in naming warm colors compared to cool colors evident by an interaction of hue and lightness, and the preeminence among colors of reddish hues. Additional experiments showed that classifiers trained on responses to color words could decode color from data obtained using colored stimuli, but only at relatively long delays after stimulus onset. These results provide evidence that perceptual representations can give rise to semantic representations, but not the reverse. Taken together, the results uncover a dynamic geometry that provides neural correlates for color appearance and generates new hypotheses about the structure of color space.

Simultaneous estimation procedure reveals the object-based, but not space-based, dependence of visual working memory representations

Visual working memory (VWM) allows us to actively represent a limited amount of visual information in mind. Although its severe capacity limit is widely accepted, researchers disagree on the nature of its representational unit. Object-based theories argue that VWM organizes feature representations into integrated representations, whereas feature-based theories argue that VWM represents visual features independently. Supporting a feature-based account of VWM, recent studies have demonstrated that features comprising an object can be forgotten independently. Although evidence of feature-based forgetting invalidates a pure object-based account of VWM that assumes perfect integration of feature representations, it is possible that feature representations may be organized in a dependent manner on the basis of objects when they exist in memory. Furthermore, many previous studies prompted participants to recall object features independently by sequentially displaying a response probe for each feature (i.e., sequential estimation procedure), and this task demand might have promoted the independence of feature representations in VWM. To test these possibilities, we created a novel task to simultaneously capture the representational quality of two features of the same object (i.e., simultaneous estimation procedure) and tested their dependence across the entire spectrum of representational quality. Here, we found that the quality of feature representations within the same object covaried reliably in both sequential and simultaneous estimation procedures, but this representational dependence was statistically stronger in the simultaneous estimation procedure than in the sequential estimation procedure. Furthermore, we confirmed that neither the shared spatial location nor simultaneous estimation of two features was sufficient to induce representational dependence in VWM. Thus, our results demonstrate that feature representations in VWM are organized in a dependent manner on the basis of objects, but the degree of dependence can vary based on the current task demand.

Color-shape associations affect feature binding

Non-synesthetic people tend to systematically associate certain shapes with particular colors (i.e., circle-red, triangle-yellow, square-blue). In the present study, we investigated whether such color-shape associations influence illusory conjunctions. Two letters were centrally presented, while two colored-shape stimuli were presented in the periphery. Participants were asked to report: (1) whether the letters were identical, (2) the color of a specific shape, and (3) the confidence of the color choice. The colored-shape stimuli were either congruent or incongruent with the color-shape associations. Results showed that participants reported more illusory conjunctions in the incongruent condition. Thus, color-shape associations might precede and subsequently affect feature binding, and/or affect binding via top-down feedback.

The Influence of Object-Color Knowledge on Emerging Object Representations in the Brain

The ability to rapidly and accurately recognize complex objects is a crucial function of the human visual system. To recognize an object, we need to bind incoming visual features, such as color and form, together into cohesive neural representations and integrate these with our preexisting knowledge about the world. For some objects, typical color is a central feature for recognition; for example, a banana is typically yellow. Here, we applied multivariate pattern analysis on time-resolved neuroimaging (MEG) data to examine how object-color knowledge affects emerging object representations over time. Our results from 20 participants (11 female) show that the typicality of object-color combinations influences object representations, although not at the initial stages of object and color processing. We find evidence that color decoding peaks later for atypical object-color combinations compared with typical object-color combinations, illustrating the interplay between processing incoming object features and stored object knowledge. Together, these results provide new insights into the integration of incoming visual information with existing conceptual object knowledge.SIGNIFICANCE STATEMENT To recognize objects, we have to be able to bind object features, such as color and shape, into one coherent representation and compare it with stored object knowledge. The MEG data presented here provide novel insights about the integration of incoming visual information with our knowledge about the world. Using color as a model to understand the interaction between seeing and knowing, we show that there is a unique pattern of brain activity for congruently colored objects (e.g., a yellow banana) relative to incongruently colored objects (e.g., a red banana). This effect of object-color knowledge only occurs after single object features are processed, demonstrating that conceptual knowledge is accessed relatively late in the visual processing hierarchy.

Multivoxel Pattern of Blood Oxygen Level Dependent Activity can be sensitive to stimulus specific fine scale responses

At ultra-high field, fMRI voxels can span the sub-millimeter range, allowing the recording of blood oxygenation level dependent (BOLD) responses at the level of fundamental units of neural computation, such as cortical columns and layers. This sub-millimeter resolution, however, is only nominal in nature as a number of factors limit the spatial acuity of functional voxels. Multivoxel Pattern Analysis (MVPA) may provide a means to detect information at finer spatial scales that may otherwise not be visible at the single voxel level due to limitations in sensitivity and specificity. Here, we evaluate the spatial scale of stimuli specific BOLD responses in multivoxel patterns exploited by linear Support Vector Machine, Linear Discriminant Analysis and Naïve Bayesian classifiers across cortical depths in V1. To this end, we artificially misaligned the testing relative to the training portion of the data in increasing spatial steps, then investigated the breakdown of the classifiers’ performances. A one voxel shift led to a significant decrease in decoding accuracy (p < 0.05) across all cortical depths, indicating that stimulus specific responses in a multivoxel pattern of BOLD activity exploited by multivariate decoders can be as precise as the nominal resolution of single voxels (here 0.8 mm isotropic). Our results further indicate that large draining vessels, prominently residing in proximity of the pial surface, do not, in this case, hinder the ability of MVPA to exploit fine scale patterns of BOLD signals. We argue that tailored analytical approaches can help overcoming limitations in high-resolution fMRI and permit studying the mesoscale organization of the human brain with higher sensitivities.

Masked blindsight in normal observers: Measuring subjective and objective responses to two features of each stimulus

Recent visual masking studies that have measured visual awareness with graded subjective scales have often failed the show any evidence for unconscious visual processing in normal observers in a paradigm similar to that used in studies on blindsight patients. Without any reported awareness of the target, normal observers typically cannot discriminate target's features better than chance. The present study examined processing of color and orientation by measuring graded awareness and forced-choice discriminations for both features in each trial. When no awareness for either feature was reported, discrimination of each feature succeed better than expected by chance, even when the other feature was incorrectly discriminated in the same trial. However, the characteristics of the mask determined whether or not masked blindsight was observed. We conclude that when the processing channels are free from intra-channel interference, unbound or weakly bound features can guide behaviour without any reported awareness in normal observers.

Late disruption of central visual field disrupts peripheral perception of form and color

Evidence from neuroimaging and brain stimulation studies suggest that visual information about objects in the periphery is fed back to foveal retinotopic cortex in a separate representation that is essential for peripheral perception. The characteristics of this phenomenon have important theoretical implications for the role fovea-specific feedback might play in perception. In this work, we employed a recently developed behavioral paradigm to explore whether late disruption to central visual space impaired perception of color. In the first experiment, participants performed a shape discrimination task on colored novel objects in the periphery while fixating centrally. Consistent with the results from previous work, a visual distractor presented at fixation ~100ms after presentation of the peripheral stimuli impaired sensitivity to differences in peripheral shapes more than a visual distractor presented at other stimulus onset asynchronies. In a second experiment, participants performed a color discrimination task on the same colored objects. In a third experiment, we further tested for this foveal distractor effect with stimuli restricted to a low-level feature by using homogenous color patches. These two latter experiments resulted in a similar pattern of behavior: a central distractor presented at the critical stimulus onset asynchrony impaired sensitivity to peripheral color differences, but, importantly, the magnitude of the effect was stronger when peripheral objects contained complex shape information. These results show a behavioral effect consistent with disrupting feedback to the fovea, in line with the foveal feedback suggested by previous neuroimaging studies.

A roadmap to integrate astrocytes into Systems Neuroscience

Systems neuroscience is still mainly a neuronal field, despite the plethora of evidence supporting the fact that astrocytes modulate local neural circuits, networks, and complex behaviors. In this article, we sought to identify which types of studies are necessary to establish whether astrocytes, beyond their well-documented homeostatic and metabolic functions, perform computations implementing mathematical algorithms that sub-serve coding and higher-brain functions. First, we reviewed Systems-like studies that include astrocytes in order to identify computational operations that these cells may perform, using Ca2+ transients as their encoding language. The analysis suggests that astrocytes may carry out canonical computations in a time scale of subseconds to seconds in sensory processing, neuromodulation, brain state, memory formation, fear, and complex homeostatic reflexes. Next, we propose a list of actions to gain insight into the outstanding question of which variables are encoded by such computations. The application of statistical analyses based on machine learning, such as dimensionality reduction and decoding in the context of complex behaviors, combined with connectomics of astrocyte-neuronal circuits, is, in our view, fundamental undertakings. We also discuss technical and analytical approaches to study neuronal and astrocytic populations simultaneously, and the inclusion of astrocytes in advanced modeling of neural circuits, as well as in theories currently under exploration such as predictive coding and energy-efficient coding. Clarifying the relationship between astrocytic Ca2+ and brain coding may represent a leap forward toward novel approaches in the study of astrocytes in health and disease.

On the brink: The demise of the item in visual search moves closer

We proposed to abandon the item as conceptual unit in visual search and adopt a fixation-based framework instead. We treat various themes raised by our commentators, including the nature of the Functional Visual Field and existing similar ideas, alongside the importance of items, covert attention, and top-down/contextual influences. We reflect on the current state of, and future directions for, visual search.

Continuous to discrete: Ensemble-based segmentation in the perception of multiple feature conjunctions

Although objects around us vary in a number of continuous dimensions (color, size, orientation, etc.), we tend to perceive the objects using more discrete, categorical descriptions (e.g., berries and leaves). Previously, we described how continuous ensemble statistics of simple features are transformed into categorical classes: The visual system tests whether the feature distribution has one or several peaks, each representing a likely "category". Here, we tested the mechanism of segmentation for more complex conjunctions of features. Observers discriminated between two textures filled with lines of various lengths and orientations, which had same distributions between the textures, but opposite directions of correlations. Critically, feature distributions could be "segmentable" (only extreme feature values and a large gap between them) or "non-segmentable" (both extreme and middle values with smooth transition are present). Segmentable displays yielded steeper psychometric functions indicating better discrimination (Experiment 1). The effect of segmentability arises early in visual processing (Experiment 2) and is likely to be provided by global sampling of the entire field (Experiment 3). Also, rapid segmentation requires both feature dimensions having a "segmentable" distribution supporting division of the textures into categorical classes of conjunctions. We propose that observers select items from one side (peak) of one dimension and sample mean differences along a second dimension within the selected subset. In this scenario, subset selection is a limiting factor (Experiment 4) of texture discrimination. Yet, segmentability provided by the sharp feature distributions seems to facilitate both subset selection and mean comparison.

Prospective motion correction improves the sensitivity of fMRI pattern decoding

We evaluated the effectiveness of prospective motion correction (PMC) on a simple visual task when no deliberate subject motion was present. The PMC system utilizes an in‐bore optical camera to track an external marker attached to the participant via a custom‐molded mouthpiece. The study was conducted at two resolutions (1.5 mm vs 3 mm) and under three conditions (PMC On and Mouthpiece On vs PMC Off and Mouthpiece On vs PMC Off and Mouthpiece Off). Multiple data analysis methods were conducted, including univariate and multivariate approaches, and we demonstrated that the benefit of PMC is most apparent for multi‐voxel pattern decoding at higher resolutions. Additional testing on two participants showed that our inexpensive, commercially available mouthpiece solution produced comparable results to a dentist‐molded mouthpiece. Our results showed that PMC is increasingly important at higher resolutions for analyses that require accurate voxel registration across time.

Human V4 Activity Patterns Predict Behavioral Performance in Imagery of Object Color

Color is special among basic visual features in that it can form a defining part of objects that are engrained in our memory. Whereas most neuroimaging research on human color vision has focused on responses related to external stimulation, the present study investigated how sensory-driven color vision is linked to subjective color perception induced by object imagery. We recorded fMRI activity in male and female volunteers during viewing of abstract color stimuli that were red, green, or yellow in half of the runs. In the other half we asked them to produce mental images of colored, meaningful objects (such as tomato, grapes, banana) corresponding to the same three color categories. Although physically presented color could be decoded from all retinotopically mapped visual areas, only hV4 allowed predicting colors of imagined objects when classifiers were trained on responses to physical colors. Importantly, only neural signal in hV4 was predictive of behavioral performance in the color judgment task on a trial-by-trial basis. The commonality between neural representations of sensory-driven and imagined object color and the behavioral link to neural representations in hV4 identifies area hV4 as a perceptual hub linking externally triggered color vision with color in self-generated object imagery.SIGNIFICANCE STATEMENT Humans experience color not only when visually exploring the outside world, but also in the absence of visual input, for example when remembering, dreaming, and during imagery. It is not known where neural codes for sensory-driven and internally generated hue converge. In the current study we evoked matching subjective color percepts, one driven by physically presented color stimuli, the other by internally generated color imagery. This allowed us to identify area hV4 as the only site where neural codes of corresponding subjective color perception converged regardless of its origin. Color codes in hV4 also predicted behavioral performance in an imagery task, suggesting it forms a perceptual hub for color perception.

Neural representation of form-contingent color filling-in in the early visual cortex

Perceptual filling-in exemplifies the constructive nature of visual processing. Color, a prominent surface property of visual objects, can appear to spread to neighboring areas that lack any color. We investigated cortical responses to a color filling-in illusion that effectively dissociates perceived color from the retinal input (van Lier, Vergeer, & Anstis, 2009). Observers adapted to a star-shaped stimulus with alternating red- and cyan-colored points to elicit a complementary afterimage. By presenting an achromatic outline that enclosed one of the two afterimage colors, perceptual filling-in of that color was induced in the unadapted central region. Visual cortical activity was monitored with fMRI, and analyzed using multivariate pattern analysis. Activity patterns in early visual areas (V1–V4) reliably distinguished between the two color-induced filled-in conditions, but only higher extrastriate visual areas showed the predicted correspondence with color perception. Activity patterns allowed for reliable generalization between filled-in colors and physical presentations of perceptually matched colors in areas V3 and V4, but not in earlier visual areas. These findings suggest that the perception of filled-in surface color likely requires more extensive processing by extrastriate visual areas, in order for the neural representation of surface color to become aligned with perceptually matched real colors.

Feature-coding transitions to conjunction-coding with progression through human visual cortex

Identifying an object and distinguishing it from similar items depends upon the ability to perceive its component parts as conjoined into a cohesive whole, but the brain mechanisms underlying this ability remain elusive. The ventral visual processing pathway in primates is organized hierarchically: Neuronal responses in early stages are sensitive to the manipulation of simple visual features, whereas neuronal responses in subsequent stages are tuned to increasingly complex stimulus attributes. It is widely assumed that feature-coding dominates in early visual cortex whereas later visual regions employ conjunction-coding in which object representations are different from the sum of their simple feature parts. However, no study in humans has demonstrated that putative object-level codes in higher visual cortex cannot be accounted for by feature-coding and that putative feature codes in regions prior to ventral temporal cortex are not equally well characterized as object-level codes. Thus the existence of a transition from feature- to conjunction-coding in human visual cortex remains unconfirmed, and if a transition does occur its location remains unknown. By employing multivariate analysis of functional imaging data, we measure both feature-coding and conjunction-coding directly, using the same set of visual stimuli, and pit them against each other to reveal the relative dominance of one vs. the other throughout cortex. Our results reveal a transition from feature-coding in early visual cortex to conjunction-coding in both inferior temporal and posterior parietal cortices. This novel method enables the use of experimentally controlled stimulus features to investigate population-level feature and conjunction codes throughout human cortex.NEW & NOTEWORTHY We use a novel analysis of neuroimaging data to assess representations throughout visual cortex, revealing a transition from feature-coding to conjunction-coding along both ventral and dorsal pathways. Occipital cortex contains more information about spatial frequency and contour than about conjunctions of those features, whereas inferotemporal and parietal cortices contain conjunction coding sites in which there is more information about the whole stimulus than its component parts.

Invariance of surface color representations across illuminant changes in the human cortex

A central problem in color vision is that the light reaching the eye from a given surface can vary dramatically depending on the illumination. Despite this, our color percept, the brain's estimate of surface reflectance, remains remarkably stable. This phenomenon is called color constancy. Here we investigated which human brain regions represent surface color in a way that is invariant with respect to illuminant changes. We used physically realistic rendering methods to display natural yet abstract 3D scenes that were displayed under three distinct illuminants. The scenes embedded, in different conditions, surfaces that differed in their surface color (i.e. in their reflectance property). We used multivariate fMRI pattern analysis to probe neural coding of surface reflectance and illuminant, respectively. While all visual regions encoded surface color when viewed under the same illuminant, we found that only in V1 and V4α surface color representations were invariant to illumination changes. Along the visual hierarchy there was a gradient from V1 to V4α to increasingly encode surface color rather than illumination. Finally, effects of a stimulus manipulation on individual behavioral color constancy indices correlated with neural encoding of the illuminant in hV4. This provides neural evidence for the Equivalent Illuminant Model. Our results provide a principled characterization of color constancy mechanisms across the visual hierarchy, and demonstrate complementary contributions in early and late processing stages.

Edge-Related Activity Is Not Necessary to Explain Orientation Decoding in Human Visual Cortex

Multivariate pattern analysis is a powerful technique; however, a significant theoretical limitation in neuroscience is the ambiguity in interpreting the source of decodable information used by classifiers. This is exemplified by the continued controversy over the source of orientation decoding from fMRI responses in human V1. Recently Carlson (2014) identified a potential source of decodable information by modeling voxel responses based on the Hubel and Wiesel (1972) ice-cube model of visual cortex. The model revealed that activity associated with the edges of gratings covaries with orientation and could potentially be used to discriminate orientation. Here we empirically evaluate whether "edge-related activity" underlies orientation decoding from patterns of BOLD response in human V1. First, we systematically mapped classifier performance as a function of stimulus location using population receptive field modeling to isolate each voxel's overlap with a large annular grating stimulus. Orientation was decodable across the stimulus; however, peak decoding performance occurred for voxels with receptive fields closer to the fovea and overlapping with the inner edge. Critically, we did not observe the expected second peak in decoding performance at the outer stimulus edge as predicted by the edge account. Second, we evaluated whether voxels that contribute most to classifier performance have receptive fields that cluster in cortical regions corresponding to the retinotopic location of the stimulus edge. Instead, we find the distribution of highly weighted voxels to be approximately random, with a modest bias toward more foveal voxels. Our results demonstrate that edge-related activity is likely not necessary for orientation decoding.

SIGNIFICANCE STATEMENT

A significant theoretical limitation of multivariate pattern analysis in neuroscience is the ambiguity in interpreting the source of decodable information used by classifiers. For example, orientation can be decoded from BOLD activation patterns in human V1, even though orientation columns are at a finer spatial scale than 3T fMRI. Consequently, the source of decodable information remains controversial. Here we test the proposal that information related to the stimulus edges underlies orientation decoding. We map voxel population receptive fields in V1 and evaluate orientation decoding performance as a function of stimulus location in retinotopic cortex. We find orientation is decodable from voxels whose receptive fields do not overlap with the stimulus edges, suggesting edge-related activity does not substantially drive orientation decoding.

Feature Confirmation in Object Perception: Feature Integration Theory 26 Years on from the Treisman Bartlett Lecture

The Treisman Bartlett lecture, reported in the Quarterly Journal of Experimental Psychology in 1988, provided a major overview of the feature integration theory of attention. This has continued to be a dominant account of human visual attention to this day. The current paper provides a summary of the work reported in the lecture and an update on critical aspects of the theory as applied to visual object perception. The paper highlights the emergence of findings that pose significant challenges to the theory and which suggest that revisions are required that allow for (a) several rather than a single form of feature integration, (b) some forms of feature integration to operate preattentively, (c) stored knowledge about single objects and interactions between objects to modulate perceptual integration, (d) the application of feature-based inhibition to object files where visual features are specified, which generates feature-based spreading suppression and scene segmentation, and (e) a role for attention in feature confirmation rather than feature integration in visual selection. A feature confirmation account of attention in object perception is outlined.

Learning to Associate Orientation with Color in Early Visual Areas by Associative Decoded fMRI Neurofeedback

Associative learning is an essential brain process where the contingency of different items increases after training. Associative learning has been found to occur in many brain regions [1-4]. However, there is no clear evidence that associative learning of visual features occurs in early visual areas, although a number of studies have indicated that learning of a single visual feature (perceptual learning) involves early visual areas [5-8]. Here, via decoded fMRI neurofeedback termed "DecNef" [9], we tested whether associative learning of orientation and color can be created in early visual areas. During 3 days of training, DecNef induced fMRI signal patterns that corresponded to a specific target color (red) mostly in early visual areas while a vertical achromatic grating was physically presented to participants. As a result, participants came to perceive "red" significantly more frequently than "green" in an achromatic vertical grating. This effect was also observed 3-5 months after the training. These results suggest that long-term associative learning of two different visual features such as orientation and color was created, most likely in early visual areas. This newly extended technique that induces associative learning is called "A-DecNef," and it may be used as an important tool for understanding and modifying brain functions because associations are fundamental and ubiquitous functions in the brain.

Unconscious Familiarity-based Color-Form Binding: Evidence from Visual Extinction

There is good evidence that early visual processing involves the coding of different features in independent brain regions. A major question, then, is how we see the world in an integrated manner, in which the different features are "bound" together. A standard account of this has been that feature binding depends on attention to the stimulus, which enables only the relevant features to be linked together [Treisman, A., & Gelade, G. A feature-integration theory of attention. Cognitive Psychology, 12, 97-136, 1980]. Here we test this influential idea by examining whether, in patients showing visual extinction, the processing of otherwise unconscious (extinguished) stimuli is modulated by presenting objects in their correct (familiar) color. Correctly colored objects showed reduced extinction when they had a learned color, and this color matched across the ipsi- and contralesional items (red strawberry + red tomato). In contrast, there was no reduction in extinction under the same conditions when the stimuli were colored incorrectly (blue strawberry + blue tomato; Experiment 1). The result was not due to the speeded identification of a correctly colored ipsilesional item, as there was no benefit from having correctly colored objects in different colors (red strawberry + yellow lemon; Experiment 2). There was also no benefit to extinction from presenting the correct colors in the background of each item (Experiment 3). The data suggest that learned color-form binding can reduce extinction even when color is irrelevant for the task. The result is consistent with preattentive binding of color and shape for familiar stimuli.

The selectivity of responses to red-green colour and achromatic contrast in the human visual cortex: an fMRI adaptation study

There is controversy as to how responses to colour in the human brain are organized within the visual pathways. A key issue is whether there are modular pathways that respond selectively to colour or whether there are common neural substrates for both colour and achromatic (Ach) contrast. We used functional magnetic resonance imaging (fMRI) adaptation to investigate the responses of early and extrastriate visual areas to colour and Ach contrast. High-contrast red-green (RG) and Ach sinewave rings (0.5 cycles/degree, 2 Hz) were used as both adapting stimuli and test stimuli in a block design. We found robust adaptation to RG or Ach contrast in all visual areas. Cross-adaptation between RG and Ach contrast occurred in all areas indicating the presence of integrated, colour and Ach responses. Notably, we revealed contrasting trends for the two test stimuli. For the RG test, unselective processing (robust adaptation to both RG and Ach contrast) was most evident in the early visual areas (V1 and V2), but selective responses, revealed as greater adaptation between the same stimuli than cross-adaptation between different stimuli, emerged in the ventral cortex, in V4 and VO in particular. For the Ach test, unselective responses were again most evident in early visual areas but Ach selectivity emerged in the dorsal cortex (V3a and hMT+). Our findings support a strong presence of integrated mechanisms for colour and Ach contrast across the visual hierarchy, with a progression towards selective processing in extrastriate visual areas.

The physiology and psychophysics of the color-form relationship: a review

The relationship between color and form has been a long standing issue in visual science. A picture of functional segregation and topographic clustering emerges from anatomical and electrophysiological studies in animals, as well as by brain imaging studies in human. However, one of the many roles of chromatic information is to support form perception, and in some cases it can do so in a way superior to achromatic (luminance) information. This occurs both at an early, contour-detection stage, as well as in late, higher stages involving spatial integration and the perception of global shapes. Pure chromatic contrast can also support several visual illusions related to form-perception. On the other hand, form seems a necessary prerequisite for the computation and assignment of color across space, and there are several respects in which the color of an object can be influenced by its form. Evidently, color and form are mutually dependent. Electrophysiological studies have revealed neurons in the visual brain able to signal contours determined by pure chromatic contrast, the spatial tuning of which is similar to that of neurons carrying luminance information. It seems that, especially at an early stage, form is processed by several, independent systems that interact with each other, each one having different tuning characteristics in color space. At later processing stages, mechanisms able to combine information coming from different sources emerge. A clear interaction between color and form is manifested by the fact that color-form contingencies can be observed in various perceptual phenomena such as adaptation aftereffects and illusions. Such an interaction suggests a possible early binding between these two attributes, something that has been verified by both electrophysiological and fMRI studies.

The role of color in motion feature-binding errors

Color-motion feature-binding errors occur in the periphery when half of the objects are red and move downward, and the other half are green and move upward. When red and green objects in the central visual field are similar but move in the opposite directions (red upward, green downward), peripheral objects often take on the perceived motion direction of the like-colored central objects (Wu, Kanai, & Shimojo, 2004). The present study determined whether color is essential to elicit these motion-binding errors, and tested two hypotheses that attempt to explain them. One hypothesis holds that binding errors occur because peripheral and central objects become linked if they have combinations of features in common. A peripheral object's link to central objects overwhelms its posited weak peripheral representation for motion feature binding, so the peripheral object appears to move in the direction of the linked central objects. Eliminating color by making all stimuli achromatic, therefore, should not increase peripheral binding errors. An alternative hypothesis is that binding errors depend on the overall feature correspondence among central and peripheral features represented at a preconjunctive level. In this case, binding errors may increase when all objects are changed to achromatic because chromatic central/peripheral correspondence is maximal (100%). Experiments showed more motion-binding errors with all-achromatic objects than with half red and half green objects. This and additional findings imply that peripheral motion-binding errors (a) can be elicited without color and (b) depend at least in part on the similarity of central and peripheral features represented preconjunctively.