Seeing versus knowing: The temporal dynamics of real and implied colour processing in the human brain

Mapping the dynamics of visual feature coding: Insights into perception and integration

The basic computations performed in the human early visual cortex are the foundation for visual perception. While we know a lot about these computations, a key missing piece is how the coding of visual features relates to our perception of the environment. To investigate visual feature coding, interactions, and their relationship to human perception, we investigated neural responses and perceptual similarity judgements to a large set of visual stimuli that varied parametrically along four feature dimensions. We measured neural responses using electroencephalography (N = 16) to 256 grating stimuli that varied in orientation, spatial frequency, contrast, and colour. We then mapped the response profiles of the neural coding of each visual feature and their interactions, and related these to independently obtained behavioural judgements of stimulus similarity. The results confirmed fundamental principles of feature coding in the visual system, such that all four features were processed simultaneously but differed in their dynamics, and there was distinctive conjunction coding for different combinations of features in the neural responses. Importantly, modelling of the behaviour revealed that every stimulus feature contributed to perceptual judgements, despite the untargeted nature of the behavioural task. Further, the relationship between neural coding and behaviour was evident from initial processing stages, signifying that the fundamental features, not just their interactions, contribute to perception. This study highlights the importance of understanding how feature coding progresses through the visual hierarchy and the relationship between different stages of processing and perception.

Testing the generalization of neural representations

Multivariate analysis methods are widely used in neuroscience to investigate the presence and structure of neural representations. Representational similarities across time or contexts are often investigated using pattern generalization, e.g. by training and testing multivariate decoders in different contexts, or by comparable pattern-based encoding methods. It is however unclear what conclusions can be validly drawn on the underlying neural representations when significant pattern generalization is found in mass signals such as LFP, EEG, MEG, or fMRI. Using simulations, we show how signal mixing and dependencies between measurements can drive significant pattern generalization even though the true underlying representations are orthogonal. We suggest that, using an accurate estimate of the expected pattern generalization given identical representations, it is nonetheless possible to test meaningful hypotheses about the generalization of neural representations. We offer such an estimate of the expected magnitude of pattern generalization and demonstrate how this measure can be used to assess the similarity and differences of neural representations across time and contexts.

Magnetoencephalography contrast adaptation reflects perceptual adaptation

Contrast adaptation is a fundamental visual process that has been extensively investigated and used to infer the selectivity of visual cortex. We recently reported an apparent disconnect between the effects of contrast adaptation on perception and functional magnetic resonance imaging BOLD response adaptation, in which adaptation between chromatic and achromatic stimuli measured psychophysically showed greater selectivity than adaptation measured using BOLD signals. Here we used magnetoencephalography (MEG) recordings of neural responses to the same chromatic and achromatic adaptation conditions to characterize the neural effects of contrast adaptation and to determine whether BOLD adaptation or MEG better reflect the measured perceptual effects. Participants viewed achromatic, L-M isolating, or S-cone isolating radial sinusoids before adaptation and after adaptation to each of the three contrast directions. We measured adaptation-related changes in the neural response to a range of stimulus contrast amplitudes using two measures of the MEG response: the overall response amplitude, and a novel time-resolved measure of the contrast response function, derived from a classification analysis combined with multidimensional scaling. Within-stimulus adaptation effects on the contrast response functions in each case showed a pattern of contrast-gain or a combination of contrast-gain and response-gain effects. Cross-stimulus adaptation conditions showed that adaptation effects were highly stimulus selective across early, ventral, and dorsal visual cortical areas, consistent with the perceptual effects.

The nature of neural object representations during dynamic occlusion

Objects disappearing briefly from sight due to occlusion is an inevitable occurrence in everyday life. Yet we generally have a strong experience that occluded objects continue to exist, despite the fact that they objectively disappear. This indicates that neural object representations must be maintained during dynamic occlusion. However, it is unclear what the nature of such representation is and in particular whether it is perception-like or more abstract, for example, reflecting limited features such as position or movement direction only. In this study, we address this question by examining how different object features such as object shape, luminance, and position are represented in the brain when a moving object is dynamically occluded. We apply multivariate decoding methods to Magnetoencephalography (MEG) data to track how object representations unfold over time. Our methods allow us to contrast the representations of multiple object features during occlusion and enable us to compare the neural responses evoked by visible and occluded objects. The results show that object position information is represented during occlusion to a limited extent while object identity features are not maintained through the period of occlusion. Together, this suggests that the nature of object representations during dynamic occlusion is different from visual representations during perception.

Temporal dynamics of the neural representation of hue and luminance polarity

Hue and luminance contrast are basic visual features. Here we use multivariate analyses of magnetoencephalography data to investigate the timing of the neural computations that extract them, and whether they depend on common neural circuits. We show that hue and luminance-contrast polarity can be decoded from MEG data and, with lower accuracy, both features can be decoded across changes in the other feature. These results are consistent with the existence of both common and separable neural mechanisms. The decoding time course is earlier and more temporally precise for luminance polarity than hue, a result that does not depend on task, suggesting that luminance contrast is an updating signal that separates visual events. Meanwhile, cross-temporal generalization is slightly greater for representations of hue compared to luminance polarity, providing a neural correlate of the preeminence of hue in perceptual grouping and memory. Finally, decoding of luminance polarity varies depending on the hues used to obtain training and testing data. The pattern of results is consistent with observations that luminance contrast is mediated by both L-M and S cone sub-cortical mechanisms.

Linking the Brain with Behavior: The Neural Dynamics of Success and Failure in Goal-directed Behavior

The human brain is extremely flexible and capable of rapidly selecting relevant information in accordance with task goals. Regions of frontoparietal cortex flexibly represent relevant task information such as task rules and stimulus features when participants perform tasks successfully, but less is known about how information processing breaks down when participants make mistakes. This is important for understanding whether and when information coding recorded with neuroimaging is directly meaningful for behavior. Here, we used magnetoencephalography to assess the temporal dynamics of information processing and linked neural responses with goal-directed behavior by analyzing how they changed on behavioral error. Participants performed a difficult stimulus–response task using two stimulus–response mapping rules. We used time-resolved multivariate pattern analysis to characterize the progression of information coding from perceptual information about the stimulus, cue and rule coding, and finally, motor response. Response-aligned analyses revealed a ramping up of perceptual information before a correct response, suggestive of internal evidence accumulation. Strikingly, when participants made a stimulus-related error, and not when they made other types of errors, patterns of activity initially reflected the stimulus presented, but later reversed, and accumulated toward a representation of the “incorrect” stimulus. This suggests that the patterns recorded at later time points reflect an internally generated stimulus representation that was used to make the (incorrect) decision. These results illustrate the orderly and overlapping temporal dynamics of information coding in perceptual decision-making and show a clear link between neural patterns in the late stages of processing and behavior.

Influence of colour on object motor representation

Colour conveys specific information about the status/quality of an object; whereas its role in object recognition has been widely studied, little is known about its role in sensorimotor processes. We performed three experiments to assess whether colour influences the motor representation of graspable objects. In Experiment 1, we used a grasp compatibility task, in which participants categorized each object as natural or artifact, by performing reach-to-grasp movements. Response grasps could be compatible or incompatible with the ones normally used to manipulate the objects. Results showed faster reaction times for natural objects displayed in the correct colour compared with both opposite colour and correct colour artifact objects. In Experiment 2, to directly assess the effect of colour on object motor representation, we used an interference task in which an irrelevant object was shown while performing a pre-specified reach-to-grasp movement (i.e., verbal cues: small vs. large). Results highlighted a reversed compatibility effect when objects were shown in their correct colour, but only at the beginning of the movement (10 ms SOA). Finally, we run a third experiment using the same task as in Experiment 2. In this experiment, we compared the grasp compatibility effect driven by natural objects with the grasp compatibility effect driven by dangerous natural objects (e.g., cactus), which are objects that should not elicit a grasping program. The results of Experiment 3 confirm those of Experiment 2, highlighting also specific processes related to dangerous objects. Taken together, these results revealed that colour can be significant for the motor system, highlighting the close link between colour and shape, and also specific processes related to dangerous objects.

Bizarreness and Typicality Effects of Color on Object Recognition Memory

The bizarreness effect on object recognition is a strong phenomenon, but its influence has been inconsistent for bizarre object color. In this study, we manipulated three factors in separate experiments to determine whether a color bizarreness effect on object recognition memory would occur and, if not, why. Participants first saw (i.e., learned) object pictures that were either bizarrely or typically colored; they then completed a recognition memory test. In three experiments, we then manipulated (a) degree of color bizarreness (Experiment 1), (b) the orientation task (Experiment 2), and (c) additional demands for object identification (Experiment 3). In Experiment 1, we provided 49 undergraduate participants with object pictures whose colors were typical, moderately atypical, or bizarre and found no color bizarreness effect on recognition memory even for extremely bizarre colors. In Experiment 2, we manipulated the orientation task in that 28 young adult participants expressed their preferences for the pictures on a three-point scale while another 28 participants judged how natural the pictures were. Each orientation task group better recognized typically-colored rather than bizarrely-colored objects (typicality effect). In Experiment 3, we asked 27 young adults to identify the objects during the learning phase to ensure that they paid attention to the objects' bizarre colors; recognition memory was then unaffected by either color bizarreness or typicality. Thus, despite a general bizarreness effect in recognition memory, bizarre colors are less likely to influence object recognition memory.

Decoding visual colour from scalp electroencephalography measurements

Recent advances have made it possible to decode various aspects of visually presented stimuli from patterns of scalp EEG measurements. As of recently, such multivariate methods have been commonly used to decode visual-spatial features such as location, orientation, or spatial frequency. In the current study, we show that it is also possible to track visual colour processing by using Linear Discriminant Analysis on patterns of EEG activity. Building on other recent demonstrations, we show that colour decoding: (1) reflects sensory qualities (as opposed to, for example, verbal labelling) with a prominent contribution from posterior electrodes contralateral to the stimulus, (2) conforms to a parametric coding space, (3) is possible in multi-item displays, and (4) is comparable in magnitude to the decoding of visual stimulus orientation. Through subsampling our data, we also provide an estimate of the approximate number of trials and participants required for robust decoding. Finally, we show that while colour decoding can be sensitive to subtle differences in luminance, our colour decoding results are primarily driven by measured colour differences between stimuli. Colour decoding opens a relevant new dimension in which to track visual processing using scalp EEG measurements, while bypassing potential confounds associated with decoding approaches that focus on spatial features.

Color Space Geometry Uncovered with Magnetoencephalography

The geometry that describes the relationship among colors, and the neural mechanisms that support color vision, are unsettled. Here, we use multivariate analyses of measurements of brain activity obtained with magnetoencephalography to reverse-engineer a geometry of the neural representation of color space. The analyses depend upon determining similarity relationships among the spatial patterns of neural responses to different colors and assessing how these relationships change in time. We evaluate the approach by relating the results to universal patterns in color naming. Two prominent patterns of color naming could be accounted for by the decoding results: the greater precision in naming warm colors compared to cool colors evident by an interaction of hue and lightness, and the preeminence among colors of reddish hues. Additional experiments showed that classifiers trained on responses to color words could decode color from data obtained using colored stimuli, but only at relatively long delays after stimulus onset. These results provide evidence that perceptual representations can give rise to semantic representations, but not the reverse. Taken together, the results uncover a dynamic geometry that provides neural correlates for color appearance and generates new hypotheses about the structure of color space.

High-pass filtering artifacts in multivariate classification of neural time series data

BACKGROUND

Traditionally, EEG/MEG data are high-pass filtered and baseline-corrected to remove slow drifts. Minor deleterious effects of high-pass filtering in traditional time-series analysis have been well-documented, including temporal displacements. However, its effects on time-resolved multivariate pattern classification analyses (MVPA) are largely unknown.

NEW METHOD

To prevent potential displacement effects, we extend an alternative method of removing slow drift noise - robust detrending - with a procedure in which we mask out all cortical events from each trial. We refer to this method as trial-masked robust detrending.

RESULTS

In both real and simulated EEG data of a working memory experiment, we show that both high-pass filtering and standard robust detrending create artifacts that result in the displacement of multivariate patterns into activity silent periods, particularly apparent in temporal generalization analyses, and especially in combination with baseline correction. We show that trial-masked robust detrending is free from such displacements.

COMPARISON WITH EXISTING METHOD(S)

Temporal displacement may emerge even with modest filter cut-off settings such as 0.05 Hz, and even in regular robust detrending. However, trial-masked robust detrending results in artifact-free decoding without displacements. Baseline correction may unwittingly obfuscate spurious decoding effects and displace them to the rest of the trial.

CONCLUSIONS

Decoding analyses benefit from trial-masked robust detrending, without the unwanted side effects introduced by filtering or regular robust detrending. However, for sufficiently clean data sets and sufficiently strong signals, no filtering or detrending at all may work adequately. Implications for other types of data are discussed, followed by a number of recommendations.

The Influence of Object-Color Knowledge on Emerging Object Representations in the Brain

The ability to rapidly and accurately recognize complex objects is a crucial function of the human visual system. To recognize an object, we need to bind incoming visual features, such as color and form, together into cohesive neural representations and integrate these with our preexisting knowledge about the world. For some objects, typical color is a central feature for recognition; for example, a banana is typically yellow. Here, we applied multivariate pattern analysis on time-resolved neuroimaging (MEG) data to examine how object-color knowledge affects emerging object representations over time. Our results from 20 participants (11 female) show that the typicality of object-color combinations influences object representations, although not at the initial stages of object and color processing. We find evidence that color decoding peaks later for atypical object-color combinations compared with typical object-color combinations, illustrating the interplay between processing incoming object features and stored object knowledge. Together, these results provide new insights into the integration of incoming visual information with existing conceptual object knowledge.SIGNIFICANCE STATEMENT To recognize objects, we have to be able to bind object features, such as color and shape, into one coherent representation and compare it with stored object knowledge. The MEG data presented here provide novel insights about the integration of incoming visual information with our knowledge about the world. Using color as a model to understand the interaction between seeing and knowing, we show that there is a unique pattern of brain activity for congruently colored objects (e.g., a yellow banana) relative to incongruently colored objects (e.g., a red banana). This effect of object-color knowledge only occurs after single object features are processed, demonstrating that conceptual knowledge is accessed relatively late in the visual processing hierarchy.

Recent advances in understanding object recognition in the human brain: deep neural networks, temporal dynamics, and context

Object recognition is the ability to identify an object or category based on the combination of visual features observed. It is a remarkable feat of the human brain, given that the patterns of light received by the eye associated with the properties of a given object vary widely with simple changes in viewing angle, ambient lighting, and distance. Furthermore, different exemplars of a specific object category can vary widely in visual appearance, such that successful categorization requires generalization across disparate visual features. In this review, we discuss recent advances in understanding the neural representations underlying object recognition in the human brain. We highlight three current trends in the approach towards this goal within the field of cognitive neuroscience. Firstly, we consider the influence of deep neural networks both as potential models of object vision and in how their representations relate to those in the human brain. Secondly, we review the contribution that time-series neuroimaging methods have made towards understanding the temporal dynamics of object representations beyond their spatial organization within different brain regions. Finally, we argue that an increasing emphasis on the context (both visual and task) within which object recognition occurs has led to a broader conceptualization of what constitutes an object representation for the brain. We conclude by identifying some current challenges facing the experimental pursuit of understanding object recognition and outline some emerging directions that are likely to yield new insight into this complex cognitive process.

Processing the Word Red and Intellectual Performance: Four Replication Attempts

Colors convey meaning and can impair intellectual performance in achievement situations. Even the processing of color words can exert similar detrimental effects. In four experiments, we tried to replicate previous findings regarding the processing of the word “red” (as compared to a control color) on cognitive test scores. Experiments 1 and 2 (Ns = 69 and 104) are direct replications of Lichtenfeld, Maier, Elliot, and Pekrun (2009). Both experiments failed to uncover a red color effect on verbal reasoning scores among high school students and undergraduates (Cohen’s d = 0.04 and –0.23). Experiments 3 and 4 (N = 103 and 1,149) failed to identify an effect of processing red on general knowledge test scores (Cohen’sd = 0.19) and 0.01) among undergraduates and adults. Together, these results do not corroborate the assumption that processing the word red impairs intellectual performance.