Probing unconscious visual processing with the McCollough effect

Are Color Experiences the Same across the Visual Field?

It seems obvious to laypeople that neurotypical humans experience color equivalently across their entire visual field. To some neuroscientists, psychologists, and philosophers, though, this claim has been met with skepticism, as neurophysiological evidence indicates the mechanisms that support color perception degrade with eccentricity. However, the argument that this entails altered color experience in peripheral vision is not universally accepted. Here, we address whether color experience is essentially equivalent between central and peripheral vision. To assess this, we will obtain similarity relationships between color experiences across the visual field using both online and laboratory-based far-field displays, while removing the confounds of saccades, memory, and expectation about color experiences. Our experiment was designed to provide clear evidence that would favor either unchanged or altered color experience relationships in the periphery. Our results are consistent with lay people's phenomenological reports: Color experiences, as probed by similarity relationships in central vision and the far field (60°), are equivalent when elicited by large stimuli. These findings challenge the widespread view in philosophy and cognitive science that peripheral color experiences are illusory, and are discussed in the context of their related neurophysiological, psychophysical, and philosophical literature.

Nonlinear cortical encoding of color predicts enhanced McCollough effects in anomalous trichromats

Nonlinear encoding of chromatic contrast by the early visual cortex predicts that anomalous trichromats will show a larger McCollough effect than normal trichromats. In Experiment 1 we employed the McCollough effect to probe the cortical representation of saturation in normal trichromats, and used the results to predict enhanced McCollough effects for anomalous trichromats, which we measured in Experiment 2. In Experiment 1 three participants adapted to red and green orthogonal gratings of four different saturations. Using nulling to measure aftereffect strength, we found that halving the saturation of the inducing gratings decreased aftereffect strength only slightly, consistent with a compressive coding of saturation in early visual cortex. In anomalous trichromats, cone contrasts between red and green are greatly decreased from those of normal trichromats, but induced aftereffects are only slightly decreased, because of the non-linearity in the cortical encoding of saturation. To null the aftereffect, however, the retinal color deficiency must be overcome by adding more color to the null than required by normal trichromats. We confirmed this prediction in Experiment 2 where four anomalous trichromats required nulling stimuli approximately four times more saturated than did normal trichromats. We consider two competing models to explain our results: in a 'pigment swap' model anomalous trichromats have an altered photopigment but process color postreceptorally in the same way as normal trichromats; in a 'postreceptoral compensation' model the cortical representation of red-green contrasts is amplified to compensate for reduced cone contrasts. The latter provided a better fit to our data.

The structure of illusory conjunctions reveals hierarchical binding of multipart objects

The world around us is filled with complex objects, full of color, motion, shape, and texture, and these features seem to be represented separately in the early visual system. Anne Treisman pointed out that binding these separate features together into coherent conscious percepts is a serious challenge, and she argued that selective attention plays a critical role in this process. Treisman also showed that, consistent with this view, outside the focus of attention we suffer from illusory conjunctions: misperceived pairings of features into objects. Here we used Treisman's logic to study the structure of pre-attentive representations of multipart, multicolor objects, by exploring the patterns of illusory conjunctions that arise outside the focus of attention. We found consistent evidence of some pre-attentive binding of colors to their parts, and weaker evidence of binding multiple colors of the same object. The extent to which such hierarchical binding occurs seems to depend on the geometric structure of multipart objects: Objects whose parts are easier to separate seem to exhibit greater pre-attentive binding. Together, these results suggest that representations outside the focus of attention are not entirely a "shapeless bundles of features," but preserve some meaningful object structure.

Learning to Associate Orientation with Color in Early Visual Areas by Associative Decoded fMRI Neurofeedback

Associative learning is an essential brain process where the contingency of different items increases after training. Associative learning has been found to occur in many brain regions [1-4]. However, there is no clear evidence that associative learning of visual features occurs in early visual areas, although a number of studies have indicated that learning of a single visual feature (perceptual learning) involves early visual areas [5-8]. Here, via decoded fMRI neurofeedback termed "DecNef" [9], we tested whether associative learning of orientation and color can be created in early visual areas. During 3 days of training, DecNef induced fMRI signal patterns that corresponded to a specific target color (red) mostly in early visual areas while a vertical achromatic grating was physically presented to participants. As a result, participants came to perceive "red" significantly more frequently than "green" in an achromatic vertical grating. This effect was also observed 3-5 months after the training. These results suggest that long-term associative learning of two different visual features such as orientation and color was created, most likely in early visual areas. This newly extended technique that induces associative learning is called "A-DecNef," and it may be used as an important tool for understanding and modifying brain functions because associations are fundamental and ubiquitous functions in the brain.

Misbinding of color and motion in human early visual cortex: Evidence from event-related potentials

One of the central tasks for the visual system is to integrate visual features into objects, which is referred to as the binding problem. To study the binding mechanisms, it has been suggested to use phenomena of feature misbinding to separate active feature binding from feature co-occurence. Taking advantage of a steady-state misbinding of color and motion, we performed psychophysical and event-related potential (ERP) adaptation experiments to investigate the neural mechanisms of the misbinding (i.e., the active color-motion binding). Human subjects adapted to the misbinding of color and motion, as well as their correct binding that was used for identifying neural processes associated with the co-occurrence of color and motion. We found that adaptation to the misbinding and the correct binding could generate color-contingent motion aftereffects (CCMAEs), but in opposite directions. ERP adaptation effects manifested in the earliest ERP component C1. The C1 latency in the misbinding condition was 11ms longer than that in the correct binding condition. In the correct binding condition, the C1 adaptation effect (i.e., the C1 amplitude reduction after adaptation) took place in the peak phase of the C1. The dipole source of the adaptation effect was located in V1. In the misbinding condition, the C1 adaptation effect occurred in the descending phase of the C1 and its dipole source was in V2. In both conditions, the C1 adaptation effects correlated with the CCMAEs across individual subjects. These findings provide human electrophysiological evidence that active feature binding takes place in early visual cortex, but at later processing stages than feature co-occurrence.

Congruent tactile stimulation reduces the strength of visual suppression during binocular rivalry

Presenting different images to each eye triggers ‘binocular rivalry’ in which one image is visible and the other suppressed, with the visible image alternating every second or so. We previously showed that binocular rivalry between cross-oriented gratings is altered when the fingertip explores a grooved stimulus aligned with one of the rivaling gratings: the matching visual grating's dominance duration was lengthened and its suppression duration shortened. In a more robust test, we here measure visual contrast sensitivity during rivalry dominance and suppression, with and without exploration of the grooved surface, to determine if rivalry suppression strength is modulated by touch. We find that a visual grating undergoes 45% less suppression when observers touch an aligned grating, compared to a cross-oriented one. Touching an aligned grating also improved visual detection thresholds for the ‘invisible’ suppressed grating by 2.4 dB, relative to a vision-only condition. These results show that congruent haptic stimulation prevents a visual stimulus from becoming deeply suppressed in binocular rivalry. Moreover, because congruent touch acted on the phenomenally invisible grating, this visuo-haptic interaction must precede awareness and likely occurs early in visual processing.

Activity in the lateral occipital cortex between 200 and 300 ms distinguishes between physically identical seen and unseen stimuli

There is converging evidence that electrophysiological responses over posterior cortical regions in the 200–300 ms range distinguish between physically identical stimuli that reach consciousness or remain unseen. Here, we attempt at determining the sources of this awareness-related activity using magneto-encephalographic (MEG). Fourteen subjects were presented with faint colored gratings at threshold for contrast and reported on each trial whether the grating was seen or unseen. Subjects were primed with a color cue that could be congruent or incongruent with the color of the grating, to probe to what extent two co-localized features (color and orientation) would be bound in consciousness. The contrast between neural responses to seen and unseen physically identical gratings revealed a sustained posterior difference between 190 and 350 ms, thereby replicating prior studies. We further show that the main sources of the awareness-related activity were localized bilaterally on the lateral convexity of the occipito-temporal region, in the Lateral Occipital (LO) complex, as well as in the right posterior infero-temporal region. No activity differentiating seen and unseen trials could be observed in frontal or parietal regions in this latency range, even at lower threshold. Color congruency did not improve grating's detection, and the awareness-related activity was independent from color congruency. However, at the neural level, color congruency was processed differently in grating-present and grating-absent trials. The pattern of results suggests the existence of a neural process of color congruency engaging left parietal regions that is affected by the mere presence of another feature, whether this feature reaches consciousness or not. Altogether, our results reveal an occipital source of visual awareness insensitive to color congruency, and a simultaneous parietal source not engaged in visual awareness, but sensitive to the manipulation of co-localized features.

Independent Sampling of Features Enables Conscious Perception of Bound Objects

Decades of research suggest that selective attention is critical for binding the features of objects together for conscious perception. A fundamental question, however, remains unresolved: How do people perceive objects, albeit with binding errors ( illusory conjunctions), when attentional resolution is poor? We used a novel technique to investigate how features are selected to create percepts of bound objects. We measured the correlation of errors (intrusions) in color and identity reports in spatial and temporal selection tasks under conditions of varying spatial or temporal uncertainty. Our findings suggest that attention selects each feature independently by randomly sampling from a probability distribution over space or time. Thus, veridical perception of bound object features arises only when attentional selection is sufficiently precise that the independently sampled features originate from a single object.

Preserved Striate Cortex is Not Sufficient to Support the McCollough Effect: Evidence from two Patients with Cerebral Achromatopsia

The McCollough effect (ME) is a colour aftereffect contingent on pattern orientation. This effect is generally thought to be mediated by primary visual cortex (V1) although this has remained the subject of some debate. To determine whether V1 is in fact sufficient to subserve the ME, we compared McCollough adaptation in controls to adaptation in two patients with damage to ventrotemporal cortex, resulting in achromatopsia, but who have spared V1. Each of these patients has some residual colour abilities of which he is unaware. Participants performed a 2AFC orientation-discrimination task for pairs of oblique and vertical/horizontal gratings both before and after adaptation to red/green oblique induction gratings. Successful ME induction would manifest itself as an improvement in oblique-orientation discrimination owing to the additional colour cue after adaptation. Indeed, in controls oblique grating discrimination improved post-adaptation. Further, a subdivision of our control group demonstrated successful ME induction despite a lack of conscious awareness of the added colour cue, indicating that conscious colour awareness is not required for ME induction. The patients, however, did not show improvement in oblique-orientation discrimination, indicating a lack of ME induction. This suggests that V1 must be connected to higher cortical colour areas to drive ME induction.

Does Adaptation of Motion-Direction Detectors Affect Bias or Sensitivity of Direction Judgments?

The question how channels tuned to different motion directions contribute to motion perception has been investigated by using motion adaptation to silence certain channels, and then measuring performance in a fine motion-discrimination task. To help constrain models of how the channels become integrated, we examined whether changes in performance stem from reduced accuracy (bias) or from reduced precision (sensitivity) in direction judgments. On a given trial, the observer first adapted to a field of dots moving coherently in a given direction (ranging ±180° from upward), then judged whether the motion of an ensuing test stimulus (ranging ±3°) was left or right of reference. Bias and sensitivity of the psychometric fits were computed for each adapter direction. Relative to baseline performance, post-adaptation judgments showed significant changes in sensitivity that were tightly correlated with overall performance. Meanwhile, bias shifts were found to be weaker and less systematic. Both performance and sensitivity suffered the largest losses at ±60°, with some enhancement at 180°. No similar trends were found in the domain of bias. A regression model, with precision as the sole predictor, captured 97% of the variation in performance; no gains were found in adding bias to the model. Our findings on fine motion-discrimination question the idealized notion of a pure feature detector, as the main impact of adaptation in such a system would be to bias direction judgments away from the adapted direction.

Seeing the invisible: the scope and limits of unconscious processing in binocular rivalry

When an image is presented to one eye and a very different image is presented to the corresponding location of the other eye, the two images compete for conscious representations, such that only one image is visible at a time while the other is suppressed. Called binocular rivalry, this phenomenon and its deviants have been extensively exploited to study the mechanism and neural correlates of consciousness. In this paper, we propose a framework - the unconscious binding hypothesis - to distinguish unconscious processing from conscious processing. According to this framework, the unconscious mind not only encodes individual features but also temporally binds distributed features to give rise to cortical representations; unlike conscious binding, however, unconscious binding is fragile. Under this framework, we review evidence from psychophysical and neuroimaging studies and come to two important conclusions. First, processing of invisible features depends on the "level" of the features as defined by their neural mechanisms. For low-level simple features, prolonged exposure to visual patterns (e.g. tilt) and simple translational motion can alter the appearance of subsequent visible features (i.e. adaptation). For invisible high-level features, complex spiral motion cannot produce adaptation, nor can objects/words enhance subsequent processing of related stimuli (i.e. priming). Yet images of tools can activate the dorsal pathway. Second, processing of invisible features has functional significance. Although invisible central cues cannot orient attention, invisible erotic pictures in the periphery can nevertheless guide attention, likely through emotional arousal; reciprocally, the processing of invisible information can be modulated by attention.

Independent, synchronous access to color and motion features

We investigated the role of attention in pairing superimposed visual features. When moving dots alternate in color and in motion direction, reports of the perceived color and motion reveal an asynchrony: the most accurate reports occur when the motion change precedes the associated color change by approximately 100ms [Moutoussis, K., & Zeki, S. (1997). A direct demonstration of perceptual asynchrony in vision. Proceedings of the Royal Society of London B, 264, 393-399]. This feature binding asynchrony was probed by manipulating endogenous and exogenous attention. First, endogenous attention was manipulated by changing which feature dimension observers were instructed to attend to first. This yielded little effect on the asynchrony. Second, exogenous attention was manipulated by briefly presenting a ring around the target, cueing the report of the color and motion seen within the ring. This reduced or eliminated the apparent latency difference between color and motion. Accuracy was best predicted by timing of each feature relative to the cue rather than the timing of the two features relative to each other, suggesting independent attentional access to the two features with an exogenous attention cue. The timing of attentional cueing affected feature pairing reports as much as the timing of the features themselves.

Illusory colors promote interocular grouping during binocular rivalry

When dissimilar monocular images are presented separately to each of a person's eyes, these images compete for visual dominance, with dominance of one image or the other alternating over time. While this phenomenon, called binocular rivalry, transpires, local image features distributed over space and between the eyes can become visually dominant at the same time; the resulting global figure implicates interocular grouping. Previous studies have suggested that color tends to influence the incidence of global dominance; in this study, we assess whether illusory color can also influence interocular grouping. To test this, we exploited the McCollough effect, an orientation-contingent color aftereffect induced by prolonged adaptation to different colors paired with different orientations. Results show that during binocular rivalry, illusory colors induced by the McCollough adaptation enhance strong interocular grouping relative to preadaptation testing, to an extent comparable in strength with the enhancement induced by real colors. Thus, illusory colors that are present only in an observer's mind are sufficiently potent to influence low-level visual processes such as binocular rivalry.

Contingent aftereffects distinguish conscious and preconscious color processing

The brain can process input without perception, but what distinguishes conscious from preconscious processing? Using aftereffects induced by quickly alternating images, we show that cortical mechanisms track color much faster than perception, responding well to color alternations that are too rapid to be perceptible. The more restricted frequency response of the conscious perception of color suggests that extra integrative steps give conscious color perception a time course substantially slower than that of early cortical mechanisms.

Motion aftereffects specific to surface depth order: beyond binocular disparity

Despite evidence for concurrent processing of motion and stereopsis from psychophysics and neurophysiology, the detailed relationship between depth and motion processing is not yet clear. Using the contingent aftereffect paradigm, we investigated how the order of surfaces presented across depth influenced motion perception. After having observers adapt to two superimposed populations of dots moving in opposite directions at different binocular disparities, we assessed how much of the motion aftereffect (MAE) was specific to absolute disparity and how much was specific to the depth order of the surfaces. The test contained two planes of moving dots at several different pairs of disparities and asked observers to report the MAE direction at one of the planes (the target). In addition to the disparity-contingent MAE (Verstraten, Verlinde, Fredericksen, & van de Grind, 1994), we found MAEs dependent on surface order. When the target surface was in front of another surface, observers more often reported the MAE in the direction opposite to the front adapting surface than the back. This effect was observed despite differences in absolute and relative disparity between the adapted and test surfaces. The results suggest that some motion information is represented in terms of surface depth order.

Implicit Attentional Selection of Bound Visual Features

Traditionally, research on visual attention has been focused on the processes involved in conscious, explicit selection of task-relevant sensory input. Recently, however, it has been shown that attending to a specific feature of an object automatically increases neural sensitivity to this feature throughout the visual field. Here we show that directing attention to a specific color of an object results in attentional modulation of the processing of task-irrelevant and not consciously perceived motion signals that are spatiotemporally associated with this color throughout the visual field. Such implicit cross-feature spreading of attention takes place according to the veridical physical associations between the color and motion signals, even under special circumstances when they are perceptually misbound. These results imply that the units of implicit attentional selection are spatiotemporally colocalized feature clusters that are automatically bound throughout the visual field.

Binding of motion and colour is early and automatic

At what stages of the human visual hierarchy different features are bound together, and whether this binding requires attention, is still highly debated. We used a colour-contingent motion after-effect (CCMAE) to study the binding of colour and motion signals. The logic of our approach was as follows: if CCMAEs can be evoked by targeted adaptation of early motion processing stages, without allowing for feedback from higher motion integration stages, then this would support our hypothesis that colour and motion are bound automatically on the basis of spatiotemporally local information. Our results show for the first time that CCMAE's can be evoked by adaptation to a locally paired opposite-motion dot display, a stimulus that, importantly, is known to trigger direction-specific responses in the primary visual cortex yet results in strong inhibition of the directional responses in area MT of macaques as well as in area MT+ in humans and, indeed, is perceived only as motionless flicker. The magnitude of the CCMAE in the locally paired condition was not significantly different from control conditions where the different directions were spatiotemporally separated (i.e. not locally paired) and therefore perceived as two moving fields. These findings provide evidence that adaptation at an early, local motion stage, and only adaptation at this stage, underlies this CCMAE, which in turn implies that spatiotemporally coincident colour and motion signals are bound automatically, most probably as early as cortical area V1, even when the association between colour and motion is perceptually inaccessible.

Adaptive modulation of color salience contingent upon global form coding and task relevance

Extensive research on local color aftereffects has revealed perceptual consequences of opponent color coding in the retina and the LGN, and of orientation-and/or spatial-frequency-contingent color coding in early cortical visual areas (e.g., V1 and V2). Here, we report a color aftereffect that depends crucially on global-form-contingent color processing. Brief viewing of colored items (passively viewed, ignored, or attended) reduced the salience of the previewed color in a subsequent task of color-based visual search. This color-salience aftereffect was relatively insensitive to variations (between color preview and search) in local image features, but was substantially affected by changes in global configuration (e.g. the presence or absence of perceptual unitization); the global-form dependence of the aftereffect was also modulated by task demands. The overall results suggest that (1) color salience is adaptively modulated (from fixation to fixation), drawing attention to a new color in visual-search contexts, and (2) these modulations seem to be mediated by global-form-and-color-selective neural processing in mid to late stages of the ventral visual pathway (e.g., V4 and IT), in combination with task-dependent feedback from higher cortical areas (e.g., prefrontal cortex).

Object-based cross-feature attentional modulation from color to motion

Object-based theories of visual attention predict that attempting to direct attention to a particular attribute of a visual object will result in an automatic selection of the whole object, including all of its features. It has been assumed, but not critically tested, that the spreading of attention from one feature to another in this manner, i.e. cross-feature attentional (CFA) effects, takes place at object-level stages of processing as opposed to early, local stages. In the present study we disambiguated these options for color-to-motion CFA by contrasting attention's effect on bivectorial transparent versus bivectorial locally paired motion displays. We found that association between features at the global, but not at the local, stage of motion processing leads to cross-feature attentional effects. These findings provide strong psychophysical evidence that such effects are indeed object-based.

Perceptual adaptation: motion parallels orientation

Adaptation phenomena provide striking examples of perceptual plasticity and offer valuable insight into the mechanisms of visual coding. Within the context of recent progress in neurobiology and computational modelling, I review evidence from studies employing psychophysical adaptation to investigate orientation and motion processing. These studies reveal marked similarities between the orientation and motion domains, raising the possibility that common computational principles underlie the processing of orientation and motion despite apparently distinct cortical substrates.

Asynchronous processing in vision

It has been demonstrated that subjects do not report changes in color and direction of motion as being co-incidental when they occur synchronously. Instead, for the changes to be reported as being synchronous, changes in direction of motion must precede changes in color. To explain this observation, some researchers have suggested that the neural processing of color and motion is asynchronous. This interpretation has been criticized on the basis that processing time may not correlate directly and invariantly with perceived time of occurrence. Here we examine this possibility by making use of the color-contingent motion aftereffect. By correlating color states disproportionately with two directions of motion, we produced and measured color-contingent motion aftereffects as a function of the range of physical correlations. The aftereffects observed are consistent with the perceptual correlation between color and motion being different from the physical correlation. These findings demonstrate asynchronous processing for different stimulus attributes, with color being processed more quickly than motion. This suggests that the time course of perceptual experience correlates directly with that of neural activity.