Recurrent processing enhances visual awareness but is not necessary for fast categorization of natural scenes

The functions of consciousness in visual processing

Conscious experiences form a relatively diverse class of psychological phenomena, supported by a range of distinct neurobiological mechanisms. This diversity suggests that consciousness occupies a variety of different functional roles across different task domains, individuals, and species; a position I call functional pluralism. In this paper, I begin to tease out some of the functional contributions that consciousness makes to (human) visual processing. Consolidating research from across the cognitive sciences, I discuss semantic and spatiotemporal processing as specific points of comparison between the functional capabilities of the visual system in the presence and absence of conscious awareness. I argue that consciousness contributes a cluster of functions to visual processing; facilitating, among other things, (i) increased capacities for semantically processing informationally complex visual stimuli, (ii) increased spatiotemporal precision, and (iii) increased capacities for representational integration over large spatiotemporal intervals. This sort of analysis should ultimately yield a plurality of functional markers that can be used to guide future research in the philosophy and science of consciousness, some of which are not captured by popular theoretical frameworks like global workspace theory and information integration theory.

Bidirectionally connected cores in a mouse connectome: towards extracting the brain subnetworks essential for consciousness

Where in the brain consciousness resides remains unclear. It has been suggested that the subnetworks supporting consciousness should be bidirectionally (recurrently) connected because both feed-forward and feedback processing are necessary for conscious experience. Accordingly, evaluating which subnetworks are bidirectionally connected and the strength of these connections would likely aid the identification of regions essential to consciousness. Here, we propose a method for hierarchically decomposing a network into cores with different strengths of bidirectional connection, as a means of revealing the structure of the complex brain network. We applied the method to a whole-brain mouse connectome. We found that cores with strong bidirectional connections consisted of regions presumably essential to consciousness (e.g. the isocortical and thalamic regions, and claustrum) and did not include regions presumably irrelevant to consciousness (e.g. cerebellum). Contrarily, we could not find such correspondence between cores and consciousness when we applied other simple methods that ignored bidirectionality. These findings suggest that our method provides a novel insight into the relation between bidirectional brain network structures and consciousness.

Revealing a competitive dynamic in rapid categorization with object substitution masking

Categorization at different levels of abstraction have distinct time courses, but the different levels are often considered separately. Superordinate-level categorization is typically faster than basic-level categorization at ultra-rapid exposure durations (< 33 ms) while basic-level categorization is faster than superordinate-level categorization at longer exposure durations. This difference may be due to a competitive dynamic between levels of categorization. By leveraging object substitution masking, we found a distinct time course of masking effects for each level of categorization. Superordinate-level categorization showed a masking effect earlier than basic-level categorization. However, when basic-level categorization first showed a masking effects, superordinate-level categorization was spared despite its earlier masking effect. This unique pattern suggests a trade-off between the two levels of categorization over time. Such an effect supports an account of categorization that depends on the interaction of perceptual encoding, selective attention, and competition between levels of category representation.

On the Necessity of Recurrent Processing during Object Recognition: It Depends on the Need for Scene Segmentation

Although feedforward activity may suffice for recognizing objects in isolation, additional visual operations that aid object recognition might be needed for real-world scenes. One such additional operation is figure-ground segmentation, extracting the relevant features and locations of the target object while ignoring irrelevant features. In this study of 60 human participants (female and male), we show objects on backgrounds of increasing complexity to investigate whether recurrent computations are increasingly important for segmenting objects from more complex backgrounds. Three lines of evidence show that recurrent processing is critical for recognition of objects embedded in complex scenes. First, behavioral results indicated a greater reduction in performance after masking objects presented on more complex backgrounds, with the degree of impairment increasing with increasing background complexity. Second, electroencephalography (EEG) measurements showed clear differences in the evoked response potentials between conditions around time points beyond feedforward activity, and exploratory object decoding analyses based on the EEG signal indicated later decoding onsets for objects embedded in more complex backgrounds. Third, deep convolutional neural network performance confirmed this interpretation. Feedforward and less deep networks showed a higher degree of impairment in recognition for objects in complex backgrounds compared with recurrent and deeper networks. Together, these results support the notion that recurrent computations drive figure-ground segmentation of objects in complex scenes.SIGNIFICANCE STATEMENT The incredible speed of object recognition suggests that it relies purely on a fast feedforward buildup of perceptual activity. However, this view is contradicted by studies showing that disruption of recurrent processing leads to decreased object recognition performance. Here, we resolve this issue by showing that how object recognition is resolved and whether recurrent processing is crucial depends on the context in which it is presented. For objects presented in isolation or in simple environments, feedforward activity could be sufficient for successful object recognition. However, when the environment is more complex, additional processing seems necessary to select the elements that belong to the object and by that segregate them from the background.

Scene complexity modulates degree of feedback activity during object detection in natural scenes

Selective brain responses to objects arise within a few hundreds of milliseconds of neural processing, suggesting that visual object recognition is mediated by rapid feed-forward activations. Yet disruption of neural responses in early visual cortex beyond feed-forward processing stages affects object recognition performance. Here, we unite these discrepant findings by reporting that object recognition involves enhanced feedback activity (recurrent processing within early visual cortex) when target objects are embedded in natural scenes that are characterized by high complexity. Human participants performed an animal target detection task on natural scenes with low, medium or high complexity as determined by a computational model of low-level contrast statistics. Three converging lines of evidence indicate that feedback was selectively enhanced for high complexity scenes. First, functional magnetic resonance imaging (fMRI) activity in early visual cortex (V1) was enhanced for target objects in scenes with high, but not low or medium complexity. Second, event-related potentials (ERPs) evoked by target objects were selectively enhanced at feedback stages of visual processing (from ~220 ms onwards) for high complexity scenes only. Third, behavioral performance for high complexity scenes deteriorated when participants were pressed for time and thus less able to incorporate the feedback activity. Modeling of the reaction time distributions using drift diffusion revealed that object information accumulated more slowly for high complexity scenes, with evidence accumulation being coupled to trial-to-trial variation in the EEG feedback response. Together, these results suggest that while feed-forward activity may suffice to recognize isolated objects, the brain employs recurrent processing more adaptively in naturalistic settings, using minimal feedback for simple scenes and increasing feedback for complex scenes.

Different Electrophysiological Correlates of Visual Awareness for Detection and Identification

Detecting the presence of an object is a different process than identifying the object as a particular object. This difference has not been taken into account in designing experiments on the neural correlates of consciousness. We compared the electrophysiological correlates of conscious detection and identification directly by measuring ERPs while participants performed either a task only requiring the conscious detection of the stimulus or a higher-level task requiring its conscious identification. Behavioral results showed that, even if the stimulus was consciously detected, it was not necessarily identified. A posterior electrophysiological signature 200–300 msec after stimulus onset was sensitive for conscious detection but not for conscious identification, which correlated with a later widespread activity. Thus, we found behavioral and neural evidence for elementary visual experiences, which are not yet enriched with higher-level knowledge. The search for the mechanisms of consciousness should focus on the early elementary phenomenal experiences to avoid the confounding effects of higher-level processes.

Top-down preparation modulates visual categorization but not subjective awareness of objects presented in natural backgrounds

Top-down processes are widely assumed to be essential in visual awareness, subjective experience of seeing. However, previous studies have not tried to separate directly the roles of different types of top-down influences in visual awareness. We studied the effects of top-down preparation and object substitution masking (OSM) on visual awareness during categorization of objects presented in natural scene backgrounds. The results showed that preparation facilitated categorization but did not influence visual awareness. OSM reduced visual awareness and impaired categorization. The dissociations between the effects of preparation and OSM on visual awareness and on categorization imply that they influence at different stages of cognitive processing. We propose that preparation influences at the top of the visual hierarchy, whereas OSM interferes with processes occurring at lower levels of the hierarchy. These lower level processes play an essential role in visual awareness.

Differential Visual Processing of Animal Images, with and without Conscious Awareness

The human visual system can quickly and efficiently extract categorical information from a complex natural scene. The rapid detection of animals in a scene is one compelling example of this phenomenon, and it suggests the automatic processing of at least some types of categories with little or no attentional requirements (Li et al., 2002, 2005). The aim of this study is to investigate whether the remarkable capability to categorize complex natural scenes exist in the absence of awareness, based on recent reports that “invisible” stimuli, which do not reach conscious awareness, can still be processed by the human visual system (Pasley et al., 2004; Williams et al., 2004; Fang and He, 2005; Jiang et al., 2006, 2007; Kaunitz et al., 2011a). In two experiments, we recorded event-related potentials (ERPs) in response to animal and non-animal/vehicle stimuli in both aware and unaware conditions in a continuous flash suppression (CFS) paradigm. Our results indicate that even in the “unseen” condition, the brain responds differently to animal and non-animal/vehicle images, consistent with rapid activation of animal-selective feature detectors prior to, or outside of, suppression by the CFS mask.

Early Local Activity in Temporal Areas Reflects Graded Content of Visual Perception

In visual cognitive neuroscience the debate on consciousness is focused on two major topics: the search for the neural correlates of the different properties of visual awareness and the controversy on the graded versus dichotomous nature of visual conscious experience. The aim of this study is to search for the possible neural correlates of different grades of visual awareness investigating the Event Related Potentials to reduced contrast visual stimuli whose perceptual clarity was rated on the four-point Perceptual Awareness Scale. Results revealed a left centro-parietal negative deflection (Visual Awareness Negativity; VAN) peaking at 280–320 ms from stimulus onset, related to the perceptual content of the stimulus, followed by a bilateral positive deflection (Late Positivity; LP) peaking at 510–550 ms over almost all electrodes, reflecting post-perceptual processes performed on such content. Interestingly, the amplitude of both deflections gradually increased as a function of visual awareness. Moreover, the intracranial generators of the phenomenal content (VAN) were found to be located in the left temporal lobe. The present data thus seem to suggest (1) that visual conscious experience is characterized by a gradual increase of perceived clarity at both behavioral and neural level and (2) that the actual content of perceptual experiences emerges from early local activation in temporal areas, without the need of later widespread frontal engagement.

Dense sampling reveals behavioral oscillations in rapid visual categorization

Perceptual systems must create discrete objects and events out of a continuous flow of sensory information. Previous studies have demonstrated oscillatory effects in the behavioral outcome of low-level visual tasks, suggesting a cyclic nature of visual processing as the solution. To investigate whether these effects extend to more complex tasks, a stream of "neutral" photographic images (not containing targets) was rapidly presented (20 ms/image). Embedded were one or two presentations of a randomly selected target image (vehicles and animals). Subjects reported the perceived target category. On dual-presentation trials, the ISI varied systematically from 0 to 600 ms. At randomized timing before first target presentation, the screen was flashed with the intent of creating a phase reset in the visual system. Sorting trials by temporal distance between flash and first target presentation revealed strong oscillations in behavioral performance, peaking at 5 Hz. On dual-target trials, longer ISIs led to reduced performance, implying a temporal integration window for object category discrimination. The "animal" trials exhibited a significant oscillatory component around 5 Hz. Our results indicate that oscillatory effects are not mere fringe effects relevant only with simple stimuli, but are resultant from the core mechanisms of visual processing and may well extend into real-life scenarios.

Behavioral and electrophysiological evidence for fast emergence of visual consciousness

A fundamental unsettled dispute concerns how fast the brain generates subjective visual experiences. Both early visual cortical activation and later activity in fronto-parietal global neuronal workspace correlate with conscious vision, but resolving which of the correlates causally triggers conscious vision has proved a methodological impasse. We show that participants can report whether or not they consciously perceived a stimulus in just over 200 ms. These fast consciousness reports were extremely reliable, and did not include reflexive, unconscious responses. The neural events that causally generate conscious vision must have occurred before these behavioral reports. Analyses on single-trial neural correlates of consciousness revealed that the late cortical processing in fronto-parietal global neuronal workspace (∼300 ms) started after the fastest consciousness reports, ruling out the possibility that this late activity directly reflects the emergence of visual consciousness. The consciousness reports were preceded by a negative amplitude difference (∼160-220 ms) that spread from occipital to frontal cortex, suggesting that this correlate underlies the emergence of conscious vision.

Probing feedforward and feedback contributions to awareness with visual masking and transcranial magnetic stimulation

A number of influential theories posit that visual awareness relies not only on the initial, stimulus-driven (i.e., feedforward) sweep of activation but also on recurrent feedback activity within and between brain regions. These theories of awareness draw heavily on data from masking paradigms in which visibility of one stimulus is reduced due to the presence of another stimulus. More recently transcranial magnetic stimulation (TMS) has been used to study the temporal dynamics of visual awareness. TMS over occipital cortex affects performance on visual tasks at distinct time points and in a manner that is comparable to visual masking. We draw parallels between these two methods and examine evidence for the neural mechanisms by which visual masking and TMS suppress stimulus visibility. Specifically, both methods have been proposed to affect feedforward as well as feedback signals when applied at distinct time windows relative to stimulus onset and as a result modify visual awareness. Most recent empirical evidence, moreover, suggests that while visual masking and TMS impact stimulus visibility comparably, the processes these methods affect may not be as similar as previously thought. In addition to reviewing both masking and TMS studies that examine feedforward and feedback processes in vision, we raise questions to guide future studies and further probe the necessary conditions for visual awareness.