Temporal structure coding with and without awareness

Taking consciousness for real: Increasing the ecological validity of the study of conscious vs. unconscious processes

The study of consciousness has developed well-controlled, rigorous methods for manipulating and measuring consciousness. Yet, in the process, experimental paradigms grew farther away from everyday conscious and unconscious processes, which raises the concern of ecological validity. In this review, we suggest that the field can benefit from adopting a more ecological approach, akin to other fields of cognitive science. There, this approach challenged some existing hypotheses, yielded stronger effects, and enabled new research questions. We argue that such a move is critical for studying consciousness, where experimental paradigms tend to be artificial and small effect sizes are relatively prevalent. We identify three paths for doing so-changing the stimuli and experimental settings, changing the measures, and changing the research questions themselves-and review works that have already started implementing such approaches. While acknowledging the inherent challenges, we call for increasing ecological validity in consciousness studies.

Temporal integration by multi-level regularities fosters the emergence of dynamic conscious experience

The relationship between integration and awareness is central to contemporary theories and research on consciousness. Here, we investigated whether and how information integration over time, by incorporating the underlying regularities, contributes to our awareness of the dynamic world. Using binocular rivalry, we demonstrated that structured visual streams, constituted by shape, motion, or idiom sequences containing perceptual- or semantic-level regularities, predominated over their nonstructured but otherwise matched counterparts in the competition for visual awareness. Despite the apparent resemblance, a substantial dissociation of the observed rivalry advantages emerged between perceptual- and semantic-level regularities. These effects stem from nonconscious and conscious temporal integration processes, respectively, with the former but not the latter being vulnerable to perturbations in the spatiotemporal integration window. These findings corroborate the essential role of structure-guided information integration in visual awareness and highlight a multi-level mechanism where temporal integration by perceptually and semantically defined regularities fosters the emergence of continuous conscious experience.

Dissociation between temporal attention and Consciousness: Unconscious temporal cue induces temporal expectation effect

The debate over the independence of attention and consciousness is ongoing. Prior studies have established that invisible spatial cues can direct attention. However, our exploration extends beyond spatial dimensions to temporal information as a potent guide for attention. A intriguing question arises: Can unconscious temporal cues trigger attentional orienting? To investigate, we employed a modified reaction-time task in Experiments 1 and 2, using Gabor stimuli or human facial stimuli as temporal cues rendered invisible through continuous flash suppression. We aimed to uncover temporal expectation effects (TE effects) without conscious awareness. Moreover, Experiments 3 and 4 probed the boundaries of this unconscious processing, assessing whether conscious temporal cues could modulate TE effects. Our results confirm that invisible temporal cues can induce TE effects, and these effects can be overridden by conscious temporal cues. This dissociation between temporal attention and consciousness provide a new perspective on our understanding of their relationship.

Cortical encoding of rhythmic kinematic structures in biological motion

Biological motion (BM) perception is of great survival value to human beings. The critical characteristics of BM information lie in kinematic cues containing rhythmic structures. However, how rhythmic kinematic structures of BM are dynamically represented in the brain and contribute to visual BM processing remains largely unknown. Here, we probed this issue in three experiments using electroencephalogram (EEG). We found that neural oscillations of observers entrained to the hierarchical kinematic structures of the BM sequences (i.e., step-cycle and gait-cycle for point-light walkers). Notably, only the cortical tracking of the higher-level rhythmic structure (i.e., gait-cycle) exhibited a BM processing specificity, manifested by enhanced neural responses to upright over inverted BM stimuli. This effect could be extended to different motion types and tasks, with its strength positively correlated with the perceptual sensitivity to BM stimuli at the right temporal brain region dedicated to visual BM processing. Modeling results further suggest that the neural encoding of spatiotemporally integrative kinematic cues, in particular the opponent motions of bilateral limbs, drives the selective cortical tracking of BM information. These findings underscore the existence of a cortical mechanism that encodes periodic kinematic features of body movements, which underlies the dynamic construction of visual BM perception.

Looking more masculine among females: Spatial context modulates gender perception of face and biological motion

Perception of visual information highly depends on spatial context. For instance, perception of a low-level visual feature, such as orientation, can be shifted away from its surrounding context, exhibiting a simultaneous contrast effect. Although previous studies have demonstrated the adaptation aftereffect of gender, a high-level visual feature, it remains largely unknown whether gender perception can also be shaped by a simultaneously presented context. In the present study, we found that the gender perception of a central face or a point-light walker was repelled away from the gender of its surrounding faces or walkers. A norm-based opponent model of lateral inhibition, which accounts for the adaptation aftereffect of high-level features, can also excellently fit the simultaneous contrast effect. But different from the reported contextual effect of low-level features, the simultaneous contrast effect of gender cannot be observed when the centre and the surrounding stimuli are from different categories, or when the surrounding stimuli are suppressed from awareness. These findings on one hand reveal a resemblance between the simultaneous contrast effect and the adaptation aftereffect of high-level features, on the other hand highlight different biological mechanisms underlying the contextual effects of low- and high-level visual features.

The functions of consciousness in visual processing

Conscious experiences form a relatively diverse class of psychological phenomena, supported by a range of distinct neurobiological mechanisms. This diversity suggests that consciousness occupies a variety of different functional roles across different task domains, individuals, and species; a position I call functional pluralism. In this paper, I begin to tease out some of the functional contributions that consciousness makes to (human) visual processing. Consolidating research from across the cognitive sciences, I discuss semantic and spatiotemporal processing as specific points of comparison between the functional capabilities of the visual system in the presence and absence of conscious awareness. I argue that consciousness contributes a cluster of functions to visual processing; facilitating, among other things, (i) increased capacities for semantically processing informationally complex visual stimuli, (ii) increased spatiotemporal precision, and (iii) increased capacities for representational integration over large spatiotemporal intervals. This sort of analysis should ultimately yield a plurality of functional markers that can be used to guide future research in the philosophy and science of consciousness, some of which are not captured by popular theoretical frameworks like global workspace theory and information integration theory.

Role of visual awareness on semantic integration of sequentially presented words: An fMRI study

Reading comprehension requires the semantic integration of words across space and time. However, it remains unclear whether comprehension requires visual awareness for such semantic integration. Compared to earlier studies that investigated semantic integration indirectly from its priming effect, we used functional magnetic resonance imaging (fMRI) to directly examine the processes of semantic integration with or without visual awareness. Specifically, we manipulated participants' visual awareness by continuous flash suppression (CFS) while they viewed a meaningful sequence of four Chinese words (i.e., an idiom) or its meaningless counterpart (i.e., a random sequence). Behaviorally, participants had better recognition memory for idioms than random sequences only when their visual awareness was interfered rather than blocked by CFS. Neurally, semantics-processing areas, such as the superior temporal gyrus and inferior frontal gyrus, were significantly activated only when participants were aware of word sequences, be they meaningful or meaningless. By contrast, orthography-processing areas, such as the fusiform gyrus and inferior occipital gyrus, were significantly activated regardless of visual awareness or word sequence. Taken together, these results suggest that visual awareness modules the functioning of the semantic neural network in the brain and facilitates reading comprehension.

The functional contributions of consciousness

The most widely endorsed philosophical and scientific theories of consciousness assume that it contributes a single functional capacity to an organism's information processing toolkit. However, conscious processes are a heterogeneous class of psychological phenomena supported by a variety of neurobiological mechanisms. This suggests a plurality of functional contributions of consciousness (FCCs), in the sense that conscious experience facilitates different functional capacities in different psychological domains. In this paper, I first develop a general methodological framework for isolating the psychological functions that are associated with conscious experience. I then use this to show that the leading accounts-Global Workspace Theories, Higher Order Thought Theory and Information Integration Theory-all fail to acknowledge this functional pluralism, which limits their applicability as theories of consciousness.

Is Neuromorphic MNIST Neuromorphic? Analyzing the Discriminative Power of Neuromorphic Datasets in the Time Domain

A major characteristic of spiking neural networks (SNNs) over conventional artificial neural networks (ANNs) is their ability to spike, enabling them to use spike timing for coding and efficient computing. In this paper, we assess if neuromorphic datasets recorded from static images are able to evaluate the ability of SNNs to use spike timings in their calculations. We have analyzed N-MNIST, N-Caltech101 and DvsGesture along these lines, but focus our study on N-MNIST. First we evaluate if additional information is encoded in the time domain in a neuromorphic dataset. We show that an ANN trained with backpropagation on frame-based versions of N-MNIST and N-Caltech101 images achieve 99.23 and 78.01% accuracy. These are comparable to the state of the art-showing that an algorithm that purely works on spatial data can classify these datasets. Second we compare N-MNIST and DvsGesture on two STDP algorithms, RD-STDP, that can classify only spatial data, and STDP-tempotron that classifies spatiotemporal data. We demonstrate that RD-STDP performs very well on N-MNIST, while STDP-tempotron performs better on DvsGesture. Since DvsGesture has a temporal dimension, it requires STDP-tempotron, while N-MNIST can be adequately classified by an algorithm that works on spatial data alone. This shows that precise spike timings are not important in N-MNIST. N-MNIST does not, therefore, highlight the ability of SNNs to classify temporal data. The conclusions of this paper open the question-what dataset can evaluate SNN ability to classify temporal data?

Windows of Integration Hypothesis Revisited

In the ongoing research of the functions of consciousness, special emphasis has been put on integration of information: the ability to combine different signals into a coherent, unified one. Several theories of consciousness hold that this ability depends on - or at least goes hand in hand with - conscious processing. Yet some empirical findings have suggested otherwise, claiming that integration of information could take place even without awareness. Trying to reconcile this apparent contradiction, the "windows of integration" (WOI) hypothesis claims that conscious access enables signal processing over large integration windows. The hypothesis applies to integration windows defined either temporally, spatially, or semantically. In this review, we explain the hypothesis and re-examine it in light of new studies published since it was suggested. In line with the hypothesis, these studies provide compelling evidence for unconscious integration, but also demonstrate its limits with respect to time, space, and semantic distance. The review further highlights open questions that still need to be pursued to demonstrate the applicability of the WOI hypothesis as a guiding principle for understanding the depth and scope of unconscious processes.

Seeing a Page in a Flipbook: Shorter Visual Temporal Integration Windows in 2-Year-Old Toddlers with Autism Spectrum Disorder

Individuals with autism spectrum disorder (ASD) experience differences in visual temporal processing, the part of vision responsible for parsing continuous input into discrete objects and events. Here we investigated temporal processing in 2-year-old toddlers diagnosed with ASD and age-matched typically developing (TD) toddlers. We used a visual search task where the visibility of the target was determined by the pace of a display sequence. On integration trials, each display viewed alone had no visible target, but if integrated over time, the target became visible. On segmentation trials, the target became visible only when displays were perceptually segmented. We measured the percent of trials when participants fixated the target as a function of the stimulus onset asynchrony (SOA) between displays. We computed the crossover point of the integration and segmentation performance functions for each group, an estimate of the temporal integration window (TIW), the period in which visual input is combined. We found that both groups of toddlers had significantly longer TIWs (125 ms) than adults (65 ms) from previous studies using the same paradigm, and that toddlers with ASD had significantly shorter TIWs (108 ms) than chronologically age-matched TD controls (142 ms). LAY SUMMARY: We investigated how young children, with and without autism, organize dynamic visual information across time, using a visual search paradigm. We found that toddlers with autism had higher temporal resolution than typically developing (TD) toddlers of the same age - that is, they are more likely to be able to detect rapid change across time, relative to TD toddlers. These differences in visual temporal processing can impact how one sees, interprets, and interacts with the world. Autism Res 2021, 14: 946-958. © 2020 International Society for Autism Research and Wiley Periodicals LLC.

Predictions from masked motion with and without obstacles

Predicting the future is essential for organisms like Homo sapiens, who live in a dynamic and ever-changing world. Previous research has established that conscious stimuli can lead to non-conscious predictions. Here we examine whether masked stimuli can also induce such predictions. We use masked movement-with and without obstacles-to examine predictions from masked stimuli. In six experiments a moving object was masked using continuous flash suppression (CFS). A few hundred milliseconds after the object had disappeared, a conscious probe appeared in a location that was either consistent with the masked stimulus or not. In Experiments 1-3 the movement was linear, and reaction times (RTs) indicated predictions that were based on direction and speed of movement. In Experiment 4, the masked moving object collided with an obstacle and then disappeared. Predictions in this case should reflect deflection, and indeed reaction times revealed predictions on the deflection route. In Experiments 5 and 6 we introduce an innovative way of using eye-tracking during continuous flash suppression (CFS) and report physiological evidence-in the forms of eye-movements-for masked stimuli induced predictions. We thus conclude that humans can use dynamic masked stimuli to generate active predictions about the future, and use these predictions to guide behavior. We also discuss the possible interpretations of these findings in light of the current scientific discussion regarding the relation between masked presentation, subliminal perception and awareness measurement methods.

B or 13? Unconscious Top-Down Contextual Effects at the Categorical but Not the Lexical Level

Contextual effects require integration of top-down predictions and bottom-up visual information. Given the widely assumed link between integration and consciousness, we asked whether contextual effects require consciousness. In two experiments (total N = 60), an ambiguous stimulus (which could be read as either B or 13) was presented alongside masked numbers (12 and 14) or letters (A and C). Context biased stimulus classification when it was consciously and unconsciously perceived. However, unconsciously perceived contexts evoked smaller effects. This finding was replicated and generalized into another language in a further experiment (N = 46) using a different set of stimuli, strengthening the claim that symbolic contextual effects can occur without awareness. Moreover, four experiments (total N = 160) suggested that these unconscious effects might be limited to the categorical level (numbers context vs. letters context) and do not extend to the lexical level (words context vs. nonwords context). Taken together, our results suggest that although consciousness may not be necessary for effects that require simple integration or none at all, it is nevertheless required for integration over larger semantic windows.

Visual temporal integration windows are adult-like in 5- to 7-year-old children

The visual system must organize dynamic input into useful percepts across time, balancing between stability and sensitivity to change. The temporal integration window (TIW) has been hypothesized to underlie this balance: If two or more stimuli fall within the same TIW, they are integrated into a single percept; those that fall in different windows are segmented (Arnett & Di Lollo, 1979; Wutz, Muschter, van Koningsbruggen, Weisz, & Melcher, 2016). Visual TIWs have been studied in adults, showing average windows of 65 ms (Wutz et al., 2016); however, it is unclear how windows develop through early childhood. Here we measured TIWs in 5- to 7-year-old children and adults, using a variant of the missing dot task (Di Lollo, 1980; Wutz et al. 2016), in which integration and segmentation thresholds were measured within the same participant, using the same stimuli. Participants saw a sequence of two displays separated by an interstimulus interval (ISI) that determined the visibility of a visual search target. Longer ISIs increased the likelihood of detecting a segmentation target (but decreased detection for the integration target) although shorter ISIs increased the likelihood of detecting the integration target (but decreased detection of the segmentation target). We could then estimate the TIW by measuring the point at which these two functions intersect. Children's TIWs (M = 68 ms) were comparable to adults' (M = 73 ms) with no appreciable age trend within our sample, indicating that TIWs reach adult levels by approximately 5 years of age.

"Real-life" continuous flash suppression (CFS)-CFS with real-world objects using augmented reality goggles

Continuous flash suppression (CFS) is a popular method for suppressing visual stimuli from awareness for relatively long periods. Thus far, this method has only been used for suppressing two-dimensional images presented on screen. We present a novel variant of CFS, termed “real-life” CFS, in which a portion of the actual immediate surroundings of an observer—including three-dimensional, real-life objects—can be rendered unconscious. Our method uses augmented reality goggles to present subjects with CFS masks to the dominant eye, leaving the nondominant eye exposed to the real world. In three experiments we demonstrated that real objects can indeed be suppressed from awareness for several seconds, on average, and that the suppression duration is comparable to that obtained using classic, on-screen CFS. As supplementary information, we further provide an example of experimental code that can be modified for future studies. This technique opens the way to new questions in the study of consciousness and its functions.

Role of consciousness in temporal integration of semantic information

Previous studies found that word meaning can be processed unconsciously. Yet it remains unknown whether temporally segregated words can be integrated into a holistic meaningful phrase without consciousness. The first four experiments were designed to examine this by sequentially presenting the first three words of Chinese four-word idioms as prime to one eye and dynamic Mondrians to the other (i.e., the continuous flash suppression paradigm; CFS). An unmasked target word followed the three masked words in a lexical decision task. Results from such invisible (CFS) condition were compared with the visible condition where the preceding words were superimposed on the Mondrians and presented to both eyes. Lower performance in behavioral experiments and larger N400 event-related potentials (ERP) component for incongruent- than congruent-ending words were found in the visible condition. However, no such congruency effect was found in the invisible condition, even with enhanced statistical power and top-down attention, and with several potential confounding factors (contrast-dependent processing, long interval, no conscious training) excluded. Experiment 5 demonstrated that familiarity of word orientation without temporal integration can be processed unconsciously, excluding the possibility of general insensitivity of our paradigm. The overall result pattern therefore suggests that consciousness plays an important role in semantic temporal integration in the conditions we tested.

The temporal frequency tuning of continuous flash suppression reveals peak suppression at very low frequencies

Continuous flash suppression (CFS) is a psychophysical technique where a rapidly changing Mondrian pattern viewed by one eye suppresses the target in the other eye for several seconds. Despite the widespread use of CFS to study unconscious visual processes, the temporal tuning of CFS suppression is currently unknown. In the present study we used spatiotemporally filtered dynamic noise as masking stimuli to probe the temporal characteristics of CFS. Surprisingly, we find that suppression in CFS peaks very prominently at approximately 1 Hz, well below the rates typically used in CFS studies (10 Hz or more). As well as a strong bias to low temporal frequencies, CFS suppression is greater for high spatial frequencies and increases with increasing masker contrast, indicating involvement of parvocellular/ventral mechanisms in the suppression process. These results are reminiscent of binocular rivalry, and unifies two phenomenon previously thought to require different explanations.

Does Temporal Integration Occur for Unrecognizable Words in Visual Crowding?

Visual crowding—the inability to see an object when it is surrounded by flankers in the periphery—does not block semantic activation: unrecognizable words due to visual crowding still generated robust semantic priming in subsequent lexical decision tasks. Based on the previous finding, the current study further explored whether unrecognizable crowded words can be temporally integrated into a phrase. By showing one word at a time, we presented Chinese four-word idioms with either a congruent or incongruent ending word in order to examine whether the three preceding crowded words can be temporally integrated to form a semantic context so as to affect the processing of the ending word. Results from both behavioral (Experiment 1) and Event-Related Potential (Experiment 2 and 3) measures showed congruency effect in only the non-crowded condition, which does not support the existence of unconscious multi-word integration. Aside from four-word idioms, we also found that two-word (modifier + adjective combination) integration—the simplest kind of temporal semantic integration—did not occur in visual crowding (Experiment 4). Our findings suggest that integration of temporally separated words might require conscious awareness, at least under the timing conditions tested in the current study.

Sustained invisibility through crowding and continuous flash suppression: a comparative review

The study of non-conscious vision benefits from several alternative methods that allow the suppression of an image from awareness. Here, we present and compare two of them that are particularly well-suited for creating sustained periods of invisibility, namely visual crowding and continuous flash suppression (CFS). In visual crowding, a peripheral image surrounded by similar flankers becomes impossible to discriminate. In CFS, an image presented to one eye becomes impossible to detect when rapidly changing patterns are presented to the other eye. After discussing the experimental specificities of each method, we give a comparative overview of the main empirical results derived from them, from the mere analysis of low-level features to the extraction of semantic contents. We conclude by proposing practical guidelines and future directions to obtain more quantitative and systematic measures of non-conscious processes under prolonged stimulation.