A method for investigating binocular rivalry in real-time with the steady-state VEP

Binocularly incongruent, multifrequency-coded SSVEP in VR: feasibility and characteristics

Objective.Steady-state visual evoked potentials (SSVEPs) in response to flickering stimuli are popular in brain-computer interfacing but their implementation in virtual reality (VR) offers new opportunities also for clinical applications. While traditional SSVEP target selection relies on single-frequency stimulation of both eyes simultaneously, further called congruent stimulation, recent studies attempted to improve the information transfer rate by using dual-frequency-coded SSVEP where each eye is presented with a stimulus flickering at a different frequency, further called incongruent stimulation. However, few studies have investigated incongruent multifrequency-coded SSVEP (MultiIncong-SSVEP).Approach.This paper reports on a systematical investigation of incongruent dual-, triple-, and quadruple-frequency-coded SSVEP for use in VR, several of which are entirely novel, and compares their performance with that of congruent dual-frequency-coded SSVEP.Main results.We were able to confirm the presence of a summation effect when comparing monocular- and binocular single-frequency congruent stimulation, and a suppression effect when comparing monocular- and binocular dual-frequency incongruent stimulation, as both tap into the binocular vision capabilities which, when hampered, could signal amblyopia.Significance.In sum, our findings not only evidence the potential of VR-based binocularly incongruent SSVEP but also underscore the importance of paradigm choice and decoder design to optimize system performance and user comfort.

Measuring attentional selection of object categories using hierarchical frequency tagging

In the present study, we used Hierarchical Frequency Tagging (Gordon et al., 2017) to investigate in electroencephalography how different levels of the neural processing hierarchy interact with category-selective attention during visual object recognition. We constructed stimulus sequences of cyclic wavelet scrambled face and house stimuli at two different frequencies (f1 = 0.8 Hz and f2 = 1 Hz). For each trial, two stimulus sequences of different frequencies were superimposed and additionally augmented by a sinusoidal contrast modulation with f3 = 12.5 Hz. This allowed us to simultaneously assess higher level processing using semantic wavelet-induced frequency-tagging (SWIFT) and processing in earlier visual levels using steady-state visually evoked potentials (SSVEPs), along with their intermodulation (IM) components. To investigate the category specificity of the SWIFT signal, we manipulated the category congruence between target and distractor by superimposing two sequences containing stimuli from the same or different object categories. Participants attended to one stimulus (target) and ignored the other (distractor). Our results showed successful tagging of different levels of the cortical hierarchy. Using linear mixed-effects modeling, we detected different attentional modulation effects on lower versus higher processing levels. SWIFT and IM components were substantially increased for target versus distractor stimuli, reflecting attentional selection of the target stimuli. In addition, distractor stimuli from the same category as targets elicited stronger SWIFT signals than distractor stimuli from a different category indicating category-selective attention. In contrast, for IM components, this category-selective attention effect was largely absent, indicating that IM components probably reflect more stimulus-specific processing.

Tracking rivalry with neural rhythms: multivariate SSVEPs reveal perception during binocular rivalry

The contents of awareness can substantially change without any modification to the external world. Such effects are exemplified in binocular rivalry, where a different stimulus is presented to each eye causing instability in perception. This phenomenon has made binocular rivalry a quintessential method for studying consciousness and the necessary neural correlates for awareness. However, to conduct research on binocular rivalry usually requires self-reports of changes in percept, which can produce confounds and exclude states and contexts where self-reports are undesirable or unreliable. Here, we use a novel multivariate spatial filter dubbed 'Rhythmic Entrainment Source Separation' to extract steady state visual evoked potentials from electroencephalography data. We show that this method can be used to quantify the perceptual switch-rate of participants during binocular rivalry and therefore may be valuable in experimental contexts where self-reports are methodologically problematic or impossible, particularly as an adjunct. Our analyses also reveal that 'no-report' conditions may affect the deployment of attention and thereby neural correlates, another important consideration for consciousness research.

Frequency-dependent dynamics of steady-state visual evoked potentials under sustained flicker stimulation

Steady-state visual evoked potentials (SSVEP) are electroencephalographic signals elicited when the brain is exposed to a visual stimulus with a steady frequency. We analyzed the temporal dynamics of SSVEP during sustained flicker stimulation at 5, 10, 15, 20 and 40 Hz. We found that the amplitudes of the responses were not stable over time. For a 5 Hz stimulus, the responses progressively increased, while, for higher flicker frequencies, the amplitude increased during the first few seconds and often showed a continuous decline afterward. We hypothesize that these two distinct sets of frequency-dependent SSVEP signal properties reflect the contribution of parvocellular and magnocellular visual pathways generating sustained and transient responses, respectively. These results may have important applications for SSVEP signals used in research and brain-computer interface technology and may contribute to a better understanding of the frequency-dependent temporal mechanisms involved in the processing of prolonged periodic visual stimuli.

Adaptation of Ocular Opponency Neurons Mediates Attention-Induced Ocular Dominance Plasticity

Previous research has shown that ocular dominance can be biased by prolonged attention to one eye. The ocular-opponency-neuron model of binocular rivalry has been proposed as a candidate account for this phenomenon. Yet direct neural evidence is still lacking. By manipulating the contrast of dichoptic testing gratings, here we measured the steady-state visually evoked potentials (SSVEPs) at the intermodulation frequencies to selectively track the activities of ocular-opponency-neurons before and after the "dichoptic-backward-movie" adaptation. One hour of adaptation caused a shift of perceptual and neural ocular dominance towards the unattended eye. More importantly, we found a decrease in the intermodulation SSVEP response after adaptation, which was significantly greater when high-contrast gratings were presented to the attended eye than when they were presented to the unattended eye. These results strongly support the view that the adaptation of ocular-opponency-neurons contributes to the ocular dominance plasticity induced by prolonged eye-based attention.

Negligible contribution of adaptation of ocular opponency neurons to the effect of short-term monocular deprivation

Introduction

Modeling work on binocular rivalry has described how ocular opponency neurons represent interocular conflict. These neurons have recently been considered to mediate an ocular dominance shift to the eye that has viewed a backward movie for long during which time the other eye is presented with a regular movie. Unlike typical short-term monocular deprivation, the visual inputs are comparable across eyes in that "dichoptic-backward-movie" paradigm. Therefore, it remains unclear whether the ocular opponency neurons are also responsible for the short-term monocular deprivation effect which is prevalently explained by the homeostatic compensation theory. We designed two experiments from distinct perspectives to investigate this question.

Methods

In Experiment 1, we mitigated the imbalance in the activity of opponency neurons between the two eyes during monocular deprivation by presenting video stimuli alternately. In Experiment 2, we directly evaluated the response of opponency neurons before and after monocular deprivation using SSVEP techniques.

Results

Consistent with each other, both experiments failed to provide reliable evidence supporting the involvement of ocular opponency neurons in the short-term monocular deprivation effect.

Discussion

Our results suggest that ocular opponency neurons may not play an essential role in the short-term monocular deprivation effect, potentially due to interference from the homeostatic plasticity mechanism.

An Accumulating Neural Signal Underlying Binocular Rivalry Dynamics

During binocular rivalry, conflicting images are presented one to each eye and perception alternates stochastically between them. Despite stable percepts between alternations, modeling suggests that neural signals representing the two images change gradually, and that the duration of stable percepts are determined by the time required for these signals to reach a threshold that triggers an alternation. However, direct physiological evidence for such signals has been lacking. Here, we identify a neural signal in the human visual cortex that shows these predicted properties. We measured steady-state visual evoked potentials (SSVEPs) in 84 human participants (62 females, 22 males) who were presented with orthogonal gratings, one to each eye, flickering at different frequencies. Participants indicated their percept while EEG data were collected. The time courses of the SSVEP amplitudes at the two frequencies were then compared across different percept durations, within participants. For all durations, the amplitude of signals corresponding to the suppressed stimulus increased and the amplitude corresponding to the dominant stimulus decreased throughout the percept. Critically, longer percepts were characterized by more gradual increases in the suppressed signal and more gradual decreases of the dominant signal. Changes in signals were similar and rapid at the end of all percepts, presumably reflecting perceptual transitions. These features of the SSVEP time courses are well predicted by a model in which perceptual transitions are produced by the accumulation of noisy signals. Identification of this signal underlying binocular rivalry should allow strong tests of neural models of rivalry, bistable perception, and neural suppression.SIGNIFICANCE STATEMENT During binocular rivalry, two conflicting images are presented to the two eyes and perception alternates between them, with switches occurring at seemingly random times. Rivalry is an important and longstanding model system in neuroscience, used for understanding neural suppression, intrinsic neural dynamics, and even the neural correlates of consciousness. All models of rivalry propose that it depends on gradually changing neural activity that on reaching some threshold triggers the perceptual switches. This manuscript reports the first physiological measurement of neural signals with that set of properties in human participants. The signals, measured with EEG in human observers, closely match the predictions of recent models of rivalry, and should pave the way for much future work.

Cortical signatures of visual body representation develop in human infancy

Human infants cannot report their experiences, limiting what we can learn about their bodily awareness. However, visual cortical responses to the body, linked to visual awareness and selective attention in adults, can be easily measured in infants and provide a promising marker of bodily awareness in early life. We presented 4- and 8-month-old infants with a flickering (7.5 Hz) video of a hand being stroked and recorded steady-state visual evoked potentials (SSVEPs). In half of the trials, the infants also received tactile stroking synchronously with visual stroking. The 8-month-old, but not the 4-month-old infants, showed a significant enhancement of SSVEP responses when they received tactile stimulation concurrent with the visually observed stroking. Follow-up experiments showed that this enhancement did not occur when the visual hand was presented in an incompatible posture with the infant's own body or when the visual stimulus was a body-irrelevant video. Our findings provide a novel insight into the development of bodily self-awareness in the first year of life.

Laplacian reference is optimal for steady-state visual-evoked potentials

Steady-state visual-evoked potentials (SSVEPs) are widely used in human neuroscience studies and applications such as brain-computer interfaces (BCIs). Surprisingly, no previous study has systematically evaluated different reference methods for SSVEP analysis, despite that signal reference is crucial for the proper assessment of neural activities. In the present study, using four datasets from our previous SSVEP studies (Chen J, Valsecchi M, Gegenfurtner KR. J Neurophysiol 118: 749-754, 2017; Chen J, Valsecchi M, Gegenfurtner KR. Neuropsychologia 102: 206-216, 2017; Chen J, McManus M, Valsecchi M, Harris LR, Gegenfurtner KR. J Vis 19: 8, 2019) and three public datasets from other studies (Baker DH, Vilidaite G, Wade AR. PLoS Comput Biol 17: e1009507, 2021; Lygo FA, Richard B, Wade AR, Morland AB, Baker DH. NeuroImage 230: 117780, 2021; Vilidaite G, Norcia AM, West RJH, Elliott CJH, Pei F, Wade AR, Baker DH. Proc R Soc B 285: 20182255, 2018), we compared four reference methods: monopolar reference, common average reference, averaged-mastoids reference, and Laplacian reference. The quality of the resulting SSVEP signals was compared in terms of both signal-to-noise ratios (SNRs) and reliability. The results showed that Laplacian reference, which uses signals at the maximally activated electrode after subtracting the average of the nearby electrodes to reduce common noise, gave rise to the highest SNRs. Furthermore, the Laplacian reference resulted in SSVEP signals that were highly reliable across recording sessions or trials. These results suggest that Laplacian reference is optimal for SSVEP studies and applications. Laplacian reference is especially advantageous for SSVEP experiments where short preparation time is preferred as it requires only data from the maximally activated electrode and a few surrounding electrodes.NEW & NOTEWORTHY The present study provides a comprehensive evaluation of the use of different reference methods for steady-state visual-evoked potentials (SSVEPs) and has found that Laplacian reference increases signal-to-noise ratios (SNRs) and enhances reliabilities of SSVEP signals. Thus, the results suggest that Laplacian reference is optimal for SSVEP analysis.

Effects of altered-reality training on interocular disinhibition in amblyopia

Training of viewing an altered-reality environment dichoptically has been found to reactivate human adult ocular dominance plasticity, allowing improvement of vision for amblyopia. One suspected mechanism for this training effect is ocular dominance rebalancing through interocular disinhibition. Here, we investigated whether the training modulated the neural responses reflecting interocular inhibition. Thirteen patients with amblyopia and 11 healthy controls participated in this study. Before and after six daily altered-reality training sessions, participants watched flickering video stimuli with their steady-state visually evoked potential (SSVEP) signals recorded simultaneously. We assessed the amplitude of SSVEP response at intermodulation frequencies, which was a potential neural indicator of interocular suppression. The results showed that training weakened the intermodulation response only in the amblyopic group, which was in agreement with the hypothesis that the training reduced interocular suppression specific to amblyopia. Moreover, even one month after the training ended, we could still observe this neural training effect. These findings provide preliminary neural evidence in support of the disinhibition account for treating amblyopia. We also explain these results with the ocular opponency model, which, to our knowledge, is the first time for this binocular rivalry model to be used in explaining long-term ocular dominance plasticity.

Steady state visual evoked potentials reveal a signature of the pitch-size crossmodal association in visual cortex

Crossmodal correspondences describe our tendency to associate sensory features from different modalities with each other, such as the pitch of a sound with the size of a visual object. While such crossmodal correspondences (or associations) are described in many behavioural studies their neurophysiological correlates remain unclear. Under the current working model of multisensory perception both a low- and a high-level account seem plausible. That is, the neurophysiological processes shaping these associations could commence in low-level sensory regions, or may predominantly emerge in high-level association regions of semantic and object identification networks. We exploited steady-state visual evoked potentials (SSVEP) to directly probe this question, focusing on the associations between pitch and the visual features of size, hue or chromatic saturation. We found that SSVEPs over occipital regions are sensitive to the congruency between pitch and size, and a source analysis pointed to an origin around primary visual cortices. We speculate that this signature of the pitch-size association in low-level visual cortices reflects the successful pairing of congruent visual and acoustic object properties and may contribute to establishing causal relations between multisensory objects. Besides this, our study also provides a paradigm can be exploited to study other crossmodal associations involving visual stimuli in the future.

New insights into binocular rivalry from the reconstruction of evolving percepts using model network dynamics

When the two eyes are presented with highly distinct stimuli, the resulting visual percept generally switches every few seconds between the two monocular images in an irregular fashion, giving rise to a phenomenon known as binocular rivalry. While a host of theoretical studies have explored potential mechanisms for binocular rivalry in the context of evoked model dynamics in response to simple stimuli, here we investigate binocular rivalry directly through complex stimulus reconstructions based on the activity of a two-layer neuronal network model with competing downstream pools driven by disparate monocular stimuli composed of image pixels. To estimate the dynamic percept, we derive a linear input-output mapping rooted in the non-linear network dynamics and iteratively apply compressive sensing techniques for signal recovery. Utilizing a dominance metric, we are able to identify when percept alternations occur and use data collected during each dominance period to generate a sequence of percept reconstructions. We show that despite the approximate nature of the input-output mapping and the significant reduction in neurons downstream relative to stimulus pixels, the dominant monocular image is well-encoded in the network dynamics and improvements are garnered when realistic spatial receptive field structure is incorporated into the feedforward connectivity. Our model demonstrates gamma-distributed dominance durations and well obeys Levelt's four laws for how dominance durations change with stimulus strength, agreeing with key recurring experimental observations often used to benchmark rivalry models. In light of evidence that individuals with autism exhibit relatively slow percept switching in binocular rivalry, we corroborate the ubiquitous hypothesis that autism manifests from reduced inhibition in the brain by systematically probing our model alternation rate across choices of inhibition strength. We exhibit sufficient conditions for producing binocular rivalry in the context of natural scene stimuli, opening a clearer window into the dynamic brain computations that vary with the generated percept and a potential path toward further understanding neurological disorders.

InFoRM (Indicate-Follow-Replay-Me): A novel method to measure perceptual multistability dynamics using continuous data tracking and validated estimates of visual introspection

Perceptual multistability, e.g. Binocular Rivalry, has been intensively used as a tool to study visual consciousness. Current methods to assess multistability do not capture all potentially occurring perceptual states, provide no estimate of introspection, and lack continuous, high-temporal resolution to resolve perceptual changes between states and within mixed perceptual states. We introduce InFoRM (Indicate-Follow-Replay-Me), a four-phase method that (1) trains a participant to self-generate estimates of perceptual introspection-maps that are (2) validated during a physical mimic task, (3) gathers perceptual multistability data, and (4) confirms their validity during a physical replay. 28 condition-blinded adults performed InFoRM while experiencing binocular rivalry evoked with orthogonal sinusoidal gratings. A 60 Hz joystick (3600 data samples/minute) was used to indicate continuously changes across six perceptual states within each 1 min trial. A polarized monitor system was used to present the stimuli dichoptically. Three contrast conditions were investigated: low vs low, high vs high, and low vs high. InFoRM replicates standard outcome measures, i.e. alternation rate, mean and relative proportions of perception, and distribution of exclusive percepts that are well fitted with gamma functions. Furthermore, InFoRM generates novel outcomes that deliver new insights in visual cognition via estimates of introspection maps, in ocular dominance via perceptual-state-specific dominance scores, in transitory dynamics between and within perceptual states, via techniques adopted from eye-tracking, and in rivalry-zone-size estimates utilizing InFoRM's ability to simulate piecemeal perception. The replay phase (physical replay of perceptual rivalry) confirmed good overall agreement (73% ±5 standard deviation). InFoRM can be applied to other multistable paradigms and can be used to study visual consciousness in typical and neuro-atypical populations.

VEP-based acuity estimation: unaffected by translucency of contralateral occlusion

Purpose

Visual evoked potential (VEP) recordings for objective visual acuity estimates are typically obtained monocularly with the contralateral eye occluded. Psychophysical studies suggest that the translucency of the occluder has only a minimal effect on the outcome of an acuity test. However, there is literature evidence for the VEP being susceptible to the type of occlusion. The present study assessed whether this has an impact on VEP-based estimates of visual acuity.

Methods

We obtained VEP-based acuity estimates with opaque, non-translucent occlusion of the contralateral eye, and with translucent occlusion that lets most of the light pass while abolishing the perception of any stimulus structure. The tested eye was measured with normal and artificially degraded vision, resulting in a total of 4 experimental conditions. Two different algorithms, a stepwise heuristic and a machine learning approach, were used to derive acuity from the VEP tuning curve.

Results

With normal vision, translucent occlusion resulted in slight, yet statistically significant better acuity estimates when analyzed with the heuristic algorithm (p = 0.014). The effect was small (mean ΔlogMAR = 0.06), not present in some participants, and without practical relevance. It was absent with the machine learning approach. With degraded vision, the difference was tiny and not statistically significant.

Conclusion

The type of occlusion for the contralateral eye does not substantially affect the outcome of VEP-based acuity estimation.

Shifting eye balance using monocularly directed attention in normal vision

In binocular vision, even without conscious awareness of eye of origin, attention can be selectively biased toward one eye by presenting a visual stimulus uniquely to that eye. Monocularly directed visual cues can bias perceptual dominance, as shown by studies using discrete measures of percept changes in continuous-flash suppression. Here, we use binocular rivalry to determine whether eye-based visual cues can modulate eye balance using continuous percept reporting. Using a dual-task versus single-task paradigm, we investigated whether the attentional load of these cues differentially modulates eye balance. Furthermore, both color-based and motion-based cue stimuli, non-overlaid and peripheral to the rivalry grating stimuli, were used to determine whether shifts in eye balance were stimulus specific. Aligned to cue stimulus onset, time series of percept reports were constructed and averaged across trials and participants. Specifically, for the monocular attention conditions, we found a significant shift in eye balance toward the cued eye and a significant difference in the time taken to switch from the dominating percept, regardless of whether the attention stimuli is color based or motion based. Although we did not find a significant main effect of attentional load, we found a significant interaction effect between the attentionally cued eye and attentional load on the shift in eye balance, indicating an influence of monocular attention on the shift in eye balance.

SSVEP phase synchronies and propagation during repetitive visual stimulation at high frequencies

Steady-state visual evoked potentials (SSVEPs), the brain response to visual flicker stimulation, have proven beneficial in both research and clinical applications. Despite the practical advantages of stimulation at high frequencies in terms of visual comfort and safety, high frequency-SSVEPs have not received enough attention and little is known about the mechanisms behind their generation and propagation in time and space. In this study, we investigated the origin and propagation of SSVEPs in the gamma frequency band (40-60 Hz) by studying the dynamic properties of EEG in 32 subjects. Using low-resolution brain electromagnetic tomography (sLORETA) we identified the cortical sources involved in SSVEP generation in that frequency range to be in the primary visual cortex, Brodmann areas 17, 18 and 19 with minor contribution from sources in central and frontal sites. We investigated the SSVEP propagation as measured on the scalp in the framework of the existing theories regarding the neurophysiological mechanism through which the SSVEP spreads through the cortex. We found a progressive phase shift from posterior parieto-occipital sites over the cortex with a phase velocity of approx. 8-14 m/s and wavelength of about 21 and 24 cm. The SSVEP spatial properties appear sensitive to input frequency with higher stimulation frequencies showing a faster propagation speed.

Growing evidence for separate neural mechanisms for attention and consciousness

Our conscious experience of the world seems to go in lockstep with our attentional focus: We tend to see, hear, taste, and feel what we attend to, and vice versa. This tight coupling between attention and consciousness has given rise to the idea that these two phenomena are indivisible. In the late 1950s, the honoree of this special issue, Charles Eriksen, was among a small group of early pioneers that sought to investigate whether a transient increase in overall level of attention (alertness) in response to a noxious stimulus can be decoupled from conscious perception using experimental techniques. Recent years saw a similar debate regarding whether attention and consciousness are two dissociable processes. Initial evidence that attention and consciousness are two separate processes primarily rested on behavioral data. However, the past couple of years witnessed an explosion of studies aimed at testing this conjecture using neuroscientific techniques. Here we provide an overview of these and related empirical studies on the distinction between the neuronal correlates of attention and consciousness, and detail how advancements in theory and technology can bring about a more detailed understanding of the two. We argue that the most promising approach will combine ever-evolving neurophysiological and interventionist tools with quantitative, empirically testable theories of consciousness that are grounded in a mathematically formalized understanding of phenomenology.

The SSVEP tracks attention, not consciousness, during perceptual filling-in

Research on the neural basis of conscious perception has almost exclusively shown that becoming aware of a stimulus leads to increased neural responses. By designing a novel form of perceptual filling-in (PFI) overlaid with a dynamic texture display, we frequency-tagged multiple disappearing targets as well as their surroundings. We show that in a PFI paradigm, the disappearance of a stimulus and subjective invisibility is associated with increases in neural activity, as measured with steady-state visually evoked potentials (SSVEPs), in electroencephalography (EEG). We also find that this increase correlates with alpha-band activity, a well-established neural measure of attention. These findings cast doubt on the direct relationship previously reported between the strength of neural activity and conscious perception, at least when measured with current tools, such as the SSVEP. Instead, we conclude that SSVEP strength more closely measures changes in attention.

Conscious perception of flickering stimuli in binocular rivalry and continuous flash suppression is not affected by tACS-induced SSR modulation

The content of conscious perception is known to correlate with steady-state responses (SSRs), yet their causal relationship remains unclear. Can we manipulate conscious perception by directly interfering with SSRs through transcranial alternating current stimulation (tACS)? Here, we directly addressed this question in three experiments involving binocular rivalry and continuous flash suppression (CFS). Specifically, while participants (N = 24) viewed either binocular rivalry or tried to detect stimuli masked by CFS, we applied sham or real tACS across parieto-occipital cortex at either the same or a different frequency and phase as an SSR eliciting flicker stimulus. We found that tACS did not differentially affect conscious perception in the forms of predominance, CFS detection accuracy, reaction time, or metacognitive sensitivity, confirmed by Bayesian statistics. We conclude that tACS application at frequencies of stimulus-induced SSRs does not have perceptual effects and that SSRs may be epiphenomenal to conscious perception.

Ring models of binocular rivalry and fusion

When similar visual stimuli are presented binocularly to both eyes, one perceives a fused single image. However, when the two stimuli are distinct, one does not perceive a single image; instead, one perceives binocular rivalry. That is, one perceives one of the stimulated patterns for a few seconds, then the other for few seconds, and so on - with random transitions between the two percepts. Most theoretical studies focus on rivalry, with few considering the coexistence of fusion and rivalry. Here we develop three distinct computational neuronal network models which capture binocular rivalry with realistic stochastic properties, fusion, and the hysteretic transition between. Each is a conductance-based point neuron model, which is multi-layer with two ocular dominance columns (L & R) and with an idealized "ring" architecture where the orientation preference of each neuron labels its location on a ring. In each model, the primary mechanism initiating binocular rivalry is cross-column inhibition, with firing rate adaptation governing the temporal properties of the transitions between percepts. Under stimulation by similar visual patterns, each of three models uses its own mechanism to overcome cross-column inhibition, and thus to prevent rivalry and allow the fusion of similar images: The first model uses cross-column feedforward inhibition from the opposite eye to "shut off" the cross-column feedback inhibition; the second model "turns on" a second layer of monocular neurons as a parallel pathway to the binocular neurons, rivaling out of phase with the first layer, and together these two pathways represent fusion; and the third model uses cross-column excitation to overcome the cross-column inhibition and enable fusion. Thus, each of the idealized ring models depends upon a different mechanism for fusion that might emerge as an underlying mechanism present in real visual cortex.

A multiple-response frequency-tagging paradigm measures graded changes in consciousness during perceptual filling-in

Perceptual filling-in (PFI) occurs when a physically present visual target disappears from conscious perception, with its location filled-in by the surrounding visual background. These perceptual changes are complete, near instantaneous, and can occur for multiple separate locations simultaneously. Here, we show that contrasting neural activity during the presence or absence of multi-target PFI can complement other findings from multistable phenomena to reveal the neural correlates of consciousness (NCC). We presented four peripheral targets over a background dynamically updating at 20 Hz. While participants reported on target disappearances/reappearances via button press/release, we tracked neural activity entrained by the background during PFI using steady-state visually evoked potentials (SSVEPs) recorded in the electroencephalogram. We found background SSVEPs closely correlated with subjective report, and increased with an increasing amount of PFI. Unexpectedly, we found that as the number of filled-in targets increased, the duration of target disappearances also increased, suggesting that facilitatory interactions exist between targets in separate visual quadrants. We also found distinct spatiotemporal correlates for the background SSVEP harmonics. Prior to genuine PFI, the response at the second harmonic (40 Hz) increased before the first (20 Hz), which we tentatively link to an attentional effect, while no such difference between harmonics was observed for physically removed stimuli. These results demonstrate that PFI can be used to study multi-object perceptual suppression when frequency-tagging the background of a visual display, and because there are distinct neural correlates for endogenously and exogenously induced changes in consciousness, that it is ideally suited to study the NCC.

Under the Mind's Hood: What We Have Learned by Watching the Brain at Work

Imagine you were asked to investigate the workings of an engine, but to do so without ever opening the hood. Now imagine the engine fueled the human mind. This is the challenge faced by cognitive neuroscientists worldwide aiming to understand the neural bases of our psychological functions. Luckily, human ingenuity comes to the rescue. Around the same time as the Society for Neuroscience was being established in the 1960s, the first tools for measuring the human brain at work were becoming available. Noninvasive human brain imaging and neurophysiology have continued developing at a relentless pace ever since. In this 50 year anniversary, we reflect on how these methods have been changing our understanding of how brain supports mind.

Unraveling nonlinear electrophysiologic processes in the human visual system with full dimension spectral analysis

Natural sensory signals have nonlinear structures dynamically composed of the carrier frequencies and the variation of the amplitude (i.e., envelope). How the human brain processes the envelope information is still poorly understood, largely due to the conventional analysis failing to quantify it directly. Here, we used a recently developed method, Holo-Hilbert spectral analysis, and steady-state visually evoked potential collected using electroencephalography (EEG) recordings to investigate how the human visual system processes the envelope of amplitude-modulated signals, in this case with a 14 Hz carrier and a 2 Hz envelope. The EEG results demonstrated that in addition to the fundamental stimulus frequencies, 4 Hz amplitude modulation residing in 14 Hz carrier and a broad range of carrier frequencies covering from 8 to 32 Hz modulated by 2 Hz amplitude modulation are also found in the two-dimensional frequency spectrum, which have not yet been recognized before. The envelope of the stimulus is also found to dominantly modulate the response to the incoming signal. The findings thus reveal that the electrophysiological response to amplitude-modulated stimuli is more complex than could be revealed by, for example, Fourier analysis. This highlights the dynamics of neural processes in the visual system.

Slower Binocular Rivalry in the Autistic Brain

Autism has traditionally been regarded as a disorder of the social brain. Recent reports of differences in visual perception have challenged this notion, but little evidence for altered visual processing in the autistic brain exists. We have previously observed slower behaviorally reported rates of a basic visual phenomenon, binocular rivalry, in autism [1, 2]. During rivalry, two images-one presented to each eye-vie for awareness, alternating back and forth in perception. This competition is modeled to rely, in part, on the balance of excitation and inhibition in visual cortex [3-8], which may be altered in autism [2, 9-14]. Yet direct neural evidence for this potential marker of excitation/inhibition (E/I) balance in autism is lacking. Here, we report a striking alteration in the neural dynamics of binocular rivalry in individuals with autism. Participants viewed true and simulated frequency-tagged binocular rivalry displays while steady-state visually evoked potentials (SSVEPs) were measured over occipital cortex using electroencephalography (EEG). First, we replicate our prior behavioral findings of slower rivalry and reduced perceptual suppression in individuals with autism compared with controls. Second, we provide direct neural evidence for slower rivalry in autism compared with controls, which strongly predicted individuals' behavioral switch rates. Finally, using neural data alone, we were able to predict autism symptom severity (ADOS) and correctly classify individuals' diagnostic status (autistic versus control; 87% accuracy). These findings clearly implicate atypical visual processing in the neurobiology of autism. Down the road, this paradigm may serve as a non-verbal marker of autism for developmental and cross-species research.

Tagged MEG measures binocular rivalry in a cortical network that predicts alternation rate

Binocular rivalry (BR) is a dynamic visual illusion that provides insight into the cortical mechanisms of visual awareness, stimulus selection, and object identification. When dissimilar binocular images cannot be fused, perception switches every few seconds between the left and right eye images. The speed at which individuals switch between alternatives is a stable, partially heritable trait. In order to isolate the monocular and binocular processes that determine the speed of rivalry, we presented stimuli tagged with a different flicker frequency in each eye and applied stimulus-phase locked MEG source imaging. We hypothesized that the strength of the evoked fundamental or intermodulation frequencies would vary when comparing Fast and Slow Switchers. Ten subjects reported perceptual alternations, with mean dominance durations between 1.2–4.0 sec. During BR, event-related monocular input in V1, and broadly in higher-tier ventral temporal cortex, waxed and waned with the periods of left or right eye dominance/suppression. In addition, we show that Slow Switchers produce greater evoked intermodulation frequency responses in a cortical network composed of V1, lateral occipital, posterior STS, retrosplenial & superior parietal cortices. Importantly, these dominance durations were not predictable from the brain responses to either of the fundamental tagging frequencies in isolation, nor from any responses to a pattern rivalry control condition, or a non-rivalrous control. The novel cortical network isolated, which overlaps with the default-mode network, may contain neurons that compute the level of endogenous monocular difference, and monitor accumulation of this conflict over extended periods of time. These findings are the first to relate the speed of rivalry across observers to the ‘efficient coding’ theory of computing binocular differences that may apply to binocular vision generally.

From intermodulation components to visual perception and cognition-a review

Perception results from complex interactions among sensory and cognitive processes across hierarchical levels in the brain. Intermodulation (IM) components, used in frequency tagging neuroimaging designs, have emerged as a promising direct measure of such neural interactions. IMs have initially been used in electroencephalography (EEG) to investigate low-level visual processing. In a more recent trend, IMs in EEG and other neuroimaging methods are being used to shed light on mechanisms of mid- and high-level perceptual processes, including the involvement of cognitive functions such as attention and expectation. Here, we provide an account of various mechanisms that may give rise to IMs in neuroimaging data, and what these IMs may look like. We discuss methodologies that can be implemented for different uses of IMs and we demonstrate how IMs can provide insights into the existence, the degree and the type of neural integration mechanisms at hand. We then review a range of recent studies exploiting IMs in visual perception research, placing an emphasis on high-level vision and the influence of awareness and cognition on visual processing. We conclude by suggesting future directions that can enhance the benefits of IM-methodology in perception research.

Frequency of alpha oscillation predicts individual differences in perceptual stability during binocular rivalry

When ambiguous visual stimuli have multiple interpretations, human perception can alternate between them, producing perceptual multistability. There is a large variation between individuals in how long stable percepts endure, on average, between switches, but the underlying neural basis of this individual difference in perceptual dynamics remains obscure. Here, we show that in one widely studied multistable paradigm-binocular rivalry-perceptual stability in individuals is predicted by the frequency of their neural oscillations within the alpha range (7-13 Hz). Our results suggest revising models of rivalry to incorporate effects of neural oscillations on perceptual alternations, and raise the possibility that a common factor may influence dynamics in many neural processes.

Perceptual reversals in binocular rivalry: Improved detection from OKN

When binocular rivalry is induced by opponent motion displays, perceptual reversals are often associated with changed oculomotor behavior (Frässle, Sommer, Jansen, Naber, & Einhäuser, 2014; Fujiwara et al., 2017). Specifically, the direction of smooth pursuit phases in optokinetic nystagmus typically corresponds to the direction of motion that dominates perceptual appearance at any given time. Here we report an improved analysis that continuously estimates perceived motion in terms of "cumulative smooth pursuit." In essence, smooth pursuit segments are identified, interpolated where necessary, and joined probabilistically into a continuous record of cumulative smooth pursuit (i.e., probability of eye position disregarding blinks, saccades, signal losses, and artefacts). The analysis is fully automated and robust in healthy, developmental, and patient populations. To validate reliability, we compare volitional reports of perceptual reversals in rivalry displays, and of physical reversals in nonrivalrous control displays. Cumulative smooth pursuit detects physical reversals and estimates eye velocity more accurately than existing methods do (Frässle et al., 2014). It also appears to distinguish dominant and transitional perceptual states, detecting changes with a precision of ±100 ms. We conclude that cumulative smooth pursuit significantly improves the monitoring of binocular rivalry by means of recording optokinetic nystagmus.

Stimulus rivalry and binocular rivalry share a common neural substrate

When two incompatible images are shown separately to each eye, a perceptual process known as binocular rivalry occurs by which the two images compete for awareness. The site of competition for binocular rivalry has been a topic of debate, and recent theories are that it may occur either at low levels of the visual system where the inputs from the two eyes are combined or at high levels of the visual system where the two images are processed. One of the major pieces of evidence for a high-level image account of rivalry is a phenomenon known as stimulus rivalry, in which two competing stimuli are swapped between the eyes at 3 Hz. However, there is little available neurophysiological evidence for a neural substrate for this high-level competition. Here, we used frequency tagging of two competing stimuli in binocular rivalry and stimulus rivalry in humans to evaluate whether the steady-state visually evoked potentials (SSVEPs) show similar signatures of neural competition for both conditions. We found that flickering the stimuli generates spectral power at the tagged frequencies in both types of rivalry in the early visual cortex. We then quantified dynamic signatures of competition by tracking amplitude changes in the frequency tags, which showed that both types of rivalry colocalized in occipital regions of the cortex. Thus, contrary to our hypothesis that stimulus rivalry was being mediated by high-level competition between the images, we find that neural competition measured by the SSVEP instead suggests that the sites of competition for stimulus rivalry and binocular rivalry may similarly include the occipital pole and middle temporal gyrus (hMT+/V5) of the visual system, consistent with a low-level, binocular interpretation.

Neural Mechanisms of High-Level Vision

The last three decades have seen major strides in our understanding of neural mechanisms of high-level vision, or visual cognition of the world around us. Vision has also served as a model system for the study of brain function. Several broad insights, as yet incomplete, have recently emerged. First, visual perception is best understood not as an end unto itself, but as a sensory process that subserves the animal's behavioral goal at hand. Visual perception is likely to be simply a side effect that reflects the readout of visual information processing that leads to behavior. Second, the brain is essentially a probabilistic computational system that produces behaviors by collectively evaluating, not necessarily consciously or always optimally, the available information about the outside world received from the senses, the behavioral goals, prior knowledge about the world, and possible risks and benefits of a given behavior. Vision plays a prominent role in the overall functioning of the brain providing the lion's share of information about the outside world. Third, the visual system does not function in isolation, but rather interacts actively and reciprocally with other brain systems, including other sensory faculties. Finally, various regions of the visual system process information not in a strict hierarchical manner, but as parts of various dynamic brain-wide networks, collectively referred to as the "connectome." Thus, a full understanding of vision will ultimately entail understanding, in granular, quantitative detail, various aspects of dynamic brain networks that use visual sensory information to produce behavior under real-world conditions. © 2017 American Physiological Society. Compr Physiol 8:903-953, 2018.

Visual Awareness in Binocular Rivalry Modulates Induced Pupil Fluctuations

When a visual stimulus oscillates in luminance, pupil size follows this oscillation. Recently, it has been demonstrated that such induced pupil oscillations can be used to tag which stimulus is covertly attended. Here we ask whether this "pupil frequency tagging" approach can be extended to visual awareness, specifically to inferring perceptual dominance in Binocular Rivalry between complex stimuli. We presented two distinct stimuli, a face and a house, to each eye and modulated their luminance at 1.7 Hz either in counter-phase (180° phase shift), with a 90° phase shift or in phase (0° control). In some conditions, we additionally asked observers to attend either of the stimuli. The luminance modulation was sufficiently subtle that rivalry dynamics did not differ among these conditions, and was also indistinguishable from unmodulated presentation of the stimuli. For the 180° and the 90° phase-shifted stimuli, we found that the phase of the pupil response relative to the stimuli was modulated by perceptual dominance; that is, the relative phase depended on the stimulus the observer was aware of. In turn, this perceptually dominant stimulus could be decoded from the phase of the pupil response significantly above chance. Neither percept dependence of the phase nor significant decoding was found for the 0° control condition. Our results show that visual awareness modulates pupil responses and provide proof of principle that dominance in rivalry for complex stimuli can be inferred from induced pupil fluctuations.

Conflict-sensitive neurons gate interocular suppression in human visual cortex

Neural suppression plays an important role in cortical function, including sensory, memory, and motor systems. It remains, however, relatively poorly understood. A paradigmatic case arises when conflicting images are presented to the two eyes. These images can compete for awareness, and one is usually strongly suppressed. The mechanisms that resolve such interocular conflict remain unclear. Suppression could arise solely from “winner-take-all” competition between neurons responsive to each eye. Alternatively, suppression could also depend upon neurons detecting interocular conflict. Here, we provide physiological evidence in human visual cortex for the latter: suppression depends upon conflict-sensitive neurons. We recorded steady-state visual evoked potentials (SSVEP), and used the logic of selective adaptation. The amplitude of SSVEP responses at intermodulation frequencies strengthened as interocular conflict in the stimulus increased, suggesting the presence of neurons responsive to conflict. Critically, adaptation to conflict both reduced this SSVEP effect, and increased the amount of conflict needed to produce perceptual suppression. The simplest account of these results is that interocular-conflict-sensitive neurons exist in human cortex: adaptation likely reduced the responsiveness of these neurons which in turn raised the amount of conflict required to produce perceptual suppression. Similar mechanisms may be used to resolve other varieties of perceptual conflict.

Combined fMRI- and eye movement-based decoding of bistable plaid motion perception

The phenomenon of bistable perception, in which perception alternates spontaneously despite constant sensory stimulation, has been particularly useful in probing the neural bases of conscious perception. The study of such bistability requires access to the observer's perceptual dynamics, which is usually achieved via active report. This report, however, constitutes a confounding factor in the study of conscious perception and can also be biased in the context of certain experimental manipulations. One approach to circumvent these problems is to track perceptual alternations using signals from the eyes or the brain instead of observers' reports. Here we aimed to optimize such decoding of perceptual alternations by combining eye and brain signals. Eye-tracking and functional magnetic resonance imaging (fMRI) was performed in twenty participants while they viewed a bistable visual plaid motion stimulus and reported perceptual alternations. Multivoxel pattern analysis (MVPA) for fMRI was combined with eye-tracking in a Support vector machine to decode participants' perceptual time courses from fMRI and eye-movement signals. While both measures individually already yielded high decoding accuracies (on average 86% and 88% correct, respectively) classification based on the two measures together further improved the accuracy (91% correct). These findings show that leveraging on both fMRI and eye movement data may pave the way for optimized no-report paradigms through improved decodability of bistable motion perception and hence for a better understanding of the neural correlates of consciousness.

Do early neural correlates of visual consciousness show the oblique effect? A binocular rivalry and event-related potential study

When dissimilar images are presented one to each eye, we do not see both images; rather, we see one at a time, alternating unpredictably. This is called binocular rivalry, and it has recently been used to study brain processes that correlate with visual consciousness, because perception changes without any change in the sensory input. Such studies have used various types of images, but the most popular have been gratings: sets of bright and dark lines of orthogonal orientations presented one to each eye. We studied whether using cardinal rival gratings (vertical, 0°, and horizontal, 90°) versus oblique rival gratings (left-oblique, -45°, and right-oblique, 45°) influences early neural correlates of visual consciousness, because of the oblique effect: the tendency for visual performance to be greater for cardinal gratings than for oblique gratings. Participants viewed rival gratings and pressed keys indicating which of the two gratings they perceived, was dominant. Next, we changed one of the gratings to match the grating shown to the other eye, yielding binocular fusion. Participants perceived the rivalry-to-fusion change to the dominant grating and not to the other, suppressed grating. Using event-related potentials (ERPs), we found neural correlates of visual consciousness at the P1 for both sets of gratings, as well as at the P1-N1 for oblique gratings, and we found a neural correlate of the oblique effect at the N1, but only for perceived changes. These results show that the P1 is the earliest neural activity associated with visual consciousness and that visual consciousness might be necessary to elicit the oblique effect.

Multistable Perception and the Role of the Frontoparietal Cortex in Perceptual Inference

A given pattern of optical stimulation can arise from countless possible real-world sources, creating a dilemma for vision: What in the world actually gives rise to the current pattern? This dilemma was pointed out centuries ago by the astronomer and mathematician Ibn Al-Haytham and was forcefully restated 150 years ago when von Helmholtz characterized perception as unconscious inference. To buttress his contention, von Helmholtz cited multistable perception: recurring changes in perception despite unchanging sensory input. Recent neuroscientific studies have exploited multistable perception to identify brain areas uniquely activated in association with these perceptual changes, but the specific roles of those activations remain controversial. This article provides an overview of theoretical models of multistable perception, a review of recent neuroimaging and brain stimulation studies focused on mechanisms associated with these perceptual changes, and a synthesis of available evidence within the context of current notions about Bayesian inference that find their historical roots in von Helmholtz's work.

Attention model of binocular rivalry

When the corresponding retinal locations in the two eyes are presented with incompatible images, a stable percept gives way to perceptual alternations in which the two images compete for perceptual dominance. As perceptual experience evolves dynamically under constant external inputs, binocular rivalry has been used for studying intrinsic cortical computations and for understanding how the brain regulates competing inputs. Converging behavioral and EEG results have shown that binocular rivalry and attention are intertwined: binocular rivalry ceases when attention is diverted away from the rivalry stimuli. In addition, the competing image in one eye suppresses the target in the other eye through a pattern of gain changes similar to those induced by attention. These results require a revision of the current computational theories of binocular rivalry, in which the role of attention is ignored. Here, we provide a computational model of binocular rivalry. In the model, competition between two images in rivalry is driven by both attentional modulation and mutual inhibition, which have distinct selectivity (feature vs. eye of origin) and dynamics (relatively slow vs. relatively fast). The proposed model explains a wide range of phenomena reported in rivalry, including the three hallmarks: (i) binocular rivalry requires attention; (ii) various perceptual states emerge when the two images are swapped between the eyes multiple times per second; (iii) the dominance duration as a function of input strength follows Levelt's propositions. With a bifurcation analysis, we identified the parameter space in which the model's behavior was consistent with experimental results.

Enhanced brain responses to color during smooth-pursuit eye movements

Eye movements alter visual perceptions in a number of ways. During smooth-pursuit eye movements, previous studies reported decreased detection threshold for colored stimuli and for high-spatial-frequency luminance stimuli, suggesting a boost in the parvocellular system. The present study investigated the underlying neural mechanism using EEG in human participants. Participants followed a moving target with smooth-pursuit eye movements while steady-state visually evoked potentials (SSVEPs) were elicited by equiluminant red-green flickering gratings in the background. SSVEP responses to colored gratings were 18.9% higher during smooth pursuit than during fixation. There was no enhancement of SSVEPs by smooth pursuit when the flickering grating was defined by luminance instead of color. This result provides physiological evidence that the chromatic response in the visual system is boosted by the execution of smooth-pursuit eye movements in humans. Because the response improvement is thought to be the result of an improved response in the parvocellular system, SSVEPs to equiluminant stimuli could provide a direct test of parvocellular signaling, especially in populations where collecting an explicit behavioral response from the participant is not feasible.NEW & NOTEWORTHY We constantly move our eyes when we explore the world. Eye movements alter visual perception in various ways. The smooth-pursuit eye movements have been shown to boost color sensitivity. We recorded steady-state visually evoked potentials to equiluminant chromatic flickering stimuli and observed increased steady-state visually evoked potentials when participants smoothly pursued a moving target compared with when they maintained fixation. This work provides direct neurophysiological evidence for the parvocellular boost by smooth-pursuit eye movements in humans.

Deactivation in the posterior mid-cingulate cortex reflects perceptual transitions during binocular rivalry: Evidence from simultaneous EEG-fMRI

Binocular rivalry is a phenomenon in which perception spontaneously shifts between two different images that are dichoptically presented to the viewer. By elucidating the cortical networks responsible for these stochastic fluctuations in perception, we can potentially learn much about the neural correlates of visual awareness. We obtained concurrent EEG-fMRI data for a group of 20 healthy human subjects during the continuous presentation of dichoptic visual stimuli. The two eyes’ images were tagged with different temporal frequencies so that eye specific steady-state visual evoked potential (SSVEP) signals could be extracted from the EEG data for direct comparison with changes in fMRI BOLD activity associated with binocular rivalry. We additionally included a smooth replay condition that emulated the perceptual transitions experienced during binocular rivalry as a control stimulus. We evaluated a novel SSVEP-informed fMRI analysis in this study in order to delineate the temporal dynamics of rivalry-related BOLD activity from both an electrophysiological and behavioral perspective. In this manner, we assessed BOLD activity during rivalry that was directly correlated with peaks and crosses of the two rivaling, frequency-tagged SSVEP signals, for comparison with BOLD activity associated with subject reported perceptual transitions. Our findings point to a critical role of a right lateralized fronto-parietal network in the processing of bistable stimuli, given that BOLD activity in the right superior/inferior parietal lobules was significantly elevated throughout binocular rivalry and in particular during perceptual transitions, compared with the replay condition. Based on the SSVEP-informed analysis, rivalry was further associated with significantly enhanced BOLD suppression in the posterior mid-cingulate cortex during perceptual transitions, compared with SSVEP crosses. Overall, this work points to a careful interplay between early visual areas, the right posterior parietal cortex and the mid-cingulate cortex in mediating the spontaneous perceptual changes associated with binocular rivalry and has significant implications for future multimodal imaging studies of perception and awareness.

SSVEP signatures of binocular rivalry during simultaneous EEG and fMRI

BACKGROUND

Binocular rivalry is a perceptual phenomenon that arises when two incompatible images are presented separately, one to each eye, and the observer experiences involuntary perceptual alternations between the two images. If the two images are flickering at two distinct frequencies, electroencephalography (EEG) can be used to track the frequency-tagged steady-state visually evoked potential (SSVEP) driven by each image as they compete for awareness, providing an objective measure of the subjective perceptual state. This spontaneous alternation in perceptual dominance is believed to be driven by neural processes across widespread regions in the brain, but the real-time mechanisms of these processes remain unclear.

NEW METHOD

The goal of this study was to determine the feasibility of investigating binocular rivalry using a simultaneous EEG-fMRI approach in order to leverage the high temporal resolution of EEG with the high spatial resolution of fMRI.

RESULTS

We have developed novel techniques for artifact removal and signal optimization for the rivalry-related SSVEP data collected simultaneously during fMRI.

COMPARISON WITH EXISTING METHODS

Our methods address several significant technical challenges of recording SSVEP data in the noisy fMRI environment, and enabled us to successfully reconstruct SSVEP signatures of rivalry in a group of healthy human subjects.

CONCLUSION

Further development and application of these techniques will enable more comprehensive integration of EEG and fMRI data collected simultaneously and could have significant implications for EEG-fMRI studies of brain activity in general.

The steady-state visual evoked potential in vision research: A review

Periodic visual stimulation and analysis of the resulting steady-state visual evoked potentials were first introduced over 80 years ago as a means to study visual sensation and perception. From the first single-channel recording of responses to modulated light to the present use of sophisticated digital displays composed of complex visual stimuli and high-density recording arrays, steady-state methods have been applied in a broad range of scientific and applied settings.The purpose of this article is to describe the fundamental stimulation paradigms for steady-state visual evoked potentials and to illustrate these principles through research findings across a range of applications in vision science.

Fast mental states decoding in mixed reality

The combination of Brain-Computer Interface (BCI) technology, allowing online monitoring and decoding of brain activity, with virtual and mixed reality (MR) systems may help to shape and guide implicit and explicit learning using ecological scenarios. Real-time information of ongoing brain states acquired through BCI might be exploited for controlling data presentation in virtual environments. Brain states discrimination during mixed reality experience is thus critical for adapting specific data features to contingent brain activity. In this study we recorded electroencephalographic (EEG) data while participants experienced MR scenarios implemented through the eXperience Induction Machine (XIM). The XIM is a novel framework modeling the integration of a sensing system that evaluates and measures physiological and psychological states with a number of actuators and effectors that coherently reacts to the user's actions. We then assessed continuous EEG-based discrimination of spatial navigation, reading and calculation performed in MR, using linear discriminant analysis (LDA) and support vector machine (SVM) classifiers. Dynamic single trial classification showed high accuracy of LDA and SVM classifiers in detecting multiple brain states as well as in differentiating between high and low mental workload, using a 5 s time-window shifting every 200 ms. Our results indicate overall better performance of LDA with respect to SVM and suggest applicability of our approach in a BCI-controlled MR scenario. Ultimately, successful prediction of brain states might be used to drive adaptation of data representation in order to boost information processing in MR.

Linking perception to neural activity as measured by visual evoked potentials

Linking propositions have played an important role in refining our understanding of the relationship between neural activity and perception. Over the last 40 years, visual evoked potentials (VEPs) have been used in many different ways to address questions of the relationship between neural activity and perception. This review organizes and discusses this research within the linking proposition framework developed by Davida Teller, and her colleagues. A series of examples from the VEP literature illustrates each of the five classes of linking propositions originally proposed by Davida Teller. The related concept of the bridge locus—the site at which neural activity can be said to first be proscriptive of perception—is discussed and a suggestion is made that the concept be expanded to include an evolution over time and cortical area.

Early visual responses predict conscious face perception within and between subjects during binocular rivalry

Previous studies indicate that conscious face perception may be related to neural activity in a large time window around 170-800 msec after stimulus presentation, yet in the majority of these studies changes in conscious experience are confounded with changes in physical stimulation. Using multivariate classification on MEG data recorded when participants reported changes in conscious perception evoked by binocular rivalry between a face and a grating, we showed that only MEG signals in the 120-320 msec time range, peaking at the M170 around 180 msec and the P2m at around 260 msec, reliably predicted conscious experience. Conscious perception could not only be decoded significantly better than chance from the sensors that showed the largest average difference, as previous studies suggest, but also from patterns of activity across groups of occipital sensors that individually were unable to predict perception better than chance. In addition, source space analyses showed that sources in the early and late visual system predicted conscious perception more accurately than frontal and parietal sites, although conscious perception could also be decoded there. Finally, the patterns of neural activity associated with conscious face perception generalized from one participant to another around the times of maximum prediction accuracy. Our work thus demonstrates that the neural correlates of particular conscious contents (here, faces) are highly consistent in time and space within individuals and that these correlates are shared to some extent between individuals.

High-level binocular rivalry effects

Binocular rivalry (BR) occurs when the brain cannot fuse percepts from the two eyes because they are different. We review results relating to an ongoing controversy regarding the cortical site of the BR mechanism. Some BR qualities suggest it is low-level: (1) BR, as its name implies, is usually between eyes and only low-levels have access to utrocular information. (2) All input to one eye is suppressed: blurring doesn’t stimulate accommodation; pupilary constrictions are reduced; probe detection is reduced. (3) Rivalry is affected by low-level attributes, contrast, spatial frequency, brightness, motion. (4) There is limited priming due to suppressed words or pictures. On the other hand, recent studies favor a high-level mechanism: (1) Rivalry occurs between patterns, not eyes, as in patchwork rivalry or a swapping paradigm. (2) Attention affects alternations. (3) Context affects dominance. There is conflicting evidence from physiological studies (single cell and fMRI) regarding cortical level(s) of conscious perception. We discuss the possibility of multiple BR sites and theoretical considerations that rule out this solution. We present new data regarding the locus of the BR switch by manipulating stimulus semantic content or high-level characteristics. Since these variations are represented at higher cortical levels, their affecting rivalry supports high-level BR intervention. In Experiment I, we measure rivalry when one eye views words and the other non-words and find significantly longer dominance durations for non-words. In Experiment II, we find longer dominance times for line drawings of simple, structurally impossible figures than for similar, possible objects. In Experiment III, we test the influence of idiomatic context on rivalry between words. Results show that generally words within their idiomatic context have longer mean dominance durations. We conclude that BR has high-level cortical influences, and may be controlled by a high-level mechanism.

Why is Binocular Rivalry Uncommon? Discrepant Monocular Images in the Real World

When different images project to corresponding points in the two eyes they can instigate a phenomenon called binocular rivalry (BR), wherein each image seems to intermittently disappear such that only one of the two images is seen at a time. Cautious readers may have noted an important caveat in the opening sentence – this situation can instigate BR, but usually it doesn’t. Unmatched monocular images are frequently encountered in daily life due to either differential occlusions of the two eyes or because of selective obstructions of just one eye, but this does not tend to induce BR. Here I will explore the reasons for this and discuss implications for BR in general. It will be argued that BR is resolved in favor of the instantaneously stronger neural signal, and that this process is driven by an adaptation that enhances the visibility of distant fixated objects over that of more proximate obstructions of an eye. Accordingly, BR would reflect the dynamics of an inherently visual operation that usually deals with real-world constraints.

Binocular rivalry requires visual attention

An interocular conflict arises when different images are presented to each eye at the same spatial location. The visual system resolves this conflict through binocular rivalry: observers consciously perceive spontaneous alternations between the two images. Visual attention is generally important for resolving competition between neural representations. However, given the seemingly spontaneous and automatic nature of binocular rivalry, the role of attention in resolving interocular competition remains unclear. Here we test whether visual attention is necessary to produce rivalry. Using an EEG frequency-tagging method to track cortical representations of the conflicting images, we show that when attention was diverted away, rivalry stopped. The EEG data further suggested that the neural representations of the dichoptic images combined without attention. Thus, attention is necessary for dichoptic images to be engaged in sustained rivalry and may be generally required for resolving conflicting, potentially ambiguous input and giving a single interpretation access to consciousness.

The neural mechanism for Latent (fusion maldevelopment) nystagmus

Latent nystagmus (LN) is the by-product of fusion maldevelopment in infancy. Because fusion maldevelopment--in the form of strabismus and amblyopia--is common, LN is a prevalent form of pathologic nystagmus encountered in clinical practice. It originates as an afferent visual pathway disorder. To unravel the mechanism for LN, we studied patients and nonhuman primates with maldeveloped fusion. These experiments have revealed that loss of binocular connections within striate cortex (area V1) in the first months of life is the necessary and sufficient cause of LN. The severity of LN increases systematically with longer durations of binocular decorrelation and greater losses of V1 connections. Decorrelation durations that exceed the equivalent of 2-3 months in human development result in an LN prevalence of 100%. No manipulation of brain stem motor pathways is required. The binocular maldevelopment originating in area V1 is passed on to downstream extrastriate regions of cerebral cortex that drive conjugate gaze, notably MSTd. Conjugate gaze is stable when MSTd neurons of the right and left cerebral hemispheres have balanced binocular activity. Fusion maldevelopment in infancy causes unbalanced monocular activity. If input from one eye dominates and the other is suppressed, MSTd in one hemisphere becomes more active. Acting through downstream projections to the ipsilateral nucleus of the optic tract, the eyes are driven conjugately to that side. The unbalanced MSTd drive is evident as the nasalward gaze-holding bias of LN when viewing with either eye.