Rapid invisible frequency tagging reveals nonlinear integration of auditory and visual information

Early parafoveal semantic integration in natural reading

Humans can read and comprehend text rapidly, implying that readers might process multiple words per fixation. However, the extent to which parafoveal words are previewed and integrated into the evolving sentence context remains disputed. We investigated parafoveal processing during natural reading by recording brain activity and eye movements using MEG and an eye tracker while participants silently read one-line sentences. The sentences contained an unpredictable target word that was either congruent or incongruent with the sentence context. To measure parafoveal processing, we flickered the target words at 60 Hz and measured the resulting brain responses (i.e. Rapid Invisible Frequency Tagging, RIFT) during fixations on the pre-target words. Our results revealed a significantly weaker tagging response for target words that were incongruent with the previous context compared to congruent ones, even within 100ms of fixating the word immediately preceding the target. This reduction in the RIFT response was also found to be predictive of individual reading speed. We conclude that semantic information is not only extracted from the parafovea but can also be integrated with the previous context before the word is fixated. This early and extensive parafoveal processing supports the rapid word processing required for natural reading. Our study suggests that theoretical frameworks of natural reading should incorporate the concept of deep parafoveal processing.

A user-friendly visual brain-computer interface based on high-frequency steady-state visual evoked fields recorded by OPM-MEG

Objective. Magnetoencephalography (MEG) shares a comparable time resolution with electroencephalography. However, MEG excels in spatial resolution, enabling it to capture even the subtlest and weakest brain signals for brain-computer interfaces (BCIs). Leveraging MEG's capabilities, specifically with optically pumped magnetometers (OPM-MEG), proves to be a promising avenue for advancing MEG-BCIs, owing to its exceptional sensitivity and portability. This study harnesses the power of high-frequency steady-state visual evoked fields (SSVEFs) to build an MEG-BCI system that is flickering-imperceptible, user-friendly, and highly accurate.Approach.We have constructed a nine-command BCI that operates on high-frequency SSVEF (58-62 Hz with a 0.5 Hz interval) stimulation. We achieved this by placing the light source inside and outside the magnetic shielding room, ensuring compliance with non-magnetic and visual stimulus presentation requirements. Five participants took part in offline experiments, during which we collected six-channel multi-dimensional MEG signals along both the vertical (Z-axis) and tangential (Y-axis) components. Our approach leveraged the ensemble task-related component analysis algorithm for SSVEF identification and system performance evaluation.Main Results.The offline average accuracy of our proposed system reached an impressive 92.98% when considering multi-dimensional conjoint analysis using data from both theZandYaxes. Our method achieved a theoretical average information transfer rate (ITR) of 58.36 bits min-1with a data length of 0.7 s, and the highest individual ITR reached an impressive 63.75 bits min-1.Significance.This study marks the first exploration of high-frequency SSVEF-BCI based on OPM-MEG. These results underscore the potential and feasibility of MEG in detecting subtle brain signals, offering both theoretical insights and practical value in advancing the development and application of MEG in BCI systems.

Attention Drives Visual Processing and Audiovisual Integration During Multimodal Communication

During communication in real-life settings, our brain often needs to integrate auditory and visual information and at the same time actively focus on the relevant sources of information, while ignoring interference from irrelevant events. The interaction between integration and attention processes remains poorly understood. Here, we use rapid invisible frequency tagging and magnetoencephalography to investigate how attention affects auditory and visual information processing and integration, during multimodal communication. We presented human participants (male and female) with videos of an actress uttering action verbs (auditory; tagged at 58 Hz) accompanied by two movie clips of hand gestures on both sides of fixation (attended stimulus tagged at 65 Hz; unattended stimulus tagged at 63 Hz). Integration difficulty was manipulated by a lower-order auditory factor (clear/degraded speech) and a higher-order visual semantic factor (matching/mismatching gesture). We observed an enhanced neural response to the attended visual information during degraded speech compared to clear speech. For the unattended information, the neural response to mismatching gestures was enhanced compared to matching gestures. Furthermore, signal power at the intermodulation frequencies of the frequency tags, indexing nonlinear signal interactions, was enhanced in the left frontotemporal and frontal regions. Focusing on the left inferior frontal gyrus, this enhancement was specific for the attended information, for those trials that benefitted from integration with a matching gesture. Together, our results suggest that attention modulates audiovisual processing and interaction, depending on the congruence and quality of the sensory input.

Dysfunction of specific auditory fibers impacts cortical oscillations, driving an autism phenotype despite near-normal hearing

Autism spectrum disorder is discussed in the context of altered neural oscillations and imbalanced cortical excitation-inhibition of cortical origin. We studied here whether developmental changes in peripheral auditory processing, while preserving basic hearing function, lead to altered cortical oscillations. Local field potentials (LFPs) were recorded from auditory, visual, and prefrontal cortices and the hippocampus of BdnfPax2 KO mice. These mice develop an autism-like behavioral phenotype through deletion of BDNF in Pax2+ interneuron precursors, affecting lower brainstem functions, but not frontal brain regions directly. Evoked LFP responses to behaviorally relevant auditory stimuli were weaker in the auditory cortex of BdnfPax2 KOs, connected to maturation deficits of high-spontaneous rate auditory nerve fibers. This was correlated with enhanced spontaneous and induced LFP power, excitation-inhibition imbalance, and dendritic spine immaturity, mirroring autistic phenotypes. Thus, impairments in peripheral high-spontaneous rate fibers alter spike synchrony and subsequently cortical processing relevant for normal communication and behavior.

Cell-type-specific propagation of visual flicker

Rhythmic flicker stimulation has gained interest as a treatment for neurodegenerative diseases and as a method for frequency tagging neural activity. Yet, little is known about the way in which flicker-induced synchronization propagates across cortical levels and impacts different cell types. Here, we use Neuropixels to record from the lateral geniculate nucleus (LGN), the primary visual cortex (V1), and CA1 in mice while presenting visual flicker stimuli. LGN neurons show strong phase locking up to 40 Hz, whereas phase locking is substantially weaker in V1 and is absent in CA1. Laminar analyses reveal an attenuation of phase locking at 40 Hz for each processing stage. Gamma-rhythmic flicker predominantly entrains fast-spiking interneurons. Optotagging experiments show that these neurons correspond to either parvalbumin (PV+) or narrow-waveform somatostatin (Sst+) neurons. A computational model can explain the observed differences based on the neurons' capacitative low-pass filtering properties. In summary, the propagation of synchronized activity and its effect on distinct cell types strongly depend on its frequency.

Spatial and temporal (non)binding of audiovisual rhythms in sensorimotor synchronisation

Human movement synchronisation with moving objects strongly relies on visual input. However, auditory information also plays an important role, since real environments are intrinsically multimodal. We used electroencephalography (EEG) frequency tagging to investigate the selective neural processing and integration of visual and auditory information during motor tracking and tested the effects of spatial and temporal congruency between audiovisual modalities. EEG was recorded while participants tracked with their index finger a red flickering (rate f_V = 15 Hz) dot oscillating horizontally on a screen. The simultaneous auditory stimulus was modulated in pitch (rate f_A = 32 Hz) and lateralised between left and right audio channels to induce perception of a periodic displacement of the sound source. Audiovisual congruency was manipulated in terms of space in Experiment 1 (no motion, same direction or opposite direction), and timing in Experiment 2 (no delay, medium delay or large delay). For both experiments, significant EEG responses were elicited at f_V and f_A tagging frequencies. It was also hypothesised that intermodulation products corresponding to the nonlinear integration of visual and auditory stimuli at frequencies f_V ± f_A would be elicited, due to audiovisual integration, especially in Congruent conditions_. However, these components were not observed. Moreover, synchronisation and EEG results were not influenced by congruency manipulations, which invites further exploration of the conditions which may modulate audiovisual processing and the motor tracking of moving objects.

Rapid invisible frequency tagging (RIFT): a promising technique to study neural and cognitive processing using naturalistic paradigms

Frequency tagging has been successfully used to investigate selective stimulus processing in electroencephalography (EEG) or magnetoencephalography (MEG) studies. Recently, new projectors have been developed that allow for frequency tagging at higher frequencies (>60 Hz). This technique, rapid invisible frequency tagging (RIFT), provides two crucial advantages over low-frequency tagging as (i) it leaves low-frequency oscillations unperturbed, and thus open for investigation, and ii) it can render the tagging invisible, resulting in more naturalistic paradigms and a lack of participant awareness. The development of this technique has far-reaching implications as oscillations involved in cognitive processes can be investigated, and potentially manipulated, in a more naturalistic manner.

Application of rapid invisible frequency tagging for brain computer interfaces

BACKGROUND

Brain-computer interfaces (BCI) based on steady-state visual evoked potentials (SSVEPs/SSVEFs) are among the most commonly used BCI systems. They require participants to covertly attend to visual objects flickering at specified frequencies. The attended location is decoded online by analysing the power of neuronal responses at the flicker frequency.

NEW METHOD

We implemented a novel rapid invisible frequency-tagging technique, utilizing a state-of-the-art projector with refresh rates of up to 1440 Hz. We flickered the luminance of visual objects at 56 and 60 Hz, which was invisible to participants but produced strong neuronal responses measurable with magnetoencephalography (MEG). The direction of covert attention, decoded from frequency-tagging responses, was used to control an online BCI PONG game.

RESULTS

Our results show that seven out of eight participants were able to play the pong game controlled by the frequency-tagging signal, with average accuracies exceeding 60 %. Importantly, participants were able to modulate the power of the frequency-tagging response within a 1-second interval, while only seven occipital sensors were required to reliably decode the neuronal response.

COMPARISON WITH EXISTING METHODS

In contrast to existing SSVEP-based BCI systems, rapid frequency-tagging does not produce a visible flicker. This extends the time-period participants can use it without fatigue, by avoiding distracting visual input. Furthermore, higher frequencies increase the temporal resolution of decoding, resulting in higher communication rates.

CONCLUSION

Using rapid invisible frequency-tagging opens new avenues for fundamental research and practical applications. In combination with novel optically pumped magnetometers (OPMs), it could facilitate the development of high-speed and mobile next-generation BCI systems.

Neural evidence for lexical parafoveal processing

In spite of the reduced visual acuity, parafoveal information plays an important role in natural reading. However, competing models on reading disagree on whether words are previewed parafoveally at the lexical level. We find neural evidence for lexical parafoveal processing by combining a rapid invisible frequency tagging (RIFT) approach with magnetoencephalography (MEG) and eye-tracking. In a silent reading task, target words are tagged (flickered) subliminally at 60 Hz. The tagging responses measured when fixating on the pre-target word reflect parafoveal processing of the target word. We observe stronger tagging responses during pre-target fixations when followed by low compared with high lexical frequency targets. Moreover, this lexical parafoveal processing is associated with individual reading speed. Our findings suggest that reading unfolds in the fovea and parafovea simultaneously to support fluent reading.

No Evidence for Entrainment: Endogenous Gamma Oscillations and Rhythmic Flicker Responses Coexist in Visual Cortex

Over the past decades, numerous studies have linked cortical gamma oscillations (∼30-100 Hz) to neurocomputational mechanisms. Their functional relevance, however, is still passionately debated. Here, we asked whether endogenous gamma oscillations in the human brain can be entrained by a rhythmic photic drive >50 Hz. Such a noninvasive modulation of endogenous brain rhythms would allow conclusions about their causal involvement in neurocognition. To this end, we systematically investigated oscillatory responses to a rapid sinusoidal flicker in the absence and presence of endogenous gamma oscillations using magnetoencephalography (MEG) in combination with a high-frequency projector. The photic drive produced a robust response over visual cortex to stimulation frequencies of up to 80 Hz. Strong, endogenous gamma oscillations were induced using moving grating stimuli as repeatedly done in previous research. When superimposing the flicker and the gratings, there was no evidence for phase or frequency entrainment of the endogenous gamma oscillations by the photic drive. Unexpectedly, we did not observe an amplification of the flicker response around participants' individual gamma frequencies (IGFs); rather, the magnitude of the response decreased monotonically with increasing frequency. Source reconstruction suggests that the flicker response and the gamma oscillations were produced by separate, coexistent generators in visual cortex. The presented findings challenge the notion that cortical gamma oscillations can be entrained by rhythmic visual stimulation. Instead, the mechanism generating endogenous gamma oscillations seems to be resilient to external perturbation.SIGNIFICANCE STATEMENT We aimed to investigate to what extent ongoing, high-frequency oscillations in the gamma-band (30-100 Hz) in the human brain can be entrained by a visual flicker. Gamma oscillations have long been suggested to coordinate neuronal firing and enable interregional communication. Our results demonstrate that rhythmic visual stimulation cannot hijack the dynamics of ongoing gamma oscillations; rather, the flicker response and the endogenous gamma oscillations coexist in different visual areas. Therefore, while a visual flicker evokes a strong neuronal response even at high frequencies in the gamma-band, it does not entrain endogenous gamma oscillations in visual cortex. This has important implications for interpreting studies investigating the causal and neuroprotective effects of rhythmic sensory stimulation in the gamma-band.