Superposition model predicts EEG occipital activity during free viewing of natural scenes

Fixational eye movements and their associated evoked potentials during natural vision are altered in schizophrenia

Background

Visual exploration is abnormal in schizophrenia; however, few studies have investigated the physiological responses during selecting objectives in more ecological scenarios. This study aimed to demonstrate that people with schizophrenia have difficulties observing the prominent elements of an image due to a deficit mechanism of sensory modulation (active sensing) during natural vision.

Methods

An electroencephalogram recording with eye tracking data was collected on 18 healthy individuals and 18 people affected by schizophrenia while looking at natural images. These had a prominent color element and blinking produced by changes in image luminance.

Results

We found fewer fixations when all images were scanned, late focus on prominent image areas, decreased amplitude in the eye-fixation-related potential, and decreased intertrial coherence in the SCZ group.

Conclusions

The decrease in the visual attention response evoked by the prominence of visual stimuli in patients affected by schizophrenia is generated by a reduction in endogenous attention mechanisms to initiate and maintain visual exploration. Further work is required to explain the relationship of this decrease with clinical indicators.

Task-irrelevant decorative pictures increase cognitive load during text processing but have no effects on learning or working memory performance: an EEG and eye-tracking study

Decorative pictures (DP) are often used in multimedia task materials and are commonly considered so-called seductive details as they are commonly not task-relevant. Typically, DP result in mixed effects on behavioral performance measures. The current study focused on the effects of DP on the cognitive load during text reading and working memory task performance. The theta and alpha frequency band power of the electroencephalogram (EEG) and pupil dilation served as proxies of cognitive load. The number of fixations, mean fixation durations, and the number of transitions served as proxies of the attentional focus. For both, text reading and n-back working memory tasks, the presence and congruency of DP were manipulated in four task conditions. DP did neither affect behavioral performance nor subjective ratings of emotional-motivational factors. However, in both tasks, DP increased the cognitive load as revealed by the EEG alpha frequency band power and (at least to some extent) by subjective effort ratings. Notably, the EEG alpha frequency band power was a quite reliable and sensitive proxy of cognitive load. Analyzing the EEG data stimulus-locked and fixation-related, the EEG alpha frequency band power revealed a difference in global and local cognitive load. In sum, the current study underlines the feasibility and use of EEG for multimedia research, especially when combined with eye-tracking.

The use of the eye-fixation-related potential to investigate visual perception in professional domains with high attentional demand: a literature review

INTRODUCTION

Visual attention is a cognitive skill related to visual perception and neural activity, and also moderated by expertise, in time-constrained professional domains (e.g., aviation, driving, sport, surgery). However, the contribution of both perceptual and neural processes on performance has been studied separately in the literature.

DEVELOPMENT

We defend an integration of visual and neural signals to offer a more complete picture of the visual attention displayed by professionals of different skill levels when performing free-viewing tasks. Specifically, we propose to zoom the analysis in data related to the quiet eye and P300 component jointly, as a novel signal processing approach to evaluate professionals' visual attention.

CONCLUSION

This review highlights the advantages of using portable eye trackers and electroencephalogram systems altogether, as a promising technique for a better understanding of early cognitive components related to attentional processes. Altogether, the eye-fixation-related potentials method may provide a better understanding of the cognitive mechanisms employed by the participants in natural settings, revealing what visual information is of interest for participants and distinguishing the neural bases of visual attention between targets and non-targets whenever they perceive a stimulus during free viewing experiments.

Refixation behavior in naturalistic viewing: Methods, mechanisms, and neural correlates

When freely viewing a scene, the eyes often return to previously visited locations. By tracking eye movements and coregistering eye movements and EEG, such refixations are shown to have multiple roles: repairing insufficient encoding from precursor fixations, supporting ongoing viewing by resampling relevant locations prioritized by precursor fixations, and aiding the construction of memory representations. All these functions of refixation behavior are understood to be underpinned by three oculomotor and cognitive systems and their associated brain structures. First, immediate saccade planning prior to refixations involves attentional selection of candidate locations to revisit. This process is likely supported by the dorsal attentional network. Second, visual working memory, involved in maintaining task-related information, is likely supported by the visual cortex. Third, higher-order relevance of scene locations, which depends on general knowledge and understanding of scene meaning, is likely supported by the hippocampal memory system. Working together, these structures bring about viewing behavior that balances exploring previously unvisited areas of a scene with exploiting visited areas through refixations.

Episodic memory formation in unrestricted viewing

The brain systems of episodic memory and oculomotor control are tightly linked, suggesting a crucial role of eye movements in memory. But little is known about the neural mechanisms of memory formation across eye movements in unrestricted viewing behavior. Here, we leverage simultaneous eye tracking and EEG recording to examine episodic memory formation in free viewing. Participants memorized multi-element events while their EEG and eye movements were concurrently recorded. Each event comprised elements from three categories (face, object, place), with two exemplars from each category, in different locations on the screen. A subsequent associative memory test assessed participants' memory for the between-category associations that specified each event. We used a deconvolution approach to overcome the problem of overlapping EEG responses to sequential saccades in free viewing. Brain activity was time-locked to the fixation onsets, and we examined EEG power in the theta and alpha frequency bands, the putative oscillatory correlates of episodic encoding mechanisms. Three modulations of fixation-related EEG predicted high subsequent memory performance: (1) theta increase at fixations after between-category gaze transitions, (2) theta and alpha increase at fixations after within-element gaze transitions, (3) alpha decrease at fixations after between-exemplar gaze transitions. Thus, event encoding with unrestricted viewing behavior was characterized by three neural mechanisms, manifested in fixation-locked theta and alpha EEG activity that rapidly turned on and off during the unfolding eye movement sequences. These three distinct neural mechanisms may be the essential building blocks that subserve the buildup of coherent episodic memories during unrestricted viewing behavior.

The policy of testing hypotheses in Chilean science. The role of a hypothesis-driven research funding programme in the installation of a hypothesis-driven experimental system in visual neuroscience

O'Malley et al. (2009) and Haufe (2013) suggest that the philosophical idea of science as hypothesis testing generates a pernicious bias towards hypothesis-driven research and against exploratory research in the review process of research proposals and the allocation of resources. This paper addresses a conceptual objection to the argument by O'Malley et al. (2009) and Haufe (2013). We argue that the funding agencies' concepts of good science do not belong to epistemological or philosophical contexts but to political and institutional contexts. This means that correcting (potential) biases in research funding does not entail correcting funding agencies' (supposed) philosophies of science. To illustrate this point, we provide an in-depth historical case study: the granting of funds to neuroscientist Pedro Maldonado by the Chilean funding programme FONDECYT. This is a relevant comparison as FONDECYT's guidelines explicitly promote hypothesis-driven research and endorse a view of "good science" as hypothesis testing. However, we will see that the overall influence of the philosophical idea of science as hypothesis testing over this funding programme, the research project, and the actual practice of hypothesis testing is somewhat limited. The concept of science as hypothesis testing seems to play a crucial institutional or political (not philosophical) role in allowing the conceptual articulation of social expectations and researchers' expectations.

Naturalistic viewing conditions can increase task engagement and aesthetic preference but have only minimal impact on EEG quality

Free gaze and moving images are typically avoided in EEG experiments due to the expected generation of artifacts and noise. Yet for a growing number of research questions, loosening these rigorous restrictions would be beneficial. Among these is research on visual aesthetic experiences, which often involve open-ended exploration of highly variable stimuli. Here we systematically compare the effect of conservative vs. more liberal experimental settings on various measures of behavior, brain activity and physiology in an aesthetic rating task. Our primary aim was to assess EEG signal quality. 43 participants either maintained fixation or were allowed to gaze freely, and viewed either static images or dynamic (video) stimuli consisting of dance performances or nature scenes. A passive auditory background task (auditory steady-state response; ASSR) was added as a proxy measure for overall EEG recording quality. We recorded EEG, ECG and eyetracking data, and participants rated their aesthetic preference and state of boredom on each trial. Whereas both behavioral ratings and gaze behavior were affected by task and stimulus manipulations, EEG SNR was barely affected and generally robust across all conditions, despite only minimal preprocessing and no trial rejection. In particular, we show that using video stimuli does not necessarily result in lower EEG quality and can, on the contrary, significantly reduce eye movements while increasing both the participants' aesthetic response and general task engagement. We see these as encouraging results indicating that - at least in the lab - more liberal experimental conditions can be adopted without significant loss of signal quality.

Revealing Whole-Brain Causality Networks During Guided Visual Searching

In our daily lives, we use eye movements to actively sample visual information from our environment ("active vision"). However, little is known about how the underlying mechanisms are affected by goal-directed behavior. In a study of 31 participants, magnetoencephalography was combined with eye-tracking technology to investigate how interregional interactions in the brain change when engaged in two distinct forms of active vision: freely viewing natural images or performing a guided visual search. Regions of interest with significant fixation-related evoked activity (FRA) were identified with spatiotemporal cluster permutation testing. Using generalized partial directed coherence, we show that, in response to fixation onset, a bilateral cluster consisting of four regions (posterior insula, transverse temporal gyri, superior temporal gyrus, and supramarginal gyrus) formed a highly connected network during free viewing. A comparable network also emerged in the right hemisphere during the search task, with the right supramarginal gyrus acting as a central node for information exchange. The results suggest that all four regions are vital to visual processing and guiding attention. Furthermore, the right supramarginal gyrus was the only region where activity during fixations on the search target was significantly negatively correlated with search response times. Based on our findings, we hypothesize that, following a fixation, the right supramarginal gyrus supplies the right supplementary eye field (SEF) with new information to update the priority map guiding the eye movements during the search task.

Neural correlates of intra-saccadic motion perception

Retinal motion of the visual scene is not consciously perceived during ocular saccades in normal everyday conditions. It has been suggested that extra-retinal signals actively suppress intra-saccadic motion perception to preserve stable perception of the visual world. However, using stimuli optimized to preferentially activate the M-pathway, Castet and Masson (2000) demonstrated that motion can be perceived during a saccade. Based on this psychophysical paradigm, we used electroencephalography and eye-tracking recordings to investigate the neural correlates related to the conscious perception of intra-saccadic motion. We demonstrated the effective involvement during saccades of the cortical areas V1-V2 and MT-V5, which convey motion information along the M-pathway. We also showed that individual motion perception was related to retinal temporal frequency.

Reading direct speech quotes increases theta phase-locking: Evidence for cortical tracking of inner speech?

Growing evidence shows that theta-band (4-7 Hz) activity in the auditory cortex phase-locks to rhythms of overt speech. Does theta activity also encode the rhythmic dynamics of inner speech? Previous research established that silent reading of direct speech quotes (e.g., Mary said: "This dress is lovely!") elicits more vivid inner speech than indirect speech quotes (e.g., Mary said that the dress was lovely). As we cannot directly track the phase alignment between theta activity and inner speech over time, we used EEG to measure the brain's phase-locked responses to the onset of speech quote reading. We found that direct (vs. indirect) quote reading was associated with increased theta phase synchrony over trials at 250-500 ms post-reading onset, with sources of the evoked activity estimated in the speech processing network. An eye-tracking control experiment confirmed that increased theta phase synchrony in direct quote reading was not driven by eye movement patterns, and more likely reflects synchronous phase resetting at the onset of inner speech. These findings suggest a functional role of theta phase modulation in reading-induced inner speech.

A simple metric to study the mechanisms generating event-related potentials

BACKGROUND

There is an active debate about the mechanism underlying the generation of event-related potentials, and, particularly, whether these are generated by additive components, independent of the background EEG, or the phase-resetting of ongoing oscillations.

METHOD

We present a new metric to evaluate trial-by-trial covariations of successive ERP components. Our main assumption is that if two successive ERP components are generated by phase-resetting of a unitary oscillation, they should be time-locked to each other and their single-trial latencies should covary. In contrast, if the components are generated by independent additive components, single-trial latency covariations should not be observed. To quantify the covariance between the single-trial latencies, we define a metric based on latency-corrected averages, which we applied to both simulated and real ERPs.

RESULTS

For the simulated data, there was a clear distinction in latency covariation between the ERPs generated with unitary phase-resetting versus additive models. For real visual and auditory ERPs, we observed a lack of latency covariation of successive components.

COMPARISON WITH EXISTING METHODS

The new metric is complementary to other approaches to study the mechanisms underlying ERP generation, and does not suffer from potential caveats due to filtering artifacts. Moreover, the method proved to be more sensitive than another estimation of single-trial latency covariations using the cross-correlation function.

CONCLUSION

The observed lack of latency covariation shows the presence of parallel, independent processing within each cortical sensory pathway.

Regression-based analysis of combined EEG and eye-tracking data: Theory and applications

Fixation-related potentials (FRPs), neural responses aligned to the end of saccades, are a promising tool for studying the dynamics of attention and cognition under natural viewing conditions. In the past, four methodological problems have complicated the analysis of such combined eye-tracking/electroencephalogram experiments: (1) the synchronization of data streams, (2) the removal of ocular artifacts, (3) the condition-specific temporal overlap between the brain responses evoked by consecutive fixations, and (4) the fact that numerous low-level stimulus and saccade properties also influence the postsaccadic neural responses. Although effective solutions exist for the first two problems, the latter two are only beginning to be addressed. In the current paper, we present and review a unified regression-based framework for FRP analysis that allows us to deconvolve overlapping potentials while also controlling for both linear and nonlinear confounds on the FRP waveform. An open software implementation is provided for all procedures. We then demonstrate the advantages of this proposed (non)linear deconvolution modeling approach for data from three commonly studied paradigms: face perception, scene viewing, and reading. First, for a traditional event-related potential (ERP) face recognition experiment, we show how this technique can separate stimulus ERPs from overlapping muscle and brain potentials produced by small (micro)saccades on the face. Second, in natural scene viewing, we model and isolate multiple nonlinear effects of saccade parameters on the FRP. Finally, for a natural sentence reading experiment using the boundary paradigm, we show how it is possible to study the neural correlates of parafoveal preview after removing spurious overlap effects caused by the associated difference in average fixation time. Our results suggest a principal way of measuring reliable eye movement-related brain activity during natural vision.

Eye movement-related brain potentials during assisted navigation in real-world environments

Conducting neuroscience research in the real-world remains challenging because of movement- and environment-related artifacts as well as missing control over stimulus presentation. The present study overcame these restrictions by mobile electroencephalography (EEG) and data-driven analysis approaches during a real-world navigation task. During assisted navigation through an unfamiliar city environment, participants received either standard or landmark-based auditory navigation instructions. EEG data were recorded continuously during navigation. Saccade- and blink-events as well as gait-related EEG activity were extracted from sensor level data. Brain activity associated with the navigation task was identified by subsequent source-based cleaning of non-brain activity and unfolding of overlapping event-related potentials. When navigators received landmark-based instructions compared to those receiving standard navigation instructions, the blink-related brain potentials during navigation revealed higher amplitudes at fronto-central leads in a time window starting at 300 ms after blinks, which was accompanied by improved spatial knowledge acquisition tested in follow-up spatial tasks. Replicating improved spatial knowledge acquisition from previous experiments, the present study revealed eye movement-related brain potentials to point to the involvement of higher cognitive processes and increased processing of incoming information during periods of landmark-based instructions. The study revealed neuronal correlates underlying visuospatial information processing during assisted navigation in the real-world providing a new analysis approach for neuroscientific research in freely moving participants in uncontrollable real-world environments.

Large-scale cortical travelling waves predict localized future cortical signals

Predicting future brain signal is highly sought-after, yet difficult to achieve. To predict the future phase of cortical activity at localized ECoG and MEG recording sites, we exploit its predominant, large-scale, spatiotemporal dynamics. The dynamics are extracted from the brain signal through Fourier analysis and principal components analysis (PCA) only, and cast in a data model that predicts future signal at each site and frequency of interest. The dominant eigenvectors of the PCA that map the large-scale patterns of past cortical phase to future ones take the form of smoothly propagating waves over the entire measurement array. In ECoG data from 3 subjects and MEG data from 20 subjects collected during a self-initiated motor task, mean phase prediction errors were as low as 0.5 radians at local sites, surpassing state-of-the-art methods of within-time-series or event-related models. Prediction accuracy was highest in delta to beta bands, depending on the subject, was more accurate during episodes of high global power, but was not strongly dependent on the time-course of the task. Prediction results did not require past data from the to-be-predicted site. Rather, best accuracy depended on the availability in the model of long wavelength information. The utility of large-scale, low spatial frequency traveling waves in predicting future phase activity at local sites allows estimation of the error introduced by failing to account for irreducible trajectories in the activity dynamics.

Prediction is an important step in scientific progress, often leading to real-world applications. Prediction of future brain activity could lead to improvements in detecting driver and pilot error or real-time brain testing using transcranial magnetic stimulation. Previous studies have either supposed that the ‘noise’ level in the cortex is high, setting the prediction bar rather low; or used localized measurements to predict future activity, with modest success. A long-held but controversial hypothesis is that the cortex is best characterized as a multi-scale dynamic structure, in which the flow of activity at one scale, say, the area responsible for motor control, is inextricably tied to activity at smaller and larger scales, for example within a cortical column and the whole cortex. We test this hypothesis by analyzing large-scale traveling waves of cortical activity. Like waves arriving on a beach, the ongoing wave motion allows better prediction of future activity compared to monitoring the local rise and fall; in the best cases the future wave cycle is predicted with as low as 20° average error angle. The prediction techniques developed for the present research rely on mathematics related to quantifying large-scale weather patterns or analysis of fluid dynamics.

Optimizing the ICA-based removal of ocular EEG artifacts from free viewing experiments

Combining EEG with eye-tracking is a promising approach to study neural correlates of natural vision, but the resulting recordings are also heavily contaminated by activity of the eye balls, eye lids, and extraocular muscles. While Independent Component Analysis (ICA) is commonly used to suppress these ocular artifacts, its performance under free viewing conditions has not been systematically evaluated and many published reports contain residual artifacts. Here I evaluated and optimized ICA-based correction for two tasks with unconstrained eye movements: visual search in images and sentence reading. In a first step, four parameters of the ICA pipeline were varied orthogonally: the (1) high-pass and (2) low-pass filter applied to the training data, (3) the proportion of training data containing myogenic saccadic spike potentials (SP), and (4) the threshold for eye tracker-based component rejection. In a second step, the eye-tracker was used to objectively quantify the correction quality of each ICA solution, both in terms of undercorrection (residual artifacts) and overcorrection (removal of neurogenic activity). As a benchmark, results were compared to those obtained with an alternative spatial filter, Multiple Source Eye Correction (MSEC). With commonly used settings, Infomax ICA not only left artifacts in the data, but also distorted neurogenic activity during eye movement-free intervals. However, correction results could be strongly improved by training the ICA on optimally filtered data in which SPs were massively overweighted. With optimized procedures, ICA removed virtually all artifacts, including the SP and its associated spectral broadband artifact from both viewing paradigms, with little distortion of neural activity. It also outperformed MSEC in terms of SP correction. Matlab code is provided.

Co-registration of eye movements and neuroimaging for studying contextual predictions in natural reading

Sixteen years ago, Sereno and Rayner (2003. Measuring word recognition in reading: eye movements and event-related potentials. Trends in Cognitive Sciences, 7(11), 489-493) illustrated how "by means of review and comparison" eye movement (EM) and event-related potential (ERP) studies may advance our understanding of visual word recognition. Attempts to simultaneously record EMs and ERPs soon followed. Recently, this co-registration approach has also been transferred to fMRI and oscillatory EEG. With experimental settings close to natural reading, co-registration enables us to directly integrate insights from EM and neuroimaging studies. This should extend current experimental paradigms by moving the field towards studying sentence-level processing including effects of context and parafoveal preview. This article will introduce the basic principles and applications of co-registration and selectively review how this approach may shed light on one of the most controversially discussed issues in reading research, contextual predictions in online language processing.

Distinct modes of top-down cognitive processing in the ventral visual cortex

Top-down cognitive control leads to changes in the sensory processing of the brain. In visual perception such changes can take place in the ventral visual cortex altering the functional asymmetry in forward and backward connections. Here we used fixation-related evoked responses of EEG measurement and dynamic causal modeling to examine hierarchical forward-backward asymmetry, while twenty-six healthy adults performed cognitive tasks that require different types of top-down cognitive control (memorizing or searching visual objects embedded in a natural scene image). The generative model revealed an enhanced asymmetry toward forward connections during memorizing, whereas enhanced backward connections were found during searching. This task-dependent modulation of forward and backward connections suggests two distinct modes of top-down cognitive processing in cortical networks. The alteration in forward-backward asymmetry might underlie the functional role in the cognitive control of visual information processing.

Brain Responses to Emotional Faces in Natural Settings: A Wireless Mobile EEG Recording Study

The detection of a human face in a visual field and correct reading of emotional expression of faces are important elements in everyday social interactions, decision making and emotional responses. Although brain correlates of face processing have been established in previous fMRI and electroencephalography (EEG)/MEG studies, little is known about how the brain representation of faces and emotional expressions of faces in freely moving humans. The present study aimed to detect brain electrical potentials that occur during the viewing of human faces in natural settings. 64-channel wireless EEG and eye-tracking data were recorded in 19 participants while they moved in a mock art gallery and stopped at times to evaluate pictures hung on the walls. Positive, negative and neutral valence pictures of objects and human faces were displayed. The time instants in which pictures first occurred in the visual field were identified in eye-tracking data and used to reconstruct the triggers in continuous EEG data after synchronizing the time axes of the EEG and eye-tracking device. EEG data showed a clear face-related event-related potential (ERP) in the latency interval ranging from 165 to 210 ms (N170); this component was not seen whilst participants were viewing non-living objects. The face ERP component was stronger during viewing disgusted compared to neutral faces. Source dipole analysis revealed an equivalent current dipole in the right fusiform gyrus (BA37) accounting for N170 potential. Our study demonstrates for the first time the possibility of recording brain responses to human faces and emotional expressions in natural settings. This finding opens new possibilities for clinical, developmental, social, forensic, or marketing research in which information about face processing is of importance.

Temporal Dynamics of Natural Static Emotional Facial Expressions Decoding: A Study Using Event- and Eye Fixation-Related Potentials

This study aims at examining the precise temporal dynamics of the emotional facial decoding as it unfolds in the brain, according to the emotions displayed. To characterize this processing as it occurs in ecological settings, we focused on unconstrained visual explorations of natural emotional faces (i.e., free eye movements). The General Linear Model (GLM; Smith and Kutas, 2015a,b; Kristensen et al., 2017a) enables such a depiction. It allows deconvolving adjacent overlapping responses of the eye fixation-related potentials (EFRPs) elicited by the subsequent fixations and the event-related potentials (ERPs) elicited at the stimuli onset. Nineteen participants were displayed with spontaneous static facial expressions of emotions (Neutral, Disgust, Surprise, and Happiness) from the DynEmo database (Tcherkassof et al., 2013). Behavioral results on participants' eye movements show that the usual diagnostic features in emotional decoding (eyes for negative facial displays and mouth for positive ones) are consistent with the literature. The impact of emotional category on both the ERPs and the EFRPs elicited by the free exploration of the emotional faces is observed upon the temporal dynamics of the emotional facial expression processing. Regarding the ERP at stimulus onset, there is a significant emotion-dependent modulation of the P2-P3 complex and LPP components' amplitude at the left frontal site for the ERPs computed by averaging. Yet, the GLM reveals the impact of subsequent fixations on the ERPs time-locked on stimulus onset. Results are also in line with the valence hypothesis. The observed differences between the two estimation methods (Average vs. GLM) suggest the predominance of the right hemisphere at the stimulus onset and the implication of the left hemisphere in the processing of the information encoded by subsequent fixations. Concerning the first EFRP, the Lambda response and the P2 component are modulated by the emotion of surprise compared to the neutral emotion, suggesting an impact of high-level factors, in parieto-occipital sites. Moreover, no difference is observed on the second and subsequent EFRP. Taken together, the results stress the significant gain obtained in analyzing the EFRPs using the GLM method and pave the way toward efficient ecological emotional dynamic stimuli analyses.

Large-Scale Traveling Waves in EEG Activity Following Eye Movement

In spontaneous, stimulus-evoked, and eye-movement evoked EEG, the oscillatory signal shows large scale, dynamically organized patterns of phase. We investigated eye-movement evoked patterns in free-viewing conditions. Participants viewed photographs of natural scenes in anticipation of a memory test. From 200 ms intervals following saccades, we estimated the EEG phase gradient over the entire scalp, and the wave activity, i.e. the goodness of fit of a wave model involving a phase gradient assumed to be smooth over the scalp. In frequencies centered at 6.5 Hz, large-scale phase organization occurred, peaking around 70 ms after fixation onset and taking the form of a traveling wave. According to the wave gradient, most of the times the wave spreads from the posterior-inferior to anterior-superior direction. In these directions, the gradients depended on the size and direction of the saccade. Wave propagation velocity decreased in the course of the fixation, particularly in the interval from 50 to 150 ms after fixation onset. This interval corresponds to the fixation-related lambda activity, which reflects early perceptual processes following fixation onset. We conclude that lambda activity has a prominent traveling wave component. This component consists of a short-term whole-head phase pattern of specific direction and velocity, which may reflect feedforward propagation of visual information at fixation.

Estimation of overlapped Eye Fixation Related Potentials: The General Linear Model, a more flexible framework than the ADJAR algorithm

The Eye Fixation Related Potential (EFRP) estimation is the average of EEG signals across epochs at ocular fixation onset. Its main limitation is the overlapping issue. Inter Fixation Intervals (IFI) - typically around 300 ms in the case of unrestricted eye movement- depend on participants’ oculomotor patterns, and can be shorter than the latency of the components of the evoked potential. If the duration of an epoch is longer than the IFI value, more than one fixation can occur, and some overlapping between adjacent neural responses ensues. The classical average does not take into account either the presence of several fixations during an epoch or overlapping. The Adjacent Response algorithm (ADJAR), which is popular for event-related potential estimation, was compared to the General Linear Model (GLM) on a real dataset from a conjoint EEG and eye-tracking experiment to address the overlapping issue. The results showed that the ADJAR algorithm was based on assumptions that were too restrictive for EFRP estimation. The General Linear Model appeared to be more robust and efficient. Different configurations of this model were compared to estimate the potential elicited at image onset, as well as EFRP at the beginning of exploration. These configurations took into account the overlap between the event-related potential at stimulus presentation and the following EFRP, and the distinction between the potential elicited by the first fixation onset and subsequent ones. The choice of the General Linear Model configuration was a tradeoff between assumptions about expected behavior and the quality of the EFRP estimation: the number of different potentials estimated by a given model must be controlled to avoid erroneous estimations with large variances.

Task modulates functional connectivity networks in free viewing behavior

In free visual exploration, eye-movement is immediately followed by dynamic reconfiguration of brain functional connectivity. We studied the task-dependency of this process in a combined visual search-change detection experiment. Participants viewed two (nearly) same displays in succession. First time they had to find and remember multiple targets among distractors, so the ongoing task involved memory encoding. Second time they had to determine if a target had changed in orientation, so the ongoing task involved memory retrieval. From multichannel EEG recorded during 200 ms intervals time-locked to fixation onsets, we estimated the functional connectivity using a weighted phase lag index at the frequencies of theta, alpha, and beta bands, and derived global and local measures of the functional connectivity graphs. We found differences between both memory task conditions for several network measures, such as mean path length, radius, diameter, closeness and eccentricity, mainly in the alpha band. Both the local and the global measures indicated that encoding involved a more segregated mode of operation than retrieval. These differences arose immediately after fixation onset and persisted for the entire duration of the lambda complex, an evoked potential commonly associated with early visual perception. We concluded that encoding and retrieval differentially shape network configurations involved in early visual perception, affecting the way the visual input is processed at each fixation. These findings demonstrate that task requirements dynamically control the functional connectivity networks involved in early visual perception.

Supervised learning for neural manifold using spatiotemporal brain activity

OBJECTIVE

Determining the means by which perceived stimuli are compactly represented in the human brain is a difficult task. This study aimed to develop techniques for the construction of the neural manifold as a representation of visual stimuli.

APPROACH

We propose a supervised locally linear embedding method to construct the embedded manifold from brain activity, taking into account similarities between corresponding stimuli. In our experiments, photographic portraits were used as visual stimuli and brain activity was calculated from magnetoencephalographic data using a source localization method.

MAIN RESULTS

The results of 10 × 10-fold cross-validation revealed a strong correlation between manifolds of brain activity and the orientation of faces in the presented images, suggesting that high-level information related to image content can be revealed in the brain responses represented in the manifold.

SIGNIFICANCE

Our experiments demonstrate that the proposed method is applicable to investigation into the inherent patterns of brain activity.

Co-registration of eye movements and event-related potentials in connected-text paragraph reading

Eyetracking during reading has provided a critical source of on-line behavioral data informing basic theory in language processing. Similarly, event-related potentials (ERPs) have provided an important on-line measure of the neural correlates of language processing. Recently there has been strong interest in co-registering eyetracking and ERPs from simultaneous recording to capitalize on the strengths of both techniques, but a challenge has been devising approaches for controlling artifacts produced by eye movements in the EEG waveform. In this paper we describe our approach to correcting for eye movements in EEG and demonstrate its applicability to reading. The method is based on independent components analysis, and uses three criteria for identifying components tied to saccades: (1) component loadings on the surface of the head are consistent with eye movements; (2) source analysis localizes component activity to the eyes, and (3) the temporal activation of the component occurred at the time of the eye movement and differed for right and left eye movements. We demonstrate this method's applicability to reading by comparing ERPs time-locked to fixation onset in two reading conditions. In the text-reading condition, participants read paragraphs of text. In the pseudo-reading control condition, participants moved their eyes through spatially similar pseudo-text that preserved word locations, word shapes, and paragraph spatial structure, but eliminated meaning. The corrected EEG, time-locked to fixation onsets, showed effects of reading condition in early ERP components. The results indicate that co-registration of eyetracking and EEG in connected-text paragraph reading is possible, and has the potential to become an important tool for investigating the cognitive and neural bases of on-line language processing in reading.

Cross-frequency phase synchrony around the saccade period as a correlate of perceiver's internal state

In active vision, eye-movements depend on perceivers' internal state. We investigated peri-fixation brain activity for internal state-specific tagging. Human participants performed a task, in which a visual object was presented for identification in lateral visual field, to which they moved their eyes as soon as possible from a central fixation point. Next, a phrase appeared in the same location; the phrase could either be an easy or hard question about the object, answered by pressing one of two alternative response buttons, or it could be an instruction to simply press one of these two buttons. Depending on whether these messages were blocked or randomly mixed, one of two different internal states was induced: either the task was known in advance or it wasn't. Eye movements and electroencephalogram (EEG) were recorded simultaneously during task performance. Using eye-event-time-locked averaging and independent component analysis, saccade- and fixation-related components were identified. Coss-frequency phase-synchrony was observed between the alpha/beta1 ranges of fixation-related and beta2/gamma1 ranges of saccade-related activity 50 ms prior to fixation onset in the mixed-phrase condition only. We interpreted this result as evidence for internal state-specific tagging.

α-band oscillations, attention, and controlled access to stored information

Alpha-band oscillations are the dominant oscillations in the human brain and recent evidence suggests that they have an inhibitory function. Nonetheless, there is little doubt that alpha-band oscillations also play an active role in information processing. In this article, I suggest that alpha-band oscillations have two roles (inhibition and timing) that are closely linked to two fundamental functions of attention (suppression and selection), which enable controlled knowledge access and semantic orientation (the ability to be consciously oriented in time, space, and context). As such, alpha-band oscillations reflect one of the most basic cognitive processes and can also be shown to play a key role in the coalescence of brain activity in different frequencies.

Combining EEG and eye tracking: identification, characterization, and correction of eye movement artifacts in electroencephalographic data

Eye movements introduce large artifacts to electroencephalographic recordings (EEG) and thus render data analysis difficult or even impossible. Trials contaminated by eye movement and blink artifacts have to be discarded, hence in standard EEG-paradigms subjects are required to fixate on the screen. To overcome this restriction, several correction methods including regression and blind source separation have been proposed. Yet, there is no automated standard procedure established. By simultaneously recording eye movements and 64-channel-EEG during a guided eye movement paradigm, we investigate and review the properties of eye movement artifacts, including corneo-retinal dipole changes, saccadic spike potentials and eyelid artifacts, and study their interrelations during different types of eye- and eyelid movements. In concordance with earlier studies our results confirm that these artifacts arise from different independent sources and that depending on electrode site, gaze direction, and choice of reference these sources contribute differently to the measured signal. We assess the respective implications for artifact correction methods and therefore compare the performance of two prominent approaches, namely linear regression and independent component analysis (ICA). We show and discuss that due to the independence of eye artifact sources, regression-based correction methods inevitably over- or under-correct individual artifact components, while ICA is in principle suited to address such mixtures of different types of artifacts. Finally, we propose an algorithm, which uses eye tracker information to objectively identify eye-artifact related ICA-components (ICs) in an automated manner. In the data presented here, the algorithm performed very similar to human experts when those were given both, the topographies of the ICs and their respective activations in a large amount of trials. Moreover it performed more reliable and almost twice as effective than human experts when those had to base their decision on IC topographies only. Furthermore, a receiver operating characteristic (ROC) analysis demonstrated an optimal balance of false positive and false negative at an area under curve (AUC) of more than 0.99. Removing the automatically detected ICs from the data resulted in removal or substantial suppression of ocular artifacts including microsaccadic spike potentials, while the relevant neural signal remained unaffected. In conclusion the present work aims at a better understanding of individual eye movement artifacts, their interrelations and the respective implications for eye artifact correction. Additionally, the proposed ICA-procedure provides a tool for optimized detection and correction of eye movement-related artifact components.

Feature-independent neural coding of target detection during search of natural scenes

Visual search requires humans to detect a great variety of target objects in scenes cluttered by other objects or the natural environment. It is unknown whether there is a general purpose neural detection mechanism in the brain that codes the presence of a wide variety of categories of objects embedded in natural scenes. We provide evidence for a feature-independent coding mechanism for detecting behaviorally relevant targets in natural scenes in the dorsal frontoparietal network. Pattern classifiers using single-trial fMRI responses in the dorsal frontoparietal network reliably predicted the presence of 368 different target objects and also the observer's choices. Other vision-related areas such as the primary visual cortex, lateral occipital complex, the parahippocampal, and the fusiform gyri did not predict target presence, while high-level association areas related to general purpose decision making, including the dorsolateral prefrontal cortex and anterior cingulate, did. Activity in the intraparietal sulcus, a main area in the dorsal frontoparietal network, correlated with observers' decision confidence and with the task difficulty of individual images. These results cannot be explained by physical differences across images or eye movements. Thus, the dorsal frontoparietal network detects behaviorally relevant targets in natural scenes independent of their defining visual features and may be the human analog of the priority map in monkey lateral intraparietal cortex.

Neural saccadic response estimation during natural viewing

Studying neural activity during natural viewing conditions is not often attempted. Isolating the neural response of a single saccade is necessary to study neural activity during natural viewing; however, the close temporal spacing of saccades that occurs during natural viewing makes it difficult to determine the response to a single saccade. Herein, a general linear model (GLM) approach is applied to estimate the EEG neural saccadic response for different segments of the saccadic main sequence separately. It is determined that, in visual search conditions, neural responses estimated by conventional event-related averaging are significantly and systematically distorted relative to GLM estimates due to the close temporal spacing of saccades during visual search. Before the GLM is applied, analyses are applied that demonstrate that saccades during visual search with intersaccadic spacings as low as 100-150 ms do not exhibit significant refractory effects. Therefore, saccades displaying different intersaccadic spacings during visual search can be modeled using the same regressor in a GLM. With the use of the GLM approach, neural responses were separately estimated for five different ranges of saccade amplitudes during visual search. Occipital responses time locked to the onsets of saccades during visual search were found to account for, on average, 79 percent of the variance of EEG activity in a window 90-200 ms after the onsets of saccades for all five saccade amplitude ranges that spanned a range of 0.2-6.0 degrees. A GLM approach was also used to examine the lateralized ocular artifacts associated with saccades. Possible extensions of the methods presented here to account for the superposition of microsaccades in event-related EEG studies conducted in nominal fixation conditions are discussed.

Evoked traveling alpha waves predict visual-semantic categorization-speed

In the present study we have tested the hypothesis that evoked traveling alpha waves are behaviorally significant. The results of a visual-semantic categorization task show that three early ERP components including the P1-N1 complex had a dominant frequency characteristic in the alpha range and behaved like traveling waves do. They exhibited a traveling direction from midline occipital to right lateral parietal sites. Phase analyses revealed that this traveling behavior of ERP components could be explained by phase-delays in the alpha but not theta and beta frequency range. Most importantly, we found that the speed of the traveling alpha wave was significantly and negatively correlated with reaction time indicating that slow traveling speed was associated with fast picture-categorization. We conclude that evoked alpha oscillations are functionally associated with early access to visual-semantic information and generate--or at least modulate--the early waveforms of the visual ERP.

Evoked alpha and early access to the knowledge system: the P1 inhibition timing hypothesis

In this article, a theory is presented which assumes that the visual P1 reflects the same cognitive and physiological functionality as alpha (with a frequency of about 10 Hz).Whereas alpha is an ongoing process, the P1 is the manifestation of an event-related process. It is suggested that alpha and the P1 reflect inhibition that is effective during early access to a complex knowledge system (KS). Most importantly, inhibition operates in two different ways. In potentially competing and task irrelevant networks, inhibition is used to block information processing. In task relevant neural networks, however, inhibition is used to increase the signal to noise ratio (SNR) by enabling precisely timed activity in neurons with a high level of excitation but silencing neurons with a comparatively low level of excitation. Inhibition is increased to modulate the SNR when processing complexity and network excitation increases and when certain types of attentional demands - such as top-down control, expectancy or reflexive attention - increase. A variety of findings are reviewed to demonstrate that they can well be interpreted on the basis of the suggested theory. One interesting aspect thereby is that attentional benefits (reflected e.g., by a larger P1 for attended as compared to unattended items at contralateral sites) and costs (reflected e.g., by a larger P1 at ipsilateral sites) can both be interpreted in terms of inhibition. In the former case an increased P1 is associated with a more effective processing of the presented item (due to an inhibition modulated increase in SNR), in the latter case, however, with a suppression of item processing (due to inhibition that blocks information processing).