Combining EEG and eye movement recording in free viewing: Pitfalls and possibilities

Neural mechanism underlying preview effects and masked priming effects in visual word processing

Two classic experimental paradigms - masked repetition priming and the boundary paradigm - have played a pivotal role in understanding the process of visual word recognition. Traditionally, these paradigms have been employed by different communities of researchers, with their own long-standing research traditions. Nevertheless, a review of the literature suggests that the brain-electric correlates of word processing established with both paradigms may show interesting similarities, in particular with regard to the location, timing, and direction of N1 and N250 effects. However, as of yet, no direct comparison has been undertaken between the two paradigms. In the current study, we used combined eye-tracking/EEG to perform such a within-subject comparison using the same materials (single Chinese characters) as stimuli. To facilitate direct comparisons, we used a simplified version of the boundary paradigm - the single word boundary paradigm. Our results show the typical early repetition effects of N1 and N250 for both paradigms. However, repetition effects in N250 (i.e., a reduced negativity following identical-word primes/previews as compared to different-word primes/previews) were larger with the single word boundary paradigm than with masked priming. For N1 effects, repetition effects were similar across the two paradigms, showing a larger N1 after repetitions as compared to alternations. Therefore, the results indicate that at the neural level, a briefly presented and masked foveal prime produces qualitatively similar facilitatory effects on visual word recognition as a parafoveal preview before a single saccade, although such effects appear to be stronger in the latter case.

Refixation behavior in naturalistic viewing: Methods, mechanisms, and neural correlates

When freely viewing a scene, the eyes often return to previously visited locations. By tracking eye movements and coregistering eye movements and EEG, such refixations are shown to have multiple roles: repairing insufficient encoding from precursor fixations, supporting ongoing viewing by resampling relevant locations prioritized by precursor fixations, and aiding the construction of memory representations. All these functions of refixation behavior are understood to be underpinned by three oculomotor and cognitive systems and their associated brain structures. First, immediate saccade planning prior to refixations involves attentional selection of candidate locations to revisit. This process is likely supported by the dorsal attentional network. Second, visual working memory, involved in maintaining task-related information, is likely supported by the visual cortex. Third, higher-order relevance of scene locations, which depends on general knowledge and understanding of scene meaning, is likely supported by the hippocampal memory system. Working together, these structures bring about viewing behavior that balances exploring previously unvisited areas of a scene with exploiting visited areas through refixations.

Combining EEG and 3D-eye-tracking to study the prediction of upcoming speech in naturalistic virtual environments: A proof of principle

EEG and eye-tracking provide complementary information when investigating language comprehension. Evidence that speech processing may be facilitated by speech prediction comes from the observation that a listener's eye gaze moves towards a referent before it is mentioned if the remainder of the spoken sentence is predictable. However, changes to the trajectory of anticipatory fixations could result from a change in prediction or an attention shift. Conversely, N400 amplitudes and concurrent spectral power provide information about the ease of word processing the moment the word is perceived. In a proof-of-principle investigation, we combined EEG and eye-tracking to study linguistic prediction in naturalistic, virtual environments. We observed increased processing, reflected in theta band power, either during verb processing - when the verb was predictive of the noun - or during noun processing - when the verb was not predictive of the noun. Alpha power was higher in response to the predictive verb and unpredictable nouns. We replicated typical effects of noun congruence but not predictability on the N400 in response to the noun. Thus, the rich visual context that accompanied speech in virtual reality influenced language processing compared to previous reports, where the visual context may have facilitated processing of unpredictable nouns. Finally, anticipatory fixations were predictive of spectral power during noun processing and the length of time fixating the target could be predicted by spectral power at verb onset, conditional on the object having been fixated. Overall, we show that combining EEG and eye-tracking provides a promising new method to answer novel research questions about the prediction of upcoming linguistic input, for example, regarding the role of extralinguistic cues in prediction during language comprehension.

Saccade execution increases the preview effect with faces: An EEG and eye-tracking coregistration study

Under naturalistic viewing conditions, humans conduct about three to four saccadic eye movements per second. These dynamics imply that in real life, humans rarely see something completely new; there is usually a preview of the upcoming foveal input from extrafoveal regions of the visual field. In line with results from the field of reading research, we have shown with EEG and eye-tracking coregistration that an extrafoveal preview also affects postsaccadic visual object processing and facilitates discrimination. Here, we ask whether this preview effect in the fixation-locked N170, and in manual responses to the postsaccadic target face (tilt discrimination), requires saccade execution. Participants performed a gaze-contingent experiment in which extrafoveal face images could change their orientation during a saccade directed to them. In a control block, participants maintained stable gaze throughout the experiment and the extrafoveal face reappeared foveally after a simulated saccade latency. Compared with this no-saccade condition, the neural and the behavioral preview effects were much larger in the saccade condition. We also found shorter first fixation durations after an invalid preview, which is in contrast to reading studies. We interpret the increased preview effect under saccade execution as the result of the additional sensorimotor processes that come with gaze behavior compared with visual perception under stable fixation. In addition, our findings call into question whether EEG studies with fixed gaze capture key properties and dynamics of active, natural vision.

Tracking gaze position from EEG: Exploring the possibility of an EEG-based virtual eye-tracker

INTRODUCTION

Ocular artifact has long been viewed as an impediment to the interpretation of electroencephalogram (EEG) signals in basic and applied research. Today, the use of blind source separation (BSS) methods, including independent component analysis (ICA) and second-order blind identification (SOBI), is considered an essential step in improving the quality of neural signals. Recently, we introduced a method consisting of SOBI and a discriminant and similarity (DANS)-based identification method, capable of identifying and extracting eye movement-related components. These recovered components can be localized within ocular structures with a high goodness of fit (>95%). This raised the possibility that such EEG-derived SOBI components may be used to build predictive models for tracking gaze position.

METHODS

As proof of this new concept, we designed an EEG-based virtual eye-tracker (EEG-VET) for tracking eye movement from EEG alone. The EEG-VET is composed of a SOBI algorithm for separating EEG signals into different components, a DANS algorithm for automatically identifying ocular components, and a linear model to transfer ocular components into gaze positions.

RESULTS

The prototype of EEG-VET achieved an accuracy of 0.920° and precision of 1.510° of a visual angle in the best participant, whereas an average accuracy of 1.008° ± 0.357° and a precision of 2.348° ± 0.580° of a visual angle across all participants (N = 18).

CONCLUSION

This work offers a novel approach that readily co-registers eye movement and neural signals from a single-EEG recording, thus increasing the ease of studying neural mechanisms underlying natural cognition in the context of free eye movement.

Episodic memory formation in unrestricted viewing

The brain systems of episodic memory and oculomotor control are tightly linked, suggesting a crucial role of eye movements in memory. But little is known about the neural mechanisms of memory formation across eye movements in unrestricted viewing behavior. Here, we leverage simultaneous eye tracking and EEG recording to examine episodic memory formation in free viewing. Participants memorized multi-element events while their EEG and eye movements were concurrently recorded. Each event comprised elements from three categories (face, object, place), with two exemplars from each category, in different locations on the screen. A subsequent associative memory test assessed participants' memory for the between-category associations that specified each event. We used a deconvolution approach to overcome the problem of overlapping EEG responses to sequential saccades in free viewing. Brain activity was time-locked to the fixation onsets, and we examined EEG power in the theta and alpha frequency bands, the putative oscillatory correlates of episodic encoding mechanisms. Three modulations of fixation-related EEG predicted high subsequent memory performance: (1) theta increase at fixations after between-category gaze transitions, (2) theta and alpha increase at fixations after within-element gaze transitions, (3) alpha decrease at fixations after between-exemplar gaze transitions. Thus, event encoding with unrestricted viewing behavior was characterized by three neural mechanisms, manifested in fixation-locked theta and alpha EEG activity that rapidly turned on and off during the unfolding eye movement sequences. These three distinct neural mechanisms may be the essential building blocks that subserve the buildup of coherent episodic memories during unrestricted viewing behavior.

WildLab: A naturalistic free viewing experiment reveals previously unknown electroencephalography signatures of face processing

Neural mechanisms of face perception are predominantly studied in well-controlled experimental settings that involve random stimulus sequences and fixed eye positions. Although powerful, the employed paradigms are far from what constitutes natural vision. Here, we demonstrate the feasibility of ecologically more valid experimental paradigms using natural viewing behaviour, by combining a free viewing paradigm on natural scenes, free of photographer bias, with advanced data processing techniques that correct for overlap effects and co-varying non-linear dependencies of multiple eye movement parameters. We validate this approach by replicating classic N170 effects in neural responses, triggered by fixation onsets (fixation event-related potentials [fERPs]). Importantly, besides finding a strong correlation between both experiments, our more natural stimulus paradigm yielded smaller variability between subjects than the classic setup. Moving beyond classic temporal and spatial effect locations, our experiment furthermore revealed previously unknown signatures of face processing: This includes category-specific modulation of the event-related potential (ERP)'s amplitude even before fixation onset, as well as adaptation effects across subsequent fixations depending on their history.

Face familiarity revealed by fixational eye movements and fixation-related potentials in free viewing

Event-related potentials (ERPs) and the oculomotor inhibition (OMI) in response to visual transients are known to be sensitive to stimulus properties, attention, and expectation. We have recently found that the OMI is also sensitive to face familiarity. In natural vision, stimulation of the visual cortex is generated primarily by saccades, and it has been recently suggested that fixation-related potentials (FRPs) share similar components with the ERPs. Here, we investigated whether FRPs and microsaccade inhibition (OMI) in free viewing are sensitive to face familiarity. Observers freely watched a slideshow of seven unfamiliar and one familiar facial images presented randomly for 4-s periods, with multiple images per identity. We measured the occipital fixation-related N1 relative to the P1 magnitude as well as the associated fixation-triggered OMI. We found that the average N1-P1 was significantly smaller and the OMI was shorter for the familiar face, compared with any of the seven unfamiliar faces. Moreover, the P1 was suppressed across saccades for the familiar but not for the unfamiliar faces. Our results highlight the sensitivity of the occipital FRPs to stimulus properties such as face familiarity and advance our understanding of the integration process across successive saccades in natural vision.

Naturalistic viewing conditions can increase task engagement and aesthetic preference but have only minimal impact on EEG quality

Free gaze and moving images are typically avoided in EEG experiments due to the expected generation of artifacts and noise. Yet for a growing number of research questions, loosening these rigorous restrictions would be beneficial. Among these is research on visual aesthetic experiences, which often involve open-ended exploration of highly variable stimuli. Here we systematically compare the effect of conservative vs. more liberal experimental settings on various measures of behavior, brain activity and physiology in an aesthetic rating task. Our primary aim was to assess EEG signal quality. 43 participants either maintained fixation or were allowed to gaze freely, and viewed either static images or dynamic (video) stimuli consisting of dance performances or nature scenes. A passive auditory background task (auditory steady-state response; ASSR) was added as a proxy measure for overall EEG recording quality. We recorded EEG, ECG and eyetracking data, and participants rated their aesthetic preference and state of boredom on each trial. Whereas both behavioral ratings and gaze behavior were affected by task and stimulus manipulations, EEG SNR was barely affected and generally robust across all conditions, despite only minimal preprocessing and no trial rejection. In particular, we show that using video stimuli does not necessarily result in lower EEG quality and can, on the contrary, significantly reduce eye movements while increasing both the participants' aesthetic response and general task engagement. We see these as encouraging results indicating that - at least in the lab - more liberal experimental conditions can be adopted without significant loss of signal quality.

Revealing Whole-Brain Causality Networks During Guided Visual Searching

In our daily lives, we use eye movements to actively sample visual information from our environment ("active vision"). However, little is known about how the underlying mechanisms are affected by goal-directed behavior. In a study of 31 participants, magnetoencephalography was combined with eye-tracking technology to investigate how interregional interactions in the brain change when engaged in two distinct forms of active vision: freely viewing natural images or performing a guided visual search. Regions of interest with significant fixation-related evoked activity (FRA) were identified with spatiotemporal cluster permutation testing. Using generalized partial directed coherence, we show that, in response to fixation onset, a bilateral cluster consisting of four regions (posterior insula, transverse temporal gyri, superior temporal gyrus, and supramarginal gyrus) formed a highly connected network during free viewing. A comparable network also emerged in the right hemisphere during the search task, with the right supramarginal gyrus acting as a central node for information exchange. The results suggest that all four regions are vital to visual processing and guiding attention. Furthermore, the right supramarginal gyrus was the only region where activity during fixations on the search target was significantly negatively correlated with search response times. Based on our findings, we hypothesize that, following a fixation, the right supramarginal gyrus supplies the right supplementary eye field (SEF) with new information to update the priority map guiding the eye movements during the search task.

Elements of exogenous attentional cueing preserved during optokinetic motion of the visual scene

Navigating through our environment raises challenges for perception by generating salient background visual motion, and eliciting prominent eye movements to stabilise the retinal image. It remains unclear if exogenous spatial attentional orienting is possible during background motion and the eye movements it causes, and whether this compromises the underlying neural processing. To test this, we combined exogenous orienting, visual scene motion, and EEG. 26 participants viewed a background of moving black and grey bars (optokinetic stimulation). We tested for effects of non-spatially predictive peripheral cueing on visual motion discrimination of a target dot, presented either at the same (valid) or opposite (invalid) location as the preceding cue. Valid cueing decreased reaction times not only when participants kept their gaze fixed on a central point (fixation blocks), but even when there was no fixation point, so that participants performed intensive, repetitive tracking eye movements (eye movements blocks). Overall, manual response reaction times were slower during eye movements. Cueing also produced reliable effects on neural activity on either block, including within the first 120 milliseconds of neural processing of the target. The key pattern with larger ERP amplitudes on invalid versus valid trials showed that the neural substrate of exogenous cueing was highly similar during eye movements or fixation. Exogenous peripheral cueing and its neural correlates are robust against distraction from the moving visual scene, important for perceptual cognition during navigation.

Recent Advances and Current Trends in Brain-Computer Interface (BCI) Research and Their Applications

Brain-Computer Interface (BCI) provides direct communication between the brain and an external device. BCI systems have become a trendy field of research in recent years. These systems can be used in a variety of applications to help both disabled and healthy people. Concerning significant BCI progress, we may assume that these systems are not very far from real-world applications. This review has taken into account current trends in BCI research. In this survey, one hundred most cited articles from the WOS database were selected over the last four years. This survey is divided into several sectors. These sectors are Medicine, Communication and Control, Entertainment, and Other BCI applications. The application area, recording method, signal acquisition types, and countries of origin have been identified in each article. This survey provides an overview of the BCI articles published from 2016 to 2020 and their current trends and advances in different application areas.

Synchronization of acquisition devices in neuroimaging: An application using co-registration of eye movements and electroencephalography

Interest in applications for the simultaneous acquisition of data from different devices is growing. In neuroscience for example, co-registration complements and overcomes some of the shortcomings of individual methods. However, precise synchronization of the different data streams involved is required before joint data analysis. Our article presents and evaluates a synchronization method which maximizes the alignment of information across time. Synchronization through common triggers is widely used in all existing methods, because it is very simple and effective. However, this solution has been found to fail in certain practical situations, namely for the spurious detection of triggers and/or when the timestamps of triggers sampled by each acquisition device are not jointly distributed linearly for the entire duration of an experiment. We propose two additional mechanisms, the "Longest Common Subsequence" algorithm and a piecewise linear regression, in order to overcome the limitations of the classical method of synchronizing common triggers. The proposed synchronization method was evaluated using both real and artificial data. Co-registrations of electroencephalographic signals (EEG) and eye movements were used for real data. We compared the effectiveness of our method to another open source method implemented using EYE-EEG toolbox. Overall, we show that our method, implemented in C++ as a DOS application, is very fast, robust and fully automatic.

IEyeGASE: An Intelligent Eye Gaze-Based Assessment System for Deeper Insights into Learner Performance

In the current education environment, learning takes place outside the physical classroom, and tutors need to determine whether learners are absorbing the content delivered to them. Online assessment has become a viable option for tutors to establish the achievement of course learning outcomes by learners. It provides real-time progress and immediate results; however, it has challenges in quantifying learner aspects like wavering behavior, confidence level, knowledge acquired, quickness in completing the task, task engagement, inattentional blindness to critical information, etc. An intelligent eye gaze-based assessment system called IEyeGASE is developed to measure insights into these behavioral aspects of learners. The system can be integrated into the existing online assessment system and help tutors re-calibrate learning goals and provide necessary corrective actions.

Neural correlates of intra-saccadic motion perception

Retinal motion of the visual scene is not consciously perceived during ocular saccades in normal everyday conditions. It has been suggested that extra-retinal signals actively suppress intra-saccadic motion perception to preserve stable perception of the visual world. However, using stimuli optimized to preferentially activate the M-pathway, Castet and Masson (2000) demonstrated that motion can be perceived during a saccade. Based on this psychophysical paradigm, we used electroencephalography and eye-tracking recordings to investigate the neural correlates related to the conscious perception of intra-saccadic motion. We demonstrated the effective involvement during saccades of the cortical areas V1-V2 and MT-V5, which convey motion information along the M-pathway. We also showed that individual motion perception was related to retinal temporal frequency.

Neurophysiological indicators of internal attention: An fMRI-eye-tracking coregistration study

Many goal-directed, as well as spontaneous everyday activities (e.g., planning, mind-wandering), rely on an internal focus of attention. This fMRI-eye-tracking coregistration study investigated brain mechanisms and eye behavior related to internally versus externally directed cognition. Building on an established paradigm, we manipulated internal attention demands within tasks utilizing conditional stimulus masking. Internally directed cognition involved bilateral activation of the lingual gyrus and inferior parietal lobe areas as well as wide-spread deactivation of visual networks. Moreover, internally directed cognition was related to greater pupil diameter, pupil diameter variance, blink duration, fixation disparity variance, and smaller amounts of microsaccades. FMRI-eye-tracking covariation analyses further revealed that larger pupil diameter was related to increased activation of basal ganglia and lingual gyrus. It can be concluded that internally and externally directed cognition are characterized by distinct neurophysiological signatures. The observed neurophysiological differences indicate that internally directed cognition is associated with reduced processing of task-irrelevant information and increased mental load. These findings shed further light on the interplay between neural and perceptual mechanisms contributing to an internal focus of attention.

Reading direct speech quotes increases theta phase-locking: Evidence for cortical tracking of inner speech?

Growing evidence shows that theta-band (4-7 Hz) activity in the auditory cortex phase-locks to rhythms of overt speech. Does theta activity also encode the rhythmic dynamics of inner speech? Previous research established that silent reading of direct speech quotes (e.g., Mary said: "This dress is lovely!") elicits more vivid inner speech than indirect speech quotes (e.g., Mary said that the dress was lovely). As we cannot directly track the phase alignment between theta activity and inner speech over time, we used EEG to measure the brain's phase-locked responses to the onset of speech quote reading. We found that direct (vs. indirect) quote reading was associated with increased theta phase synchrony over trials at 250-500 ms post-reading onset, with sources of the evoked activity estimated in the speech processing network. An eye-tracking control experiment confirmed that increased theta phase synchrony in direct quote reading was not driven by eye movement patterns, and more likely reflects synchronous phase resetting at the onset of inner speech. These findings suggest a functional role of theta phase modulation in reading-induced inner speech.

Eye Tracking Measures for Studying Language Comprehension Deficits in Aphasia: A Systematic Search and Scoping Review

Aim The aim of this scoping review is to identify the eye tracking paradigms and eye movement measures used to investigate auditory and reading comprehension deficits in persons with aphasia (PWA). Method MEDLINE via PubMed, Cochrane, CINAHL, Embase, PsycINFO, OTseeker, Scopus, Google Scholar, Grey Literature Database, and ProQuest Search (Dissertations & Theses) were searched for relevant studies. The Covidence software was used to manage the initial and full-text screening process for the search. Results and Discussion From a total of 1,803 studies, 68 studies were included for full-text screening. In addition, 418 records from gray literature were also screened. After full-text screening, 16 studies were included for this review-12 studies for auditory comprehension in PWA and four studies for reading comprehension in PWA. The review highlights the use of common eye tracking paradigms used to study language comprehension in PWA. We also discusse eye movement measures and how they help in assessing auditory and reading comprehension. Methodological challenges of using eye tracking are discussed. Conclusion The studies summarized in this scoping review provide evidence that the eye tracking methods are beneficial for studying auditory and reading comprehension in PWA.

Closed-loop EEG study on visual recognition during driving

OBJECTIVE

In contrast to the classical visual BCI paradigms, which adhere to a rigid trial structure and restricted user behavior, EEG-based visual recognition decoding during our daily activities remains challenging. The objective of this study is to explore the feasibility of decoding the EEG signature of visual recognition in experimental conditions promoting our natural ocular behavior when interacting with our dynamic environment.

APPROACH

In our experiment, subjects visually search for a target object among suddenly appearing objects in the environment while driving a car-simulator. Given that subjects exhibit an unconstrained overt visual behavior, we based our study on eye fixation-related potentials (EFRP). We report on gaze behavior and single-trial EFRP decoding performance (fixations on visually similar target vs. non-target objects). In addition, we demonstrate the application of our approach in a closed-loop BCI setup.

MAIN RESULTS

To identify the target out of four symbol types along a road segment, the BCI system integrated decoding probabilities of multiple EFRP and achieved the average online accuracy of 0.37 ± 0.06 (12 subjects), statistically significantly above the chance level. Using the acquired data, we performed a comparative study of classification algorithms (discriminating target vs. non-target) and feature spaces in a simulated online scenario. The EEG approaches yielded similar moderate performances of at most 0.6 AUC, yet statistically significantly above the chance level. In addition, the gaze duration (dwell time) appears to be an additional informative feature in this context.

SIGNIFICANCE

These results show that visual recognition of sudden events can be decoded during active driving. Therefore, this study lays a foundation for assistive and recommender systems based on the driver's brain signals.

A passive BCI for monitoring the intentionality of the gaze-based moving object selection

OBJECTIVE

The use of an electroencephalogram (EEG) anticipation-related component, the expectancy wave (E-wave), in brain-machine interaction was proposed more than 50 years ago. This possibility was not explored for decades, but recently it was shown that voluntary attempts to select items using eye fixations, but not spontaneous eye fixations, are accompanied by the E-wave. Thus, the use of the E-wave detection was proposed for the enhancement of gaze interaction technology, which has a strong need for a mean to decide if a gaze behaviour is voluntary or not. Here, we attempted at estimating whether this approach can be used in the context of moving object selection through smooth pursuit eye movements.

APPROACH

18 participants selected, one by one, items which moved on a computer screen, by gazing at them. In separate runs, the participants performed tasks not related to voluntary selection but also provoking smooth pursuit. A low-cost consumer-grade eye tracker was used for item selection.

MAIN RESULTS

A component resembling the E-wave was found in the averaged EEG segments time-locked to voluntary selection events of every participant. Linear discriminant analysis with shrinkage regularization (sLDA) classified the intentional and spontaneous smooth pursuit eye movements, using single-trial 300 ms long EEG segments, significantly above chance in eight participants. When the classifier output was averaged over ten subsequent data segments, median group ROC AUC of 0.75 was achieved.

SIGNIFICANCE

The results suggest the possible usefulness of the E-wave detection in the gaze-based selection of moving items, e.g., in video games. This technique might be more effective when trial data can be averaged, thus it could be considered for use in passive interfaces, for example, in estimating the degree of the user's involvement during gaze-based interaction.

Regression-based analysis of combined EEG and eye-tracking data: Theory and applications

Fixation-related potentials (FRPs), neural responses aligned to the end of saccades, are a promising tool for studying the dynamics of attention and cognition under natural viewing conditions. In the past, four methodological problems have complicated the analysis of such combined eye-tracking/electroencephalogram experiments: (1) the synchronization of data streams, (2) the removal of ocular artifacts, (3) the condition-specific temporal overlap between the brain responses evoked by consecutive fixations, and (4) the fact that numerous low-level stimulus and saccade properties also influence the postsaccadic neural responses. Although effective solutions exist for the first two problems, the latter two are only beginning to be addressed. In the current paper, we present and review a unified regression-based framework for FRP analysis that allows us to deconvolve overlapping potentials while also controlling for both linear and nonlinear confounds on the FRP waveform. An open software implementation is provided for all procedures. We then demonstrate the advantages of this proposed (non)linear deconvolution modeling approach for data from three commonly studied paradigms: face perception, scene viewing, and reading. First, for a traditional event-related potential (ERP) face recognition experiment, we show how this technique can separate stimulus ERPs from overlapping muscle and brain potentials produced by small (micro)saccades on the face. Second, in natural scene viewing, we model and isolate multiple nonlinear effects of saccade parameters on the FRP. Finally, for a natural sentence reading experiment using the boundary paradigm, we show how it is possible to study the neural correlates of parafoveal preview after removing spurious overlap effects caused by the associated difference in average fixation time. Our results suggest a principal way of measuring reliable eye movement-related brain activity during natural vision.

Modeling Cognitive Activity of the Human Brain by the Mathematical Apparatus of Quantum Mechanics

This paper discusses the possible approaches to modeling the cognitive activity of the human brain using the mathematical apparatus of quantum mechanics (primarily – potential wells and virtual particles) in terms of the theory of information representations. The article briefly describes the proposed theory of information representations, draws analogies, and identifies common features of information representations of the human mind and Feynman’s virtual particles. The human mind is represented as a one-dimensional potential well with finite walls of different sizes and internal potential barrier simulating the boundary between consciousness and subconsciousness. This creates a foundation for a mathematical apparatus that can make it possible to forecast particular cognitive functions of the human brain. The results of these studies can be used to create predictive models of various cognitive disorders (diseases) and to be used in diagnostics.

Using Fixation-Related Potentials for Inspecting Natural Interactions

Brain-Computer Interfaces (BCI) offer unique windows into the cognitive processes underlying human-machine interaction. Identifying and analyzing the appropriate brain activity to have access to such windows is often difficult due to technical or psycho-physiological constraints. Indeed, studying interactions through this approach frequently requires adapting them to accommodate specific BCI-related paradigms which change the functioning of their interface on both the human-side and the machine-side. The combined examination of Electroencephalography and Eyetracking recordings, mainly by means of studying Fixation-Related Potentials, can help to circumvent the necessity for these adaptations by determining interaction-relevant moments during natural manipulation. In this contribution, we examine how properties contained within the bi-modal recordings can be used to assess valuable information about the interaction. Practically, three properties are studied which can be obtained solely through data obtained from analysis of the recorded biosignals. Namely, these properties consist of relative gaze metrics, being abstractions of the gaze patterns, the amplitude variations in the early brain activity potentials and the brain activity frequency band differences between fixations. Through their observation, information about three different aspects of the explored interface are obtained. Respectively, the properties provide insights about general perceived task difficulty, locate moments of higher attentional effort and discriminate between moments of exploration and moments of active interaction.

Neurophysiological indicators of internal attention: An electroencephalography-eye-tracking coregistration study

INTRODUCTION

Many goal-directed and spontaneous everyday activities (e.g., planning, mind wandering) rely on an internal focus of attention. Internally directed cognition (IDC) was shown to differ from externally directed cognition in a range of neurophysiological indicators such as electroencephalogram (EEG) alpha activity and eye behavior.

METHODS

In this EEG-eye-tracking coregistration study, we investigated effects of attention direction on EEG alpha activity and various relevant eye parameters. We used an established paradigm to manipulate internal attention demands in the visual domain within tasks by means of conditional stimulus masking.

RESULTS

Consistent with previous research, IDC involved relatively higher EEG alpha activity (lower alpha desynchronization) at posterior cortical sites. Moreover, IDC was characterized by greater pupil diameter (PD), fewer microsaccades, fixations, and saccades. These findings show that internal versus external cognition is associated with robust differences in several indicators at the neural and perceptual level. In a second line of analysis, we explored the intrinsic temporal covariation between EEG alpha activity and eye parameters during rest. This analysis revealed a positive correlation of EEG alpha power with PD especially in bilateral parieto-occipital regions.

CONCLUSION

Together, these findings suggest that EEG alpha activity and PD represent time-sensitive indicators of internal attention demands, which may be involved in a neurophysiological gating mechanism serving to shield internal cognition from irrelevant sensory information.

Neural correlates of task-related refixation behavior

Eye movement research has shown that attention shifts from the currently fixated location to the next before a saccade is executed. We investigated whether the cost of the attention shift depends on higher-order processing at the time of fixation, in particular on visual working memory load differences between fixations and refixations on task-relevant items. The attention shift is reflected in EEG activity in the saccade-related potential (SRP). In a free viewing task involving visual search and memorization of multiple targets amongst distractors, we compared the SRP in first fixations versus refixations on targets and distractors. The task-relevance of targets implies that more information will be loaded in memory (e.g. both identity and location) than for distractors (e.g. location only). First fixations will involve greater memory load than refixations, since first fixations involve loading of new items, while refixations involve rehearsal of previously visited items. The SRP in the interval preceding the saccade away from a target or distractor revealed that saccade preparation is affected by task-relevance and refixation behavior. For task-relevant items only, we found longer fixation duration and higher SRP amplitudes for first fixations than for refixations over the occipital region and the opposite effect over the frontal region. Our findings provide first neurophysiological evidence that working memory loading of task-relevant information at fixation affects saccade planning.

Ocular fixations and presaccadic potentials to explain pareidolias in Parkinson's disease

In Parkinson's disease, a precursor phenomenon to visual hallucinations presents as 'pareidolias' which make ambiguous forms appear meaningful. To evoke and detect pareidolias in patients, a noise pareidolia test was recently developed, although its task-dependent mechanisms are yet to be revealed. When subjected to this test, we hypothesized that patients exhibiting pareidolias would show altered top-down influence of visual processing allowing us to demonstrate the influence of pareidolic illusionary behaviour in Parkinson's disease patients. To that end, we evaluated eye-movement strategies and fixation-related presaccadic activity on scalp EEG when participants performed the test. Twelve healthy controls and 21 Parkinson's disease patients, evaluated for cognitive, visuo-spatial and executive functions, took a modified computer-based version of the noise pareidolia test in a free-viewing EEG eye-tracking experiment. Eye-tracking metrics (fixation-related durations and counts) documented the eye movement behaviour employed in correct responses (face/noise) and misperceptions (pareidolia/missed) during early and late visual search conditions. Simultaneously, EEG recorded the presaccadic activity in frontal and parietal areas of the brain. Based on the noise pareidolia test scores, we found certain Parkinson's disease patients exhibited pareidolias whereas others did not. ANOVA on eye-tracking data showed that patients dwelled significantly longer to detect faces and pareidolias which affected both global and local search dynamics depending on their visuo-perceptual status. Presaccadic activity in parietal electrodes for the groups was positive for faces and pareidolias, and negative for noise, though these results depended mainly on saccade size. However, patients sensitive to pareidolias showed a significantly higher presaccadic potential on frontal electrodes independent of saccade sizes, suggesting a stronger frontal activation for pareidolic stimuli. We concluded with the following interpretations (i) the noise pareidolia test specifically characterizes visuo-perceptual inadequacies in patients despite their wide range of cognitive scores, (ii) Parkinson's disease patients dwell longer to converge attention to pareidolic stimuli due to abnormal saccade generation proportional to their visuo-perceptual deficit during early search, and during late search, due to time-independent alteration of visual attentional network and (iii) patients with pareidolias show increased frontal activation reflecting the allocation of attention to irrelevant targets that express the pareidolic phenomenon. While the disease per se alters the visuo-perceptual and oculomotor dynamics, pareidolias occur in Parkinson's disease due to an abnormal top-down modulation of visual processing that affects visual attention and guidance to ambiguous stimuli.

Best practices in eye tracking research

This guide describes best practices in using eye tracking technology for research in a variety of disciplines. A basic outline of the anatomy and physiology of the eyes and of eye movements is provided, along with a description of the sorts of research questions eye tracking can address. We then explain how eye tracking technology works and what sorts of data it generates, and provide guidance on how to select and use an eye tracker as well as selecting appropriate eye tracking measures. Challenges to the validity of eye tracking studies are described, along with recommendations for overcoming these challenges. We then outline correct reporting standards for eye tracking studies.

Fixation-related Brain Potentials during Semantic Integration of Object–Scene Information

In vision science, a particularly controversial topic is whether and how quickly the semantic information about objects is available outside foveal vision. Here, we aimed at contributing to this debate by coregistering eye movements and EEG while participants viewed photographs of indoor scenes that contained a semantically consistent or inconsistent target object. Linear deconvolution modeling was used to analyze the ERPs evoked by scene onset as well as the fixation-related potentials (FRPs) elicited by the fixation on the target object (t) and by the preceding fixation (t − 1). Object–scene consistency did not influence the probability of immediate target fixation or the ERP evoked by scene onset, which suggests that object–scene semantics was not accessed immediately. However, during the subsequent scene exploration, inconsistent objects were prioritized over consistent objects in extrafoveal vision (i.e., looked at earlier) and were more effortful to process in foveal vision (i.e., looked at longer). In FRPs, we demonstrate a fixation-related N300/N400 effect, whereby inconsistent objects elicit a larger frontocentral negativity than consistent objects. In line with the behavioral findings, this effect was already seen in FRPs aligned to the pretarget fixation t − 1 and persisted throughout fixation t, indicating that the extraction of object semantics can already begin in extrafoveal vision. Taken together, the results emphasize the usefulness of combined EEG/eye movement recordings for understanding the mechanisms of object–scene integration during natural viewing.

Face Selective Neural Activity: Comparisons Between Fixed and Free Viewing

Event Related Potentials (ERPs) are widely used to study category-selective EEG responses to visual stimuli, such as the face-selective N170 component. Typically, this is done by flashing stimuli at the point of static gaze fixation. While allowing for good experimental control, these paradigms ignore the dynamic role of eye-movements in natural vision. Fixation-related potentials (FRPs), obtained using simultaneous EEG and eye-tracking, overcome this limitation. Various studies have used FRPs to study processes such as lexical processing, target detection and attention allocation. The goal of this study was to carefully compare face-sensitive activity time-locked to an abrupt stimulus onset at fixation, with that time-locked to a self-generated fixation on a stimulus. Twelve participants participated in three experimental conditions: Free-viewing (FRPs), Cued-viewing (FRPs) and Control (ERPs). We used a multiple regression approach to disentangle overlapping activity components. Our results show that the N170 face-effect is evident for the first fixation on a stimulus, whether it follows a self-generated saccade or stimulus appearance at fixation point. The N170 face-effect has similar topography across viewing conditions, but there were major differences within each stimulus category. We ascribe these differences to an overlap of the fixation-related lambda response and the N170. We tested the plausibility of this account using dipole simulations. Finally, the N170 exhibits category-specific adaptation in free viewing. This study establishes the comparability of the free-viewing N170 face-effect with the classic event-related effect, while highlighting the importance of accounting for eye-movement related effects.

Eye Movements and Fixation-Related Potentials in Reading: A Review

The present review is addressed to researchers in the field of reading and psycholinguistics who are both familiar with and new to co-registration research of eye movements (EMs) and fixation related-potentials (FRPs) in reading. At the outset, we consider a conundrum relating to timing discrepancies between EM and event related potential (ERP) effects. We then consider the extent to which the co-registration approach might allow us to overcome this and thereby discriminate between formal theoretical and computational accounts of reading. We then describe three phases of co-registration research before evaluating the existing body of such research in reading. The current, ongoing phase of co-registration research is presented in comprehensive tables which provide a detailed summary of the existing findings. The thorough appraisal of the published studies allows us to engage with issues such as the reliability of FRP components as correlates of cognitive processing in reading and the advantages of analysing both data streams (i.e., EMs and FRPs) simultaneously relative to each alone, as well as the current, and limited, understanding of the relationship between EM and FRP measures. Finally, we consider future directions and in particular the potential of analytical methods involving deconvolution and the potential of measurement of brain oscillatory activity.

Measurement of saccadic eye movements by electrooculography for simultaneous EEG recording

Eye movements are an important index of the neural functions of visual information processing, decision making, visuomotor coordination, sports performance, and so forth. However, the available optical tracking methods are impractical in many situations, such as the wearing of eyeglasses or the presence of ophthalmic disorders, and this can be overcome by accurate recording of eye movements by electrooculography (EOG). In this study we recorded eye movements by EOG simultaneously with high-density electroencephalogram (EEG) recording using a 128-channel EGI electrode net at a 500-Hz sampling rate, including appropriate facial electrodes. The participants made eye movements over a calibration target consisting of a 5×5 grid of stimulus targets. The results showed that the EOG methodology allowed accurate analysis of the amplitude and direction of the fixation locations and saccadic dynamics with a temporal resolution of 500 Hz, under both cued and uncued analysis regimes. Blink responses could be identified separately and were shown to have a more complex source derivation than has previously been recognized. The results also showed that the EOG signals recorded through the EEG net can achieve results as accurate as typical optical eye-tracking devices, and also allow for simultaneous assessment of neural activity during all types of eye movements. Moreover, the EOG method effectively avoids the technical difficulties related to eye-tracker positioning and the synchronization between EEG and eye movements. We showed that simultaneous EOG/EEG recording is a convenient means of measuring eye movements, with an accuracy comparable to that of many specialized eye-tracking systems.

Neural underpinnings of value-guided choice during auction tasks: An eye-fixation related potentials study

Values are attributed to goods during free viewing of objects which entails multi- and trans-saccadic cognitive processes. Using electroencephalographic eye-fixation related potentials, the present study investigated how neural signals related to value-guided choice evolved over time when viewing household and office products during an auction task. Participants completed a Becker-DeGroot-Marschak auction task whereby half of the stimuli were presented in either a free or forced bid protocol to obtain willingness-to-pay. Stimuli were assigned to three value categories of low, medium and high value based on subjective willingness-to-pay. Eye fixations were organised into five 800 ms time-bins spanning the objects total viewing time. Independent component analysis was applied to eye-fixation related potentials. One independent component (IC) was found to represent fixations for high value products with increased activation over the left parietal region of the scalp. An IC with a spatial maximum over a fronto-central region of the scalp coded the intermediate values. Finally, one IC displaying activity that extends over the right frontal scalp region responded to intermediate- and low-value items. Each of these components responded early on during viewing an object and remained active over the entire viewing period, both during free and forced bid trials. Results suggest that the subjective value of goods are encoded using sets of brain activation patterns which are tuned to respond uniquely to either low, medium, or high values. Data indicates that the right frontal region of the brain responds to low and the left frontal region to high values. Values of goods are determined at an early point in the decision making process and carried for the duration of the decision period via trans-saccadic processes.

Social Visual Perception Under the Eye of Bayesian Theories in Autism Spectrum Disorder Using Advanced Modeling of Spatial and Temporal Parameters

Social interaction in individuals with Autism Spectrum Disorder (ASD) is characterized by qualitative impairments that highly impact quality of life. Bayesian theories in ASD frame an understanding of underlying mechanisms suggesting atypicalities in the evaluation of probabilistic links within the perceptual environment of the affected individual. To address these theories, the present study explores the applicability of an innovative Bayesian framework on social visual perception in ASD and demonstrates the use of gaze transitions between different parts of social scenes. We applied advanced analyses with Bayesian Hidden Markov Modeling (BHMM) to track gaze movements while presenting real-life scenes to typically developing (TD) children and adolescents (N = 25) and participants with ASD and Attention-Deficit/Hyperactivity Disorder (ASD+ADHD, N = 15) and ASD without comorbidity (ASD, N = 12). Regions of interest (ROIs) were generated by BHMM based both on spatial and temporal gaze behavior. Social visual perception was compared between groups using transition and fixation variables for social (faces, bodies) and non-social ROIs. Transition variables between faces, namely gaze transitions between faces and likelihood of linking faces, were reduced in the ASD+ADHD compared to TD participants. Fixation count to faces was also reduced in this group. The ASD group showed similar performance to TD in the studied variables. There was no difference between groups for non-social ROIs. Our study provides an innovative, interpretable example of applying Bayesian theories of social visual perception in ASD. BHMM analyses and gaze transitions have the potential to reveal fundamental social perception components in ASD, contributing thus to amelioration of social-skill interventions.

The impact of driver sleepiness on fixation-related brain potentials

The effects of driver sleepiness are often quantified as deteriorated driving performance, increased blink durations and high levels of subjective sleepiness. Driver sleepiness has also been associated with increasing levels of electroencephalogram (EEG) power, especially in the alpha range. The present exploratory study investigated a new measure of driver sleepiness, the EEG fixation-related lambda response. Thirty young male drivers (23.6 ± 1.7 years old) participated in a driving simulator experiment in which they drove on rural and suburban roads in simulated daylight versus darkness during both the daytime (full sleep) and night-time (sleep deprived). The results show lower lambda responses during night driving and with longer time on task, indicating that sleep deprivation and time on task cause a general decrement in cortical responsiveness to incoming visual stimuli. Levels of subjective sleepiness and line crossings were higher under the same conditions. Furthermore, results of a linear mixed-effects model showed that low lambda responses are associated with high subjective sleepiness and more line crossings. We suggest that the fixation-related lambda response can be used to investigate driving impairment induced by sleep deprivation while driving and that, after further refinement, it may be useful as an objective measure of driver sleepiness.

EEG & Eye Tracking User Experiments for Spatial Memory Task on Maps

The aim of this research is to evaluate the use of ET and EEG for studying the cognitive processes of expert and novice map users and to explore these processes by comparing two types of spatial memory experiments through cognitive load measurements. The first experiment consisted of single trials and participants were instructed to study a map stimulus without any time constraints in order to draw a sketch map afterwards. According to the ET metrics (i.e., average fixation duration and the number of fixations per second), no statistically significant differences emerged between experts and novices. A similar result was also obtained with EEG Frontal Alpha Asymmetry calculations. On the contrary, in terms of alpha power across all electrodes, novices exhibited significantly lower alpha power, indicating a higher cognitive load. In the second experiment, a larger number of stimuli were used to study the effect of task difficulty. The same ET metrics used in the first experiment indicated that the difference between these user groups was not statistically significant. The cognitive load was also extracted using EEG event-related spectral power changes at alpha and theta frequency bands. Preliminary data exploration mostly suggested an increase in theta power and a decrease in alpha power.

Influence of reading skill and word length on fixation-related brain activity in school-aged children during natural reading

Word length is one of the main determinants of eye movements during reading and has been shown to influence slow readers more strongly than typical readers. The influence of word length on reading in individuals with different reading skill levels has been shown in separate eye-tracking and electroencephalography studies. However, the influence of reading difficulty on cortical correlates of word length effect during natural reading is unknown. To investigate how reading skill is related to brain activity during natural reading, we performed an exploratory analysis on our data set from a previous study, where slow reading (N = 27) and typically reading (N = 65) 12-to-13.5-year-old children read sentences while co-registered ET-EEG was recorded. We extracted fixation-related potentials (FRPs) from the sentences using the linear deconvolution approach. We examined standard eye-movement variables and deconvoluted FRP estimates: intercept of the response, categorical effect of first fixation versus additional fixation and continuous effect of word length. We replicated the pattern of stronger word length effect in eye movements for slow readers. We found a difference between typical readers and slow readers in the FRP intercept, which contains activity that is common to all fixations, within a fixation time-window of 50-300 ms. For both groups, the word length effect was present in brain activity during additional fixations; however, this effect was not different between groups. This suggests that stronger word length effect in the eye movements of slow readers might be mainly due re-fixations, which are more probable due to the lower efficiency of visual processing.

Optimizing the ICA-based removal of ocular EEG artifacts from free viewing experiments

Combining EEG with eye-tracking is a promising approach to study neural correlates of natural vision, but the resulting recordings are also heavily contaminated by activity of the eye balls, eye lids, and extraocular muscles. While Independent Component Analysis (ICA) is commonly used to suppress these ocular artifacts, its performance under free viewing conditions has not been systematically evaluated and many published reports contain residual artifacts. Here I evaluated and optimized ICA-based correction for two tasks with unconstrained eye movements: visual search in images and sentence reading. In a first step, four parameters of the ICA pipeline were varied orthogonally: the (1) high-pass and (2) low-pass filter applied to the training data, (3) the proportion of training data containing myogenic saccadic spike potentials (SP), and (4) the threshold for eye tracker-based component rejection. In a second step, the eye-tracker was used to objectively quantify the correction quality of each ICA solution, both in terms of undercorrection (residual artifacts) and overcorrection (removal of neurogenic activity). As a benchmark, results were compared to those obtained with an alternative spatial filter, Multiple Source Eye Correction (MSEC). With commonly used settings, Infomax ICA not only left artifacts in the data, but also distorted neurogenic activity during eye movement-free intervals. However, correction results could be strongly improved by training the ICA on optimally filtered data in which SPs were massively overweighted. With optimized procedures, ICA removed virtually all artifacts, including the SP and its associated spectral broadband artifact from both viewing paradigms, with little distortion of neural activity. It also outperformed MSEC in terms of SP correction. Matlab code is provided.

Refixation patterns reveal memory-encoding strategies in free viewing

We investigated visual working memory encoding across saccadic eye movements, focusing our analysis on refixation behavior. Over 10-s periods, participants performed a visual search for three, four, or five targets and remembered their orientations for a subsequent change-detection task. In 50% of the trials, one of the targets had its orientation changed. From the visual search period, we scored three types of refixations and applied measures for quantifying eye-fixation recurrence patterns. Repeated fixations on the same regions as well as repeated fixation patterns increased with memory load. Correct change detection was associated with more refixations on targets and less on distractors, with increased frequency of recurrence, and with longer intervals between refixations. The results are in accordance with the view that patterns of eye movement are an integral part of visual working memory representation.

Distinct modes of top-down cognitive processing in the ventral visual cortex

Top-down cognitive control leads to changes in the sensory processing of the brain. In visual perception such changes can take place in the ventral visual cortex altering the functional asymmetry in forward and backward connections. Here we used fixation-related evoked responses of EEG measurement and dynamic causal modeling to examine hierarchical forward-backward asymmetry, while twenty-six healthy adults performed cognitive tasks that require different types of top-down cognitive control (memorizing or searching visual objects embedded in a natural scene image). The generative model revealed an enhanced asymmetry toward forward connections during memorizing, whereas enhanced backward connections were found during searching. This task-dependent modulation of forward and backward connections suggests two distinct modes of top-down cognitive processing in cortical networks. The alteration in forward-backward asymmetry might underlie the functional role in the cognitive control of visual information processing.

Scene Buildup From Latent Memory Representations Across Eye Movements

An unresolved problem in eye movement research is how a representation is constructed on-line from several consecutive fixations of a scene. Such a scene representation is generally understood to be sparse; yet, for meeting behavioral goals a certain level of detail is needed. We propose that this is achieved through the buildup of latent representations acquired at fixation. Latent representations are retained in an activity-silent manner, require minimal energy expenditure for their maintenance, and thus allow a larger storage capacity than traditional, activation based, visual working memory. The latent representations accumulate and interact in working memory to form to the scene representation. The result is rich in detail while sparse in the sense that it is restricted to the task-relevant aspects of the scene sampled through fixations. Relevant information can quickly and flexibly be retrieved by dynamical attentional prioritization. Latent representations are observable as transient functional connectivity patterns, which emerge due to short-term changes in synaptic weights. We discuss how observing latent representations could benefit from recent methodological developments in EEG-eye movement co-registration.

Use of Electroencephalography (EEG) for the Analysis of Emotional Perception and Fear to Nightscapes

As the necessity for safety and aesthetic of nightscape have arisen, the importance of nightscapes (i.e., nighttime landscape) planning has garnered the attention of mainstream consciousness. Therefore, this study was to suggest the guideline for nightscape planning using electroencephalography (EEG) technology and survey for recognizing the characteristics of a nightscape. Furthermore, we verified the electroencephalography (EEG) method as a tool for landscape evaluation. Therefore, this study analyzed the change of relative alpha wave and relative beta wave and perceived fear of participants depending on twelve nightscape settings (four types of settings: Built nightscape images group with an adult; Built nightscape images groups without an adult; Nature-dominant nightscape images with an adult; and Nature-dominant nightscape images without an adult). Our findings indicate that the most fearful nightscape setting was recorded in Built nightscape images groups without an adult figure in perceived fear result depending on four types of nightscape settings. In Nature-dominant nightscape images, on the other hand, the nightscape setting with an adult figure was more fearful than the setting without an adult. The interaction effect between landscape type (built and nature-dominant) and adult presence towards perceived fear was verified and it showed that the image with adult affects landscape type. For electroencephalography (EEG) results, several brain activities in the relative alpha and beta wave showed significant differences depending on nightscape settings, which situates electroencephalography (EEG) as an invaluable tool for evaluating landscapes. Based on our physiological electroencephalography (EEG) experiment, we provide a new analytic view of the nightscape. The approach we utilized enables a deeper understanding of emotional perception and fear among human subjects by identifying the physical environment which impacts how they experience nightscapes.

Improving free-viewing fixation-related EEG potentials with continuous-time regression

BACKGROUND

In the analysis of combined ET-EEG data, there are several issues with estimating FRPs by averaging. Neural responses associated with fixations will likely overlap with one another in the EEG recording and neural responses change as a function of eye movement characteristics. Especially in tasks that do not constrain eye movements in any way, these issues can become confounds.

NEW METHOD

Here, we propose the use of regression based estimates as an alternative to averaging. Multiple regression can disentangle different influences on the EEG and correct for overlap. It thereby accounts for potential confounds in a way that averaging cannot. Specifically, we test the applicability of the rERP framework, as proposed by Smith and Kutas (2015b), (2017), or Sassenhagen (2018) to combined eye tracking and EEG data from a visual search and a scene memorization task.

RESULTS

Results show that the method successfully estimates eye movement related confounds in real experimental data, so that these potential confounds can be accounted for when estimating experimental effects.

COMPARISON WITH EXISTING METHODS

The rERP method successfully corrects for overlapping neural responses in instances where averaging does not. As a consequence, baselining can be applied without risking distortions. By estimating a known experimental effect, we show that rERPs provide an estimate with less variance and more accuracy than averaged FRPs. The method therefore provides a practically feasible and favorable alternative to averaging.

CONCLUSIONS

We conclude that regression based ERPs provide novel opportunities for estimating fixation related EEG in free-viewing experiments.

Tracking Economic Value of Products in Natural Settings: A Wireless EEG Study

Economic decision making refers to the process of individuals translating their preference into subjective value (SV). Little is known about the dynamics of the neural processes that underpin this form of value-based decision making and no studies have investigated these processes outside of controlled laboratory settings. The current study investigated the spatio-temporal dynamics that accompany economic valuation of products using mobile electroencephalography (EEG) and eye tracking techniques. Participants viewed and rated images of household products in a gallery setting while EEG and eye tracking data were collected wirelessly. A Becker-DeGroot-Marschak (BDM) auction task was subsequently used to quantify the individual's willingness to pay (WTP) for each product. WTP was used to classify products into low, low medium, high medium and high economic value conditions. Eye movement related potentials (EMRP) were examined, and independent component analysis (ICA) was used to separate sources of activity from grand averaged EEG data. Four independent components (ICs) of EMRPs were modulated by WTP (i.e., SV) in the latency range of 150-250 ms. Of the four value-sensitive ICs, one IC displayed enhanced amplitude for all value conditions excluding low value, and another IC presented enhanced amplitude for low value products only. The remaining two value-sensitive ICs resolved inter-mediate levels of SV. Our study quantified, for the first time, the neural processes involved in economic value based decisions in a natural setting. Results suggest that multiple spatio-temporal brain activation patterns mediate the attention and aversion of products which could reflect an early valuation system. The EMRP parietal P200 component could reflect an attention allocation mechanism that separates the lowest-value products (IC7) from products of all other value (IC4), suggesting that low-value items are categorized early on as being aversive. While none of the ICs showed linear amplitude changes that parallel SV's of products, results suggest that a combination of multiple components may sub-serve a fine-grained resolution of the SV of products.

Refixation control in free viewing: a specialized mechanism divulged by eye-movement-related brain activity

In free viewing, the eyes return to previously visited locations rather frequently, even though the attentional and memory-related processes controlling eye-movement show a strong antirefixation bias. To overcome this bias, a special refixation triggering mechanism may have to be recruited. We probed the neural evidence for such a mechanism by combining eye tracking with EEG recording. A distinctive signal associated with refixation planning was observed in the EEG during the presaccadic interval: the presaccadic potential was reduced in amplitude before a refixation compared with normal fixations. The result offers direct evidence for a special refixation mechanism that operates in the saccade planning stage of eye movement control. Once the eyes have landed on the revisited location, acquisition of visual information proceeds indistinguishably from ordinary fixations. NEW & NOTEWORTHY A substantial proportion of eye fixations in human natural viewing behavior are revisits of recently visited locations, i.e., refixations. Our recently developed methods enabled us to study refixations in a free viewing visual search task, using combined eye movement and EEG recording. We identified in the EEG a distinctive refixation-related signal, signifying a control mechanism specific to refixations as opposed to ordinary eye fixations.

Brain Responses to Emotional Faces in Natural Settings: A Wireless Mobile EEG Recording Study

The detection of a human face in a visual field and correct reading of emotional expression of faces are important elements in everyday social interactions, decision making and emotional responses. Although brain correlates of face processing have been established in previous fMRI and electroencephalography (EEG)/MEG studies, little is known about how the brain representation of faces and emotional expressions of faces in freely moving humans. The present study aimed to detect brain electrical potentials that occur during the viewing of human faces in natural settings. 64-channel wireless EEG and eye-tracking data were recorded in 19 participants while they moved in a mock art gallery and stopped at times to evaluate pictures hung on the walls. Positive, negative and neutral valence pictures of objects and human faces were displayed. The time instants in which pictures first occurred in the visual field were identified in eye-tracking data and used to reconstruct the triggers in continuous EEG data after synchronizing the time axes of the EEG and eye-tracking device. EEG data showed a clear face-related event-related potential (ERP) in the latency interval ranging from 165 to 210 ms (N170); this component was not seen whilst participants were viewing non-living objects. The face ERP component was stronger during viewing disgusted compared to neutral faces. Source dipole analysis revealed an equivalent current dipole in the right fusiform gyrus (BA37) accounting for N170 potential. Our study demonstrates for the first time the possibility of recording brain responses to human faces and emotional expressions in natural settings. This finding opens new possibilities for clinical, developmental, social, forensic, or marketing research in which information about face processing is of importance.

The Effect of Art Expertise on Eye Fixation-Related Potentials During Aesthetic Judgment Task in Focal and Ambient Modes

This study aimed to determine the effect of expertise on the eye fixation-related potentials (EFRPs) during the aesthetic evaluation of images, independently in focal and ambient modes of visual processing. Focal and ambient modes were identified by averaging EFRP waveforms about the beginning of long eye fixations followed by short saccades and short fixations followed by long saccades, respectively. Thirty experts with formal training in visual arts and thirty-two non-experts freely viewed 150 figurative paintings presented for 20 s, each. After viewing the painting, the participant answered the question: “Is this painting beautiful?” Differences were found between the group of experts and non-experts due to the amplitude of EFRPs but only in focal mode, which is related to top-down, focused attention on the objects. Long fixations of experts had a higher amplitude of the parietal P2 recorded from right site than non-experts. In the group of experts, the frontal P2 was higher for long fixations on not beautiful paintings in comparison to long fixation on beautiful paintings. Moreover, in focal mode, there were higher occipital lambda response and N1-P2 complex for not beautiful than beautiful paintings. These results are discussed in the light of the results of studies on the effect of visual art expertise on event-related potentials (ERPs), ERP studies during aesthetic judgment task, and the knowledge of different modes of visual processing and EFRPs.

Target probability modulates fixation-related potentials in visual search

This study investigated the influence of target probability on the neural response to target detection in free viewing visual search. Participants were asked to indicate the number of targets (one or two) among distractors in a visual search task while EEG and eye movements were co-registered. Target probability was manipulated by varying the set size of the displays between 10, 22, and 30 items. Fixation-related potentials time-locked to first target fixations revealed a pronounced P300 at the centro-parietal cortex with larger amplitudes for set sizes 22 and 30 than for set size 10. With increasing set size, more distractor fixations preceded the detection of the target, resulting in a decreased target probability and, consequently, a larger P300. For distractors, no increase of P300 amplitude with set size was observed. The findings suggest that set size specifically affects target but not distractor processing in overt serial visual search.

Presaccadic EEG activity predicts visual saliency in free-viewing contour integration

While viewing a scene, the eyes are attracted to salient stimuli. We set out to identify the brain signals controlling this process. In a contour integration task, in which participants searched for a collinear contour in a field of randomly oriented Gabor elements, a previously established model was applied to calculate a visual saliency value for each fixation location. We studied brain activity related to the modeled saliency values, using coregistered eye tracking and EEG. To disentangle EEG signals reflecting salience in free viewing from overlapping EEG responses to sequential eye movements, we adopted generalized additive mixed modeling (GAMM) to single epochs of saccade-related EEG. We found that, when saliency at the next fixation location was high, amplitude of the presaccadic EEG activity was low. Since presaccadic activity reflects covert attention to the saccade target, our results indicate that larger attentional effort is needed for selecting less salient saccade targets than more salient ones. This effect was prominent in contour-present conditions (half of the trials), but ambiguous in the contour-absent condition. Presaccadic EEG activity may thus be indicative of bottom-up factors in saccade guidance. The results underscore the utility of GAMM for EEG-eye movement coregistration research.