Visual encoding and fixation target selection in free viewing: presaccadic brain potentials

Mammals Achieve Common Neural Coverage of Visual Scenes Using Distinct Sampling Behaviors

Most vertebrates use head and eye movements to quickly change gaze orientation and sample different portions of the environment with periods of stable fixation. Visual information must be integrated across fixations to construct a complete perspective of the visual environment. In concert with this sampling strategy, neurons adapt to unchanging input to conserve energy and ensure that only novel information from each fixation is processed. We demonstrate how adaptation recovery times and saccade properties interact and thus shape spatiotemporal tradeoffs observed in the motor and visual systems of mice, cats, marmosets, macaques, and humans. These tradeoffs predict that in order to achieve similar visual coverage over time, animals with smaller receptive field sizes require faster saccade rates. Indeed, we find comparable sampling of the visual environment by neuronal populations across mammals when integrating measurements of saccadic behavior with receptive field sizes and V1 neuronal density. We propose that these mammals share a common statistically driven strategy of maintaining coverage of their visual environment over time calibrated to their respective visual system characteristics.

Refixation behavior in naturalistic viewing: Methods, mechanisms, and neural correlates

When freely viewing a scene, the eyes often return to previously visited locations. By tracking eye movements and coregistering eye movements and EEG, such refixations are shown to have multiple roles: repairing insufficient encoding from precursor fixations, supporting ongoing viewing by resampling relevant locations prioritized by precursor fixations, and aiding the construction of memory representations. All these functions of refixation behavior are understood to be underpinned by three oculomotor and cognitive systems and their associated brain structures. First, immediate saccade planning prior to refixations involves attentional selection of candidate locations to revisit. This process is likely supported by the dorsal attentional network. Second, visual working memory, involved in maintaining task-related information, is likely supported by the visual cortex. Third, higher-order relevance of scene locations, which depends on general knowledge and understanding of scene meaning, is likely supported by the hippocampal memory system. Working together, these structures bring about viewing behavior that balances exploring previously unvisited areas of a scene with exploiting visited areas through refixations.

Neural correlates of attentional orienting with neutral and fearful gaze cues in 12-month-olds

Infants use information on gaze direction and facial expressions for social referencing when encountering various objects in their environment. However, it remains unclear how these social cues influence attentional orienting in infants. Using event-related potentials (ERPs), we investigated the neural correlates of attentional orienting cued by an averted gaze with neutral and fearful expressions in 12-month-olds. We focused on the ERPs in response to a face (N290, P400, and Nc) as well as a saccade toward the target (the presaccadic spike potential: SP) and found that the amplitudes of the face-sensitive ERPs (N290 and P400) were larger for directed than averted gaze direction irrespective of facial expression. Furthermore, the amplitude of the SP involved in overt orienting was larger for fearful expressions than for neutral expressions, irrespective of gaze congruency. These results suggest that information on gaze direction and facial expression, specifically neutral and fearful expressions, may be processed independently, and that fearful expressions dominantly influence the neural correlates of attentional orienting in infants around 12 months of age.

The Visual Attention and Psychological Responses from Older Customers to Wellness Service Pictures of Hotels

Understanding the visual attention and psychological responses of consumers to marketing pictures allows hotel managers to design more attractive advertisements. Therefore, the purposes of the present study were to use an eye-tracking analysis to explore whether there were different effects from tourist hotels’ wellness service pictures based on whether they had natural or built clues. The psychological responses with regard to perceived well-being and willingness to pay were also examined. Eighty-five older consumers were recruited. Their eye movement performance while observing marketing pictures with different visual clues and their subsequent psychological responses were measured. It was found that wellness service pictures with natural clues captured more visual attention and induced higher willingness-to-pay perceptions than those with built clues in these older consumers. The present results suggest that marketing pictures with natural clues may create positive visual responses in older customers and further enhance their purchase intention.

Visual responses of patients with generalized anxiety disorder who cycling in the virtual sportscapes with different tree cover densities

The tree density of virtual sportscape is the main factor that determines the benefits that generalized anxiety disorder (GAD) patients can obtain when they exercise with virtual environment. By using pupil size, fixation count and time as metrics, this research aimed to clarify the relationship between tree cover density and stress in the virtual environment. Ninety GAD patients were randomly grouped into the 36-60% tree density (high tree density, HTDS), 20-35% tree density (medium tree density, MTDS), or control groups (n = 30). Researchers used eye-tracking technology to analyze fixation time, fixation count and changes in pupil size to evaluate the stress changes of participants after 20 min of aerobic exercise in a virtual environment. The results showed that pupil size expanded in GAD patients after exercising in the virtual environment. Furthermore, GAD patient cycling in the MTDS group can show smaller pupil size than those in HTDS. Those results suggest that GAD patient cycling 20 min in the MTDS group can perceived lower stress. The results of eye tracking analysis showed that GAD patients spend more time and counts observing tree elements in HTDS and MTDS sportscapes. Specifically, they spent more 48% and 27% time on tree and green plants in the HTDS condition and MTDS condition, respectively, than in non-natural sportsscapes. Although 36-60% tree density of virtual natural sportscape can get more visual attention from GAD patients, 20-35% tree density of virtual natural sportscape is more capable of reducing their stress.

Prestimulus Low-Alpha Frontal Networks Are Associated with Pareidolias in Parkinson's Disease

Background: Pareidolias are visual phenomena wherein ambiguous, abstract forms or shapes appear meaningful due to incorrect perception. In Parkinson's disease (PD), patients susceptible to visual hallucinations experience visuo-perceptual deficits in the form of pareidolias. Although pareidolias necessitate top-down modulation of visual processing, the cortical dynamics of internally generated perceptual priors on these visual misperceptions is unknown. Objectives: To study prestimulus-related electroencephalography (EEG) spectral and network abnormalities in PD patients experiencing pareidolias. Methods: Twenty-one PD in-patients and 10 age-matched controls were evaluated. Neuropsychological assessments included tests for cognition, attention, and executive functions. Pareidolias were quantified by using the "noise pareidolia test" with simultaneous EEG recording. The PD patients were subdivided into two groups-those with high pareidolia counts (n = 10) and those without (n = 11). The EEG was analyzed 1000 msec before stimulus presentation in the spectral domain (theta, low-alpha, and high-alpha frequencies) with corresponding graph networks to evaluate network properties. Statistical analysis included analysis of variance and multiple regression to evaluate the differences. Results: The PD patients with high pareidolia counts were older with lower scores on neuropsychological tests. Their prestimulus EEG low-alpha band showed a tendency toward higher frontal activity (p = 0.07). Graph networks showed increased normalized clustering coefficient (p = 0.05) and lower frontal degree centrality (p = 0.005). These network indices correlated positively to patients' pareidolia scores. Discussion: We suggest that pareidolias in PD are a consequence of an abnormal top-down modulation of visual processing; they are defined by their frontal low-alpha spectral and network alterations in the prestimulus phase due to a dissonance between patients' internally generated mental processing with external stimuli. Impact statement Pareidolias in Parkinson's disease (PD) are considered to be promising early markers of visual hallucinations and an indicator of PD prognosis. In certain susceptible PD patients, pareidolias can be evoked and studied. Here, via electroencephalography, we aimed at understanding this visual phenomenon by studying how neural information is processed before stimulus presentation in such patients. Using spectral and graph network measures, we revealed how top-down modulated internally generated processes affect visual perception in patients with pareidolias. Our findings highlight how prestimulus network alterations in the frontal cortex shape poststimulus pareidolic manifestations in PD.

Regression-based analysis of combined EEG and eye-tracking data: Theory and applications

Fixation-related potentials (FRPs), neural responses aligned to the end of saccades, are a promising tool for studying the dynamics of attention and cognition under natural viewing conditions. In the past, four methodological problems have complicated the analysis of such combined eye-tracking/electroencephalogram experiments: (1) the synchronization of data streams, (2) the removal of ocular artifacts, (3) the condition-specific temporal overlap between the brain responses evoked by consecutive fixations, and (4) the fact that numerous low-level stimulus and saccade properties also influence the postsaccadic neural responses. Although effective solutions exist for the first two problems, the latter two are only beginning to be addressed. In the current paper, we present and review a unified regression-based framework for FRP analysis that allows us to deconvolve overlapping potentials while also controlling for both linear and nonlinear confounds on the FRP waveform. An open software implementation is provided for all procedures. We then demonstrate the advantages of this proposed (non)linear deconvolution modeling approach for data from three commonly studied paradigms: face perception, scene viewing, and reading. First, for a traditional event-related potential (ERP) face recognition experiment, we show how this technique can separate stimulus ERPs from overlapping muscle and brain potentials produced by small (micro)saccades on the face. Second, in natural scene viewing, we model and isolate multiple nonlinear effects of saccade parameters on the FRP. Finally, for a natural sentence reading experiment using the boundary paradigm, we show how it is possible to study the neural correlates of parafoveal preview after removing spurious overlap effects caused by the associated difference in average fixation time. Our results suggest a principal way of measuring reliable eye movement-related brain activity during natural vision.

Ocular fixations and presaccadic potentials to explain pareidolias in Parkinson's disease

In Parkinson's disease, a precursor phenomenon to visual hallucinations presents as 'pareidolias' which make ambiguous forms appear meaningful. To evoke and detect pareidolias in patients, a noise pareidolia test was recently developed, although its task-dependent mechanisms are yet to be revealed. When subjected to this test, we hypothesized that patients exhibiting pareidolias would show altered top-down influence of visual processing allowing us to demonstrate the influence of pareidolic illusionary behaviour in Parkinson's disease patients. To that end, we evaluated eye-movement strategies and fixation-related presaccadic activity on scalp EEG when participants performed the test. Twelve healthy controls and 21 Parkinson's disease patients, evaluated for cognitive, visuo-spatial and executive functions, took a modified computer-based version of the noise pareidolia test in a free-viewing EEG eye-tracking experiment. Eye-tracking metrics (fixation-related durations and counts) documented the eye movement behaviour employed in correct responses (face/noise) and misperceptions (pareidolia/missed) during early and late visual search conditions. Simultaneously, EEG recorded the presaccadic activity in frontal and parietal areas of the brain. Based on the noise pareidolia test scores, we found certain Parkinson's disease patients exhibited pareidolias whereas others did not. ANOVA on eye-tracking data showed that patients dwelled significantly longer to detect faces and pareidolias which affected both global and local search dynamics depending on their visuo-perceptual status. Presaccadic activity in parietal electrodes for the groups was positive for faces and pareidolias, and negative for noise, though these results depended mainly on saccade size. However, patients sensitive to pareidolias showed a significantly higher presaccadic potential on frontal electrodes independent of saccade sizes, suggesting a stronger frontal activation for pareidolic stimuli. We concluded with the following interpretations (i) the noise pareidolia test specifically characterizes visuo-perceptual inadequacies in patients despite their wide range of cognitive scores, (ii) Parkinson's disease patients dwell longer to converge attention to pareidolic stimuli due to abnormal saccade generation proportional to their visuo-perceptual deficit during early search, and during late search, due to time-independent alteration of visual attentional network and (iii) patients with pareidolias show increased frontal activation reflecting the allocation of attention to irrelevant targets that express the pareidolic phenomenon. While the disease per se alters the visuo-perceptual and oculomotor dynamics, pareidolias occur in Parkinson's disease due to an abnormal top-down modulation of visual processing that affects visual attention and guidance to ambiguous stimuli.

Best practices in eye tracking research

This guide describes best practices in using eye tracking technology for research in a variety of disciplines. A basic outline of the anatomy and physiology of the eyes and of eye movements is provided, along with a description of the sorts of research questions eye tracking can address. We then explain how eye tracking technology works and what sorts of data it generates, and provide guidance on how to select and use an eye tracker as well as selecting appropriate eye tracking measures. Challenges to the validity of eye tracking studies are described, along with recommendations for overcoming these challenges. We then outline correct reporting standards for eye tracking studies.

Face Selective Neural Activity: Comparisons Between Fixed and Free Viewing

Event Related Potentials (ERPs) are widely used to study category-selective EEG responses to visual stimuli, such as the face-selective N170 component. Typically, this is done by flashing stimuli at the point of static gaze fixation. While allowing for good experimental control, these paradigms ignore the dynamic role of eye-movements in natural vision. Fixation-related potentials (FRPs), obtained using simultaneous EEG and eye-tracking, overcome this limitation. Various studies have used FRPs to study processes such as lexical processing, target detection and attention allocation. The goal of this study was to carefully compare face-sensitive activity time-locked to an abrupt stimulus onset at fixation, with that time-locked to a self-generated fixation on a stimulus. Twelve participants participated in three experimental conditions: Free-viewing (FRPs), Cued-viewing (FRPs) and Control (ERPs). We used a multiple regression approach to disentangle overlapping activity components. Our results show that the N170 face-effect is evident for the first fixation on a stimulus, whether it follows a self-generated saccade or stimulus appearance at fixation point. The N170 face-effect has similar topography across viewing conditions, but there were major differences within each stimulus category. We ascribe these differences to an overlap of the fixation-related lambda response and the N170. We tested the plausibility of this account using dipole simulations. Finally, the N170 exhibits category-specific adaptation in free viewing. This study establishes the comparability of the free-viewing N170 face-effect with the classic event-related effect, while highlighting the importance of accounting for eye-movement related effects.

Co-registration of eye movements and neuroimaging for studying contextual predictions in natural reading

Sixteen years ago, Sereno and Rayner (2003. Measuring word recognition in reading: eye movements and event-related potentials. Trends in Cognitive Sciences, 7(11), 489-493) illustrated how "by means of review and comparison" eye movement (EM) and event-related potential (ERP) studies may advance our understanding of visual word recognition. Attempts to simultaneously record EMs and ERPs soon followed. Recently, this co-registration approach has also been transferred to fMRI and oscillatory EEG. With experimental settings close to natural reading, co-registration enables us to directly integrate insights from EM and neuroimaging studies. This should extend current experimental paradigms by moving the field towards studying sentence-level processing including effects of context and parafoveal preview. This article will introduce the basic principles and applications of co-registration and selectively review how this approach may shed light on one of the most controversially discussed issues in reading research, contextual predictions in online language processing.

Eye fixation-related fronto-parietal neural network correlates of memory retrieval

Eye movements are considered to be informative with regard to the underlying cognitive processes of human beings. Previous studies have reported that eye movements are associated with which scientific concepts are retrieved correctly. Moreover, other studies have also suggested that eye movements involve the cooperative activity of the human brain's fronto-parietal circuits. Less research has been conducted to investigate whether fronto-parietal EEG oscillations are associated with the retrieval processing of scientific concepts. Our findings in this study demonstrated that the fronto-parietal network is indeed crucial for successful memory retrieval. In short, significantly lower theta augmentation in the frontal midline and lower alpha suppression in the right parietal region were observed at the 5th eye fixation for physics concepts that were correctly retrieved than for those that were incorrectly retrieved. Moreover, the visual cortex in the occipital lobe exhibits a significantly greater theta augmentation followed by an alpha suppression following each eye fixation, while a right fronto-parietal asymmetry was also found for the successful retrieval of presentations of physics concepts. In particular, the study results showed that eye fixation-related frontal midline theta power and right parietal alpha power at the 5th eye fixation have the greatest predictive power regarding the correctness of the retrieval of physics concepts.

Refixation control in free viewing: a specialized mechanism divulged by eye-movement-related brain activity

In free viewing, the eyes return to previously visited locations rather frequently, even though the attentional and memory-related processes controlling eye-movement show a strong antirefixation bias. To overcome this bias, a special refixation triggering mechanism may have to be recruited. We probed the neural evidence for such a mechanism by combining eye tracking with EEG recording. A distinctive signal associated with refixation planning was observed in the EEG during the presaccadic interval: the presaccadic potential was reduced in amplitude before a refixation compared with normal fixations. The result offers direct evidence for a special refixation mechanism that operates in the saccade planning stage of eye movement control. Once the eyes have landed on the revisited location, acquisition of visual information proceeds indistinguishably from ordinary fixations. NEW & NOTEWORTHY A substantial proportion of eye fixations in human natural viewing behavior are revisits of recently visited locations, i.e., refixations. Our recently developed methods enabled us to study refixations in a free viewing visual search task, using combined eye movement and EEG recording. We identified in the EEG a distinctive refixation-related signal, signifying a control mechanism specific to refixations as opposed to ordinary eye fixations.

Target probability modulates fixation-related potentials in visual search

This study investigated the influence of target probability on the neural response to target detection in free viewing visual search. Participants were asked to indicate the number of targets (one or two) among distractors in a visual search task while EEG and eye movements were co-registered. Target probability was manipulated by varying the set size of the displays between 10, 22, and 30 items. Fixation-related potentials time-locked to first target fixations revealed a pronounced P300 at the centro-parietal cortex with larger amplitudes for set sizes 22 and 30 than for set size 10. With increasing set size, more distractor fixations preceded the detection of the target, resulting in a decreased target probability and, consequently, a larger P300. For distractors, no increase of P300 amplitude with set size was observed. The findings suggest that set size specifically affects target but not distractor processing in overt serial visual search.

Presaccadic EEG activity predicts visual saliency in free-viewing contour integration

While viewing a scene, the eyes are attracted to salient stimuli. We set out to identify the brain signals controlling this process. In a contour integration task, in which participants searched for a collinear contour in a field of randomly oriented Gabor elements, a previously established model was applied to calculate a visual saliency value for each fixation location. We studied brain activity related to the modeled saliency values, using coregistered eye tracking and EEG. To disentangle EEG signals reflecting salience in free viewing from overlapping EEG responses to sequential eye movements, we adopted generalized additive mixed modeling (GAMM) to single epochs of saccade-related EEG. We found that, when saliency at the next fixation location was high, amplitude of the presaccadic EEG activity was low. Since presaccadic activity reflects covert attention to the saccade target, our results indicate that larger attentional effort is needed for selecting less salient saccade targets than more salient ones. This effect was prominent in contour-present conditions (half of the trials), but ambiguous in the contour-absent condition. Presaccadic EEG activity may thus be indicative of bottom-up factors in saccade guidance. The results underscore the utility of GAMM for EEG-eye movement coregistration research.

Large-Scale Traveling Waves in EEG Activity Following Eye Movement

In spontaneous, stimulus-evoked, and eye-movement evoked EEG, the oscillatory signal shows large scale, dynamically organized patterns of phase. We investigated eye-movement evoked patterns in free-viewing conditions. Participants viewed photographs of natural scenes in anticipation of a memory test. From 200 ms intervals following saccades, we estimated the EEG phase gradient over the entire scalp, and the wave activity, i.e. the goodness of fit of a wave model involving a phase gradient assumed to be smooth over the scalp. In frequencies centered at 6.5 Hz, large-scale phase organization occurred, peaking around 70 ms after fixation onset and taking the form of a traveling wave. According to the wave gradient, most of the times the wave spreads from the posterior-inferior to anterior-superior direction. In these directions, the gradients depended on the size and direction of the saccade. Wave propagation velocity decreased in the course of the fixation, particularly in the interval from 50 to 150 ms after fixation onset. This interval corresponds to the fixation-related lambda activity, which reflects early perceptual processes following fixation onset. We conclude that lambda activity has a prominent traveling wave component. This component consists of a short-term whole-head phase pattern of specific direction and velocity, which may reflect feedforward propagation of visual information at fixation.

Isolating Discriminant Neural Activity in the Presence of Eye Movements and Concurrent Task Demands

A growing number of studies use the combination of eye-tracking and electroencephalographic (EEG) measures to explore the neural processes that underlie visual perception. In these studies, fixation-related potentials (FRPs) are commonly used to quantify early and late stages of visual processing that follow the onset of each fixation. However, FRPs reflect a mixture of bottom-up (sensory-driven) and top-down (goal-directed) processes, in addition to eye movement artifacts and unrelated neural activity. At present there is little consensus on how to separate this evoked response into its constituent elements. In this study we sought to isolate the neural sources of target detection in the presence of eye movements and over a range of concurrent task demands. Here, participants were asked to identify visual targets (Ts) amongst a grid of distractor stimuli (Ls), while simultaneously performing an auditory N-back task. To identify the discriminant activity, we used independent components analysis (ICA) for the separation of EEG into neural and non-neural sources. We then further separated the neural sources, using a modified measure-projection approach, into six regions of interest (ROIs): occipital, fusiform, temporal, parietal, cingulate, and frontal cortices. Using activity from these ROIs, we identified target from non-target fixations in all participants at a level similar to other state-of-the-art classification techniques. Importantly, we isolated the time course and spectral features of this discriminant activity in each ROI. In addition, we were able to quantify the effect of cognitive load on both fixation-locked potential and classification performance across regions. Together, our results show the utility of a measure-projection approach for separating task-relevant neural activity into meaningful ROIs within more complex contexts that include eye movements.

EEG Negativity in Fixations Used for Gaze-Based Control: Toward Converting Intentions into Actions with an Eye-Brain-Computer Interface

We usually look at an object when we are going to manipulate it. Thus, eye tracking can be used to communicate intended actions. An effective human-machine interface, however, should be able to differentiate intentional and spontaneous eye movements. We report an electroencephalogram (EEG) marker that differentiates gaze fixations used for control from spontaneous fixations involved in visual exploration. Eight healthy participants played a game with their eye movements only. Their gaze-synchronized EEG data (fixation-related potentials, FRPs) were collected during game's control-on and control-off conditions. A slow negative wave with a maximum in the parietooccipital region was present in each participant's averaged FRPs in the control-on conditions and was absent or had much lower amplitude in the control-off condition. This wave was similar but not identical to stimulus-preceding negativity, a slow negative wave that can be observed during feedback expectation. Classification of intentional vs. spontaneous fixations was based on amplitude features from 13 EEG channels using 300 ms length segments free from electrooculogram contamination (200-500 ms relative to the fixation onset). For the first fixations in the fixation triplets required to make moves in the game, classified against control-off data, a committee of greedy classifiers provided 0.90 ± 0.07 specificity and 0.38 ± 0.14 sensitivity. Similar (slightly lower) results were obtained for the shrinkage Linear Discriminate Analysis (LDA) classifier. The second and third fixations in the triplets were classified at lower rate. We expect that, with improved feature sets and classifiers, a hybrid dwell-based Eye-Brain-Computer Interface (EBCI) can be built using the FRP difference between the intended and spontaneous fixations. If this direction of BCI development will be successful, such a multimodal interface may improve the fluency of interaction and can possibly become the basis for a new input device for paralyzed and healthy users, the EBCI "Wish Mouse."