Co-registration of eye movements and event-related potentials in connected-text paragraph reading

Naturalistic reading of multi-page texts elicits spatially extended modulation of oscillatory activity in the right hemisphere

The study of the cortical basis of reading has greatly benefited from the use of naturalistic paradigms that permit eye movements. However, due to the short stimulus lengths used in most naturalistic reading studies, it remains unclear how reading of texts comprising more than isolated sentences modulates cortical processing. To address this question, we used magnetoencephalography to study the spatiospectral distribution of oscillatory activity during naturalistic reading of multi-page texts. In contrast to previous results, we found abundant activity in the right hemisphere in several frequency bands, whereas reading-related modulation of neural activity in the left hemisphere was quite limited. Our results show that the role of the right hemisphere may be importantly emphasized as the reading process extends beyond single sentences.

Combining EEG and eye-tracking in virtual reality: Obtaining fixation-onset event-related potentials and event-related spectral perturbations

Extensive research conducted in controlled laboratory settings has prompted an inquiry into how results can be generalized to real-world situations influenced by the subjects' actions. Virtual reality lends itself ideally to investigating complex situations but requires accurate classification of eye movements, especially when combining it with time-sensitive data such as EEG. We recorded eye-tracking data in virtual reality and classified it into gazes and saccades using a velocity-based classification algorithm, and we cut the continuous data into smaller segments to deal with varying noise levels, as introduced in the REMoDNav algorithm. Furthermore, we corrected for participants' translational movement in virtual reality. Various measures, including visual inspection, event durations, and the velocity and dispersion distributions before and after gaze onset, indicate that we can accurately classify the continuous, free-exploration data. Combining the classified eye-tracking with the EEG data, we generated fixation-onset event-related potentials (ERPs) and event-related spectral perturbations (ERSPs), providing further evidence for the quality of the eye-movement classification and timing of the onset of events. Finally, investigating the correlation between single trials and the average ERP and ERSP identified that fixation-onset ERSPs are less time sensitive, require fewer repetitions of the same behavior, and are potentially better suited to study EEG signatures in naturalistic settings. We modified, designed, and tested an algorithm that allows the combination of EEG and eye-tracking data recorded in virtual reality.

The extrafoveal preview paradigm as a measure of predictive, active sampling in visual perception

A key feature of visual processing in humans is the use of saccadic eye movements to look around the environment. Saccades are typically used to bring relevant information, which is glimpsed with extrafoveal vision, into the high-resolution fovea for further processing. With the exception of some unusual circumstances, such as the first fixation when walking into a room, our saccades are mainly guided based on this extrafoveal preview. In contrast, the majority of experimental studies in vision science have investigated "passive" behavioral and neural responses to suddenly appearing and often temporally or spatially unpredictable stimuli. As reviewed here, a growing number of studies have investigated visual processing of objects under more natural viewing conditions in which observers move their eyes to a stationary stimulus, visible previously in extrafoveal vision, during each trial. These studies demonstrate that the extrafoveal preview has a profound influence on visual processing of objects, both for behavior and neural activity. Starting from the preview effect in reading research we follow subsequent developments in vision research more generally and finally argue that taking such evidence seriously leads to a reconceptualization of the nature of human visual perception that incorporates the strong influence of prediction and action on sensory processing. We review theoretical perspectives on visual perception under naturalistic viewing conditions, including theories of active vision, active sensing, and sampling. Although the extrafoveal preview paradigm has already provided useful information about the timing of, and potential mechanisms for, the close interaction of the oculomotor and visual systems while reading and in natural scenes, the findings thus far also raise many new questions for future research.

Regression-based analysis of combined EEG and eye-tracking data: Theory and applications

Fixation-related potentials (FRPs), neural responses aligned to the end of saccades, are a promising tool for studying the dynamics of attention and cognition under natural viewing conditions. In the past, four methodological problems have complicated the analysis of such combined eye-tracking/electroencephalogram experiments: (1) the synchronization of data streams, (2) the removal of ocular artifacts, (3) the condition-specific temporal overlap between the brain responses evoked by consecutive fixations, and (4) the fact that numerous low-level stimulus and saccade properties also influence the postsaccadic neural responses. Although effective solutions exist for the first two problems, the latter two are only beginning to be addressed. In the current paper, we present and review a unified regression-based framework for FRP analysis that allows us to deconvolve overlapping potentials while also controlling for both linear and nonlinear confounds on the FRP waveform. An open software implementation is provided for all procedures. We then demonstrate the advantages of this proposed (non)linear deconvolution modeling approach for data from three commonly studied paradigms: face perception, scene viewing, and reading. First, for a traditional event-related potential (ERP) face recognition experiment, we show how this technique can separate stimulus ERPs from overlapping muscle and brain potentials produced by small (micro)saccades on the face. Second, in natural scene viewing, we model and isolate multiple nonlinear effects of saccade parameters on the FRP. Finally, for a natural sentence reading experiment using the boundary paradigm, we show how it is possible to study the neural correlates of parafoveal preview after removing spurious overlap effects caused by the associated difference in average fixation time. Our results suggest a principal way of measuring reliable eye movement-related brain activity during natural vision.

Neurophysiological, Oculomotor, and Computational Modeling of Impaired Reading Ability in Schizophrenia

Schizophrenia (Sz) is associated with deficits in fluent reading ability that compromise functional outcomes. Here, we utilize a combined eye-tracking, neurophysiological, and computational modeling approach to analyze underlying visual and oculomotor processes. Subjects included 26 Sz patients (SzP) and 26 healthy controls. Eye-tracking and electroencephalography data were acquired continuously during the reading of passages from the Gray Oral Reading Tests reading battery, permitting between-group evaluation of both oculomotor activity and fixation-related potentials (FRP). Schizophrenia patients showed a marked increase in time required per word (d = 1.3, P < .0001), reflecting both a moderate increase in fixation duration (d = .7, P = .026) and a large increase in the total saccade number (d = 1.6, P < .0001). Simulation models that incorporated alterations in both lower-level visual and oculomotor function as well as higher-level lexical processing performed better than models that assumed either deficit-type alone. In neurophysiological analyses, amplitude of the fixation-related P1 potential (P1f) was significantly reduced in SzP (d = .66, P = .013), reflecting reduced phase reset of ongoing theta-alpha band activity (d = .74, P = .019). In turn, P1f deficits significantly predicted increased saccade number both across groups (P = .017) and within SzP alone (P = .042). Computational and neurophysiological methods provide increasingly important approaches for investigating sensory contributions to impaired cognition during naturalistic processing in Sz. Here, we demonstrate deficits in reading rate that reflect both sensory/oculomotor- and semantic-level impairments and that manifest, respectively, as alterations in saccade number and fixation duration. Impaired P1f generation reflects impaired fixation-related reset of ongoing brain rhythms and suggests inefficient information processing within the early visual system as a basis for oculomotor dyscontrol during fluent reading in Sz.

Best practices in eye tracking research

This guide describes best practices in using eye tracking technology for research in a variety of disciplines. A basic outline of the anatomy and physiology of the eyes and of eye movements is provided, along with a description of the sorts of research questions eye tracking can address. We then explain how eye tracking technology works and what sorts of data it generates, and provide guidance on how to select and use an eye tracker as well as selecting appropriate eye tracking measures. Challenges to the validity of eye tracking studies are described, along with recommendations for overcoming these challenges. We then outline correct reporting standards for eye tracking studies.

Eye Movements and Fixation-Related Potentials in Reading: A Review

The present review is addressed to researchers in the field of reading and psycholinguistics who are both familiar with and new to co-registration research of eye movements (EMs) and fixation related-potentials (FRPs) in reading. At the outset, we consider a conundrum relating to timing discrepancies between EM and event related potential (ERP) effects. We then consider the extent to which the co-registration approach might allow us to overcome this and thereby discriminate between formal theoretical and computational accounts of reading. We then describe three phases of co-registration research before evaluating the existing body of such research in reading. The current, ongoing phase of co-registration research is presented in comprehensive tables which provide a detailed summary of the existing findings. The thorough appraisal of the published studies allows us to engage with issues such as the reliability of FRP components as correlates of cognitive processing in reading and the advantages of analysing both data streams (i.e., EMs and FRPs) simultaneously relative to each alone, as well as the current, and limited, understanding of the relationship between EM and FRP measures. Finally, we consider future directions and in particular the potential of analytical methods involving deconvolution and the potential of measurement of brain oscillatory activity.

A co-registration investigation of inter-word spacing and parafoveal preview: Eye movements and fixation-related potentials

Participants' eye movements (EMs) and EEG signal were simultaneously recorded to examine foveal and parafoveal processing during sentence reading. All the words in the sentence were manipulated for inter-word spacing (intact spaces vs. spaces replaced by a random letter) and parafoveal preview (identical preview vs. random letter string preview). We observed disruption for unspaced text and invalid preview conditions in both EMs and fixation-related potentials (FRPs). Unspaced and invalid preview conditions received longer reading times than spaced and valid preview conditions. In addition, the FRP data showed that unspaced previews disrupted reading in earlier time windows of analysis, compared to string preview conditions. Moreover, the effect of parafoveal preview was greater for spaced relative to unspaced conditions, in both EMs and FRPs. These findings replicate well-established preview effects, provide novel insight into the neural correlates of reading with and without inter-word spacing and suggest that spatial selection precedes lexical processing.

Return sweeps in reading: Processing implications of undersweep-fixations

Models of eye-movement control during reading focus on reading single lines of text. However, with multiline texts, return sweeps, which bring fixation from the end of one line to the beginning of the next, occur regularly and influence ~20% of all reading fixations. Our understanding of return sweeps is still limited. One common feature of return sweeps is the prevalence of oculomotor errors. Return sweeps, often initially undershoot the start of the line. Corrective saccades then bring fixation closer to the line start. The fixation occurring between the undershoot and the corrective saccade (undersweep-fixation) has important theoretical implications for the serial nature of lexical processing during reading, as they occur on words ahead of the intended attentional target. Furthermore, since the attentional target of a return sweep will lie far outside the parafovea during the prior fixation, it cannot be lexically preprocessed during this prior fixation. We explore the implications of undersweep-fixations for ongoing processing and models of eye movements during reading by analysing two existing eye-movement data sets of multiline reading.

Optimizing the ICA-based removal of ocular EEG artifacts from free viewing experiments

Combining EEG with eye-tracking is a promising approach to study neural correlates of natural vision, but the resulting recordings are also heavily contaminated by activity of the eye balls, eye lids, and extraocular muscles. While Independent Component Analysis (ICA) is commonly used to suppress these ocular artifacts, its performance under free viewing conditions has not been systematically evaluated and many published reports contain residual artifacts. Here I evaluated and optimized ICA-based correction for two tasks with unconstrained eye movements: visual search in images and sentence reading. In a first step, four parameters of the ICA pipeline were varied orthogonally: the (1) high-pass and (2) low-pass filter applied to the training data, (3) the proportion of training data containing myogenic saccadic spike potentials (SP), and (4) the threshold for eye tracker-based component rejection. In a second step, the eye-tracker was used to objectively quantify the correction quality of each ICA solution, both in terms of undercorrection (residual artifacts) and overcorrection (removal of neurogenic activity). As a benchmark, results were compared to those obtained with an alternative spatial filter, Multiple Source Eye Correction (MSEC). With commonly used settings, Infomax ICA not only left artifacts in the data, but also distorted neurogenic activity during eye movement-free intervals. However, correction results could be strongly improved by training the ICA on optimally filtered data in which SPs were massively overweighted. With optimized procedures, ICA removed virtually all artifacts, including the SP and its associated spectral broadband artifact from both viewing paradigms, with little distortion of neural activity. It also outperformed MSEC in terms of SP correction. Matlab code is provided.

Cortical control of eye movements in natural reading: Evidence from MVPA

Language comprehension during reading requires fine-grained management of saccadic eye movements. A critical question, therefore, is how the brain controls eye movements in reading. Neural correlates of simple eye movements have been found in multiple cortical regions, but little is known about how this network operates in reading. To investigate this question in the present study, participants were presented with normal text, pseudo-word text, and consonant string text in a magnetic resonance imaging (MRI) scanner with eyetracking. Participants read naturally in the normal text condition and moved their eyes "as if they were reading" in the other conditions. Multi-voxel pattern analysis was used to analyze the fMRI signal in the oculomotor network. We found that activation patterns in a subset of network regions differentiated between stimulus types. These results suggest that the oculomotor network reflects more than simple saccade generation and are consistent with the hypothesis that specific network areas interface with cognitive systems.

Eye movement desensitization and reprocessing for post-traumatic stress disorder from the perspective of three-dimensional model of the experiential selfhood

Eye Movement Desensitization and Reprocessing (EMDR) therapy is included in many international trauma treatment guidelines and is also shortlisted as an evidence-based practice for the treatment of psychological trauma and Post-Traumatic Stress Disorder (PTSD). However, its neurobiological mechanisms have not yet been fully understood. In this brief article we propose a hypothesis that a recently introduced neurophysiologically based three-dimensional construct model for experiential selfhood may help to fill this gap by providing the necessary neurobiological rationale of EMDR. In support of this proposal we briefly overview the neurophysiology of eye movements and the triad selfhood components, as well as EMDR therapy neuroimaging studies.

Event-Related Potential Evidence of Implicit Metric Structure during Silent Reading

Under the Implicit Prosody Hypothesis, readers generate prosodic structures during silent reading that can direct their real-time interpretations of the text. In the current study, we investigated the processing of implicit meter by recording event-related potentials (ERPs) while participants read a series of 160 rhyming couplets, where the rhyme target was always a stress-alternating noun–verb homograph (e.g., permit, which is pronounced PERmit as a noun and perMIT as a verb). The target had a strong–weak or weak–strong stress pattern, which was either consistent or inconsistent with the stress expectation generated by the couplet. Inconsistent strong–weak targets elicited negativities between 80–155 ms and 325–375 ms relative to consistent strong–weak targets; inconsistent weak–strong targets elicited a positivity between 365–435 ms relative to consistent weak–strong targets. These results are largely consistent with effects of metric violations during listening, demonstrating that implicit prosodic representations are similar to explicit prosodic representations.

Co-registration of eye movements and neuroimaging for studying contextual predictions in natural reading

Sixteen years ago, Sereno and Rayner (2003. Measuring word recognition in reading: eye movements and event-related potentials. Trends in Cognitive Sciences, 7(11), 489-493) illustrated how "by means of review and comparison" eye movement (EM) and event-related potential (ERP) studies may advance our understanding of visual word recognition. Attempts to simultaneously record EMs and ERPs soon followed. Recently, this co-registration approach has also been transferred to fMRI and oscillatory EEG. With experimental settings close to natural reading, co-registration enables us to directly integrate insights from EM and neuroimaging studies. This should extend current experimental paradigms by moving the field towards studying sentence-level processing including effects of context and parafoveal preview. This article will introduce the basic principles and applications of co-registration and selectively review how this approach may shed light on one of the most controversially discussed issues in reading research, contextual predictions in online language processing.

Parafoveal previews and lexical frequency in natural reading: Evidence from eye movements and fixation-related potentials

Participants' eye movements and electroencephalogram (EEG) signal were recorded as they read sentences displayed according to the gaze-contingent boundary paradigm. Two target words in each sentence were manipulated for lexical frequency (high vs. low frequency) and parafoveal preview of each target word (identical vs. string of random letters vs. string of Xs). Eye movement data revealed visual parafoveal-on-foveal (PoF) effects, as well as foveal visual and orthographic preview effects and word frequency effects. Fixation-related potentials (FRPs) showed visual and orthographic PoF effects as well as foveal visual and orthographic preview effects. Our results replicated the early preview positivity effect (Dimigen, Kliegl, & Sommer, 2012) in the X-string preview condition, and revealed different neural correlates associated with a preview comprised of a string of random letters relative to a string of Xs. The former effects seem likely to reflect difficulty associated with the integration of parafoveal and foveal information, as well as feature overlap, while the latter reflect inhibition, and potentially disruption, to processing underlying reading. Interestingly, and consistent with Kretzschmar, Schlesewsky, and Staub (2015), no frequency effect was reflected in the FRP measures. The findings provide insight into the neural correlates of parafoveal processing and written word recognition in reading and demonstrate the value of utilizing ecologically valid paradigms to study well established phenomena that occur as text is read naturally. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Time-Course of Motor Involvement in Literal and Metaphoric Action Sentence Processing: A TMS Study

There is evidence that the motor cortex is involved in reading sentences containing an action verb ("The spike was hammered into the ground") as well as metaphoric sentences ("The army was hammered in the battle"). Verbs such as 'hammered' may be homonyms, with separate meanings belonging to the literal action and metaphoric action, or they may be polysemous, with the metaphoric sense grounded in the literal sense. We investigated the time course of the effects of single-pulse transcranial magnetic stimulation to primary motor cortex on literal and metaphoric sentence comprehension. Stimulation 300 ms post-verb presentation impaired comprehension of both literal and metaphoric sentences, supporting a causal role of sensory-motor areas in comprehension. Results suggest that the literal meaning of an action verb remains activated during metaphor comprehension, even after the temporal window of homonym disambiguation. This suggests that such verbs are polysemous, and both senses are related and grounded in motor cortex.

ZuCo, a simultaneous EEG and eye-tracking resource for natural sentence reading

We present the Zurich Cognitive Language Processing Corpus (ZuCo), a dataset combining electroencephalography (EEG) and eye-tracking recordings from subjects reading natural sentences. ZuCo includes high-density EEG and eye-tracking data of 12 healthy adult native English speakers, each reading natural English text for 4-6 hours. The recordings span two normal reading tasks and one task-specific reading task, resulting in a dataset that encompasses EEG and eye-tracking data of 21,629 words in 1107 sentences and 154,173 fixations. We believe that this dataset represents a valuable resource for natural language processing (NLP). The EEG and eye-tracking signals lend themselves to train improved machine-learning models for various tasks, in particular for information extraction tasks such as entity and relation extraction and sentiment analysis. Moreover, this dataset is useful for advancing research into the human reading and language understanding process at the level of brain activity and eye-movement.

Understanding Moment-to-Moment Processing of Visual Narratives

What role do moment‐to‐moment comprehension processes play in visual attentional selection in picture stories? The current work uniquely tested the role of bridging inference generation processes on eye movements while participants viewed picture stories. Specific components of the Scene Perception and Event Comprehension Theory (SPECT) were tested. Bridging inference generation was induced by manipulating the presence of highly inferable actions embedded in picture stories. When inferable actions are missing, participants have increased viewing times for the immediately following critical image (Magliano, Larson, Higgs, & Loschky, 2016). This study used eye‐tracking to test competing hypotheses about the increased viewing time: (a) Computational Load: inference generation processes increase overall computational load, producing longer fixation durations; (b) Visual Search: inference generation processes guide eye‐movements to pick up inference‐relevant information, producing more fixations. Participants had similar fixation durations, but they made more fixations while generating inferences, with that process starting from the fifth fixation. A follow‐up hypothesis predicted that when generating inferences, participants fixate scene regions important for generating the inference. A separate group of participants rated the inferential‐relevance of regions in the critical images, and results showed that these inferentially relevant regions predicted differences in other viewers’ eye movements. Thus, viewers’ event models in working memory affect visual attentional selection while viewing visual narratives.

Successful Encoding during Natural Reading Is Associated with Fixation-Related Potentials and Large-Scale Network Deactivation

Reading literature (e.g., an entire book) is an enriching experience that qualitatively differs from reading a single sentence; however, the brain dynamics of such context-dependent memory remains unclear. This study aimed to elucidate mnemonic neural dynamics during natural reading of literature by performing electroencephalogram (EEG) and functional magnetic resonance imaging (fMRI). Brain activities of human participants recruited on campus were correlated with their subsequent memory, which was quantified by semantic correlation between the read text and reports subsequently written by them based on state of the art natural language processing procedures. The results of the EEG data analysis showed a significant positive relationship between subsequent memory and fixation-related EEG. Sentence-length and paragraph-length mnemonic processes were associated with N1-P2 and P3 fixation-related potential (FRP) components and fixation-related θ-band (4-8 Hz) EEG power, respectively. In contrast, the results of fMRI analysis showed a significant negative relationship between subsequent memory and blood oxygenation level-dependent (BOLD) activation. Sentence-length and paragraph-length mnemonic processes were associated with networks of regions forming part of the salience network and the default mode network (DMN), respectively. Taken together with the EEG results, these memory-related deactivations in the salience network and the DMN were thought to reflect the reading of sentences characterized by low mnemonic load and the suppression of task-irreverent thoughts, respectively. It was suggested that the context-dependent mnemonic process during literature reading requires large-scale network deactivation, which might reflect coordination of a range of voluntary processes during reading.

Refixation control in free viewing: a specialized mechanism divulged by eye-movement-related brain activity

In free viewing, the eyes return to previously visited locations rather frequently, even though the attentional and memory-related processes controlling eye-movement show a strong antirefixation bias. To overcome this bias, a special refixation triggering mechanism may have to be recruited. We probed the neural evidence for such a mechanism by combining eye tracking with EEG recording. A distinctive signal associated with refixation planning was observed in the EEG during the presaccadic interval: the presaccadic potential was reduced in amplitude before a refixation compared with normal fixations. The result offers direct evidence for a special refixation mechanism that operates in the saccade planning stage of eye movement control. Once the eyes have landed on the revisited location, acquisition of visual information proceeds indistinguishably from ordinary fixations. NEW & NOTEWORTHY A substantial proportion of eye fixations in human natural viewing behavior are revisits of recently visited locations, i.e., refixations. Our recently developed methods enabled us to study refixations in a free viewing visual search task, using combined eye movement and EEG recording. We identified in the EEG a distinctive refixation-related signal, signifying a control mechanism specific to refixations as opposed to ordinary eye fixations.

Target probability modulates fixation-related potentials in visual search

This study investigated the influence of target probability on the neural response to target detection in free viewing visual search. Participants were asked to indicate the number of targets (one or two) among distractors in a visual search task while EEG and eye movements were co-registered. Target probability was manipulated by varying the set size of the displays between 10, 22, and 30 items. Fixation-related potentials time-locked to first target fixations revealed a pronounced P300 at the centro-parietal cortex with larger amplitudes for set sizes 22 and 30 than for set size 10. With increasing set size, more distractor fixations preceded the detection of the target, resulting in a decreased target probability and, consequently, a larger P300. For distractors, no increase of P300 amplitude with set size was observed. The findings suggest that set size specifically affects target but not distractor processing in overt serial visual search.

Development of the error-monitoring system from ages 9-35: Unique insight provided by MRI-constrained source localization of EEG

The ability to self-detect errors and dynamically adapt behavior is a cornerstone of higher-level cognition, requiring coordinated activity from a network of neural regions. However, disagreement exists over how the error-monitoring system develops throughout adolescence and early adulthood. The present report leveraged MRI-constrained EEG source localization to detail typical development of the error-monitoring system in a sample of 9-35 year-olds (n = 43). Participants performed a flanker task while high-density EEG was recorded; structural MRIs were also acquired for all participants. Analysis of the scalp-recorded EEG data revealed a frontocentral negativity (error-related negativity; ERN) immediately following errors for all participants, although the topography of the ERN varied with age. Source localization of the ERN time range revealed maximal activity within the posterior cingulate cortex (PCC) for all ages, consistent with recent evidence that the PCC provides a substantial contribution to the scalp-recorded ERN. Activity within a network of brain regions, including dorsal anterior cingulate, PCC, and parietal cortex, was predictive of improved performance following errors, regardless of age. However, additional activity within insula, orbitofrontal cortex and inferior frontal gyrus linearly increased with age. Together, these data suggest that the core error-monitoring system is online by early adolescence and remains relatively stable into adulthood. However, additional brain regions become embedded within this core network with age. These results serve as a model of typical development of the error-monitoring system from early adolescence into adulthood.

Neural Correlates of Word Recognition: A Systematic Comparison of Natural Reading and Rapid Serial Visual Presentation

Neural correlates of word recognition are commonly studied with (rapid) serial visual presentation (RSVP), a condition that eliminates three fundamental properties of natural reading: parafoveal preprocessing, saccade execution, and the fast changes in attentional processing load occurring from fixation to fixation. We combined eye-tracking and EEG to systematically investigate the impact of all three factors on brain-electric activity during reading. Participants read lists of words either actively with eye movements (eliciting fixation-related potentials) or maintained fixation while the text moved passively through foveal vision at a matched pace (RSVP-with-flankers paradigm, eliciting ERPs). The preview of the upcoming word was manipulated by changing the number of parafoveally visible letters. Processing load was varied by presenting words of varying lexical frequency. We found that all three factors have strong interactive effects on the brain's responses to words: Once a word was fixated, occipitotemporal N1 amplitude decreased monotonically with the amount of parafoveal information available during the preceding fixation; hence, the N1 component was markedly attenuated under reading conditions with preview. Importantly, this preview effect was substantially larger during active reading (with saccades) than during passive RSVP with flankers, suggesting that the execution of eye movements facilitates word recognition by increasing parafoveal preprocessing. Lastly, we found that the N1 component elicited by a word also reflects the lexical processing load imposed by the previously inspected word. Together, these results demonstrate that, under more natural conditions, words are recognized in a spatiotemporally distributed and interdependent manner across multiple eye fixations, a process that is mediated by active motor behavior.

Visual processing during natural reading

Reading is a unique human ability that plays a pivotal role in the development and functioning of our modern society. However, its neural basis remains poorly understood since previous research was focused on reading words with fixed gaze. Here we developed a methodological framework for single-trial analysis of fixation onset-related EEG activity (FOREA) that enabled us to investigate visual information processing during natural reading. To reveal the effect of reading skills on orthographic processing during natural reading, we measured how altering the configural properties of the written text by modifying inter-letter spacing affects FOREA. We found that orthographic processing is reflected in FOREA in three consecutive time windows (120–175 ms, 230–265 ms, 345–380 ms after fixation onset) and the magnitude of FOREA effects in the two later time intervals showed a close association with the participants’ reading speed: FOREA effects were larger in fast than in slow readers. Furthermore, these expertise-driven configural effects were clearly dissociable from the FOREA signatures of visual perceptual processes engaged to handle the increased crowding (155–220 ms) as a result of decreasing letter spacing. Our findings revealed that with increased reading skills orthographic processing becomes more sensitive to the configural properties of the written text.

Oscillatory Brain Dynamics during Sentence Reading: A Fixation-Related Spectral Perturbation Analysis

The present study investigated oscillatory brain dynamics during self-paced sentence-level processing. Participants read fully correct sentences, sentences containing a semantic violation and "sentences" in which the order of the words was randomized. At the target word level, fixations on semantically unrelated words elicited a lower-beta band (13-18 Hz) desynchronization. At the sentence level, gamma power (31-55 Hz) increased linearly for syntactically correct sentences, but not when the order of the words was randomized. In the 300-900 ms time window after sentence onsets, theta power (4-7 Hz) was greater for syntactically correct sentences as compared to sentences where no syntactic structure was preserved (random words condition). We interpret our results as conforming with a recently formulated predictive-coding framework for oscillatory neural dynamics during sentence-level language comprehension. Additionally, we discuss how our results relate to previous findings with serial visual presentation vs. self-paced reading.

Direction and viewing area-sensitive influence of EOG artifacts revealed in the EEG topographic pattern analysis

The influence of eye movement-related artifacts on electroencephalography (EEG) signals of human subjects, who were requested to perform a direction or viewing area dependent saccade task, was investigated by using a simultaneous recording with ocular potentials as electro-oculography (EOG). In the past, EOG artifact removals have been studied in tasks with a single fixation point in the screen center, with less attention to the sensitivity of cornea-retinal dipole orientations to the EEG head map. In the present study, we hypothesized the existence of a systematic EOG influence that differs according to coupling conditions of eye-movement directions with viewing areas including different fixation points. The effect was validated in the linear regression analysis by using 12 task conditions combining horizontal/vertical eye-movement direction and three segregated zones of gaze in the screen. In the first place, event-related potential topographic patterns were analyzed to compare the 12 conditions and propagation coefficients of the linear regression analysis were successively calculated in each condition. As a result, the EOG influences were significantly different in a large number of EEG channels, especially in the case of horizontal eye-movements. In the cross validation, the linear regression analysis using the appropriate dataset of the target direction/viewing area combination demonstrated an improved performance compared with the traditional methods using a single fixation at the center. This result may open a potential way to improve artifact correction methods by considering the systematic EOG influence that can be predicted according to the view angle such as using eye-tracker systems.

The Influence of Content Meaningfulness on Eye Movements across Tasks: Evidence from Scene Viewing and Reading

The present study investigated the influence of content meaningfulness on eye-movement control in reading and scene viewing. Texts and scenes were manipulated to make them uninterpretable, and then eye-movements in reading and scene-viewing were compared to those in pseudo-reading and pseudo-scene viewing. Fixation durations and saccade amplitudes were greater for pseudo-stimuli. The effect of the removal of meaning was seen exclusively in the tail of the fixation duration distribution in both tasks, and the size of this effect was the same across tasks. These findings suggest that eye movements are controlled by a common mechanism in reading and scene viewing. They also indicate that not all eye movements are responsive to the meaningfulness of stimulus content. Implications for models of eye movement control are discussed.

Toward FRP-Based Brain-Machine Interfaces-Single-Trial Classification of Fixation-Related Potentials

The co-registration of eye tracking and electroencephalography provides a holistic measure of ongoing cognitive processes. Recently, fixation-related potentials have been introduced to quantify the neural activity in such bi-modal recordings. Fixation-related potentials are time-locked to fixation onsets, just like event-related potentials are locked to stimulus onsets. Compared to existing electroencephalography-based brain-machine interfaces that depend on visual stimuli, fixation-related potentials have the advantages that they can be used in free, unconstrained viewing conditions and can also be classified on a single-trial level. Thus, fixation-related potentials have the potential to allow for conceptually different brain-machine interfaces that directly interpret cortical activity related to the visual processing of specific objects. However, existing research has investigated fixation-related potentials only with very restricted and highly unnatural stimuli in simple search tasks while participant’s body movements were restricted. We present a study where we relieved many of these restrictions while retaining some control by using a gaze-contingent visual search task. In our study, participants had to find a target object out of 12 complex and everyday objects presented on a screen while the electrical activity of the brain and eye movements were recorded simultaneously. Our results show that our proposed method for the classification of fixation-related potentials can clearly discriminate between fixations on relevant, non-relevant and background areas. Furthermore, we show that our classification approach generalizes not only to different test sets from the same participant, but also across participants. These results promise to open novel avenues for exploiting fixation-related potentials in electroencephalography-based brain-machine interfaces and thus providing a novel means for intuitive human-machine interaction.

Brains for all the ages: structural neurodevelopment in infants and children from a life-span perspective

Magnetic resonance imaging (MRI) is a noninvasive method to measure brain structure and function that may be applied to human participants of all ages. This chapter reviews our recent work creating a life-span Neurodevelopmental MRI Database. It provides age-specific reference data in fine-grained age intervals from 2 weeks through 89 years. The reference data include average MRI templates, segmented tissue priors, and a common stereotaxic atlas for pediatric and adult participants. The database will be useful for neuroimaging research over a wide range of ages and may be used to make life-span comparisons. The chapter reviews the application of this database to the study of neurostructural development, including a new volumetric study of segmented brain tissue over the life span. We also show how this database could be used to create "study-specific" MRI templates for special groups and apply this to the MRIs of Chinese children. Finally, we review recent use of the database in the study of brain activity in pediatric populations.

Morphology of Primary Visual Cortex Predicts Individual Differences in Fixation Duration during Text Reading

In skilled reading, fixations are brief periods of time in which the eyes settle on words. E-Z Reader, a computational model of dynamic reading, posits that fixation durations are under real-time control of lexical processing. Lexical processing, in turn, requires efficient visual encoding. Here we tested the hypothesis that individual differences in fixation durations are related to individual differences in the efficiency of early visual encoding. To test this hypothesis, we recorded participants' eye movements during reading. We then examined individual differences in fixation duration distributions as a function of individual differences in the morphology of primary visual cortex measured from MRI scans. The results showed that greater gray matter surface area and volume in visual cortex predicted shorter and less variable fixation durations in reading. These results suggest that individual differences in eye movements during skilled reading are related to initial visual encoding, consistent with models such as E-Z Reader that emphasize lexical control over fixation time.