Tracking the mind during reading: the influence of past, present, and future words on fixation durations

Linguistic and attentional factors - Not statistical regularities - Contribute to word-selective neural responses with FPVS-oddball paradigms

Studies using frequency-tagging in electroencephalography (EEG) have dramatically increased in the past 10 years, in a variety of domains and populations. Here we used Fast Periodic Visual Stimulation (FPVS) combined with an oddball design to explore visual word recognition. Given the paradigm's high sensitivity, it is crucial for future basic research and clinical application to prove its robustness across variations of designs, stimulus types and tasks. This paradigm uses periodicity of brain responses to measure discrimination between two experimentally defined categories of stimuli presented periodically. EEG was recorded in 22 adults who viewed words inserted every 5 stimuli (at 2 Hz) within base stimuli presented at 10 Hz. Using two discrimination levels (deviant words among nonwords or pseudowords), we assessed the impact of relative frequency of item repetition (set size or item repetition controlled for deviant versus base stimuli), and of the orthogonal task (focused or deployed spatial attention). Word-selective occipito-temporal responses were robust at the individual level (significant in 95% of participants), left-lateralized, larger for the prelexical (nonwords) than lexical (pseudowords) contrast, and stronger with a deployed spatial attention task as compared to the typically used focused task. Importantly, amplitudes were not affected by item repetition. These results help understanding the factors influencing word-selective EEG responses and support the validity of FPVS-EEG oddball paradigms, as they confirm that word-selective responses are linguistic. Second, they show its robustness against design-related factors that could induce statistical (ir)regularities in item rate. They also confirm its high individual sensitivity and demonstrate how it can be optimized, using a deployed rather than focused attention task, to measure implicit word recognition processes in typical and atypical populations.

Multi-Algorithm Artifact Correction (MAAC) procedure part one: Algorithm and example

The Multi-Algorithm Artifact Correction (MAAC) procedure is presented for electroencephalographic (EEG) data, as made freely available in the open-source EP Toolkit (Dien, 2010). First the major EEG artifact correction methods (regression, spatial filters, principal components analysis, and independent components analysis) are reviewed. Contrary to the dominant approach of picking one method that is thought to be most effective, this review concludes that none are globally superior, but rather each has strengths and weaknesses. Then each of the major artifact types are reviewed (Blink, Corneo-Retinal Dipole, Saccadic Spike Potential, and Movement). For each one, it is proposed that one of the major correction methods is best matched to address it, resulting in the MAAC procedure. The MAAC itself is then presented, as implemented in the EP Toolkit, in order to provide a sense of the user experience. The primary goal of this present paper is to make the conceptual argument for the MAAC approach.

A Deep Learning Approach to Analyzing Continuous-Time Cognitive Processes

The dynamics of the mind are complex. Mental processes unfold continuously in time and may be sensitive to a myriad of interacting variables, especially in naturalistic settings. But statistical models used to analyze data from cognitive experiments often assume simplistic dynamics. Recent advances in deep learning have yielded startling improvements to simulations of dynamical cognitive processes, including speech comprehension, visual perception, and goal-directed behavior. But due to poor interpretability, deep learning is generally not used for scientific analysis. Here, we bridge this gap by showing that deep learning can be used, not just to imitate, but to analyze complex processes, providing flexible function approximation while preserving interpretability. To do so, we define and implement a nonlinear regression model in which the probability distribution over the response variable is parameterized by convolving the history of predictors over time using an artificial neural network, thereby allowing the shape and continuous temporal extent of effects to be inferred directly from time series data. Our approach relaxes standard simplifying assumptions (e.g., linearity, stationarity, and homoscedasticity) that are implausible for many cognitive processes and may critically affect the interpretation of data. We demonstrate substantial improvements on behavioral and neuroimaging data from the language processing domain, and we show that our model enables discovery of novel patterns in exploratory analyses, controls for diverse confounds in confirmatory analyses, and opens up research questions in cognitive (neuro)science that are otherwise hard to study.

TURead: An eye movement dataset of Turkish reading

In this study, we present TURead, an eye movement dataset of silent and oral sentence reading in Turkish, an agglutinative language with a shallow orthography understudied in reading research. TURead provides empirical data to investigate the relationship between morphology and oculomotor control. We employ a target-word approach in which target words are manipulated by word length and by the addition of two commonly used suffixes in Turkish. The dataset contains well-established eye movement variables; prelexical characteristics such as vowel harmony and bigram-trigram frequencies and word features, such as word length, predictability, frequency, eye voice span measures, Cloze test scores of the root word and suffix predictabilities, as well as the scores obtained from two working memory tests. Our findings on fixation parameters and word characteristics are in line with the patterns reported in the relevant literature.

An updating-based working memory load alters the dynamics of eye movements but not their spatial extent during free viewing of natural scenes

The relationship between spatial deployments of attention and working memory load is an important topic of study, with clear implications for real-world tasks such as driving. Previous research has generally shown that attentional breadth broadens under higher load, while exploratory eye-movement behaviour also appears to change with increasing load. However, relatively little research has compared the effects of working memory load on different kinds of spatial deployment, especially in conditions that require updating of the contents of working memory rather than simple retrieval. The present study undertook such a comparison by measuring participants' attentional breadth (via an undirected Navon task) and their exploratory eye-movement behaviour (a free-viewing recall task) under low and high updating working memory loads. While spatial aspects of task performance (attentional breadth, and peripheral extent of image exploration in the free-viewing task) were unaffected by the load manipulation, the exploratory dynamics of the free-viewing task (including fixation durations and scan-path lengths) changed under increasing load. These findings suggest that temporal dynamics, rather than the spatial extent of exploration, are the primary mechanism affected by working memory load during the spatial deployment of attention. Further, individual differences in exploratory behaviour were observed on the free-viewing task: all metrics were highly correlated across working memory load blocks. These findings suggest a need for further investigation of individual differences in eye-movement behaviour; potential factors associated with these individual differences, including working memory capacity and persistence versus flexibility orientations, are discussed.

An eye on semantics: a study on the influence of concreteness and predictability on early fixation durations

We used eye-tracking during natural reading to study how semantic control and representation mechanisms interact for the successful comprehension of sentences, by manipulating sentence context and single-word meaning. Specifically, we examined whether a word's semantic characteristic (concreteness) affects first fixation and gaze durations (FFDs and GDs) and whether it interacts with the predictability of a word. We used a linear mixed effects model including several possible psycholinguistic covariates. We found a small but reliable main effect of concreteness and replicated a predictability effect on FFDs, but we found no interaction between the two. The results parallel previous findings of additive effects of predictability (context) and frequency (lexical level) in fixation times. Our findings suggest that the semantics of a word and the context created by the preceding words additively influence early stages of word processing in natural sentence reading.

The role of syllables and morphemes in silent reading: An eye-tracking study

German skilled readers have been found to engage in morphological and syllable-based processing in visual word recognition. However, the relative reliance on syllables and morphemes in reading multi-syllabic complex words is still unresolved. This study aimed to unveil which of these sublexical units are the preferred units of reading by employing eye-tracking technology. Participants silently read sentences while their eye-movements were recorded. Words were visually marked using colour alternation (Experiment 1) or hyphenation (Experiment 2)-at syllable boundary (e.g., Kir-schen), at morpheme boundary (e.g., Kirsch-en), or within the units themselves (e.g., Ki-rschen). A control condition without disruptions was used as a baseline (e.g., Kirschen). The results of Experiment 1 showed that eye-movements were not modulated by colour alternations. The results of Experiment 2 indicated that hyphens disrupting syllables had a larger inhibitory effect on reading times than hyphens disrupting morphemes, suggesting that eye-movements in German skilled readers are more influenced by syllabic than morphological structure.

Lexical Processing Strongly Affects Reading Times But Not Skipping During Natural Reading

In a typical text, readers look much longer at some words than at others, even skipping many altogether. Historically, researchers explained this variation via low-level visual or oculomotor factors, but today it is primarily explained via factors determining a word's lexical processing ease, such as how well word identity can be predicted from context or discerned from parafoveal preview. While the existence of these effects is well established in controlled experiments, the relative importance of prediction, preview and low-level factors in natural reading remains unclear. Here, we address this question in three large naturalistic reading corpora (n = 104, 1.5 million words), using deep neural networks and Bayesian ideal observers to model linguistic prediction and parafoveal preview from moment to moment in natural reading. Strikingly, neither prediction nor preview was important for explaining word skipping-the vast majority of explained variation was explained by a simple oculomotor model, using just fixation position and word length. For reading times, by contrast, we found strong but independent contributions of prediction and preview, with effect sizes matching those from controlled experiments. Together, these results challenge dominant models of eye movements in reading, and instead support alternative models that describe skipping (but not reading times) as largely autonomous from word identification, and mostly determined by low-level oculomotor information.

Fixation-related fMRI analysis reveals the neural basis of natural reading of unspaced and spaced Chinese sentences

Although there are many eye-movement studies focusing on natural sentence reading and functional magnetic resonance imaging research on reading with serial visual presentation paradigms, there is a scarcity of investigations into the neural mechanism of natural sentence reading. The present study recruited 33 adults to read unspaced and spaced Chinese sentences with the eye tracking and functional magnetic resonance imaging data recorded simultaneously. By using fixation-related functional magnetic resonance imaging analysis, this study showed that natural reading of Chinese sentences produced activations in ventral visual, dorsal attention, and semantic brain regions, which were modulated by the properties of words such as word length and word frequency. The multivoxel pattern analysis showed that the activity pattern in the left middle temporal gyrus could significantly predict the visual layout categories (i.e. unspaced vs. spaced conditions). Dynamic causal modeling analysis showed that there were bidirectional brain connections between the left middle temporal gyrus and the left inferior occipital cortex in the unspaced Chinese sentence reading but not in the spaced reading. These results provide a neural mechanism for the natural reading of Chinese sentences from the perspective of word segmentation.

Eye movement corpora in Adyghe and Russian: an eye-tracking study of sentence reading in bilinguals

The present study expands the eye-tracking-while reading research toward less studied languages of different typological classes (polysynthetic Adyghe vs. synthetic Russian) that use a Cyrillic script. In the corpus reading data from the two languages, we confirmed the widely studied effects of word frequency and word length on eye movements in Adyghe-Russian bilingual individuals for both languages. We also confirmed morphological effects in Adyghe reading (part-of-speech class and the number of lexical affixes) that were previously shown in some morphologically-rich languages. Importantly, we demonstrated that bilinguals' reading in Adyghe does differ quantitatively (the effect of language on reading times) and qualitatively (different effects of landing and previous/upcoming words on the eye movements within a current word) from their reading in Russian.

Joint effects of individual reading skills and word properties on Chinese children's eye movements during sentence reading

Word recognition during the reading of continuous text has received much attention. While a large body of research has investigated how linguistic properties of words affect eye movements during reading, it remains to be established how individual differences in reading skills affect momentary cognitive processes during sentence reading among typically developing Chinese readers. The present study set out to test the joint influences of word properties and individual reading skills on eye movements during reading among Chinese children. We recorded eye movements of 30 grade 3 (G3) children and 27 grade 5 (G5) children when they read sentences silently for comprehension. Predictors of linear mixed models included word frequency, visual complexity, and launch site distance, in addition to the participants' offline psychometric performances in rapid naming, morphological awareness, word segmenting, and character recognition. The results showed that word properties affected word recognition during sentence reading in both G3 and G5 children. Moreover, word segmenting predicted the G3 children's fixation durations and the G5 children's fixation location, whereas rapid naming predicted the G5 children's fixation duration. Implications are discussed based on the current findings, in light of how different literacy skills contribute to reading development.

GECO-CN: Ghent Eye-tracking COrpus of sentence reading for Chinese-English bilinguals

The current work presents the very first eye-tracking corpus of natural reading by Chinese-English bilinguals, whose two languages entail different writing systems and orthographies. Participants read an entire novel in these two languages, presented in paragraphs on screen. Half of the participants first read half of the novel in their native language (Simplified Chinese) and then the rest of the novel in their second language (English), while the other half read in the reverse language order. This article presents some important basic descriptive statistics of reading times and compares the difference between reading in the two languages. However, this unique eye-tracking corpus also allows the exploration of theories of language processing and bilingualism. Importantly, it provides a solid and reliable ground for studying the difference between Eastern and Western languages, understanding the impact and consequences of having a completely different first language on bilingual processing. The materials are freely available for use by researchers interested in (bilingual) reading.

Stronger functional connectivity during reading contextually predictable words in slow readers

The effect of word predictability is well-documented in terms of local brain activation, but less is known about the functional connectivity among those regions associated with processing predictable words. Evidence from eye movement studies showed that the effect is much more pronounced in slow than in fast readers, suggesting that speed-impaired readers rely more on sentence context to compensate for their difficulties with visual word recognition. The present study aimed to investigate differences in functional connectivity of fast and slow readers within core regions associated with processing predictable words. We hypothesize a stronger synchronization between higher-order language areas, such as the left middle temporal (MTG) and inferior frontal gyrus (IFG), and the left occipito-temporal cortex (OTC) in slow readers. Our results show that slow readers exhibit more functional correlations among these connections; especially between the left IFG and OTC. We interpret our results in terms of the lexical quality hypothesis which postulates a stronger involvement of semantics on orthographic processing in (speed-)impaired readers.

Neural bases of predictions during natural reading of known statements: An EEG and eye movements co-registration study

Predictions of incoming words performed during reading have an impact on how the reader moves their eyes and on the electrical brain potentials. Eye tracking (ET) experiments show that less predictable words are fixated for longer periods of times. Electroencephalography (EEG) experiments show that these words elicit a more negative potential around 400ms (N400) after the word onset when reading one word at a time (foveated reading). Nevertheless, there was no N400 potential during the foveated reading of previously known sentences (memory-encoded), which suggests that the prediction of words from memory-encoded sentences is based on different mechanisms than predictions performed on common sentences. Here, we performed an ET-EEG co-registration experiment where participants read common and memory-encoded sentences. Our results show that the N400 potential disappear when the reader recognises the sentence. Furthermore, time-frequency analyses show a larger alpha lateralisation and a beta power increase for memory-encoded sentences. This suggests a more distributed attention and an active maintenance of the cognitive set, in concordance to the predictive coding framework.

Reading vertically and horizontally mirrored text: An eye movement investigation

This study examined the cognitive processes involved in reading vertically and horizontally mirrored text. We tracked participants' eye movements while they were reading the Potsdam Sentence Corpus which consists of 144 sentences with target words that are manipulated for length and frequency. Sentences were presented in three different conditions: In the normal condition, text was presented with upright letters, in the vertical condition, each letter was flipped around its vertical (left-right) axis while in the horizontal condition, letters were flipped around their horizontal (up-down) axis. Results show that reading was slowed down in both mirror conditions and that horizontal mirroring was particularly disruptive. In both conditions, we found larger effects of word length than in the normal condition indicating that participants read the sentences more serially and effortfully. Similarly, frequency effects were larger in both mirror conditions in later reading measures (gaze duration, go-past time, and total reading time) and particularly pronounced in the horizontal condition. This indicates that reading mirrored script involves a late checking mechanism that is particularly important for reading a horizontally mirrored script. Together, our findings demonstrate that mirroring affects both early visual identification and later linguistic processes.

From eye movements to scanpath networks: A method for studying individual differences in expository text reading

Eye movements have been examined as an index of attention and comprehension during reading in the literature for over 30 years. Although eye-movement measurements are acknowledged as reliable indicators of readers' comprehension skill, few studies have analyzed eye-movement patterns using network science. In this study, we offer a new approach to analyze eye-movement data. Specifically, we recorded visual scanpaths when participants were reading expository science text, and used these to construct scanpath networks that reflect readers' processing of the text. Results showed that low ability and high ability readers' scanpath networks exhibited distinctive properties, which are reflected in different network metrics including density, centrality, small-worldness, transitivity, and global efficiency. Such patterns provide a new way to show how skilled readers, as compared with less skilled readers, process information more efficiently. Implications of our analyses are discussed in light of current theories of reading comprehension.

Rational speech comprehension: Interaction between predictability, acoustic signal, and noise

Introduction

During speech comprehension, multiple sources of information are available to listeners, which are combined to guide the recognition process. Models of speech comprehension posit that when the acoustic speech signal is obscured, listeners rely more on information from other sources. However, these models take into account only word frequency information and local contexts (surrounding syllables), but not sentence-level information. To date, empirical studies investigating predictability effects in noise did not carefully control the tested speech sounds, while the literature investigating the effect of background noise on the recognition of speech sounds does not manipulate sentence predictability. Additionally, studies on the effect of background noise show conflicting results regarding which noise type affects speech comprehension most. We address this in the present experiment.

Methods

We investigate how listeners combine information from different sources when listening to sentences embedded in background noise. We manipulate top-down predictability, type of noise, and characteristics of the acoustic signal, thus creating conditions which differ in the extent to which a specific speech sound is masked in a way that is grounded in prior work on the confusability of speech sounds in noise. Participants complete an online word recognition experiment.

Results and discussion

The results show that participants rely more on the provided sentence context when the acoustic signal is harder to process. This is the case even when interactions of the background noise and speech sounds lead to small differences in intelligibility. Listeners probabilistically combine top-down predictions based on context with noisy bottom-up information from the acoustic signal, leading to a trade-off between the different types of information that is dependent on the combination of a specific type of background noise and speech sound.

Speech register influences listeners' word expectations

We utilized the N400 effect to investigate the influence of speech register on predictive language processing. Participants listened to long stretches (4 - 15 min) of naturalistic speech from different registers (dialogues, news broadcasts, and read-aloud books), totalling approximately 50,000 words, while the EEG signal was recorded. We estimated the surprisal of words in the speech materials with the aid of a statistical language model in such a manner that it reflected different predictive processing strategies; generic, register-specific, or recency-based. The N400 amplitude was best predicted with register-specific word surprisal, indicating that the statistics of the wider context (i.e., register) influences predictive language processing. Furthermore, adaptation to speech register cannot merely be explained by recency effects; instead, listeners adapt their word anticipations to the presented speech register.

Expanding horizons of cross-linguistic research on reading: The Multilingual Eye-movement Corpus (MECO)

Scientific studies of language behavior need to grapple with a large diversity of languages in the world and, for reading, a further variability in writing systems. Yet, the ability to form meaningful theories of reading is contingent on the availability of cross-linguistic behavioral data. This paper offers new insights into aspects of reading behavior that are shared and those that vary systematically across languages through an investigation of eye-tracking data from 13 languages recorded during text reading. We begin with reporting a bibliometric analysis of eye-tracking studies showing that the current empirical base is insufficient for cross-linguistic comparisons. We respond to this empirical lacuna by presenting the Multilingual Eye-Movement Corpus (MECO), the product of an international multi-lab collaboration. We examine which behavioral indices differentiate between reading in written languages, and which measures are stable across languages. One of the findings is that readers of different languages vary considerably in their skipping rate (i.e., the likelihood of not fixating on a word even once) and that this variability is explained by cross-linguistic differences in word length distributions. In contrast, if readers do not skip a word, they tend to spend a similar average time viewing it. We outline the implications of these findings for theories of reading. We also describe prospective uses of the publicly available MECO data, and its further development plans.

Reading Behaviors through Patterns of Finger-Tracking in Italian Children with Autism Spectrum Disorder

The paper proposes an ecological and portable protocol for the large-scale collection of reading data in high-functioning autism spectrum disorder (ASD) children based on recording the finger movements of a subject reading a text displayed on a tablet touchscreen. By capitalizing on recent evidence that movements of a finger that points to a scene or text during visual exploration or reading may approximate eye fixations, we focus on recognition of written content and function words, pace of reading, and accuracy in reading comprehension. The analysis showed significant differences between typically developing and ASD children, with the latter group exhibiting greater variation in levels of reading ability, slower developmental pace in reading speed, less accurate comprehension, greater dependency on word length and word frequency, less significant prediction-based processing, as well as a monotonous, steady reading pace with reduced attention to weak punctuation. Finger-tracking patterns provides evidence that ASD readers may fail to integrate single word processing into major syntactic structures and lends support to the hypothesis of an impaired use of contextual information to predict upcoming stimuli, suggesting that difficulties in perception may arise as difficulties in prediction.

Eye movement control in reading Chinese: A matter of strength of character?

This paper explores the processes underlying eye movement control in Chinese reading among a population of young 4th and 5th grade readers. Various proposals to explain the underlying mechanisms involved in eye movement control are examined and the paper concludes that the most likely account is of a two-factor process whereby the character is the main driver for longer saccades and that the word plays a role in shorter ones. A computational model is proposed to provide an integrated account of the dynamic interaction of these two factors.

Fixation-related fMRI analysis reveals the neural basis of parafoveal processing in self-paced reading of Chinese words

While parafoveal word processing plays an important role in natural reading, the underlying neural mechanism remains unclear. The present study investigated the neural basis of parafoveal processing during Chinese word reading with the co-registration of eye-tracking and functional magnetic resonance imaging (fMRI) using fixation-related fMRI analysis. In the gaze-contingent boundary paradigm, preview conditions (words that are identical, orthographically similar, and unrelated to target words), pre-target word frequency and target word frequency were manipulated. When fixating the pre-target word, the identical preview condition elicited lower brain activation in the left fusiform gyrus relative to unrelated and orthographically similar preview conditions and there were significant interactions of preview condition and pre-target word frequency on brain activation of the left middle frontal gyrus, left fusiform gyrus and supplementary motor area. When fixating the target word, there was a significant main effect of preview condition on brain activation of the right fusiform gyrus and a significant interaction of preview condition and pre-target word frequency on brain activation of the left middle frontal gyrus. These results suggest that fixation-related brain activation provides immediate measures and new perspectives to understand the mechanism of parafoveal processing in self-paced reading.

RastrOS Project: Natural Language Processing contributions to the development of an eye-tracking corpus with predictability norms for Brazilian Portuguese

This article presents RastrOS, a new eye-tracking corpus of eye movement data from university students during silent reading of paragraphs of texts in Brazilian Portuguese (BP). The article shows the potential of the corpus for natural language processing (NLP) using it to evaluate the sentence complexity prediction task in BP and it also focuses on the description of NLP resources and methods developed to create the corpus. Specifically, we present: (i) the method used to select the corpus paragraphs from large corpora, using linguistic metrics and clustering algorithms; (ii) the platform for collecting the Cloze test, which is also responsible for creating the project datasets, and (iii) the hybrid semantic similarity method, based on word embedding models and contextualised word representations, used to generate semantic predictability norms. RastrOS can be downloaded from the open science framework repository with the computational infrastructure mentioned above. Datasets with predictability norms of 393 participants and eye-tracking data of 37 participants are available in the OSF repository for this work (https://osf.io/9jxg3/).

The database of eye-movement measures on words in Chinese reading

Eye movements are one of the most fundamental behaviors during reading. A growing number of Chinese reading studies have used eye-tracking techniques in the last two decades. The accumulated data provide a rich resource that can reflect the complex cognitive mechanisms underlying Chinese reading. This article reports a database of eye-movement measures of words during Chinese sentence reading. The database contains nine eye-movement measures of 8,551 Chinese words obtained from 1,718 participants across 57 Chinese sentence reading experiments. All data were collected in the same experimental environment and from homogenous participants, using the same protocols and parameters. This database enables researchers to test their theoretical or computational hypotheses concerning Chinese reading efficiently using a large number of words. The database can also indicate the processing difficulty of Chinese words during text reading, thus providing a way to control or manipulate the difficulty level of Chinese texts.

CELER: A 365-Participant Corpus of Eye Movements in L1 and L2 English Reading

We present CELER (Corpus of Eye Movements in L1 and L2 English Reading), a broad coverage eye-tracking corpus for English. CELER comprises over 320,000 words, and eye-tracking data from 365 participants. Sixty-nine participants are L1 (first language) speakers, and 296 are L2 (second language) speakers from a wide range of English proficiency levels and five different native language backgrounds. As such, CELER has an order of magnitude more L2 participants than any currently available eye movements dataset with L2 readers. Each participant in CELER reads 156 newswire sentences from the Wall Street Journal (WSJ), in a new experimental design where half of the sentences are shared across participants and half are unique to each participant. We provide analyses that compare L1 and L2 participants with respect to standard reading time measures, as well as the effects of frequency, surprisal, and word length on reading times. These analyses validate the corpus and demonstrate some of its strengths. We envision CELER to enable new types of research on language processing and acquisition, and to facilitate interactions between psycholinguistics and natural language processing (NLP).

Shared computational principles for language processing in humans and deep language models

Departing from traditional linguistic models, advances in deep learning have resulted in a new type of predictive (autoregressive) deep language models (DLMs). Using a self-supervised next-word prediction task, these models generate appropriate linguistic responses in a given context. In the current study, nine participants listened to a 30-min podcast while their brain responses were recorded using electrocorticography (ECoG). We provide empirical evidence that the human brain and autoregressive DLMs share three fundamental computational principles as they process the same natural narrative: (1) both are engaged in continuous next-word prediction before word onset; (2) both match their pre-onset predictions to the incoming word to calculate post-onset surprise; (3) both rely on contextual embeddings to represent words in natural contexts. Together, our findings suggest that autoregressive DLMs provide a new and biologically feasible computational framework for studying the neural basis of language.

Deep language models have revolutionized natural language processing. The paper discovers three computational principles shared between deep language models and the human brain, which can transform our understanding of the neural basis of language.

Semantic feature activation takes time: longer SOA elicits earlier priming effects during reading

While most previous studies of "semantic" priming confound associative and semantic relations, here we use a simple co-occurrence-based approach to examine "pure" semantic priming, while experimentally controlling for associative relations. We define associative relations by the co-occurrence of words in the sentences of a large text corpus. Contextual-semantic feature overlap, in contrast, is defined by the number of common associates that the prime shares with the target. Then we revisit the spreading activation theory and examine whether a long vs. short time available for semantic feature activation leads to early vs. late viewing time effects on the target words of a sentence reading experiment. We independently manipulate contextual-semantic feature overlap of two primes with one target word in sentences of the form pronoun, verb prime, article, adjective prime and target noun, e. g., "She rides the gray elephant." The results showed that long-SOA (verb-noun) overlap reduces early single and first fixation durations of the target noun, and short-SOA (adjective-noun) overlap reduces late go-past durations. This result pattern can be explained by the spreading activation theory: The semantic features of the prime words need some time to become sufficiently active before they can reliably affect target processing. Therefore, the verb can act on the target noun's early eye-movement measures presented three words later, while the adjective is presented immediately prior to the target-thus a difficult adjective-noun semantic integration leads to a late sentence re-examination of the preceding words.

Lexical Frequency and Sentence Context Influence the Brain's Response to Single Words

Typical adults read remarkably quickly. Such fast reading is facilitated by brain processes that are sensitive to both word frequency and contextual constraints. It is debated as to whether these attributes have additive or interactive effects on language processing in the brain. We investigated this issue by analysing existing magnetoencephalography data from 99 participants reading intact and scrambled sentences. Using a cross-validated model comparison scheme, we found that lexical frequency predicted the word-by-word elicited MEG signal in a widespread cortical network, irrespective of sentential context. In contrast, index (ordinal word position) was more strongly encoded in sentence words, in left front-temporal areas. This confirms that frequency influences word processing independently of predictability, and that contextual constraints affect word-by-word brain responses. With a conservative multiple comparisons correction, only the interaction between lexical frequency and surprisal survived, in anterior temporal and frontal cortex, and not between lexical frequency and entropy, nor between lexical frequency and index. However, interestingly, the uncorrected index × frequency interaction revealed an effect in left frontal and temporal cortex that reversed in time and space for intact compared to scrambled sentences. Finally, we provide evidence to suggest that, in sentences, lexical frequency and predictability may independently influence early (<150 ms) and late stages of word processing, but also interact during late stages of word processing (>150-250 ms), thus helping to converge previous contradictory eye-tracking and electrophysiological literature. Current neurocognitive models of reading would benefit from accounting for these differing effects of lexical frequency and predictability on different stages of word processing.

Language Models Explain Word Reading Times Better Than Empirical Predictability

Though there is a strong consensus that word length and frequency are the most important single-word features determining visual-orthographic access to the mental lexicon, there is less agreement as how to best capture syntactic and semantic factors. The traditional approach in cognitive reading research assumes that word predictability from sentence context is best captured by cloze completion probability (CCP) derived from human performance data. We review recent research suggesting that probabilistic language models provide deeper explanations for syntactic and semantic effects than CCP. Then we compare CCP with three probabilistic language models for predicting word viewing times in an English and a German eye tracking sample: (1) Symbolic n-gram models consolidate syntactic and semantic short-range relations by computing the probability of a word to occur, given two preceding words. (2) Topic models rely on subsymbolic representations to capture long-range semantic similarity by word co-occurrence counts in documents. (3) In recurrent neural networks (RNNs), the subsymbolic units are trained to predict the next word, given all preceding words in the sentences. To examine lexical retrieval, these models were used to predict single fixation durations and gaze durations to capture rapidly successful and standard lexical access, and total viewing time to capture late semantic integration. The linear item-level analyses showed greater correlations of all language models with all eye-movement measures than CCP. Then we examined non-linear relations between the different types of predictability and the reading times using generalized additive models. N-gram and RNN probabilities of the present word more consistently predicted reading performance compared with topic models or CCP. For the effects of last-word probability on current-word viewing times, we obtained the best results with n-gram models. Such count-based models seem to best capture short-range access that is still underway when the eyes move on to the subsequent word. The prediction-trained RNN models, in contrast, better predicted early preprocessing of the next word. In sum, our results demonstrate that the different language models account for differential cognitive processes during reading. We discuss these algorithmically concrete blueprints of lexical consolidation as theoretically deep explanations for human reading.

Is AI Augmenting or Substituting Humans?

In this paper, the authors focus on Artificial Intelligence as a tangible technology that is designed to sense, comprehend, act, and learn. There are two manifestations of AI in the medical service: an algorithm that analyzes and interprets the test result and a virtual assistant that communicates the result to the patient. The aim of this paper is to consider how AI can substitute a doctor in measuring human health and how the interaction with virtual assistant impacts one’s visual attention processes. Theoretically, the article refers to the following research strands: Human-Computer Interaction, technology in services, implementation of AI in the medical sector, and behavioral economy. By conducting an eye-tracking experimental study, it is demonstrated that the perception of medical diagnosis does not differ across experimental groups (human vs. AI). However, it is observed that participants exposed to AI-based assistant focused more on button allowing to contact a real doctor.

Eye-tracking as a proxy for coherence and complexity of texts

Reading is a complex cognitive process that involves primary oculomotor function and high-level activities like attention focus and language processing. When we read, our eyes move by primary physiological functions while responding to language-processing demands. In fact, the eyes perform discontinuous twofold movements, namely, successive long jumps (saccades) interposed by small steps (fixations) in which the gaze “scans” confined locations. It is only through the fixations that information is effectively captured for brain processing. Since individuals can express similar as well as entirely different opinions about a given text, it is therefore expected that the form, content and style of a text could induce different eye-movement patterns among people. A question that naturally arises is whether these individuals’ behaviours are correlated, so that eye-tracking while reading can be used as a proxy for text subjective properties. Here we perform a set of eye-tracking experiments with a group of individuals reading different types of texts, including children stories, random word generated texts and excerpts from literature work. In parallel, an extensive Internet survey was conducted for categorizing these texts in terms of their complexity and coherence, considering a large number of individuals selected according to different ages, gender and levels of education. The computational analysis of the fixation maps obtained from the gaze trajectories of the subjects for a given text reveals that the average “magnetization” of the fixation configurations correlates strongly with their complexity observed in the survey. Moreover, we perform a thermodynamic analysis using the Maximum-Entropy Model and find that coherent texts were closer to their corresponding “critical points” than non-coherent ones, as computed from the Pairwise Maximum-Entropy method, suggesting that different texts may induce distinct cohesive reading activities.

Eye movements during text reading align with the rate of speech production

Across languages, the speech signal is characterized by a predominant modulation of the amplitude spectrum between about 4.3 and 5.5 Hz, reflecting the production and processing of linguistic information chunks (syllables and words) every ~200 ms. Interestingly, ~200 ms is also the typical duration of eye fixations during reading. Prompted by this observation, we demonstrate that German readers sample written text at ~5 Hz. A subsequent meta-analysis of 142 studies from 14 languages replicates this result and shows that sampling frequencies vary across languages between 3.9 Hz and 5.2 Hz. This variation systematically depends on the complexity of the writing systems (character-based versus alphabetic systems and orthographic transparency). Finally, we empirically demonstrate a positive correlation between speech spectrum and eye movement sampling in low-skilled non-native readers, with tentative evidence from post hoc analysis suggesting the same relationship in low-skilled native readers. On the basis of this convergent evidence, we propose that during reading, our brain's linguistic processing systems imprint a preferred processing rate-that is, the rate of spoken language production and perception-onto the oculomotor system.

An oscillatory pipelining mechanism supporting previewing during visual exploration and reading

Humans have a remarkable ability to efficiently explore visual scenes and text using eye movements. Humans typically make eye movements (saccades) every ~250 ms. Since saccade initiation and execution take 100 ms, this leaves only ~150 ms to recognize the fixated object (or word) while simultaneously previewing candidates for the next saccade goal. We propose a pipelining mechanism where serial processing occurs within a specific brain region, whereas parallel processing occurs across different brain regions. The mechanism is timed by alpha oscillations that coordinate the saccades, visual recognition, and previewing in the cortical hierarchy. Consequently, the neuronal mechanism supporting natural vision and saccades must be studied in unison to uncover the brain mechanisms supporting visual exploration and reading.

Does diacritics-based lexical disambiguation modulate word frequency, length, and predictability effects? An eye-movements investigation of processing Arabic diacritics

In Arabic, a predominantly consonantal script that features a high incidence of lexical ambiguity (heterophonic homographs), glyph-like marks called diacritics supply vowel information that clarifies how each consonant should be pronounced, and thereby disambiguate the pronunciation of consonantal strings. Diacritics are typically omitted from print except in situations where a particular homograph is not sufficiently disambiguated by the surrounding context. In three experiments we investigated whether the presence of disambiguating diacritics on target homographs modulates word frequency, length, and predictability effects during reading. In all experiments, the subordinate representation of the target homographs was instantiated by the diacritics (in the diacritized conditions), and by the context subsequent to the target homographs. The results replicated the effects of word frequency (Experiment 1), word length (Experiment 2), and predictability (Experiment 3). However, there was no evidence that diacritics-based disambiguation modulated these effects in the current study. Rather, diacritized targets in all experiments attracted longer first pass and later (go past and/or total fixation count) processing. These costs are suggested to be a manifestation of the subordinate bias effect. Furthermore, in all experiments, the diacritics-based disambiguation facilitated later sentence processing, relative to when the diacritics were absent. The reported findings expand existing knowledge about processing of diacritics, their contribution towards lexical ambiguity resolution, and sentence processing.

The Natural Stories corpus: a reading-time corpus of English texts containing rare syntactic constructions

It is now a common practice to compare models of human language processing by comparing how well they predict behavioral and neural measures of processing difficulty, such as reading times, on corpora of rich naturalistic linguistic materials. However, many of these corpora, which are based on naturally-occurring text, do not contain many of the low-frequency syntactic constructions that are often required to distinguish between processing theories. Here we describe a new corpus consisting of English texts edited to contain many low-frequency syntactic constructions while still sounding fluent to native speakers. The corpus is annotated with hand-corrected Penn Treebank-style parse trees and includes self-paced reading time data and aligned audio recordings. We give an overview of the content of the corpus, review recent work using the corpus, and release the data.

The Reading Signatures of Agreement Attraction

The comprehension of subject-verb agreement shows "attraction effects," which reveal that number computations can be derailed by nouns that are grammatically unlicensed to control agreement with a verb. However, previous results are mixed regarding whether attraction affects the processing of grammatical and ungrammatical sentences alike. In a large-sample eye-tracking replication of Lago et al. (2015), we support this "grammaticality asymmetry" by showing that the reading profiles associated with attraction depend on sentence grammaticality. In ungrammatical sentences, attraction affected both fixation durations and regressive eye-movements at the critical disagreeing verb. Meanwhile, both grammatical and ungrammatical sentences showed effects of the attractor noun number prior to the verb, in the first- and second-pass reading of the subject phrase. This contrast suggests that attraction effects in comprehension have at least two different sources: the first reflects verb-triggered processes that operate mainly in ungrammatical sentences. The second source reflects difficulties in the encoding of the subject phrase, which disturb comprehension in both grammatical and ungrammatical sentences.

Neural evidence for lexical parafoveal processing

In spite of the reduced visual acuity, parafoveal information plays an important role in natural reading. However, competing models on reading disagree on whether words are previewed parafoveally at the lexical level. We find neural evidence for lexical parafoveal processing by combining a rapid invisible frequency tagging (RIFT) approach with magnetoencephalography (MEG) and eye-tracking. In a silent reading task, target words are tagged (flickered) subliminally at 60 Hz. The tagging responses measured when fixating on the pre-target word reflect parafoveal processing of the target word. We observe stronger tagging responses during pre-target fixations when followed by low compared with high lexical frequency targets. Moreover, this lexical parafoveal processing is associated with individual reading speed. Our findings suggest that reading unfolds in the fovea and parafovea simultaneously to support fluent reading.

Deafness enhances perceptual span size in Chinese reading: Evidence from a gaze-contingent moving-window paradigm

Using a gaze-contingent moving-window paradigm, we investigated whether/how deafness affects perceptual processing in Chinese reading. Besides the manipulation of window size, word length of sentences used in the experiment was also manipulated to check whether deafness enhanced the word length effect on perceptual span. Significant interactions of window constraints and deafness and a three-way interaction were observed on reading rate. Smaller effects of window constraints for deaf Chinese readers and nonreliable three-way interactions were observed on forward saccade length. This suggests that deaf Chinese readers exhibit a larger perceptual span, and word length affected the span from which information was acquired for comprehension whereas both deafness and word length might have little impact on the span from which information is acquired for oculomotor targeting during natural reading of Chinese.

Individuals with dyslexia use a different visual sampling strategy to read text

Individuals with dyslexia present with reading-related deficits including inaccurate and/or less fluent word recognition and poor decoding abilities. Slow reading speed and worse text comprehension can occur as secondary consequences of these deficits. Reports of visual symptoms such as atypical eye movements during reading gave rise to a search for these deficits' underlying mechanisms. This study sought to replicate established behavioral deficits in reading and cognitive processing speed while investigating their underlying mechanisms in more detail by developing a comprehensive profile of eye movements specific to reading in adult dyslexia. Using a validated standardized reading assessment, our findings confirm a reading speed deficit among adults with dyslexia. We observed different eye movements in readers with dyslexia across numerous eye movement metrics including the duration of a stop (i.e., fixation), the length of jumps (i.e., saccades), and the number of times a reader's eyes expressed a jump atypical for reading. We conclude that individuals with dyslexia visually sample written information in a laborious and more effortful manner that is fundamentally different from those without dyslexia. Our findings suggest a mix of aberrant cognitive linguistic and oculomotor processes being present in adults with dyslexia.

Effects of age on memory for pragmatic implications in advertising

People employ automatic inferential processing when confronting pragmatically implied claims in advertising. However, whether comprehension and memorization of pragmatic implications differ between young and older adults is unclear. In the present study, we used eye-tracking technology to investigate online cognitive processes during reading of misleading advertisements. We found an interaction between age and advertising content, manifested as our older participants generated higher misleading rates in health-related than in health-irrelevant products, whereas this content-bias did not appear in their younger counterparts. Eye movement data further showed that the older adults spent more time processing critical claims for the health-related products than for the health-irrelevant products. Moreover, the correlations between fixation duration on pragmatic implications and misleading rates showed opposite trends in the two groups. The eye-tracking evidence novelly suggests that young and older adults may adopt different information processing strategies to comprehend pragmatic implications in advertising: More reading possibly enhances young adults’ gist memory whereas it facilitates older adults’ verbatim memory instead.

Morphosyntactic but not lexical corpus-based probabilities can substitute for cloze probabilities in reading experiments

During reading or listening, people can generate predictions about the lexical and morphosyntactic properties of upcoming input based on available context. Psycholinguistic experiments that study predictability or control for it conventionally rely on a human-based approach and estimate predictability via the cloze task. Our study investigated an alternative corpus-based approach for estimating predictability via language predictability models. We obtained cloze and corpus-based probabilities for all words in 144 Russian sentences, correlated the two measures, and found a strong correlation between them. Importantly, we estimated how much variance in eye movements registered while reading the same sentences was explained by each of the two probabilities and whether the two probabilities explain the same variance. Along with lexical predictability (the activation of a particular word form), we analyzed morphosyntactic predictability (the activation of morphological features of words) and its effect on reading times over and above lexical predictability. We found that for predicting reading times, cloze and corpus-based measures of both lexical and morphosyntactic predictability explained the same amount of variance. However, cloze and corpus-based lexical probabilities both independently contributed to a better model fit, whereas for morphosyntactic probabilities, the contributions of cloze and corpus-based measures were interchangeable. Therefore, morphosyntactic but not lexical corpus-based probabilities can substitute for cloze probabilities in reading experiments. Our results also indicate that in languages with rich inflectional morphology, such as Russian, when people engage in prediction, they are much more successful in predicting isolated morphosyntactic features than predicting the particular lexeme and its full morphosyntactic markup.

Do standard optometric measures predict binocular coordination during reading?

In reading, binocular eye movements are required for optimal visual processing and thus, in case of asthenopia or reading problems, standard orthoptic and optometric routines check individual binocular vision by a variety of tests. The present study therefore examines the predictive value of such standard measures of heterophoria, accommodative and vergence facility, AC/A-ratio, NPC and symptoms for binocular coordination parameters during read-ing. Binocular eye movements were recorded (EyeLink II) for 65 volunteers during a typical reading task and linear regression analyses related all parameters of binocular coordination to all above-mentioned optometric measures: while saccade disconjugacy was weakly pre-dicted by vergence facility (15% explained variance), vergence facility, AC/A and symp-toms scores predicted vergence drift (31%). Heterophoria, vergence facility and NPC ex-plained 31% of fixation disparity and first fixation duration showed minor relations to symp-toms (18%). In sum, we found only weak to moderate relationships, with expected, selective associations: dynamic parameter related to optometric tests addressing vergence dynamics, whereas the static parameter (fixation disparity) related mainly to heterophoria. Most sur-prisingly, symptoms were only loosely related to vergence drift and fixation duration, re-flecting associations to a dynamic aspect of binocular eye movements in reading and poten-tially non-specific, overall but slight reading deficiency. Thus, the efficiency of optometric tests to predict binocular coordination during reading was low - questioning a simple, straightforward extrapolation of such test results to an overlearned, complex task.

Fixation durations in natural scene viewing are guided by peripheral scene content

Fixation durations provide insights into processing demands. We investigated factors controlling fixation durations during scene viewing in two experiments. In Experiment 1, we tested the degree to which fixation durations adapt to global scene processing difficulty by manipulating the contrast (from original contrast to isoluminant) and saturation (original vs. grayscale) of the entire scene. We observed longer fixation durations for lower levels of contrast, and longer fixation durations for grayscale than for color scenes. Thus fixation durations were globally slowed as visual information became more and more degraded, making scene processing increasingly more difficult. In Experiment 2, we investigated two possible sources for this slow-down. We used "checkerboard" stimuli in which unmodified patches alternated with patches from which luminance information had been removed (isoluminant patches). Fixation durations showed an inverted immediacy effect (longer, rather than shorter, fixation durations on unmodified patches) along with a parafoveal-on-foveal effect (shorter fixation durations, when an unmodified patch was fixated next). This effect was stronger when the currently fixated patch was isoluminant as opposed to unmodified. Our results suggest that peripheral scene information substantially affects fixation durations and are consistent with the notion of competition among the current and potential future fixation locations.

Cloze enough? Hemodynamic effects of predictive processing during natural reading

Evidence accrues that readers form multiple hypotheses about upcoming words. The present study investigated the hemodynamic effects of predictive processing during natural reading by means of combining fMRI and eye movement recordings. In particular, we investigated the neural and behavioral correlates of precision-weighted prediction errors, which are thought to be indicative of subsequent belief updating. Participants silently read sentences in which we manipulated the cloze probability and the semantic congruency of the final word that served as an index for precision and prediction error respectively. With respect to the neural correlates, our findings indicate an enhanced activation within the left inferior frontal and middle temporal gyrus suggesting an effect of precision on prediction update in higher (lexico-)semantic levels. Despite being evident at the neural level, we did not observe any evidence that this mechanism resulted in disproportionate reading times on participants' eye movements. The results speak against discrete predictions, but favor the notion that multiple words are activated in parallel during reading.

Recognition times for 62 thousand English words: Data from the English Crowdsourcing Project

We present a new dataset of English word recognition times for a total of 62 thousand words, called the English Crowdsourcing Project. The data were collected via an internet vocabulary test in which more than one million people participated. The present dataset is limited to native English speakers. Participants were asked to indicate which words they knew. Their response times were registered, although at no point were the participants asked to respond as quickly as possible. Still, the response times correlate around .75 with the response times of the English Lexicon Project for the shared words. Also, the results of virtual experiments indicate that the new response times are a valid addition to the English Lexicon Project. This not only means that we have useful response times for some 35 thousand extra words, but we now also have data on differences in response latencies as a function of education and age.

Distraction by auditory novelty during reading: Evidence for disruption in saccade planning, but not saccade execution

Novel or unexpected sounds that deviate from an otherwise repetitive sequence of the same sound cause behavioural distraction. Recent work has suggested that distraction also occurs during reading as fixation durations increased when a deviant sound was presented at the fixation onset of words. The present study tested the hypothesis that this increase in fixation durations occurs due to saccadic inhibition. This was done by manipulating the temporal onset of sounds relative to the fixation onset of words in the text. If novel sounds cause saccadic inhibition, they should be more distracting when presented during the second half of fixations when saccade programming usually takes place. Participants read single sentences and heard a 120 ms sound when they fixated five target words in the sentence. On most occasions (p = .9), the same sine wave tone was presented (“standard”), while on the remaining occasions (p = .1) a new sound was presented (“novel”). Critically, sounds were played, on average, either during the first half of the fixation (0 ms delay) or during the second half of the fixation (120 ms delay). Consistent with the saccadic inhibition hypothesis (SIH), novel sounds led to longer fixation durations in the 120 ms compared to the 0 ms delay condition. However, novel sounds did not generally influence the execution of the subsequent saccade. These results suggest that unexpected sounds have a rapid influence on saccade planning, but not saccade execution.

Atomoxetine modulates the contribution of low-level signals during free viewing of natural images in rhesus monkeys

Visuo-spatial attentional orienting is fundamental to selectively process behaviorally relevant information, depending on both low-level visual attributes of stimuli in the environment and higher-level factors, such as goals, expectations and prior knowledge. Growing evidence suggests an impact of the locus-cœruleus-norepinephrine (LC-NE) system in attentional orienting that depends on taskcontext. Nonetheless, most of previous studies used visual displays encompassing a target and various distractors, often preceded by cues to orient the attentional focus. This emphasizes the contribution of goal-driven processes, at the expense of other factors related to the stimulus content. Here, we aimed to determine the impact of NE on attentional orienting in more naturalistic conditions, using complex images and without any explicit task manipulation. We tested the effects of atomoxetine (ATX) injections, a NE reuptake inhibitor, on four monkeys during free viewing of images belonging to three categories: landscapes, monkey faces and scrambled images. Analyses of the gaze exploration patterns revealed, first, that the monkeys spent more time on each fixation under ATX compared to the control condition, regard less of the image content. Second, we found that, depending on the image content, ATX modulated the impact of low-level visual salience on attentional orienting. This effect correlated with the effect of ATX on the number and duration of fixations. Taken together, our results demonstrate that ATX adjusts the contribution of salience on attentional orienting depending on the image content, indicative of its role in balancing the role of stimulus-driven and top-down control during free viewing of complex stimuli.

The Information Gathering Framework - a Cognitive Model of Regressive Eye Movements during Reading

In this article we present a new eye movement control framework that describes the interaction between fixation durations and regressive saccades during reading: The Information Gathering Framework (IGF). Based on the FC model proposed by Bicknell and Levy [15], the basic idea of the IGF is that a confidence level for each word is computed while being monitored by three independent thresholds. These thresholds shape eye movement behavior by increasing fixation duration, triggering a regression, or guiding regression target selection. In this way, the IGF does not only account for regressive eye movements but also provides a framework able to model eye movement control during reading across different scenarios. Importantly, within the IGF it is assumed that two different types of regressive eye movements exist which differ with regard to their releases (integrations difficulties vs. missing evidence) but also with regard to their time course. We tested the predictions of the IGF by re-analyzing an experiment of Weiss et al. [57] and found, inter alia, clear evidence for shorter fixation durations before regressive saccades relative to progressive saccades, with the exception of the last region. This clearly supports the assumptions of the IGF. In addition, we found evidence that there exists a window of about 15-20 characters to the left of the current fixation that plays an important role in target selection, probably indicating the perceptual span during a regressive saccade.