Eye movements as time-series random variables: A stochastic model of eye movement control in reading

Understanding the Role of Eye Movement Pattern and Consistency in Isolated English Word Reading Through Hidden Markov Modeling

In isolated English word reading, readers have the optimal performance when their initial eye fixation is directed to the area between the beginning and word center, that is, the optimal viewing position (OVP). Thus, how well readers voluntarily direct eye gaze to this OVP during isolated word reading may be associated with reading performance. Using Eye Movement analysis with Hidden Markov Models, we discovered two representative eye movement patterns during lexical decisions through clustering, which focused at the OVP and the word center, respectively. Higher eye movement similarity to the OVP-focusing pattern predicted faster lexical decision time in addition to cognitive abilities and lexical knowledge. However, the OVP-focusing pattern was associated with longer isolated single letter naming time, suggesting conflicting visual abilities required for identifying isolated letters and multi-letter words. In contrast, in both word and pseudoword naming, although clustering did not reveal an OVP-focused pattern, higher consistency of the first fixation as measured in entropy predicted faster naming time in addition to cognitive abilities and lexical knowledge. Thus, developing a consistent eye movement pattern focusing on the OVP is essential for word orthographic processing and reading fluency. This finding has important implications for interventions for reading difficulties.

Modelling the eye movements of dyslexic children during reading as a continuous time random walk

The study of eye movements during reading is considered a valuable tool for understanding the underlying cognitive processes and for its ability to detect alterations that could be associated with neurocognitive deficiencies or visual conditions. During reading, the gaze moves from one position to the next on the text performing a saccade-fixation sequence. This dynamics resembles processes usually described as continuous time random walk, where the jumps are the saccadic movements and waiting times are the duration of fixations. The time between jumps (intersaccadic time) consists of stochastic waiting time and flight time, which is a function of the jump length (the amplitude of the saccade). This motivates the present proposal of a model of eye movements during reading in the framework of the intermittent random walk but considering the time between jumps as a combined stochastic-deterministic process. The parameters used in this model were obtained from records of eye movements of children with dyslexia and typically developed for children performing a reading task. The jump lengths arise from the characteristics of the selected text. The time required for the flights was obtained based on a previously proposed model. Synthetic signals were generated and compared with actual eye movement signals in a complexity-entropy plane.

Intra-individual variation in the songs of humpback whales suggests they are sonically searching for conspecifics

Observations of animals' vocal actions can provide important clues about how they communicate and about how they perceive and react to changing situations. Here, analyses of consecutive songs produced by singing humpback whales recorded off the coast of Hawaii revealed that singers constantly vary the acoustic qualities of their songs within prolonged song sessions. Unlike the progressive changes in song structure that singing humpback whales make across months and years, intra-individual acoustic variations within song sessions appear to be largely stochastic. Additionally, four sequentially produced song components (or "themes") were each found to vary in unique ways. The most extensively used theme was highly variable in overall duration within and across song sessions, but varied relatively little in frequency content. In contrast, the remaining themes varied greatly in frequency content, but showed less variation in duration. Analyses of variations in the amount of time singers spent producing the four themes suggest that the mechanisms that determine when singers transition between themes may be comparable to those that control when terrestrial animals move their eyes to fixate on different positions as they examine visual scenes. The dynamic changes that individual whales make to songs within song sessions are counterproductive if songs serve mainly to provide conspecifics with indications of a singer's fitness. Instead, within-session changes to the acoustic features of songs may serve to enhance a singer's capacity to echoically detect, localize, and track conspecifics from long distances.

Continuous Non-Invasive Eye Tracking In Intensive Care

Delirium, an acute confusional state, is a common occurrence in Intensive Care Units (ICUs). Patients who develop delirium have globally worse outcomes than those who do not and thus the diagnosis of delirium is of importance. Current diagnostic methods have several limitations leading to the suggestion of eye-tracking for its diagnosis through in-attention. To ascertain the requirements for an eye-tracking system in an adult ICU, measurements were carried out at Chelsea & Westminster Hospital NHS Foundation Trust. Clinical criteria guided empirical requirements of invasiveness and calibration methods while accuracy and precision were measured. A non-invasive system was then developed utilising a patient-facing RGB camera and a scene-facing RGBD camera. The system's performance was measured in a replicated laboratory environment with healthy volunteers revealing an accuracy and precision that outperforms what is required while simultaneously being non-invasive and calibration-free The system was then deployed as part of CONfuSED, a clinical feasibility study where we report aggregated data from 5 patients as well as the acceptability of the system to bedside nursing staff. To the best of our knowledge, the system is the first eye-tracking systems to be deployed in an ICU for delirium monitoring.

Visual Scanpath Prediction Using IOR-ROI Recurrent Mixture Density Network

A visual scanpath represents the human eye movements when scanning the visual field for acquiring and receiving visual information. Predicting visual scanpaths when a certain stimulus is presented plays an important role in modeling overt human visual attention and search behavior. In this paper, we presented an 'Inhibition of Return - Region of Interest' (IOR-ROI) recurrent mixture density network based framework learning to produce human-like visual scanpaths under task-free viewing conditions. The proposed model simultaneously predicts a sequence of ordered fixation positions and their corresponding fixation durations. Our model integrates bottom-up features and semantic features extracted by convolutional neural networks. Then the integrated feature maps are fed into the IOR-ROI Long Short-Term Memory (LSTM) which is the core component of the proposed model. The IOR-ROI LSTM is a dual LSTM unit, i.e., the IOR-LSTM and the ROI-LSTM, capturing IOR dynamics and gaze shift behavior simultaneously. IOR-LSTM simulates the visual working memory to adaptively maintain and update visual information regarding previously fixated regions. ROI-LSTM is responsible for predicting the next possible ROIs given the spatially inhibited image feature maps on the feature-wise basis. Fixation duration is predicted by a regression neural network given the viewing history and image feature maps corresponding to currently fixated ROI. Considering the eye movement pattern variations among subjects, a mixture density network is adopted to model the next fixation distribution as Gaussian mixtures and the fixation duration is also modeled using Gaussian distribution. Our model is evaluated on the OSIE and MIT low resolution eye-tracking datasets and experimental results indicate that the proposed method can achieve superior performance in predicting visual scanpaths. The code will be publicly available on URL: https://github.com/sunwj/scanpath.

Human visual search behaviour is far from ideal

Evolutionary pressures have made foraging behaviours highly efficient in many species. Eye movements during search present a useful instance of foraging behaviour in humans. We tested the efficiency of eye movements during search using homogeneous and heterogeneous arrays of line segments. The search target is visible in the periphery on the homogeneous array, but requires central vision to be detected on the heterogeneous array. For a compound search array that is heterogeneous on one side and homogeneous on the other, eye movements should be directed only to the heterogeneous side. Instead, participants made many fixations on the homogeneous side. By comparing search of compound arrays to an estimate of search performance based on uniform arrays, we isolate two contributions to search inefficiency. First, participants make superfluous fixations, sacrificing speed for a perceived (but not actual) gain in response certainty. Second, participants fixate the homogeneous side even more frequently than predicted by inefficient search of uniform arrays, suggesting they also fail to direct fixations to locations that yield the most new information.

Discrete states of attention during active visual fixation revealed by Markovian analysis of the time series of intrusive saccades

The frequency of intrusive saccades during maintenance of active visual fixation has been used as a measure of sustained visual attention in studies of healthy subjects as well as of neuropsychiatric patient populations. In this study, the mechanism that generates intrusive saccades during active visual fixation was investigated in a population of young healthy men performing three sustained fixation tasks (fixation to a visual target, fixation to a visual target with visual distracters, and fixation straight ahead in the dark). Markov Chain modeling of inter-saccade intervals (ISIs) was utilized. First- and second-order Markov modeling provided indications for the existence of a non-random pattern in the production of intrusive saccades. Accordingly, the system of intrusive saccade generation may operate in two "attractor" states, one in which intrusive saccades occur at short consecutive ISIs and another in which intrusive saccades occur at long consecutive ISIs. These states might correspond to two distinct states of the attention system, one of low focused - high distractibility and another of high focused - low distractibility, such as those proposed in the adaptive gain theory for the control of attention by the noradrenergic system in the brain. To the authors knowledge, this is the first time that Markov Chain modeling has been applied to the analysis of the ISIs of intrusive saccades.

A geometric method for computing ocular kinematics and classifying gaze events using monocular remote eye tracking in a robotic environment

BACKGROUND

Robotic and virtual-reality systems offer tremendous potential for improving assessment and rehabilitation of neurological disorders affecting the upper extremity. A key feature of these systems is that visual stimuli are often presented within the same workspace as the hands (i.e., peripersonal space). Integrating video-based remote eye tracking with robotic and virtual-reality systems can provide an additional tool for investigating how cognitive processes influence visuomotor learning and rehabilitation of the upper extremity. However, remote eye tracking systems typically compute ocular kinematics by assuming eye movements are made in a plane with constant depth (e.g. frontal plane). When visual stimuli are presented at variable depths (e.g. transverse plane), eye movements have a vergence component that may influence reliable detection of gaze events (fixations, smooth pursuits and saccades). To our knowledge, there are no available methods to classify gaze events in the transverse plane for monocular remote eye tracking systems. Here we present a geometrical method to compute ocular kinematics from a monocular remote eye tracking system when visual stimuli are presented in the transverse plane. We then use the obtained kinematics to compute velocity-based thresholds that allow us to accurately identify onsets and offsets of fixations, saccades and smooth pursuits. Finally, we validate our algorithm by comparing the gaze events computed by the algorithm with those obtained from the eye-tracking software and manual digitization.

RESULTS

Within the transverse plane, our algorithm reliably differentiates saccades from fixations (static visual stimuli) and smooth pursuits from saccades and fixations when visual stimuli are dynamic.

CONCLUSIONS

The proposed methods provide advancements for examining eye movements in robotic and virtual-reality systems. Our methods can also be used with other video-based or tablet-based systems in which eye movements are performed in a peripersonal plane with variable depth.

A further examination of the lexical-processing stages hypothesized by the E-Z Reader model

Participants' eye movements were monitored while they read sentences in which high- and low-frequency target words were presented normally (i.e., the normal condition) or with either reduced stimulus quality (i.e., the faint condition) or alternating lower- and uppercase letters (i.e., the case-alternated condition). Both the stimulus quality and case alternation manipulations interacted with word frequency for the gaze duration measure, such that the magnitude of word frequency effects was increased relative to the normal condition. However, stimulus quality (but not case alternation) interacted with word frequency for the early fixation time measures (i.e., first fixation, single fixation), whereas case alternation (but not stimulus quality) interacted with word frequency for the later fixation time measures (i.e., total time, go-past time). We interpret this pattern of results as evidence that stimulus quality influences an earlier stage of lexical processing than does case alternation, and we discuss the implications of our results for models of eye movement control during reading.

Ecological sampling of gaze shifts

Visual attention guides our gaze to relevant parts of the viewed scene, yet the moment-to-moment relocation of gaze can be different among observers even though the same locations are taken into account. Surprisingly, the variability of eye movements has been so far overlooked by the great majority of computational models of visual attention. In this paper we present the ecological sampling model, a stochastic model of eye guidance explaining such variability. The gaze shift mechanism is conceived as an active random sampling that the foraging eye carries out upon the visual landscape, under the constraints set by the observable features and the global complexity of the landscape. By drawing on results reported in the foraging literature, the actual gaze relocation is eventually driven by a stochastic differential equation whose noise source is sampled from a mixture of α-stable distributions. This way, the sampling strategy proposed here allows to mimic a fundamental property of the eye guidance mechanism: where we choose to look next at any given moment in time, it is not completely deterministic, but neither is it completely random To show that the model yields gaze shift motor behaviors that exhibit statistics similar to those displayed by human observers, we compare simulation outputs with those obtained from eye-tracked subjects while viewing complex dynamic scenes.

Using E-Z Reader to examine the concurrent development of eye-movement control and reading skill

Compared to skilled adult readers, children typically make more fixations that are longer in duration, shorter saccades, and more regressions, thus reading more slowly (Blythe & Joseph, 2011). Recent attempts to understand the reasons for these differences have discovered some similarities (e.g., children and adults target their saccades similarly; Joseph, Liversedge, Blythe, White, & Rayner, 2009) and some differences (e.g., children's fixation durations are more affected by lexical variables; Blythe, Liversedge, Joseph, White, & Rayner, 2009) that have yet to be explained. In this article, the E-Z Reader model of eye-movement control in reading (Reichle, 2011; Reichle, Pollatsek, Fisher, & Rayner, 1998) is used to simulate various eye-movement phenomena in adults vs. children in order to evaluate hypotheses about the concurrent development of reading skill and eye-movement behavior. These simulations suggest that the primary difference between children and adults is their rate of lexical processing, and that different rates of (post-lexical) language processing may also contribute to some phenomena (e.g., children's slower detection of semantic anomalies; Joseph et al., 2008). The theoretical implications of this hypothesis are discussed, including possible alternative accounts of these developmental changes, how reading skill and eye movements change across the entire lifespan (e.g., college-aged vs. older readers), and individual differences in reading ability.

Using reinforcement learning to examine dynamic attention allocation during reading

A fundamental question in reading research concerns whether attention is allocated strictly serially, supporting lexical processing of one word at a time, or in parallel, supporting concurrent lexical processing of two or more words (Reichle, Liversedge, Pollatsek, & Rayner, 2009). The origins of this debate are reviewed. We then report three simulations to address this question using artificial reading agents (Liu & Reichle, 2010; Reichle & Laurent, 2006) that learn to dynamically allocate attention to 1-4 words to "read" as efficiently as possible. These simulation results indicate that the agents strongly preferred serial word processing, although they occasionally attended to more than one word concurrently. The reason for this preference is discussed, along with implications for the debate about how humans allocate attention during reading.

Direct lexical control of eye movements in reading: evidence from a survival analysis of fixation durations

Participants' eye movements were monitored in an experiment that manipulated the frequency of target words (high vs. low) as well as their availability for parafoveal processing during fixations on the pre-target word (valid vs. invalid preview). The influence of the word-frequency by preview validity manipulation on the distributions of first fixation duration was examined by using ex-Gaussian fitting as well as a novel survival analysis technique which provided precise estimates of the timing of the first discernible influence of word frequency on first fixation duration. Using this technique, we found a significant influence of word frequency on fixation duration in normal reading (valid preview) as early as 145ms from the start of fixation. We also demonstrated an equally rapid non-lexical influence on first fixation duration as a function of initial landing position (location) on target words. The time-course of frequency effects, but not location effects was strongly influenced by preview validity, demonstrating the crucial role of parafoveal processing in enabling direct lexical control of reading fixation times. Implications for models of eye-movement control are discussed.

Is there a common control mechanism for anti-saccades and reading eye movements? Evidence from distributional analyses

In the saccadic literature, the voluntary control of eye movement involves inhibiting automatic saccadic plans. In contrast, the dominant view in reading is that linguistic processes trigger saccade planning. The present study explores the possibility of a common control mechanism, in which cognitively driven responses compete to inhibit automatic, perceptually driven saccade plans. A probabilistic model is developed to account for empirical distributions of saccadic response time in anti-saccade tasks (Studies 1 and 2) and fixation duration in reading and reading-like tasks (Studies 3 and 4). In all cases the distributions can be decomposed into a perceptually based component and a component sensitive to cognitive demands. Parametric similarities among the models strongly suggest a shared cognitive control mechanism between reading and other voluntary saccadic tasks.

Statistical modelling of gaze behaviour as categorical time series: what you should watch to save soccer penalties

Previous research on gaze behaviour in sport has typically reported summary fixation statistics thereby largely ignoring the temporal sequencing of gaze. In the present study on penalty kicking in soccer, our aim was to apply a Markov chain modelling method to eye movement data obtained from goalkeepers. Building on the discrete analysis of gaze employed by Dicks et al. (Atten Percept Psychophys 72(3):706-720, 2010b), we wanted to statistically model the relative probabilities of the goalkeeper's gaze being directed to different locations throughout the penalty taker's approach (Dicks et al. in Atten Percept Psychophys 72(3):706-720, 2010b). Examination of gaze behaviours under in situ and video-simulation task constraints reveals differences in information pickup for perception and action (Attention, Perception and Psychophysics 72(3), 706-720). The probabilities of fixating anatomical locations of the penalty taker were high under simulated movement response conditions. In contrast, when actually required to intercept kicks, the goalkeepers initially favoured watching the penalty taker's head but then rapidly shifted focus directly to the ball for approximately the final second prior to foot-ball contact. The increased spatio-temporal demands of in situ interceptive actions over laboratory-based simulated actions lead to different visual search strategies being used. When eye movement data are modelled as time series, it is possible to discern subtle but important behavioural characteristics that are less apparent with discrete summary statistics alone.

Models of the reading process

Reading is a complex skill involving the orchestration of a number of components. Researchers often talk about a 'model of reading' when talking about only one aspect of the reading process (e.g., models of word identification are often referred to as 'models of reading'). Here, we review prominent models that are designed to account for (1) word identification, (2) syntactic parsing, (3) discourse representations, and (4) how certain aspects of language processing (e.g., word identification), in conjunction with other constraints (e.g., limited visual acuity, saccadic error) guide readers' eyes. Unfortunately, it is the case that these various models addressing specific aspects of the reading process seldom make contact with models dealing with other aspects of reading. Thus, for example, the models of word identification seldom make contact with models of eye-movement control, and vice versa. Although this may be unfortunate in some ways, it is quite understandable in other ways because reading itself is a very complex process. We discuss prototypical models of aspects of the reading process in the order mentioned above. We do not review all possible models but rather focus on those we view as being representative and most highly recognized. WIREs Cogn Sci 2010 1 787-799 This article is categorized under: Linguistics > Computational Models of Language Psychology > Language.

Eye Movements During Mindless Reading

Mindless reading occurs when the eyes continue moving across the page even though the mind is thinking about something unrelated to the text. Despite how commonly it occurs, very little is known about mindless reading. The present experiment examined eye movements during mindless reading. Comparisons of fixation-duration measures collected during intervals of normal reading and intervals of mindless reading indicate that fixations during the latter were longer and less affected by lexical and linguistic variables than fixations during the former. Also, eye movements immediately preceding self-caught mind wandering were especially erratic. These results suggest that the cognitive processes that guide eye movements during normal reading are not engaged during mindless reading. We discuss the implications of these findings for theories of eye movement control in reading, for the distinction between experiential awareness and meta-awareness, and for reading comprehension.

Oculomotor and linguistic determinants of reading development: a longitudinal study

We longitudinally assessed the development of oculomotor control in reading from second to fourth grade by having children read sentences with embedded target words of varying length and frequency. Additionally, participants completed oculomotor (pro-/anti-saccades) and linguistic tasks (word/picture naming), the latter containing the same item material as the reading task. Results revealed a 36% increase of reading efficiency. Younger readers utilized a global refixation strategy to gain more time for word decoding. Linguistic rather than oculomotor skills determined the development of reading abilities, although naming latencies of fourth graders did not reliably reflect word decoding processes in normal sentence reading.

The 35th Sir Frederick Bartlett Lecture: Eye movements and attention in reading, scene perception, and visual search

Eye movements are now widely used to investigate cognitive processes during reading, scene perception, and visual search. In this article, research on the following topics is reviewed with respect to reading: (a) the perceptual span (or span of effective vision), (b) preview benefit, (c) eye movement control, and (d) models of eye movements. Related issues with respect to eye movements during scene perception and visual search are also reviewed. It is argued that research on eye movements during reading has been somewhat advanced over research on eye movements in scene perception and visual search and that some of the paradigms developed to study reading should be more widely adopted in the study of scene perception and visual search. Research dealing with “real-world” tasks and research utilizing the visual-world paradigm are also briefly discussed.

Visuomotor characterization of eye movements in a drawing task

Understanding visuomotor coordination requires the study of tasks that engage mechanisms for the integration of visual and motor information; in this paper we choose a paradigmatic yet little studied example of such a task, namely realistic drawing. On the one hand, our data indicate that the motor task has little influence on which regions of the image are overall most likely to be fixated: salient features are fixated most often. Viceversa, the effect of motor constraints is revealed in the temporal aspect of the scanpaths: (1) subjects direct their gaze to an object mostly when they are acting upon (drawing) it; and (2) in support of graphically continuous hand movements, scanpaths resemble edge-following patterns along image contours. For a better understanding of such properties, a computational model is proposed in the form of a novel kind of Dynamic Bayesian Network, and simulation results are compared with human eye-hand data.

Lexical Predictability Exerts Robust Effects on Fixation Duration, but not on Initial Landing Position During Reading

An eye movement experiment was conducted to examine effects of local lexical predictability on fixation durations and fixation locations during sentence reading. In the high-predictability condition, a verb strongly constrained the lexical identity of the following word, while in the low-predictability condition the target word could not be predicted on the basis of the verb. The results showed that first fixation and gaze duration on the target noun were reliably shorter in the high-predictability than in the low-predictability condition. However, initial fixation location was not affected by lexical predictability. As regards eye guidance in reading, the present study indicates that local lexical predictability influences when decisions but not where the initial fixation lands in a word.