Fixational eye movements and their associated evoked potentials during natural vision are altered in schizophrenia

Background

Visual exploration is abnormal in schizophrenia; however, few studies have investigated the physiological responses during selecting objectives in more ecological scenarios. This study aimed to demonstrate that people with schizophrenia have difficulties observing the prominent elements of an image due to a deficit mechanism of sensory modulation (active sensing) during natural vision.

Methods

An electroencephalogram recording with eye tracking data was collected on 18 healthy individuals and 18 people affected by schizophrenia while looking at natural images. These had a prominent color element and blinking produced by changes in image luminance.

Results

We found fewer fixations when all images were scanned, late focus on prominent image areas, decreased amplitude in the eye-fixation-related potential, and decreased intertrial coherence in the SCZ group.

Conclusions

The decrease in the visual attention response evoked by the prominence of visual stimuli in patients affected by schizophrenia is generated by a reduction in endogenous attention mechanisms to initiate and maintain visual exploration. Further work is required to explain the relationship of this decrease with clinical indicators.

Saccade size predicts onset time of object processing during visual search of an open world virtual environment

OBJECTIVE

To date the vast majority of research in the visual neurosciences have been forced to adopt a highly constrained perspective of the vision system in which stimuli are processed in an open-loop reactive fashion (i.e., abrupt stimulus presentation followed by an evoked neural response). While such constraints enable high construct validity for neuroscientific investigation, the primary outcomes have been a reductionistic approach to isolate the component processes of visual perception. In electrophysiology, of the many neural processes studied under this rubric, the most well-known is, arguably, the P300 evoked response. There is, however, relatively little known about the real-world corollary of this component in free-viewing paradigms where visual stimuli are connected to neural function in a closed-loop. While growing evidence suggests that neural activity analogous to the P300 does occur in such paradigms, it is an open question when this response occurs and what behavioral or environmental factors could be used to isolate this component.

APPROACH

The current work uses convolutional networks to decode neural signals during a free-viewing visual search task in a closed-loop paradigm within an open-world virtual environment. From the decoded activity we construct fixation-locked response profiles that enable estimations of the variable latency of any P300 analogue around the moment of fixation. We then use these estimates to investigate which factors best reduce variable latency and, thus, predict the onset time of the response. We consider measurable, search-related factors encompassing top-down (i.e., goal driven) and bottom-up (i.e., stimulus driven) processes, such as fixation duration and salience. We also consider saccade size as an intermediate factor reflecting the integration of these two systems.

MAIN RESULTS

The results show that of these factors only saccade size reliably determines the onset time of P300 analogous activity for this task. Specifically, we find that for large saccades the variability in response onset is small enough to enable analysis using traditional ensemble averaging methods.

SIGNIFICANCE

The results show that P300 analogous activity does occur during closed-loop, free-viewing visual search while highlighting distinct differences between the open-loop version of this response and its real-world analogue. The results also further establish saccades, and saccade size, as a key factor in real-world visual processing.

Objective assessment of visual attention in orthognathic surgery training based on eye tracking

The aim of this study was to investigate the differences in visual attention between novices and orthognathic experts, as well as to provide evidence for use in developing and optimizing training strategies for orthognathic surgery. Novice and orthognathic experts were recruited, and their distributions of visual attention were monitored via an eye-tracking device while they watched orthognathic surgery videos. The percentages of visual fixation duration devoted to the areas of interest - surgical objects, instruments controlled by the main surgeon, and instruments controlled by the assistants - in each orthognathic surgery section were analyzed and compared between the two groups using repeated-measures factorial analysis of variance (ANOVA). In total, there were 18 participants, comprising both novices (n = 9) and experts (n = 9). For all sections of orthognathic surgery, the percentage of fixation duration on surgical objects was significantly higher for the novices than for the experts (p = 0.031, p = 0.005, p = 0.026, p = 0.047, p = 0.047, p = 0.031, p = 0.027, p = 0.034, p = 0.008, and p = 0.016). During the maxillary segment separation as part of Le Fort I osteotomy and the splitting of the mandible as part of bilateral sagittal split osteotomy, the novices also had a higher percentage of fixation duration on the instruments controlled by the main surgeon, as compared with the experts (p = 0.007 and p = 0.048, respectively). Novices invested great cognitive effort into the surgical objects in each section of orthognathic surgery, including the instruments controlled by the main surgeon in the maxillary segment separation and the splitting of the mandible. Strengthening this aspect of instruction could help novices reduce their cognitive load and achieve mastery more efficiently.

A computational modeling approach to investigating mind wandering-related adjustments to gaze behavior during scene viewing

Research on gaze control has long shown that increased visual-cognitive processing demands in scene viewing are associated with longer fixation durations. More recently, though, longer durations have also been linked to mind wandering, a perceptually decoupled state of attention marked by decreased visual-cognitive processing. Toward better understanding the relationship between fixation durations and visual-cognitive processing, we ran simulations using an established random-walk model for saccade timing and programming and assessed which model parameters best predicted modulations in fixation durations associated with mind wandering compared to attentive viewing. Mind wandering-related fixation durations were best described as an increase in the variability of the fixation-generating process, leading to more variable-sometimes very long-durations. In contrast, past research showed that increased processing demands increased the mean duration of the fixation-generating process. The findings thus illustrate that mind wandering and processing demands modulate fixation durations through different mechanisms in scene viewing. This suggests that processing demands cannot be inferred from changes in fixation durations without understanding the underlying mechanism by which these changes were generated.

The influence of resource-gaining capacity on mate preferences: an eye tracking study

To investigate whether resource-gaining capacity influences mate preferences, explicit (self-report data) and implicit tasks (eye tracking data) were used to explore whether individuals' resource-gaining capacity influences mate preferences and whether there are sex differences in mate preferences under two different conditions (short-term and long-term strategies). A total of 59 college students completed a questionnaire collecting basic demographic information, the Resource-Gaining Capacity Scale and the two above tasks. The results showed that (1) in the short-term mating, individuals with higher resource-gaining capacity paid more attention to "good parent" than those with lower resource-gaining capacity, while individuals with lower resource-gaining capacity preferred "good provider" than those with higher resource-gaining capacity. (2) In the long-term mating, women valued "good provider" traits more than men, and they paid more attention to "good parent" traits than men in the short-term. In addition, no matter in the short-term or the long-term mating, men placed more value on "good genes" traits than women. (3) Compared with long-term mating, individuals of both sexes had preferences based on "good genes" in short-term mating, while they had preferences based on "good parent" and "good provider" in long-term mating compared with short-term mating. (4) Regarding explicit mate selection, "good parent" traits were most preferred by the participants, while the implicit eye tracking data indicated that participants preferred partners who were "good providers" and had "good genes".

Test-retest reliability of eye tracking measures in a computerized Trail Making Test

The Trail Making Test (TMT) is a frequently applied neuropsychological test that evaluates participants' executive functions based on their time to connect a sequence of numbers (TMT-A) or alternating numbers and letters (TMT-B). Test performance is associated with various cognitive functions ranging from visuomotor speed to working memory capabilities. However, although the test can screen for impaired executive functioning in a variety of neuropsychiatric disorders, it provides only little information about which specific cognitive impairments underlie performance detriments. To resolve this lack of specificity, recent cognitive research combined the TMT with eye tracking so that eye movements could help uncover reasons for performance impairments. However, using eye-tracking-based test scores to examine differences between persons, and ultimately apply the scores for diagnostics, presupposes that the reliability of the scores is established. Therefore, we investigated the test-retest reliabilities of scores in an eye-tracking version of the TMT recently introduced by Recker et al. (2022). We examined two healthy samples performing an initial test and then a retest 3 days (n = 31) or 10 to 30 days (n = 34) later. Results reveal that, although reliabilities of classic completion times were overall good, comparable with earlier versions, reliabilities of eye-tracking-based scores ranged from excellent (e.g., durations of fixations) to poor (e.g., number of fixations guiding manual responses). These findings indicate that some eye-tracking measures offer a strong basis for assessing interindividual differences beyond classic behavioral measures when examining processes related to information accumulation processes but are less suitable to diagnose differences in eye-hand coordination.

Eye movements dissociate between perceiving, sensing, and unconscious change detection in scenes

Detecting visual changes can be based on perceiving, whereby one can identify a specific detail that has changed, on sensing, whereby one knows that there is a change but is unable to identify what changed, or on unconscious change detection, whereby one is unaware of any change even though the change influences one's behavior. Prior work has indicated that the processes underlying these different types of change detection are functionally and neurally distinct, but the attentional mechanisms that are related to these different types of change detection remain largely unknown. In the current experiment, we examined eye movements during a change detection task in globally manipulated scenes, and participants indicated their change detection confidence on a scale that allowed us to isolate perceiving, sensing, and unconscious change detection. For perceiving-based change detection, but not sensing-based or unconscious change detection, participants were more likely to preferentially revisit highly changed scene regions across the first and second presentation of the scene (i.e., resampling). This increase in resampling started within 250 ms of the test scene onset, suggesting that the effect began within the first two fixations. In addition, changed scenes were related to more clustered (i.e., less dispersed) eye movements than unchanged scenes, particularly when the subjects were highly confident that no change had occurred - providing evidence for change detection outside of conscious awareness. The results indicate that perceiving, sensing, and unconscious change detection responses are related to partially distinct patterns of eye movements.

Combining fMRI and Eye-tracking for the Study of Social Cognition

The study of social cognition with functional magnetic resonance imaging (fMRI) affords the use of complex stimulus material. Visual attention to distinct aspects of these stimuli can result in the involvement of remarkably different neural systems. Usually, the influence of gaze on neural signal is either disregarded or dealt with by controlling gaze of participants through instructions or tasks. However, behavioral restrictions like this limit the study’s ecological validity. Thus, it would be preferable if participants freely look at the stimuli while their gaze traces are measured. Yet several impediments hamper a combination of fMRI and eye-tracking. In our recent work on neural Theory of Mind processes in alexithymia, we propose a simple way of integrating dwell time on specific stimulus features into general linear models of fMRI data. By parametrically modeling fixations, we were able to distinguish neural processes asssociated with specific stimulus features looked at. Here, I discuss opportunities and obstacles of this approach in more detail. My goal is to motivate a wider use of parametric models — usually implemented in common fMRI software packages — to combine fMRI and eye-tracking data.

The Incomplete Tyranny of Dynamic Stimuli: Gaze Similarity Predicts Response Similarity in Screen-Captured Instructional Videos

Although eye tracking has been used extensively to assess cognitions for static stimuli, recent research suggests that the link between gaze and cognition may be more tenuous for dynamic stimuli such as videos. Part of the difficulty in convincingly linking gaze with cognition is that in dynamic stimuli, gaze position is strongly influenced by exogenous cues such as object motion. However, tests of the gaze-cognition link in dynamic stimuli have been done on only a limited range of stimuli often characterized by highly organized motion. Also, analyses of cognitive contrasts between participants have been mostly been limited to categorical contrasts among small numbers of participants that may have limited the power to observe more subtle influences. We, therefore, tested for cognitive influences on gaze for screen-captured instructional videos, the contents of which participants were tested on. Between-participant scanpath similarity predicted between-participant similarity in responses on test questions, but with imperfect consistency across videos. We also observed that basic gaze parameters and measures of attention to centers of interest only inconsistently predicted learning, and that correlations between gaze and centers of interest defined by other-participant gaze and cursor movement did not predict learning. It, therefore, appears that the search for eye movement indices of cognition during dynamic naturalistic stimuli may be fruitful, but we also agree that the tyranny of dynamic stimuli is real, and that links between eye movements and cognition are highly dependent on task and stimulus properties.

The influence of item familiarization on neural discriminability during associative memory encoding and retrieval

Associative memory requires one to encode and form memory representations not just for individual items, but for the association or link between those items. Past work has suggested that associative memory is facilitated when individual items are familiar rather than simultaneously learning the items and their associative link. The current study employed multivoxel pattern analyses (MVPA) to investigate whether item familiarization prior to associative encoding affects the distinctiveness of neural patterns, and whether that distinctiveness is also present during associative retrieval. Our results suggest that prior exposure to item stimuli impacts the representations of their shared association compared to stimuli that are novel at the time of associative encoding throughout most of the associative memory network. While this distinction was also present at retrieval, the overall extent of the difference was diminished. Overall the results suggest that stimulus familiarity influences the representation of associative pairings during memory encoding and retrieval, and the pair-specific representation is maintained across memory phases irrespective of this distinction.

Seeing things differently: Gaze shapes neural signal during mentalizing according to emotional awareness

Studies on social cognition often use complex visual stimuli to asses neural processes attributed to abilities like "mentalizing" or "Theory of Mind" (ToM). During the processing of these stimuli, eye gaze, however, shapes neural signal patterns. Individual differences in neural operations on social cognition may therefore be obscured if individuals' gaze behavior differs systematically. These obstacles can be overcome by the combined analysis of neural signal and natural viewing behavior. Here, we combined functional magnetic resonance imaging (fMRI) with eye-tracking to examine effects of unconstrained gaze on neural ToM processes in healthy individuals with differing levels of emotional awareness, i.e. alexithymia. First, as previously described for emotional tasks, people with higher alexithymia levels look less at eyes in both ToM and task-free viewing contexts. Further, we find that neural ToM processes are not affected by individual differences in alexithymia per se. Instead, depending on alexithymia levels, gaze on critical stimulus aspects reversely shapes the signal in medial prefrontal cortex (MPFC) and anterior temporoparietal junction (TPJ) as distinct nodes of the ToM system. These results emphasize that natural selective attention affects fMRI patterns well beyond the visual system. Our study implies that, whenever using a task with multiple degrees of freedom in scan paths, ignoring the latter might obscure important conclusions.

Using aversive conditioning with near-real-time feedback to shape eye movements during naturalistic viewing

Strategically shaping patterns of eye movements through training has manifold promising applications, with the potential to improve the speed and efficiency of visual search, improve the ability of humans to extract information from complex displays, and help correct disordered eye movement patterns. However, training how a person moves their eyes when viewing an image or scene is notoriously difficult, with typical approaches relying on explicit instruction and strategy, which have notable limitations. The present study introduces a novel approach to eye movement training using aversive conditioning with near-real-time feedback. Participants viewed indoor scenes (eight scenes presented over 48 trials) with the goal of remembering those scenes for a later memory test. During viewing, saccades meeting specific amplitude and direction criteria probabilistically triggered an aversive electric shock, which was felt within 50 ms after the eliciting eye movement, allowing for a close temporal coupling between an oculomotor behavior and the feedback intended to shape it. Results demonstrate a bias against performing an initial saccade in the direction paired with shock (Experiment 1) or generally of the amplitude paired with shock (Experiment 2), an effect that operates without apparent awareness of the relationship between shocks and saccades, persists into extinction, and generalizes to the viewing of novel images. The present study serves as a proof of concept concerning the implementation of near-real-time feedback in eye movement training.

Harnessing Visual Imagery and Oculomotor Behaviour to Understand Prospection

Much of the rich internal world constructed by humans is derived from, and experienced through, visual mental imagery. Despite growing appreciation of visual exploration in guiding episodic memory processes, extant theories of prospection have yet to accommodate the precise role of visual mental imagery in the service of future-oriented thinking. We propose that the construction of future events relies on the assimilation of perceptual details originally experienced, and subsequently reinstantiated, predominantly in the visual domain. Individual differences in the capacity to summon discrete aspects of visual imagery can therefore account for the diversity of content generated by humans during future simulation. Our integrative framework provides a novel testbed to query alterations in future thinking in health and disease.

Target processing in overt serial visual search involves the dorsal attention network: A fixation-based event-related fMRI study

In serial visual search we shift attention successively from location to location in search for the target. Although such search has been investigated using fMRI, overt attention (i.e., eye movements) was usually neglected or discouraged. As a result, it is unclear what happens in the instant when our gaze falls upon a target as compared to a distractor. In the present experiment, we used a multiple target search task that required eye movements and employed an analysis based on fixations as events of interest to investigate differences between target and distractor processing. Twenty young healthy adults indicated the number of targets (0-3) among distractors in a 20-item display. Compared to distractor fixations, we found that target fixations gave rise to wide-spread activation in the dorsal attention system, as well as in the visual cortex. Targets that were found later during the search activated the left inferior frontal gyrus and the left supramarginal gyrus more strongly than those that were found earlier. Finally, areas associated with visual and verbal working memory showed increased activation with a larger number of targets in the display.

Cloze enough? Hemodynamic effects of predictive processing during natural reading

Evidence accrues that readers form multiple hypotheses about upcoming words. The present study investigated the hemodynamic effects of predictive processing during natural reading by means of combining fMRI and eye movement recordings. In particular, we investigated the neural and behavioral correlates of precision-weighted prediction errors, which are thought to be indicative of subsequent belief updating. Participants silently read sentences in which we manipulated the cloze probability and the semantic congruency of the final word that served as an index for precision and prediction error respectively. With respect to the neural correlates, our findings indicate an enhanced activation within the left inferior frontal and middle temporal gyrus suggesting an effect of precision on prediction update in higher (lexico-)semantic levels. Despite being evident at the neural level, we did not observe any evidence that this mechanism resulted in disproportionate reading times on participants' eye movements. The results speak against discrete predictions, but favor the notion that multiple words are activated in parallel during reading.

Restricting Visual Exploration Directly Impedes Neural Activity, Functional Connectivity, and Memory

We move our eyes to explore the visual world, extract information, and create memories. The number of gaze fixations-the stops that the eyes make-has been shown to correlate with activity in the hippocampus, a region critical for memory, and with later recognition memory. Here, we combined eyetracking with fMRI to provide direct evidence for the relationships between gaze fixations, neural activity, and memory during scene viewing. Compared to free viewing, fixating a single location reduced: 1) subsequent memory, 2) neural activity along the ventral visual stream into the hippocampus, 3) neural similarity between effects of subsequent memory and visual exploration, and 4) functional connectivity among the hippocampus, parahippocampal place area, and other cortical regions. Gaze fixations were uniquely related to hippocampal activity, even after controlling for neural effects due to subsequent memory. Therefore, this study provides key causal evidence supporting the notion that the oculomotor and memory systems are intrinsically related at both the behavioral and neural level. Individual gaze fixations may provide the basic unit of information on which memory binding processes operate.

Neural Correlates of Fixated Low- and High-level Scene Properties during Active Scene Viewing

During real-world scene perception, viewers actively direct their attention through a scene in a controlled sequence of eye fixations. During each fixation, local scene properties are attended, analyzed, and interpreted. What is the relationship between fixated scene properties and neural activity in the visual cortex? Participants inspected photographs of real-world scenes in an MRI scanner while their eye movements were recorded. Fixation-related fMRI was used to measure activation as a function of lower- and higher-level scene properties at fixation, operationalized as edge density and meaning maps, respectively. We found that edge density at fixation was most associated with activation in early visual areas, whereas semantic content at fixation was most associated with activation along the ventral visual stream including core object and scene-selective areas (lateral occipital complex, parahippocampal place area, occipital place area, and retrosplenial cortex). The observed activation from semantic content was not accounted for by differences in edge density. The results are consistent with active vision models in which fixation gates detailed visual analysis for fixated scene regions, and this gating influences both lower and higher levels of scene analysis.

Brain mechanisms of eye contact during verbal communication predict autistic traits in neurotypical individuals

Atypical eye contact in communication is a common characteristic in autism spectrum disorders. Autistic traits vary along a continuum extending into the neurotypical population. The relation between autistic traits and brain mechanisms underlying spontaneous eye contact during verbal communication remains unexplored. Here, we used simultaneous functional magnetic resonance imaging and eye tracking to investigate this relation in neurotypical people within a naturalistic verbal context. Using multiple regression analyses, we found that brain response in the posterior superior temporal sulcus (pSTS) and its connectivity with the fusiform face area (FFA) during eye contact with a speaker predicted the level of autistic traits measured by Autism-spectrum Quotient (AQ). Further analyses for different AQ subclusters revealed that these two predictors were negatively associated with attention to detail. The relation between FFA–pSTS connectivity and the attention to detail ability was mediated by individuals’ looking preferences for speaker’s eyes. This study identified the role of an individual eye contact pattern in the relation between brain mechanisms underlying natural eye contact during verbal communication and autistic traits in neurotypical people. The findings may help to increase our understanding of the mechanisms of atypical eye contact behavior during natural communication.

Two hours in Hollywood: A manually annotated ground truth data set of eye movements during movie clip watching

In this short article we present our manual annotation of the eye movement events in a subset of the large-scale eye tracking data set Hollywood2. Our labels include fixations, saccades, and smooth pursuits, as well as a noise event type (the latter representing either blinks, loss of tracking, or physically implausible signals). In order to achieve more consistent annotations, the gaze samples were labelled by a novice rater based on rudimentary algorithmic suggestions, and subsequently corrected by an expert rater. Overall, we annotated eye movement events in the recordings corresponding to 50 randomly selected test set clips and 6 training set clips from Hollywood2, which were viewed by 16 observers and amount to a total of approximately 130 minutes of gaze data. In these labels, 62.4% of the samples were attributed to fixations, 9.1% – to saccades, and, notably, 24.2% – to pursuit (the remainder marked as noise). After evaluation of 15 published eye movement classification algorithms on our newly collected annotated data set, we found that the most recent algorithms perform very well on average, and even reach human-level labelling quality for fixations and saccades, but all have a much larger room for improvement when it comes to smooth pursuit classification. The data set is made available at https://gin.g-node.org/ioannis.agtzidis/hollywood2_em.

3-Dimensional magnetic resonance imaging of the freely moving human eye

Eye motion is a major confound for magnetic resonance imaging (MRI) in neuroscience or ophthalmology. Currently, solutions toward eye stabilisation include participants fixating or administration of paralytics/anaesthetics. We developed a novel MRI protocol for acquiring 3-dimensional images while the eye freely moves. Eye motion serves as the basis for image reconstruction, rather than an impediment. We fully reconstruct videos of the moving eye and head. We quantitatively validate data quality with millimetre resolution in two ways for individual participants. First, eye position based on reconstructed images correlated with simultaneous eye-tracking. Second, the reconstructed images preserve anatomical properties; the eye's axial length measured from MRI images matched that obtained with ocular biometry. The technique operates on a standard clinical setup, without necessitating specialized hardware, facilitating wide deployment. In clinical practice, we anticipate that this may help reduce burdens on both patients and infrastructure, by integrating multiple varieties of assessments into a single comprehensive session. More generally, our protocol is a harbinger for removing the necessity of fixation, thereby opening new opportunities for ethologically-valid, naturalistic paradigms, the inclusion of populations typically unable to stably fixate, and increased translational research such as in awake animals whose eye movements constitute an accessible behavioural readout.

Best practices in eye tracking research

This guide describes best practices in using eye tracking technology for research in a variety of disciplines. A basic outline of the anatomy and physiology of the eyes and of eye movements is provided, along with a description of the sorts of research questions eye tracking can address. We then explain how eye tracking technology works and what sorts of data it generates, and provide guidance on how to select and use an eye tracker as well as selecting appropriate eye tracking measures. Challenges to the validity of eye tracking studies are described, along with recommendations for overcoming these challenges. We then outline correct reporting standards for eye tracking studies.

Neural dynamics of sentiment processing during naturalistic sentence reading

When we read, our eyes move through the text in a series of fixations and high-velocity saccades to extract visual information. This process allows the brain to obtain meaning, e.g., about sentiment, or the emotional valence, expressed in the written text. How exactly the brain extracts the sentiment of single words during naturalistic reading is largely unknown. This is due to the challenges of naturalistic imaging, which has previously led researchers to employ highly controlled, timed word-by-word presentations of custom reading materials that lack ecological validity. Here, we aimed to assess the electrical neural correlates of word sentiment processing during naturalistic reading of English sentences. We used a publicly available dataset of simultaneous electroencephalography (EEG), eye-tracking recordings, and word-level semantic annotations from 7129 words in 400 sentences (Zurich Cognitive Language Processing Corpus; Hollenstein et al., 2018). We computed fixation-related potentials (FRPs), which are evoked electrical responses time-locked to the onset of fixations. A general linear mixed model analysis of FRPs cleaned from visual- and motor-evoked activity showed a topographical difference between the positive and negative sentiment condition in the 224-304 ​ms interval after fixation onset in left-central and right-posterior electrode clusters. An additional analysis that included word-, phrase-, and sentence-level sentiment predictors showed the same FRP differences for the word-level sentiment, but no additional FRP differences for phrase- and sentence-level sentiment. Furthermore, decoding analysis that classified word sentiment (positive or negative) from sentiment-matched 40-trial average FRPs showed a 0.60 average accuracy (95% confidence interval: [0.58, 0.61]). Control analyses ruled out that these results were based on differences in eye movements or linguistic features other than word sentiment. Our results extend previous research by showing that the emotional valence of lexico-semantic stimuli evoke a fast electrical neural response upon word fixation during naturalistic reading. These results provide an important step to identify the neural processes of lexico-semantic processing in ecologically valid conditions and can serve to improve computer algorithms for natural language processing.

The intersection between the oculomotor and hippocampal memory systems: empirical developments and clinical implications

Decades of cognitive neuroscience research has shown that where we look is intimately connected to what we remember. In this article, we review findings from human and nonhuman animals, using behavioral, neuropsychological, neuroimaging, and computational modeling methods, to show that the oculomotor and hippocampal memory systems interact in a reciprocal manner, on a moment-to-moment basis, mediated by a vast structural and functional network. Visual exploration serves to efficiently gather information from the environment for the purpose of creating new memories, updating existing memories, and reconstructing the rich, vivid details from memory. Conversely, memory increases the efficiency of visual exploration. We call for models of oculomotor control to consider the influence of the hippocampal memory system on the cognitive control of eye movements, and for models of hippocampal and broader medial temporal lobe function to consider the influence of the oculomotor system on the development and expression of memory. We describe eye movement-based applications for the detection of neurodegeneration and delivery of therapeutic interventions for mental health disorders for which the hippocampus is implicated and memory dysfunctions are at the forefront.

Where Do We Look? Assessing Gaze Patterns in Cosmetic Face-Lift Surgery with Eye Tracking Technology

BACKGROUND

Aesthetics plays a central role in determining success in plastic surgery. Understanding perceptions of favorable aesthetics is critical to ensure patient satisfaction. Eye-tracking technology offers an objective way of evaluating attention and understanding how viewers direct their focus on patients who undergo cosmetic face-lift procedures.

METHODS

Thirty-six subjects ranging from layperson to attending plastic surgeon viewed 15 sets of photographs before and after patients underwent an elective face-lift procedure. They were instructed to evaluate the aesthetic quality on a Likert scale while eye-tracking equipment tracked their gaze and analyzed their distribution of attention.

RESULTS

Postoperative images showed a Likert score improvement of 0.51 ± 0.26, with the greatest difference in attending cosmetic plastic surgeons (1.36 ± 0.22; p < 0.05). The nose was the most common first fixation location (31 percent of first fixations) and the most viewed area (16 ± 3 percent of fixation time) for all subjects. Experienced subjects spent less time in nonrelevant areas (30 ± 11 percent for attending cosmetic plastic surgeons and 37 ± 10 percent for attending noncosmetic plastic surgeons) compared with less experienced subjects (50 ± 15 percent for laypersons).

CONCLUSIONS

This study demonstrates that viewers with greater experience in cosmetic surgery focus quickly on the cheeks, chin, and neck and have evenly distributed gaze across the entire face. These results suggest that a layperson's gaze is drawn to the center of the face (because of both unfamiliarity with the face-lift procedure and the natural tendency to look at the central face), whereas attending plastic surgeons exhibit holistic gaze patterns and are more aware of the impact of the procedure.

Cortical control of eye movements in natural reading: Evidence from MVPA

Language comprehension during reading requires fine-grained management of saccadic eye movements. A critical question, therefore, is how the brain controls eye movements in reading. Neural correlates of simple eye movements have been found in multiple cortical regions, but little is known about how this network operates in reading. To investigate this question in the present study, participants were presented with normal text, pseudo-word text, and consonant string text in a magnetic resonance imaging (MRI) scanner with eyetracking. Participants read naturally in the normal text condition and moved their eyes "as if they were reading" in the other conditions. Multi-voxel pattern analysis was used to analyze the fMRI signal in the oculomotor network. We found that activation patterns in a subset of network regions differentiated between stimulus types. These results suggest that the oculomotor network reflects more than simple saccade generation and are consistent with the hypothesis that specific network areas interface with cognitive systems.

The effect of convolving word length, word frequency, function word predictability and first pass reading time in the analysis of a fixation-related fMRI dataset

The data presented in this document was created to explore the effect of including or excluding word length, word frequency, the lexical predictability of function words and first pass reading time (or the duration of the first fixation on a word) as either baseline regressors or duration modulators on the final analysis for a fixation-related fMRI investigation of linguistic processing. The effect of these regressors was a central question raised during the review of Linguistic networks associated with lexical, semantic and syntactic predictability in reading: A fixation-related fMRI study [1]. Three datasets were created and compared to the original dataset to determine their effect. The first examines the effect of adding word length and word frequency as baseline regressors. The second examines the effect of removing first pass reading time as a duration modulator. The third examines the inclusion of function word predictability into the baseline hemodynamic response function. Statistical maps were created for each dataset and compared to the primary dataset (published in [1]) across the linguistic conditions of the initial dataset (lexical predictability, semantic predictability or syntax predictability).

Co-registration of eye movements and neuroimaging for studying contextual predictions in natural reading

Sixteen years ago, Sereno and Rayner (2003. Measuring word recognition in reading: eye movements and event-related potentials. Trends in Cognitive Sciences, 7(11), 489-493) illustrated how "by means of review and comparison" eye movement (EM) and event-related potential (ERP) studies may advance our understanding of visual word recognition. Attempts to simultaneously record EMs and ERPs soon followed. Recently, this co-registration approach has also been transferred to fMRI and oscillatory EEG. With experimental settings close to natural reading, co-registration enables us to directly integrate insights from EM and neuroimaging studies. This should extend current experimental paradigms by moving the field towards studying sentence-level processing including effects of context and parafoveal preview. This article will introduce the basic principles and applications of co-registration and selectively review how this approach may shed light on one of the most controversially discussed issues in reading research, contextual predictions in online language processing.

Research on the Quantitative Method of Cognitive Loading in a Virtual Reality System

Aimed at the problem of how to objectively obtain the threshold of a user’s cognitive load in a virtual reality interactive system, a method for user cognitive load quantification based on an eye movement experiment is proposed. Eye movement data were collected in the virtual reality interaction process by using an eye movement instrument. Taking the number of fixation points, the average fixation duration, the average saccade length, and the number of the first mouse clicking fixation points as the independent variables, and the number of backward-looking times and the value of user cognitive load as the dependent variables, a cognitive load evaluation model was established based on the probabilistic neural network. The model was validated by using eye movement data and subjective cognitive load data. The results show that the absolute error and relative mean square error were 6.52%–16.01% and 6.64%–23.21%, respectively. Therefore, the model is feasible.

Word Frequency Effects in Naturalistic Reading

Word frequency is a central psycholinguistic variable that accounts for substantial variance in language processing. A number of neuroimaging studies have examined frequency at a single word level, typically demonstrating a strong negative, and sometimes positive correlation between frequency and hemodynamic response. Here, 40 subjects read passages of text in an MRI scanner while their eye movements were recorded. We used fixation-related analysis to identify neural activity tied to the frequency of each fixated word. We found that negative correlations with frequency were reduced, while strong positive correlations were found in the temporal and parietal areas associated with semantics. We propose that the processing cost of low frequency words is reduced due to contextual cues. Meanings of high frequency words are more readily accessed and integrated with context resulting in enhanced processing in the semantic system. The results demonstrate similarities and differences between single word and naturalistic text processing.

Neural mechanisms of eye contact when listening to another person talking

Eye contact occurs frequently and voluntarily during face-to-face verbal communication. However, the neural mechanisms underlying eye contact when it is accompanied by spoken language remain unexplored to date. Here we used a novel approach, fixation-based event-related functional magnetic resonance imaging (fMRI), to simulate the listener making eye contact with a speaker during verbal communication. Participants’ eye movements and fMRI data were recorded simultaneously while they were freely viewing a pre-recorded speaker talking. The eye tracking data were then used to define events for the fMRI analyses. The results showed that eye contact in contrast to mouth fixation involved visual cortical areas (cuneus, calcarine sulcus), brain regions related to theory of mind/intentionality processing (temporoparietal junction, posterior superior temporal sulcus, medial prefrontal cortex) and the dorsolateral prefrontal cortex. In addition, increased effective connectivity was found between these regions for eye contact in contrast to mouth fixations. The results provide first evidence for neural mechanisms underlying eye contact when watching and listening to another person talking. The network we found might be well suited for processing the intentions of communication partners during eye contact in verbal communication.

Effects of Scene Properties and Emotional Valence on Brain Activations: A Fixation-Related fMRI Study

Temporal and spatial characteristics of fixations are affected by image properties, including high-level scene characteristics, such as object-background composition, and low-level physical characteristics, such as image clarity. The influence of these factors is modulated by the emotional content of an image. Here, we aimed to establish whether brain correlates of fixations reflect these modulatory effects. To this end, we simultaneously scanned participants and measured their eye movements, while presenting negative and neutral images in various image clarity conditions, with controlled object-background composition. The fMRI data were analyzed using a novel fixation-based event-related (FIBER) method, which allows the tracking of brain activity linked to individual fixations. The results revealed that fixating an emotional object was linked to greater deactivation in the right lingual gyrus than fixating the background of an emotional image, while no difference between object and background was found for neutral images. We suggest that deactivation in the lingual gyrus might be linked to inhibition of saccade execution. This was supported by fixation duration results, which showed that in the negative condition, fixations falling on the object were longer than those falling on the background. Furthermore, increase in the image clarity was correlated with fixation-related activity within the lateral occipital complex, the structure linked to object recognition. This correlation was significantly stronger for negative images, presumably due to greater deployment of attention towards emotional objects. Our eye-tracking results are in line with these observations, showing that the chance of fixating an object rose faster for negative images over neutral ones as the level of noise decreased. Overall, our study demonstrated that emotional value of an image changes the way that low and high-level scene properties affect the characteristics of fixations. The fixation-related brain activity is affected by the low-level scene properties and this impact differs between negative and neutral images. The high-level scene properties also affect brain correlates of fixations, but only in the case of the negative images.

Look me in the eyes: constraining gaze in the eye-region provokes abnormally high subcortical activation in autism

Individuals with Autism Spectrum Disorder (ASD) seem to have difficulties looking others in the eyes, but the substrate for this behavior is not well understood. The subcortical pathway, which consists of superior colliculus, pulvinar nucleus of the thalamus, and amygdala, enables rapid and automatic face processing. A specific component of this pathway – i.e., the amygdala – has been shown to be abnormally activated in paradigms where individuals had to specifically attend to the eye-region; however, a direct examination of the effect of manipulating the gaze to the eye-regions on all the components of the subcortical system altogether has never been performed. The subcortical system is particularly important as it shapes the functional specialization of the face-processing cortex during development. Using functional MRI, we investigated the effect of constraining gaze in the eye-region during dynamic emotional face perception in groups of participants with ASD and typical controls. We computed differences in activation in the subcortical face processing system (superior colliculus, pulvinar nucleus of the thalamus and amygdala) for the same stimuli seen freely or with the gaze constrained in the eye-region. Our results show that when constrained to look in the eyes, individuals with ASD show abnormally high activation in the subcortical system, which may be at the basis of their eye avoidance in daily life.

Neural correlates of individual differences in fixation duration during natural reading

Reading requires integration of language and cognitive processes with attention and eye movement control. Individuals differ in their reading ability, but little is known about the neurocognitive processes associated with these individual differences. To investigate this issue, we combined eyetracking and fMRI, simultaneously recording eye movements and BOLD activity while subjects read text passages. We found that the variability and skew of fixation duration distributions across individuals, as assessed by ex-Gaussian analyses, decreased with increasing neural activity in regions associated with the cortical eye movement control network (Left FEF, Left IPS, Left IFG, and Right IFG). The results suggest that individual differences in fixation duration during reading are related to underlying neurocognitive processes associated with the eye movement control system and its relationship to language processing. The results also show that eye movements and fMRI can be combined to investigate the neural correlates of individual differences in natural reading.

Eye Movements during Auditory Attention Predict Individual Differences in Dorsal Attention Network Activity

The neural mechanisms supporting auditory attention are not fully understood. A dorsal frontoparietal network of brain regions is thought to mediate the spatial orienting of attention across all sensory modalities. Key parts of this network, the frontal eye fields (FEF) and the superior parietal lobes (SPL), contain retinotopic maps and elicit saccades when stimulated. This suggests that their recruitment during auditory attention might reflect crossmodal oculomotor processes; however this has not been confirmed experimentally. Here we investigate whether task-evoked eye movements during an auditory task can predict the magnitude of activity within the dorsal frontoparietal network. A spatial and non-spatial listening task was used with on-line eye-tracking and functional magnetic resonance imaging (fMRI). No visual stimuli or cues were used. The auditory task elicited systematic eye movements, with saccade rate and gaze position predicting attentional engagement and the cued sound location, respectively. Activity associated with these separate aspects of evoked eye-movements dissociated between the SPL and FEF. However these observed eye movements could not account for all the activation in the frontoparietal network. Our results suggest that the recruitment of the SPL and FEF during attentive listening reflects, at least partly, overt crossmodal oculomotor processes during non-visual attention. Further work is needed to establish whether the network’s remaining contribution to auditory attention is through covert crossmodal processes, or is directly involved in the manipulation of auditory information.

The Influence of Content Meaningfulness on Eye Movements across Tasks: Evidence from Scene Viewing and Reading

The present study investigated the influence of content meaningfulness on eye-movement control in reading and scene viewing. Texts and scenes were manipulated to make them uninterpretable, and then eye-movements in reading and scene-viewing were compared to those in pseudo-reading and pseudo-scene viewing. Fixation durations and saccade amplitudes were greater for pseudo-stimuli. The effect of the removal of meaning was seen exclusively in the tail of the fixation duration distribution in both tasks, and the size of this effect was the same across tasks. These findings suggest that eye movements are controlled by a common mechanism in reading and scene viewing. They also indicate that not all eye movements are responsive to the meaningfulness of stimulus content. Implications for models of eye movement control are discussed.