Active inhibition and memory promote exploration and search of natural scenes

Effects of lorazepam on saccadic eye movements - evidence from prosaccade and free viewing tasks

RATIONALE

Peak velocities of saccadic eye movements are reduced after benzodiazepine administration. Even though this is an established effect, past research has only examined it in horizontal prosaccade tasks.

OBJECTIVES

The spectrum of saccadic eye movements, however, is much larger. Therefore, we aimed to make a first attempt at filling this research gap by testing benzodiazepine effects on saccades under different experimental task conditions.

METHODS

1 mg lorazepam or placebo was administered (within-subjects, double-blind, in randomised order) to n = 30 healthy adults. Participants performed an extended version of the prosaccade task, including vertical saccade directions and different stimulus eccentricities, as well as a free viewing task.

RESULTS

Results from the prosaccade task confirmed established effects of benzodiazepines as well as saccade direction on saccadic parameters but additionally showed that the drug effect on peak velocity was independent of saccade direction. Remarkably, in the free viewing task peak velocities as well as other saccade parameters were unaffected by lorazepam. Furthermore, exploration patterns during free viewing did not change under lorazepam.

CONCLUSIONS

Overall, our findings further consolidate the peak velocity of prosaccades as a biomarker of sedation. Additionally, we suggest that sedative effects of low doses of benzodiazepines may be compensated in tasks that more closely resemble natural eye movement behaviour, possibly due to the lack of time constraints or via neurophysiological processes related to volition.

Motor "laziness" constrains fixation selection in real-world tasks

Humans coordinate their eye, head, and body movements to gather information from a dynamic environment while maximizing reward and minimizing biomechanical and energetic costs. However, such natural behavior is not possible in traditional experiments employing head/body restraints and artificial, static stimuli. Therefore, it is unclear to what extent mechanisms of fixation selection discovered in lab studies, such as inhibition-of-return (IOR), influence everyday behavior. To address this gap, participants performed nine real-world tasks, including driving, visually searching for an item, and building a Lego set, while wearing a mobile eye tracker (169 recordings; 26.6 h). Surprisingly, in all tasks, participants most often returned to what they just viewed and saccade latencies were shorter preceding return than forward saccades, i.e., consistent with facilitation, rather than inhibition, of return. We hypothesize that conservation of eye and head motor effort ("laziness") contributes. Correspondingly, we observed center biases in fixation position and duration relative to the head's orientation. A model that generates scanpaths by randomly sampling these distributions reproduced all return phenomena we observed, including distinct 3-fixation sequences for forward versus return saccades. After controlling for orbital eccentricity, one task (building a Lego set) showed evidence for IOR. This, along with small discrepancies between model and data, indicates that the brain balances minimization of motor costs with maximization of rewards (e.g., accomplished by IOR and other mechanisms) and that the optimal balance varies according to task demands. Supporting this account, the orbital range of motion used in each task traded off lawfully with fixation duration.

Refixation behavior in naturalistic viewing: Methods, mechanisms, and neural correlates

When freely viewing a scene, the eyes often return to previously visited locations. By tracking eye movements and coregistering eye movements and EEG, such refixations are shown to have multiple roles: repairing insufficient encoding from precursor fixations, supporting ongoing viewing by resampling relevant locations prioritized by precursor fixations, and aiding the construction of memory representations. All these functions of refixation behavior are understood to be underpinned by three oculomotor and cognitive systems and their associated brain structures. First, immediate saccade planning prior to refixations involves attentional selection of candidate locations to revisit. This process is likely supported by the dorsal attentional network. Second, visual working memory, involved in maintaining task-related information, is likely supported by the visual cortex. Third, higher-order relevance of scene locations, which depends on general knowledge and understanding of scene meaning, is likely supported by the hippocampal memory system. Working together, these structures bring about viewing behavior that balances exploring previously unvisited areas of a scene with exploiting visited areas through refixations.

Objects guide human gaze behavior in dynamic real-world scenes

The complexity of natural scenes makes it challenging to experimentally study the mechanisms behind human gaze behavior when viewing dynamic environments. Historically, eye movements were believed to be driven primarily by space-based attention towards locations with salient features. Increasing evidence suggests, however, that visual attention does not select locations with high saliency but operates on attentional units given by the objects in the scene. We present a new computational framework to investigate the importance of objects for attentional guidance. This framework is designed to simulate realistic scanpaths for dynamic real-world scenes, including saccade timing and smooth pursuit behavior. Individual model components are based on psychophysically uncovered mechanisms of visual attention and saccadic decision-making. All mechanisms are implemented in a modular fashion with a small number of well-interpretable parameters. To systematically analyze the importance of objects in guiding gaze behavior, we implemented five different models within this framework: two purely spatial models, where one is based on low-level saliency and one on high-level saliency, two object-based models, with one incorporating low-level saliency for each object and the other one not using any saliency information, and a mixed model with object-based attention and selection but space-based inhibition of return. We optimized each model's parameters to reproduce the saccade amplitude and fixation duration distributions of human scanpaths using evolutionary algorithms. We compared model performance with respect to spatial and temporal fixation behavior, including the proportion of fixations exploring the background, as well as detecting, inspecting, and returning to objects. A model with object-based attention and inhibition, which uses saliency information to prioritize between objects for saccadic selection, leads to scanpath statistics with the highest similarity to the human data. This demonstrates that scanpath models benefit from object-based attention and selection, suggesting that object-level attentional units play an important role in guiding attentional processing.

Hierarchical Bayesian LSTM for Head Trajectory Prediction on Omnidirectional Images

When viewing omnidirectional images (ODIs), viewers can access different viewports via head movement (HM), which sequentially forms head trajectories in spatial-temporal domain. Thus, head trajectories play a key role in modeling human attention on ODIs. In this paper, we establish a large-scale dataset collecting 21,600 head trajectories on 1,080 ODIs. By mining our dataset, we find two important factors influencing head trajectories, i.e., temporal dependency and subject-specific variance. Accordingly, we propose a novel approach integrating hierarchical Bayesian inference into long short-term memory (LSTM) network for head trajectory prediction on ODIs, which is called HiBayes-LSTM. In HiBayes-LSTM, we develop a mechanism of Future Intention Estimation (FIE), which captures the temporal correlations from previous, current and estimated future information, for predicting viewport transition. Additionally, a training scheme called Hierarchical Bayesian inference (HBI) is developed for modeling inter-subject uncertainty in HiBayes-LSTM. For HBI, we introduce a joint Gaussian distribution in a hierarchy, to approximate the posterior distribution over network weights. By sampling subject-specific weights from the approximated posterior distribution, our HiBayes-LSTM approach can yield diverse viewport transition among different subjects and obtain multiple head trajectories. Extensive experiments validate that our HiBayes-LSTM approach significantly outperforms 9 state-of-the-art approaches for trajectory prediction on ODIs, and then it is successfully applied to predict saliency on ODIs.

Look twice: A generalist computational model predicts return fixations across tasks and species

Primates constantly explore their surroundings via saccadic eye movements that bring different parts of an image into high resolution. In addition to exploring new regions in the visual field, primates also make frequent return fixations, revisiting previously foveated locations. We systematically studied a total of 44,328 return fixations out of 217,440 fixations. Return fixations were ubiquitous across different behavioral tasks, in monkeys and humans, both when subjects viewed static images and when subjects performed natural behaviors. Return fixations locations were consistent across subjects, tended to occur within short temporal offsets, and typically followed a 180-degree turn in saccadic direction. To understand the origin of return fixations, we propose a proof-of-principle, biologically-inspired and image-computable neural network model. The model combines five key modules: an image feature extractor, bottom-up saliency cues, task-relevant visual features, finite inhibition-of-return, and saccade size constraints. Even though there are no free parameters that are fine-tuned for each specific task, species, or condition, the model produces fixation sequences resembling the universal properties of return fixations. These results provide initial steps towards a mechanistic understanding of the trade-off between rapid foveal recognition and the need to scrutinize previous fixation locations.

Between the Scenes

We constantly move our eyes to new information while inspecting a scene, but these patterns of eye movements change based on the task and goals of the observer. Inhibition of return (IOR) may facilitate visual search by reducing the likelihood of revisiting previously attended locations. However, IOR may present in any visual task, or it may be search-specific. We investigated the presence of IOR in foraging, memorization, change detection, and two versions of visual search. One version of search used a static search array that remained stable throughout the trial, but the second used a scene flickering paradigm similar to the change detection task. IOR was observed in both versions of visual search, memorization, and foraging, but not in change detection. Visual search and change detection both had temporal nonscene components, and we observed that IOR could be maintained despite the scene removal but only for search. Although IOR is maintained in scene coordinates, short disruptions to this scene are insufficient to completely remove the inhibitory tags. Finally, we compare return saccades in trials without a probe and observe fewer return saccades in tasks for which IOR was observed, providing further evidence that IOR might serve as a novelty drive.

The Spatial Leaky Competing Accumulator Model

The Leaky Competing Accumulator model (LCA) of Usher and McClelland is able to simulate the time course of perceptual decision making between an arbitrary number of stimuli. Reaction times, such as saccadic latencies, produce a typical distribution that is skewed toward longer latencies and accumulator models have shown excellent fit to these distributions. We propose a new implementation called the Spatial Leaky Competing Accumulator (SLCA), which can be used to predict the timing of subsequent fixation durations during a visual task. SLCA uses a pre-existing saliency map as input and represents accumulation neurons as a two-dimensional grid to generate predictions in visual space. The SLCA builds on several biologically motivated parameters: leakage, recurrent self-excitation, randomness and non-linearity, and we also test two implementations of lateral inhibition. A global lateral inhibition, as implemented in the original model of Usher and McClelland, is applied to all competing neurons, while a local implementation allows only inhibition of immediate neighbors. We trained and compared versions of the SLCA with both global and local lateral inhibition with use of a genetic algorithm, and compared their performance in simulating human fixation latency distribution in a foraging task. Although both implementations were able to produce a positively skewed latency distribution, only the local SLCA was able to match the human data distribution from the foraging task. Our model is discussed for its potential in models of salience and priority, and its benefits as compared to other models like the Leaky integrate and fire network.

Maintaining rejected distractors in working memory during visual search depends on search stimuli: Evidence from contralateral delay activity

The presence of memory for rejected distractors during visual search has been heavily debated in the literature and has proven challenging to investigate behaviorally. In this research, we used an electrophysiological index of working memory (contralateral delay activity) to passively measure working memory activity during visual search. Participants were asked to indicate whether a novel target was present or absent in a lateralized search array with three visual set sizes (2, 4, or 6). If rejected distractors are maintained in working memory during search, working memory activity should increase with the number of distractors that need to be evaluated. Therefore, we predicted the amplitude of the contralateral delay activity would be larger for target-absent trials and would increase with visual set size until WM capacity was reached. In Experiment 1, we found no evidence for distractor maintenance in working memory during search for real-world stimuli. In Experiment 2, we found partial evidence in support of distractor maintenance during search for stimuli with high target/distractor similarity. In both experiments, working memory capacity did not appear to be a limiting factor during visual search. These results suggest the role of working memory during search may depend on the visual search task in question. Maintaining distractors in working memory appears to be unnecessary during search for realistic stimuli. However, there appears to be a limited role for distractor maintenance during search for artificial stimuli with a high degree of feature overlap.

Impaired inhibition of return during free-viewing behaviour in patients with schizophrenia

Schizophrenia affects various aspects of cognitive and behavioural functioning. Eye movement abnormalities are commonly observed in patients with schizophrenia (SZs). Here we examined whether such abnormalities reflect an anomaly in inhibition of return (IOR), the mechanism that inhibits orienting to previously fixated or attended locations. We analyzed spatiotemporal patterns of eye movement during free-viewing of visual images including natural scenes, geometrical patterns, and pseudorandom noise in SZs and healthy control participants (HCs). SZs made saccades to previously fixated locations more frequently than HCs. The time lapse from the preceding saccade was longer for return saccades than for forward saccades in both SZs and HCs, but the difference was smaller in SZs. SZs explored a smaller area than HCs. Generalized linear mixed-effect model analysis indicated that the frequent return saccades served to confine SZs' visual exploration to localized regions. The higher probability of return saccades in SZs was related to cognitive decline after disease onset but not to the dose of prescribed antipsychotics. We conclude that SZs exhibited attenuated IOR under free-viewing conditions, which led to restricted scene scanning. IOR attenuation will be a useful clue for detecting impairment in attention/orienting control and accompanying cognitive decline in schizophrenia.

Predicting artificial visual field losses: A gaze-based inference study

Visual field defects are a world-wide concern, and the proportion of the population experiencing vision loss is ever increasing. Macular degeneration and glaucoma are among the four leading causes of permanent vision loss. Identifying and characterizing visual field losses from gaze alone could prove crucial in the future for screening tests, rehabilitation therapies, and monitoring. In this experiment, 54 participants took part in a free-viewing task of visual scenes while experiencing artificial scotomas (central and peripheral) of varying radii in a gaze-contingent paradigm. We studied the importance of a set of gaze features as predictors to best differentiate between artificial scotoma conditions. Linear mixed models were utilized to measure differences between scotoma conditions. Correlation and factorial analyses revealed redundancies in our data. Finally, hidden Markov models and recurrent neural networks were implemented as classifiers in order to measure the predictive usefulness of gaze features. The results show separate saccade direction biases depending on scotoma type. We demonstrate that the saccade relative angle, amplitude, and peak velocity of saccades are the best features on the basis of which to distinguish between artificial scotomas in a free-viewing task. Finally, we discuss the usefulness of our protocol and analyses as a gaze-feature identifier tool that discriminates between artificial scotomas of different types and sizes.

What do we know about volumetric medical image interpretation?: a review of the basic science and medical image perception literatures

Interpretation of volumetric medical images represents a rapidly growing proportion of the workload in radiology. However, relatively little is known about the strategies that best guide search behavior when looking for abnormalities in volumetric images. Although there is extensive literature on two-dimensional medical image perception, it is an open question whether the conclusions drawn from these images can be generalized to volumetric images. Importantly, volumetric images have distinct characteristics (e.g., scrolling through depth, smooth-pursuit eye-movements, motion onset cues, etc.) that should be considered in future research. In this manuscript, we will review the literature on medical image perception and discuss relevant findings from basic science that can be used to generate predictions about expertise in volumetric image interpretation. By better understanding search through volumetric images, we may be able to identify common sources of error, characterize the optimal strategies for searching through depth, or develop new training and assessment techniques for radiology residents.

Spatial statistics for gaze patterns in scene viewing: Effects of repeated viewing

Scene viewing is used to study attentional selection in complex but still controlled environments. One of the main observations on eye movements during scene viewing is the inhomogeneous distribution of fixation locations: While some parts of an image are fixated by almost all observers and are inspected repeatedly by the same observer, other image parts remain unfixated by observers even after long exploration intervals. Here, we apply spatial point process methods to investigate the relationship between pairs of fixations. More precisely, we use the pair correlation function, a powerful statistical tool, to evaluate dependencies between fixation locations along individual scanpaths. We demonstrate that aggregation of fixation locations within 4° is stronger than expected from chance. Furthermore, the pair correlation function reveals stronger aggregation of fixations when the same image is presented a second time. We use simulations of a dynamical model to show that a narrower spatial attentional span may explain differences in pair correlations between the first and the second inspection of the same image.

Searchers adjust their eye-movement dynamics to target characteristics in natural scenes

When searching a target in a natural scene, it has been shown that both the target's visual properties and similarity to the background influence whether and how fast humans are able to find it. So far, it was unclear whether searchers adjust the dynamics of their eye movements (e.g., fixation durations, saccade amplitudes) to the target they search for. In our experiment, participants searched natural scenes for six artificial targets with different spatial frequency content throughout eight consecutive sessions. High-spatial frequency targets led to smaller saccade amplitudes and shorter fixation durations than low-spatial frequency targets if target identity was known. If a saccade was programmed in the same direction as the previous saccade, fixation durations and successive saccade amplitudes were not influenced by target type. Visual saliency and empirical fixation density at the endpoints of saccades which maintain direction were comparatively low, indicating that these saccades were less selective. Our results suggest that searchers adjust their eye movement dynamics to the search target efficiently, since previous research has shown that low-spatial frequencies are visible farther into the periphery than high-spatial frequencies. We interpret the saccade direction specificity of our effects as an underlying separation into a default scanning mechanism and a selective, target-dependent mechanism.

Re-fixation and perseveration patterns in neglect patients during free visual exploration

The literature suggests that neglect patients not only show impairments in directing attention toward the left, contralesional space, but also present with perseverative behavior. Moreover, previous studies described re-fixations during visual search tasks, and interpreted this finding as an impairment of spatial working memory. The aim of the present study was to study re-fixations and perseverations (i.e., recurrent re-fixations to same locations) during free visual exploration, a task with high ecological validity. We hypothesized that: (1) neglect patient would perform re-fixations more frequently than healthy controls within the right hemispace; and, (2) the re-fixation behavior of neglect patients would be characterized by perseverative fixations. To test these hypotheses, we assessed 22 neglect patients and 23 healthy controls, measuring their eye movements during free exploration of naturalistic pictures. The results showed that neglect patients tend to re-fixate locations within the ipsilesional hemispace when they freely explore naturalistic pictures. Importantly, the saliency of discrete locations within the pictures has a stronger influence on fixation behavior within the contralesional than within the ipsilesional hemispace in neglect patients. Finally, the results indicated that, for re-fixations, saliency plays a more important role within the contralesional than the ipsilesional hemispace. Moreover, we found evidence that re-fixation behavior of neglect patients is characterized by frequent recurrent re-fixations back to the same spatial locations which may be interpreted as perseverations. Hence, with the present study, we could better elucidate the mechanism leading to re-fixations and perseverative behavior during free visual exploration in neglect patients.

Saccadic momentum and attentive control in V4 neurons during visual search

Saccadic momentum refers to the increased probability of making a saccade in a forward direction relative to the previous saccade. During visual search and free viewing conditions saccadic probability falls in a gradient from forward to backward directions. It has been considered to reflect an oculomotor bias for a continuing motor plan. Here we report that a saccadic momentum gradient is observed in nonhuman primate behavior and in the visual responses of cortical area V4 neurons during a conjunction style visual search task. This result suggests that saccadic momentum arises in part from a biased spatial distribution of visual responses to stimuli. The effect is independent of feature-based selective attention and overridden by directed spatial attention. The implications of saccadic momentum for search guidance are much broader and robust than the inhibition-of-return's presumed role in preventing refixation of recent locations.

No supplementary evidence of attention to a spatial cue when saccadic facilitation is absent

Attending a location in space facilitates responses to targets at that location when the time between cue and target is short. Certain types of exogenous cues – such as sudden peripheral onsets – have been described as reflexive and automatic. Recent studies however, have been showing many cases where exogenous cues are less automatic than previously believed and do not always result in facilitation. A lack of the behavioral facilitation, however, does not automatically necessitate a lack of underlying attention to that location. We test exogenous cueing in two experiments where facilitation is and is not likely to be observed with saccadic responses. We also test alternate measures linked to the allocation of attention such as saccadic curvature, microsaccades and pupil size. As expected, we find early facilitation as measured by saccadic reaction time when CTOAs are predictable but not when they are randomized within a block. We find no impact of the cue on microsaccade direction for either experiment, and only a slight dip in the frequency of microsaccades after the cue. We do find that change in pupil size to the cue predicts the magnitude of the validity effect, but only in the experiment where facilitation was observed. In both experiments, we observed a tendency for saccadic curvature to deviate away from the cued location and this was stronger for early CTOAs and toward vertical targets. Overall, we find that only change in pupil size is consistent with observed facilitation. Saccadic curvature is influenced by the onset of the cue, buts its direction is indicative of oculomotor inhibition whether we see RT facilitation or not. Microsaccades were not diagnostic in either experiment. Finally, we see little to no evidence of attention at the cued location in any additional measures when facilitation of saccadic responses is absent.

Visual short-term memory guides infants' visual attention

Adults' visual attention is guided by the contents of visual short-term memory (VSTM). Here we asked whether 10-month-old infants' (N = 41) visual attention is also guided by the information stored in VSTM. In two experiments, we modified the one-shot change detection task (Oakes, Baumgartner, Barrett, Messenger, & Luck, 2013) to create a simplified cued visual search task to ask how information stored in VSTM influences where infants look. A single sample item (e.g., a colored circle) was presented at fixation for 500 ms, followed by a brief (300 ms) retention interval and then a test array consisting of two items, one on each side of fixation. One item in the test array matched the sample stimulus and the other did not. Infants were more likely to look at the non-matching item than at the matching item, demonstrating that the information stored rapidly in VSTM guided subsequent looking behavior.

Temporal ambiguity of onsets in a cueing task prevents facilitation but not inhibition of return

Cueing effects, i.e., early facilitation of reaction time and inhibition of return (IOR), are well-established and robust phenomena characterizing exogenous orienting and are widely observed in experiments with a traditional Posner cueing paradigm. Krüger, MacInnes, and Hunt (2014) proposed that facilitatory effects of peripheral cues are the result of a cue-target perceptual merging due to re-entrant visual processing. To test the role and timing of these feedback mechanisms in peripheral cueing effects, we modified the traditional cueing task in Experiments 1-3 by interleaving pre- and post-cue trials at the valid and invalid location and random cue-target onset asynchrony (CTOA) ranging from -300 to +1,000 ms. Analysis of the manual reaction time distribution over CTOA showed well-pronounced IOR in the valid pre-cue condition and a small cost of perceptual merging in the post-cue condition, but no early facilitation of reaction time was observed in the pre-cue condition. In Experiment 4, we tested directly whether temporal ambiguity eliminated facilitation by restricting CTOAs to only the pre-cue time range and including a between-subject manipulation of a) random, b) mixed discrete, and c) blocked discrete CTOAs. Results obtained in the continuous and binned conditions showed no facilitation but robust IOR. We found both early facilitation and IOR in the blocked condition. Overall, the present findings show a small perceptual merging result without accompanying facilitation, suggesting different underlying mechanisms. Second, they demonstrate that early facilitation is likely to be affected by the presence or absence of temporal expectations and that the early onset of IOR might be masked by stronger facilitation in traditional cueing experiments.

Neural field model of memory-guided search

Many organisms can remember locations they have previously visited during a search. Visual search experiments have shown exploration is guided away from these locations, reducing redundancies in the search path before finding a hidden target. We develop and analyze a two-layer neural field model that encodes positional information during a search task. A position-encoding layer sustains a bump attractor corresponding to the searching agent's current location, and search is modeled by velocity input that propagates the bump. A memory layer sustains persistent activity bounded by a wave front, whose edges expand in response to excitatory input from the position layer. Search can then be biased in response to remembered locations, influencing velocity inputs to the position layer. Asymptotic techniques are used to reduce the dynamics of our model to a low-dimensional system of equations that track the bump position and front boundary. Performance is compared for different target-finding tasks.

Trajectory curvature in saccade sequences: spatiotopic influences vs. residual motor activity

When decisions drive saccadic eye movements, traces of the decision process can be inferred from the movement trajectories. For example, saccades can curve away from distractor stimuli, which was thought to reflect cortical inhibition biasing activity in the superior colliculus. Recent neurophysiological work does not support this theory, and two recent models have replaced top-down inhibition with lateral interactions in the superior colliculus or neural fatigue in the brainstem saccadic burst generator. All current models operate in retinotopic coordinates and are based on single saccade paradigms. To extend these models to sequences of saccades, we assessed whether and how saccade curvature depends on previously fixated locations and the direction of previous saccades. With a two-saccade paradigm, we first demonstrated that second saccades curved away from the initial fixation stimulus. Furthermore, by varying the time from fixation offset and the intersaccadic duration, we distinguished the extent of curvature originating from the spatiotopic representation of the previous fixation location or residual motor activity of the previous saccade. Results suggest that both factors drive curvature, and we discuss how these effects could be implemented in current models. In particular, we propose that the collicular retinotopic maps receive an excitatory spatiotopic update from the lateral interparial region.NEW & NOTEWORTHY Saccades curve away from locations of previous fixation. Varying stimulus timing demonstrates the effects of both 1) spatiotopic representation and 2) motor residual activity from previous saccades. The spatiotopic effect can be explained if current models are augmented with an excitatory top-down spatiotopic signal.

Influence of initial fixation position in scene viewing

During scene perception our eyes generate complex sequences of fixations. Predictors of fixation locations are bottom-up factors such as luminance contrast, top-down factors like viewing instruction, and systematic biases, e.g., the tendency to place fixations near the center of an image. However, comparatively little is known about the dynamics of scanpaths after experimental manipulation of specific fixation locations. Here we investigate the influence of initial fixation position on subsequent eye-movement behavior on an image. We presented 64 colored photographs to participants who started their scanpaths from one of two experimentally controlled positions in the right or left part of an image. Additionally, we used computational models to predict the images' fixation locations and classified them as balanced images or images with high conspicuity on either the left or right side of a picture. The manipulation of the starting position influenced viewing behavior for several seconds and produced a tendency to overshoot to the image side opposite to the starting position. Possible mechanisms for the generation of this overshoot were investigated using numerical simulations of statistical and dynamical models. Our model comparisons show that inhibitory tagging is a viable mechanism for dynamical planning of scanpaths.

Scanpath estimation based on foveated image saliency

The estimation of gaze shift has been an important research area in saliency modeling. Gaze movement is a dynamic progress, yet existing estimation methods are limited to estimating scanpaths within only one saliency map, providing results with unsatisfactory accuracy. A bio-inspired method for gaze shift prediction is thus proposed. We take the effect of foveation into account in the proposed model, which plays an important role in the search for dynamic salient regions. The saccadic bias of gaze shifts and the mechanism of inhibition of return in short-term memory are also considered. Based on the probability map derived from these factors, candidates for the next fixation can be randomly generated, and the final scanpath can be acquired point by point. By the evaluation of objective measures, experimental results show that this method possesses better performance in several datasets than many existing models do.

Revisiting the global effect and inhibition of return

Saccades toward previously cued locations have longer latencies than saccades toward other locations, a phenomenon known as inhibition of return (IOR). Watanabe (Exp Brain Res 138:330–342. doi:10.1007/s002210100709, 2001) combined IOR with the global effect (where saccade landing points fall in between neighboring objects) to investigate whether IOR can also have a spatial component. When one of two neighboring targets was cued, there was a clear bias away from the cued location. In a condition where both targets were cued, it appeared that the global effect magnitude was similar to the condition without any cues. However, as the latencies in the double cue condition were shorter compared to the no cue condition, it is still an open question whether these results are representative for IOR. Considering the double cue condition can provide valuable insight into the interaction of the mechanisms underlying the two phenomena, here, we revisit this condition in an adapted paradigm. Our paradigm does result in longer latencies for the cued locations, and we find that the magnitude of the global effect is reduced significantly. Unexpectedly, this holds even when only including saccades with the same latencies for both conditions. Thus, the increased latencies associated with IOR cannot directly explain the reduction in global effect. The global effect reduction can likely best be seen as either a result of short-term depression of exogenous visual signals or a result of IOR established at the center of gravity of cues.

Spatial inhibition of return as a function of fixation history, task, and spatial references

In oculomotor selection, each saccade is thought to be automatically biased toward uninspected locations, inhibiting the inefficient behavior of repeatedly refixating the same objects. This automatic bias is related to inhibition of return (IOR). Although IOR seems an appealing property that increases efficiency in visual search, such a mechanism would not be efficient in other tasks. Indeed, evidence for additional, more flexible control over refixations has been provided. Here, we investigated whether task demands implicitly affect the rate of refixations. We measured the probability of refixations after series of six binary saccadic decisions under two conditions: visual search and free viewing. The rate of refixations seems influenced by two effects. One effect is related to the rate of intervening fixations, specifically, more refixations were observed with more intervening fixations. In addition, we observed an effect of task set, with fewer refixations in visual search than in free viewing. Importantly, the history-related effect was more pronounced when sufficient spatial references were provided, suggesting that this effect is dependent on spatiotopic encoding of previously fixated locations. This known history-related bias in gaze direction is not the primary influence on the refixation rate. Instead, multiple factors, such as task set and spatial references, assert strong influences as well.

Scanpaths of Complex Image Viewing: Insights From Experimental and Modeling Studies

From the first works of Buswell, Yarbus, and Noton and Stark, the scan path for viewing complex images has been considered as a possible key to objective estimation of cognitive processes and their dynamics. However, evidences both pro and con were revealed in the modern research. In this article, the results supporting the Yarbus-Stark concept are presented. In psychophysical tests, two types of images (three paintings from Yarbus` works and four textures) were used with two instructions, namely, “free viewing” and “search for modified image regions.” The focus of the analysis of experimental results and modeling has been given to local elements of the scan path. It was shown that each parameter used (square of viewing area, S; distance between center of mass of viewing area and image center, R; parameter Xi, based on duration of the current fixation and angle between preceding and following saccades), reflects the specificity of both visual task and image properties. Additionally, the return gaze fixations which have a set of specific properties and mainly address to the areas of interest on image were revealed. Evidently these facts can be formalized in an advanced mathematical model as additional instrument to study the mechanisms of complex image viewing.

Effects of task and task-switching on temporal inhibition of return, facilitation of return, and saccadic momentum during scene viewing

During scene viewing, saccades directed toward a recently fixated location tend to be delayed relative to saccades in other directions ("delay effect"), an effect attributable to inhibition of return (IOR) and/or saccadic momentum (SM). Previous work indicates this effect may be task-specific, suggesting that gaze control parameters are task-relevant and potentially affected by task-switching. Accordingly, the present study investigated task-set control of gaze behavior using the delay effect as a measure of task performance. The delay effect was measured as the effect of relative saccade direction on preceding fixation duration. Participants were cued on each trial to perform either a search, memory, or rating task. Tasks were performed either in pure-task or mixed-task blocks. This design allowed separation of switch-cost and mixing-cost. The critical result was that expression of the delay effect at 2-back locations was reversed on switch versus repeat trials such that return was delayed in repeat trials but speeded in switch trials. This difference between repeat and switch trials suggests that gaze-relevant parameters may be represented and switched as part of a task-set. Existing and new tests for dissociating IOR and SM accounts of the delay effect converged on the conclusion that the delay at 2-back locations was due to SM, and that task-switching affects SM. Additionally, the new test simultaneously replicated noncorroborating results in the literature regarding facilitation-of-return (FOR), which confirmed its existence and showed that FOR is "reversed" SM that occurs when preceding and current saccades are both directed toward the 2-back location.

Saccadic model of eye movements for free-viewing condition

In this paper, we propose a new framework to predict visual scanpaths of observers while they freely watch a visual scene. The visual fixations are inferred from bottom-up saliency and several oculomotor biases. Bottom-up saliency is represented by a saliency map whereas the oculomotor biases (saccade amplitudes and saccade orientations) are modeled using public eye tracking datasets. Our experiments show that the simulated scanpaths exhibit similar trends of human eye movements in a free-viewing condition. The generated scanpaths are more similar to human scanpaths than those generated by two existing methods. In addition, we show that computing saliency maps from simulated visual scanpaths allows to outperform existing saliency models.

Just passing through? Inhibition of return in saccadic sequences

Responses tend to be slower to previously fixated spatial locations, an effect known as “inhibition of return” (IOR). Saccades cannot be assumed to be independent, however, and saccade sequences programmed in parallel differ from independent eye movements. We measured the speed of both saccadic and manual responses to probes appearing in previously fixated locations when those locations were fixated as part of either parallel or independent saccade sequences. Saccadic IOR was observed in independent but not parallel saccade sequences, while manual IOR was present in both parallel and independent sequence types. Saccadic IOR was also short-lived, and dissipated with delays of more than ∼1500 ms between the intermediate fixation and the probe onset. The results confirm that the characteristics of IOR depend critically on the response modality used for measuring it, with saccadic and manual responses giving rise to motor and attentional forms of IOR, respectively. Saccadic IOR is relatively short-lived and is not observed at intermediate locations of parallel saccade sequences, while attentional IOR is long-lasting and consistent for all sequence types.

In search of a reliable electrophysiological marker of oculomotor inhibition of return

Inhibition of return (IOR) operationalizes a behavioral phenomenon characterized by slower responding to cued, relative to uncued, targets. Two independent forms of IOR have been theorized: input-based IOR occurs when the oculomotor system is quiescent, while output-based IOR occurs when the oculomotor system is engaged. EEG studies forbidding eye movements have demonstrated that reductions of target-elicited P1 components are correlated with IOR magnitude, but when eye movements occur, P1 effects bear no relationship to behavior. We expand on this work by adapting the cueing paradigm and recording event-related potentials: IOR is caused by oculomotor responses to central arrows or peripheral onsets and measured by key presses to peripheral targets. Behavioral IOR is observed in both conditions, but P1 reductions are absent in the central arrow condition. By contrast, arrow and peripheral cues enhance Nd, especially over contralateral electrode sites.

Novelty enhances visual salience independently of reward in the parietal lobe

Novelty modulates sensory and reward processes, but it remains unknown how these effects interact, i.e., how the visual effects of novelty are related to its motivational effects. A widespread hypothesis, based on findings that novelty activates reward-related structures, is that all the effects of novelty are explained in terms of reward. According to this idea, a novel stimulus is by default assigned high reward value and hence high salience, but this salience rapidly decreases if the stimulus signals a negative outcome. Here we show that, contrary to this idea, novelty affects visual salience in the monkey lateral intraparietal area (LIP) in ways that are independent of expected reward. Monkeys viewed peripheral visual cues that were novel or familiar (received few or many exposures) and predicted whether the trial will have a positive or a negative outcome--i.e., end in a reward or a lack of reward. We used a saccade-based assay to detect whether the cues automatically attracted or repelled attention from their visual field location. We show that salience--measured in saccades and LIP responses--was enhanced by both novelty and positive reward associations, but these factors were dissociable and habituated on different timescales. The monkeys rapidly recognized that a novel stimulus signaled a negative outcome (and withheld anticipatory licking within the first few presentations), but the salience of that stimulus remained high for multiple subsequent presentations. Therefore, novelty can provide an intrinsic bonus for attention that extends beyond the first presentation and is independent of physical rewards.

Synchronous contextual irregularities affect early scene processing: replication and extension

Whether contextual regularities facilitate perceptual stages of scene processing is widely debated, and empirical evidence is still inconclusive. Specifically, it was recently suggested that contextual violations affect early processing of a scene only when the incongruent object and the scene are presented a-synchronously, creating expectations. We compared event-related potentials (ERPs) evoked by scenes that depicted a person performing an action using either a congruent or an incongruent object (e.g., a man shaving with a razor or with a fork) when scene and object were presented simultaneously. We also explored the role of attention in contextual processing by using a pre-cue to direct subjects׳ attention towards or away from the congruent/incongruent object. Subjects׳ task was to determine how many hands the person in the picture used in order to perform the action. We replicated our previous findings of frontocentral negativity for incongruent scenes that started ~ 210 ms post stimulus presentation, even earlier than previously found. Surprisingly, this incongruency ERP effect was negatively correlated with the reaction times cost on incongruent scenes. The results did not allow us to draw conclusions about the role of attention in detecting the regularity, due to a weak attention manipulation. By replicating the 200-300 ms incongruity effect with a new group of subjects at even earlier latencies than previously reported, the results strengthen the evidence for contextual processing during this time window even when simultaneous presentation of the scene and object prevent the formation of prior expectations. We discuss possible methodological limitations that may account for previous failures to find this an effect, and conclude that contextual information affects object model selection processes prior to full object identification, with semantic knowledge activation stages unfolding only later on.

Attention as foraging for information and value

What is the purpose of attention? One avenue of research has led to the proposal that attention might be crucial for gathering information about the environment, while other lines of study have demonstrated how attention may play a role in guiding behavior to rewarded options. Many experiments that study attention require participants to make a decision based on information acquired discretely at one point in time. In real-world situations, however, we are usually not presented with information about which option to select in such a manner. Rather we must initially search for information, weighing up reward values of options before we commit to a decision. Here, we propose that attention plays a role in both foraging for information and foraging for value. When foraging for information, attention is guided toward the unknown. When foraging for reward, attention is guided toward high reward values, allowing decision-making to proceed by accept-or-reject decisions on the currently attended option. According to this account, attention can be regarded as a low-cost alternative to moving around and physically interacting with the environment—“teleforaging”—before a decision is made to interact physically with the world. To track the timecourse of attention, we asked participants to seek out and acquire information about two gambles by directing their gaze, before choosing one of them. Participants often made multiple refixations on items before making a decision. Their eye movements revealed that early in the trial, attention was guided toward information, i.e., toward locations that reduced uncertainty about value. In contrast, late in the trial, attention was guided by expected value of the options. At the end of the decision period, participants were generally attending to the item they eventually chose. We suggest that attentional foraging shifts from an uncertainty-driven to a reward-driven mode during the evolution of a decision, permitting decisions to be made by an engage-or-search strategy.

Optimal and Preferred Eye Landing Positions in Objects and Scenes

Viewing position effects are commonly observed in reading, but they have only rarely been investigated in object perception or in the realistic context of a natural scene. In two experiments, we explored where people fixate within photorealistic objects and the effects of this landing position on recognition and subsequent eye movements. The results demonstrate an optimal viewing position—objects are processed more quickly when fixation is in the centre of the object. Viewers also prefer to saccade to the centre of objects within a natural scene, even when making a large saccade. A central landing position is associated with an increased likelihood of making a refixation, a result that differs from previous reports and suggests that multiple fixations within objects, within scenes, occur for a range of reasons. These results suggest that eye movements within scenes are systematic and are made with reference to an early parsing of the scene into constituent objects.

Saccadic momentum and facilitation of return saccades contribute to an optimal foraging strategy

The interest in saccadic IOR is funneled by the hypothesis that it serves a clear functional purpose in the selection of fixation points: the facilitation of foraging. In this study, we arrive at a different interpretation of saccadic IOR. First, we find that return saccades are performed much more often than expected from the statistical properties of saccades and saccade pairs. Second, we find that fixation durations before a saccade are modulated by the relative angle of the saccade, but return saccades show no sign of an additional temporal inhibition. Thus, we do not find temporal saccadic inhibition of return. Interestingly, we find that return locations are more salient, according to empirically measured saliency (locations that are fixated by many observers) as well as stimulus dependent saliency (defined by image features), than regular fixation locations. These results and the finding that return saccades increase the match of individual trajectories with a grand total priority map evidences the return saccades being part of a fixation selection strategy that trades off exploration and exploitation.

Sometimes humans look at the same location twice. To appreciate the importance of this inconspicuous statement you have to consider that we move our eyes several billion (10⁹) times during our lives and that looking at something is a necessary condition to enable conscious visual awareness. Thus, understanding why and how we move our eyes provides a window into our mental life. Here we investigate one heavily discussed aspect of human's fixation selection strategy: whether it inhibits returning to previously fixated locations. We analyze a large data set (more than 550,000 fixations from 235 subjects) and find that, returning to previously fixated locations happens much more often than expected from the statistical properties of eye-movement trajectories. Furthermore, those locations that we return to are not ordinary – they are more salient than locations that we do not return to. Thus, the inconspicuous statement that we look at the same locations twice reveals an important aspect of our strategy to select fixation points: That we trade off exploring our environment against making sure that we have fully comprehended the relevant parts of our environment.