Eye can read your mind: decoding gaze fixations to reveal categorical search targets

Exploring power-law behavior in human gaze shifts across tasks and populations

Visual perception is an integral part of human cognition. Vision comprises sampling information and processing them. Tasks and stimuli influence human sampling behavior, while cognitive and neurological processing mechanisms remain unchanged. A question still controversial today is whether the components interact with each other. Some theories see the components of visual cognition as separate and their influence on gaze behavior as additive. Others see gaze behavior as an emergent structure of visual cognition that emerges through multiplicative interactions. One way to approach this problem is to examine the magnitude of gaze shifts. Demonstrating that gaze shifts show a constant behavior across tasks would argue for the existence of an independent component in human visual behavior. However, studies attempting to generally describe gaze shift magnitudes deliver contradictory results. In this work, we analyze data from numerous experiments to advance the debate on visual cognition by providing a more comprehensive view of visual behavior. The data show that the magnitude of eye movements, also called saccades, cannot be described by a consistent distribution across different experiments. However, we also propose a new way of measuring the magnitude of saccades: relative saccade lengths. We find that a saccade's length relative to the preceding saccade's length consistently follows a power-law distribution. We observe this distribution for all datasets we analyze, regardless of the task, stimulus, age, or native language of the participants. Our results indicate the existence of an independent component utilized by other cognitive processes without interacting with them. This suggests that a part of human visual cognition is based on an additive component that does not depend on stimulus features.

Teaching through their eyes: effects on optometry teachers' adaptivity and students' learning when teachers see students' gaze

Adaptive teacher support fosters effective learning in one-to-one teaching sessions, which are a common way of learning complex visual tasks in the health sciences. Adaptive support is tailored to student needs, and this is difficult in complex visual tasks as visual problem-solving processes are covert and thus cannot be directly observed by the teacher. Eye-tracking apparatus can measure covert processes and make them visible in gaze displays: visualizations of where a student looks while executing a task. We investigate whether live dynamic gaze displays help teachers in being more adaptive to students' needs when teaching optical coherence tomography interpretation in one-to-one teaching sessions and whether this fosters learning. Forty-nine students and 10 teachers participated in a one-to-one teaching session in clinical optometry. In the control condition, teachers saw the learning task of the student and could discuss it with them, whereas in the gaze-display condition, teachers could additionally see where the student looked. After the 15-minute teaching session, a test was administered to examine achievement. Furthermore, students filled in the 'questionnaire on teacher support adaptivity', and teachers rated how adaptive their support was. Bayesian analyses provide some initial evidence that students did not experience support to be more adaptive in the gaze-display condition versus the control condition, nor were their post-test scores higher. Teachers rated their provided support as being more adaptive in the gaze-display versus the control condition. Further research could investigate if live dynamic gaze displays impact adaptive teaching when used over longer periods or with more teacher training.

Nonverbal behavioral patterns predict social rejection elicited aggression

Aggression elicited by social rejection is costly, prevalent, and often lethal. Attempts to predict rejection-elicited aggression using trait-based data have had little success. This may be because in-the-moment aggression is a complex process influenced by current states of attention, arousal, and affect which are poorly predicted by trait-level characteristics. In a study of young adults (N = 89; 18-25 years), machine learning tested the extent to which nonverbal behavioral indices of attention (eye gaze), arousal (pupillary reactivity), and affect (facial expressions) during a novel social interaction paradigm predicted subsequent aggression towards rejecting and accepting peers. Eye gaze and pupillary reactivity predicted aggressive behavior; predictions were more successful than measures of trait-based aggression and harsh parenting. These preliminary results suggest that nonverbal behavior may elucidate underlying mechanisms of in-the-moment aggression.

Eyes don't lie: Eye movements differ during covert and overt autobiographical recall

In everyday life, autobiographical memories are revisited silently (i.e., covert recall) or shared with others (i.e., overt recall), yet most research regarding eye movements and autobiographical recall has focused on overt recall. With that in mind, the aim of the current study was to evaluate eye movements during the retrieval of autobiographical memories (with a focus on emotion), recollected during covert and overt recall. Forty-three participants recalled personal memories out loud and silently, while wearing eye-tracking glasses, and rated these memories in terms of mental imagery and emotional intensity. Analyses showed fewer and longer fixations, fewer and shorter saccades, and fewer blinks during covert recall compared with overt recall. Participants perceived more mental images and had a more intense emotional experience during covert recall. These results are discussed considering cognitive load theories and the various functions of autobiographical recall. We theorize that fewer and longer fixations during covert recall may be due to more intense mental imagery. This study enriches the field of research on eye movements and autobiographical memory by addressing how we retrieve memories silently, a common activity of everyday life. More broadly, our results contribute to building objective tools to measure autobiographical memory, alongside already existing subjective scales.

Did You Get That? Predicting Learners' Comprehension of a Video Lecture from Visualizations of Their Gaze Data

In online lectures, unlike in face-to-face lectures, teachers lack access to (nonverbal) cues to check if their students are still "with them" and comprehend the lecture. The increasing availability of low-cost eye-trackers provides a promising solution. These devices measure unobtrusively where students look and can visualize these data to teachers. These visualizations might inform teachers about students' level of "with-me-ness" (i.e., do students look at the information that the teacher is currently talking about) and comprehension of the lecture, provided that (1) gaze measures of "with-me-ness" are related to comprehension, (2) people not trained in eye-tracking can predict students' comprehension from gaze visualizations, (3) we understand how different visualization techniques impact this prediction. We addressed these issues in two studies. In Study 1, 36 students watched a video lecture while being eye-tracked. The extent to which students looked at relevant information and the extent to which they looked at the same location as the teacher both correlated with students' comprehension (score on an open question) of the lecture. In Study 2, 50 participants watched visualizations of students' gaze (from Study 1), using six visualization techniques (dynamic and static versions of scanpaths, heatmaps, and focus maps) and were asked to predict students' posttest performance and to rate their ease of prediction. We found that people can use gaze visualizations to predict learners' comprehension above chance level, with minor differences between visualization techniques. Further research should investigate if teachers can act on the information provided by gaze visualizations and thereby improve students' learning.

Two-dimensional and three-dimensional multiple object tracking learning performance in adolescent female soccer players: The role of flow experience reflected by heart rate variability

Three-dimensional multiple object tracking (3D-MOT) has been used in various fields to mimic real-life tracking, especially in perceptual-cognitive skills training for soccer. Yet, the learning efficiency in 3D-MOT tasks has not been compared with 2D-MOT. Further, whether the advantage can be reflected by heart rate variability (HRV) based on the neurovisceral integration model should also be examined. Therefore, we used both 2D- and 3D-MOT in a brief adaptive task procedure for adolescent female soccer players with HRV measurement. A faster tracking speed threshold of participants was found in the 3D- compared to 2D-MOT, as well as average tracking speed in the last training period of 3D-MOT. Moreover, lower low frequency (LF) components of HRV in the 3D-MOT indicated a flow experience, demonstrating the provision of more attentional resources. Therefore, we observed that adolescent female soccer players demonstrated higher learning efficiency in 3D-MOT tasks in virtual reality (VR) through a higher flow experience. This study examined the learning efficiency between the two MOT tasks in the soccer domain using evidence from HRV and highlighted the utility and applicability of 3D-MOT application.

Target-absent Human Attention

The prediction of human gaze behavior is important for building human-computer interaction systems that can anticipate the user's attention. Computer vision models have been developed to predict the fixations made by people as they search for target objects. But what about when the target is not in the image? Equally important is to know how people search when they cannot find a target, and when they would stop searching. In this paper, we propose a data-driven computational model that addresses the search-termination problem and predicts the scanpath of search fixations made by people searching for targets that do not appear in images. We model visual search as an imitation learning problem and represent the internal knowledge that the viewer acquires through fixations using a novel state representation that we call Foveated Feature Maps (FFMs). FFMs integrate a simulated foveated retina into a pretrained ConvNet that produces an in-network feature pyramid, all with minimal computational overhead. Our method integrates FFMs as the state representation in inverse reinforcement learning. Experimentally, we improve the state of the art in predicting human target-absent search behavior on the COCO-Search18 dataset. Code is available at: https://github.com/cvlab-stonybrook/Target-absent-Human-Attention.

Weighting the factors affecting attention guidance during free viewing and visual search: The unexpected role of object recognition uncertainty

The factors determining how attention is allocated during visual tasks have been studied for decades, but few studies have attempted to model the weighting of several of these factors within and across tasks to better understand their relative contributions. Here we consider the roles of saliency, center bias, target features, and object recognition uncertainty in predicting the first nine changes in fixation made during free viewing and visual search tasks in the OSIE and COCO-Search18 datasets, respectively. We focus on the latter-most and least familiar of these factors by proposing a new method of quantifying uncertainty in an image, one based on object recognition. We hypothesize that the greater the number of object categories competing for an object proposal, the greater the uncertainty of how that object should be recognized and, hence, the greater the need for attention to resolve this uncertainty. As expected, we found that target features best predicted target-present search, with their dominance obscuring the use of other features. Unexpectedly, we found that target features were only weakly used during target-absent search. We also found that object recognition uncertainty outperformed an unsupervised saliency model in predicting free-viewing fixations, although saliency was slightly more predictive of search. We conclude that uncertainty in object recognition, a measure that is image computable and highly interpretable, is better than bottom-up saliency in predicting attention during free viewing.

Human classifier: Observers can deduce task solely from eye movements

Computer classifiers have been successful at classifying various tasks using eye movement statistics. However, the question of human classification of task from eye movements has rarely been studied. Across two experiments, we examined whether humans could classify task based solely on the eye movements of other individuals. In Experiment 1, human classifiers were shown one of three sets of eye movements: Fixations, which were displayed as blue circles, with larger circles meaning longer fixation durations; Scanpaths, which were displayed as yellow arrows; and Videos, in which a neon green dot moved around the screen. There was an additional Scene manipulation in which eye movement properties were displayed either on the original scene where the task (Search, Memory, or Rating) was performed or on a black background in which no scene information was available. Experiment 2 used similar methods but only displayed Fixations and Videos with the same Scene manipulation. The results of both experiments showed successful classification of Search. Interestingly, Search was best classified in the absence of the original scene, particularly in the Fixation condition. Memory also was classified above chance with the strongest classification occurring with Videos in the presence of the scene. Additional analyses on the pattern of correct responses in these two conditions demonstrated which eye movement properties successful classifiers were using. These findings demonstrate conditions under which humans can extract information from eye movement characteristics in addition to providing insight into the relative success/failure of previous computer classifiers.

Scanpath modeling and classification with hidden Markov models

How people look at visual information reveals fundamental information about them; their interests and their states of mind. Previous studies showed that scanpath, i.e., the sequence of eye movements made by an observer exploring a visual stimulus, can be used to infer observer-related (e.g., task at hand) and stimuli-related (e.g., image semantic category) information. However, eye movements are complex signals and many of these studies rely on limited gaze descriptors and bespoke datasets. Here, we provide a turnkey method for scanpath modeling and classification. This method relies on variational hidden Markov models (HMMs) and discriminant analysis (DA). HMMs encapsulate the dynamic and individualistic dimensions of gaze behavior, allowing DA to capture systematic patterns diagnostic of a given class of observers and/or stimuli. We test our approach on two very different datasets. Firstly, we use fixations recorded while viewing 800 static natural scene images, and infer an observer-related characteristic: the task at hand. We achieve an average of 55.9% correct classification rate (chance = 33%). We show that correct classification rates positively correlate with the number of salient regions present in the stimuli. Secondly, we use eye positions recorded while viewing 15 conversational videos, and infer a stimulus-related characteristic: the presence or absence of original soundtrack. We achieve an average 81.2% correct classification rate (chance = 50%). HMMs allow to integrate bottom-up, top-down, and oculomotor influences into a single model of gaze behavior. This synergistic approach between behavior and machine learning will open new avenues for simple quantification of gazing behavior. We release SMAC with HMM, a Matlab toolbox freely available to the community under an open-source license agreement.

Learning from where 'eye' remotely look or point: Impact on number line estimation error in adults

In this article, we present an investigation into the use of visual cues during number line estimation and their influence on cognitive processes for reducing number line estimation error. Participants completed a 0-1000 number line estimation task before and after a brief intervention in which they observed static-visual or dynamic-visual cues (control, anchor, gaze cursor, mouse cursor) and also made estimation marks to test effective number-target estimation. Results indicated that a significant pre-test to post-test reduction in estimation error was present for dynamic-visual cues of modelled eye-gaze and mouse cursor. However, there was no significant performance difference between pre- and post-test for the control or static anchor conditions. Findings are discussed in relation to the extent to which anchor points alone are meaningful in promoting successful segmentation of the number line and whether dynamic cues promote the utility of these locations in reducing error through attentional guidance.

What Am I Looking at? Interpreting Dynamic and Static Gaze Displays

Displays of eye movements may convey information about cognitive processes but require interpretation. We investigated whether participants were able to interpret displays of their own or others' eye movements. In Experiments 1 and 2, participants observed an image under three different viewing instructions. Then they were shown static or dynamic gaze displays and had to judge whether it was their own or someone else's eye movements and what instruction was reflected. Participants were capable of recognizing the instruction reflected in their own and someone else's gaze display. Instruction recognition was better for dynamic displays, and only this condition yielded above chance performance in recognizing the display as one's own or another person's (Experiments 1 and 2). Experiment 3 revealed that order information in the gaze displays facilitated instruction recognition when transitions between fixated regions distinguish one viewing instruction from another. Implications of these findings are discussed.

A Model of the Superior Colliculus Predicts Fixation Locations during Scene Viewing and Visual Search

Modern computational models of attention predict fixations using saliency maps and target maps, which prioritize locations for fixation based on feature contrast and target goals, respectively. But whereas many such models are biologically plausible, none have looked to the oculomotor system for design constraints or parameter specification. Conversely, although most models of saccade programming are tightly coupled to underlying neurophysiology, none have been tested using real-world stimuli and tasks. We combined the strengths of these two approaches in MASC, a model of attention in the superior colliculus (SC) that captures known neurophysiological constraints on saccade programming. We show that MASC predicted the fixation locations of humans freely viewing naturalistic scenes and performing exemplar and categorical search tasks, a breadth achieved by no other existing model. Moreover, it did this as well or better than its more specialized state-of-the-art competitors. MASC's predictive success stems from its inclusion of high-level but core principles of SC organization: an over-representation of foveal information, size-invariant population codes, cascaded population averaging over distorted visual and motor maps, and competition between motor point images for saccade programming, all of which cause further modulation of priority (attention) after projection of saliency and target maps to the SC. Only by incorporating these organizing brain principles into our models can we fully understand the transformation of complex visual information into the saccade programs underlying movements of overt attention. With MASC, a theoretical footing now exists to generate and test computationally explicit predictions of behavioral and neural responses in visually complex real-world contexts.SIGNIFICANCE STATEMENT The superior colliculus (SC) performs a visual-to-motor transformation vital to overt attention, but existing SC models cannot predict saccades to visually complex real-world stimuli. We introduce a brain-inspired SC model that outperforms state-of-the-art image-based competitors in predicting the sequences of fixations made by humans performing a range of everyday tasks (scene viewing and exemplar and categorical search), making clear the value of looking to the brain for model design. This work is significant in that it will drive new research by making computationally explicit predictions of SC neural population activity in response to naturalistic stimuli and tasks. It will also serve as a blueprint for the construction of other brain-inspired models, helping to usher in the next generation of truly intelligent autonomous systems.

Using space to represent categories: insights from gaze position

We investigated the boundaries among imagery, memory, and perception by measuring gaze during retrieved versus imagined visual information. Eye fixations during recall were bound to the location at which a specific stimulus was encoded. However, eye position information generalized to novel objects of the same category that had not been seen before. For example, encoding an image of a dog in a specific location enhanced the likelihood of looking at the same location during subsequent mental imagery of other mammals. The results suggest that eye movements can also be launched by abstract representations of categories and not exclusively by a single episode or a specific visual exemplar.

Using multidimensional scaling to quantify similarity in visual search and beyond

Visual search is one of the most widely studied topics in vision science, both as an independent topic of interest, and as a tool for studying attention and visual cognition. A wide literature exists that seeks to understand how people find things under varying conditions of difficulty and complexity, and in situations ranging from the mundane (e.g., looking for one's keys) to those with significant societal importance (e.g., baggage or medical screening). A primary determinant of the ease and probability of success during search are the similarity relationships that exist in the search environment, such as the similarity between the background and the target, or the likeness of the non-targets to one another. A sense of similarity is often intuitive, but it is seldom quantified directly. This presents a problem in that similarity relationships are imprecisely specified, limiting the capacity of the researcher to examine adequately their influence. In this article, we present a novel approach to overcoming this problem that combines multi-dimensional scaling (MDS) analyses with behavioral and eye-tracking measurements. We propose a method whereby MDS can be repurposed to successfully quantify the similarity of experimental stimuli, thereby opening up theoretical questions in visual search and attention that cannot currently be addressed. These quantifications, in conjunction with behavioral and oculomotor measures, allow for critical observations about how similarity affects performance, information selection, and information processing. We provide a demonstration and tutorial of the approach, identify documented examples of its use, discuss how complementary computer vision methods could also be adopted, and close with a discussion of potential avenues for future application of this technique.

Effects of target typicality on categorical search

The role of target typicality in a categorical visual search task was investigated by cueing observers with a target name, followed by a five-item target present/absent search array in which the target images were rated in a pretest to be high, medium, or low in typicality with respect to the basic-level target cue. Contrary to previous work, we found that search guidance was better for high-typicality targets compared to low-typicality targets, as measured by both the proportion of immediate target fixations and the time to fixate the target. Consistent with previous work, we also found an effect of typicality on target verification times, the time between target fixation and the search judgment; as target typicality decreased, verification times increased. To model these typicality effects, we trained Support Vector Machine (SVM) classifiers on the target categories, and tested these on the corresponding specific targets used in the search task. This analysis revealed significant differences in classifier confidence between the high-, medium-, and low-typicality groups, paralleling the behavioral results. Collectively, these findings suggest that target typicality broadly affects both search guidance and verification, and that differences in typicality can be predicted by distance from an SVM classification boundary.