An inverse Yarbus process: predicting observers' task from eye movement patterns

In this paper we develop a probabilistic method to infer the visual-task of a viewer given measured eye movement trajectories. This method is based on the theory of hidden Markov models (HMM) that employs a first order Markov process to predict the coordinates of fixations given the task. The prediction confidence level of each task-dependent model is used in a Bayesian inference formulation, whereby the task with the maximum a posteriori (MAP) probability is selected. We applied this technique to a challenging dataset consisting of eye movement trajectories obtained from subjects viewing monochrome images of real scenes tasked with answering questions regarding the scenes. The results show that the HMM approach, combined with a clustering technique, can be a reliable way to infer visual-task from eye movements data.

A computational model for task inference in visual search

We develop a probabilistic framework to infer the ongoing task in visual search by revealing what the subject is looking for during a search process. Based on the level of difficulty, two types of tasks, easy and difficult, are investigated in this work, and individual models are customized for them according to their specific dynamics. We use Hidden Markov Models (HMMs) to serve as a model for the human cognitive process that is responsible for directing the center of gaze (COG) according to the task at hand during visual search and generating task-dependent eye trajectories. This generative model, then, is used to estimate the likelihood term in a Bayesian inference formulation to infer the task given the eye trajectory. In the easy task, focus of attention (FOA) often lands on targets, whereas in the difficult one, in addition to the on-target foci of attention, deployment of attention on nontarget objects happens very often. Therefore, we suggest a single-state and a multi-state HMM to serve as the cognitive process model of attention for the easy and difficult tasks, respectively.