Spatial and temporal visual attention prediction in videos using eye movement data

Mining attention distribution paradigm: Discover gaze patterns and their association rules behind the visual image

BACKGROUND AND OBJECTIVE

Attention allocation reflects the way of humans filtering and organizing the information. On one hand, different task scenarios seriously affect human's rule of attention distribution, on the other hand, visual attention reflecting the cognitive and psychological process. Most of the previous studies on visual attention allocation are based on cognitive models, predicted models, or statistical analysis of eye movement data or visual images, however, these methods are inadequate to provide an inside view of gaze behavior to reveal the attention distribution pattern within scenario context. Moreover, they seldom study the association rules of these patterns. Therefore, we adopted the big data mining approach to discover the paradigm of visual attention distribution.

METHODS

We applied the data mining method to extract the gaze patterns to discover the regularities of attention distribution behavior within the scenario context. The proposed method consists of three components, tasks scenario segmented and clustered, gaze pattern mining, and association rule of frequent pattern mining.

RESULTS

The proposed approach is tested on the operation platform. The complex operation task is simultaneously segmented and clustered with the TICC-based method and evaluated by the BCI index. The operator's eye movement frequent patterns and their association rule are discovered. The results demonstrate that our method can associate the eye-tracking data with the task-oriented scene data.

DISCUSSION

The proposed method provides the benefits of being able to explicitly express and quantitatively analyze people's visual attention patterns. The proposed method can not only be applied in the field of aerospace medicine and aviation psychology, but also can likely be applied to computer-aided diagnosis and follow-up tool for neurological disease and cognitive impairment related disease, such as ADHD (Attention Deficit Hyperactivity Disorder), neglect syndrome, social attention differences in ASD (Autism spectrum disorder).

Towards tacit knowledge mining within context: Visual cognitive graph model and eye movement image interpretation

Visual attention is one of the most important brain cognitive functions, which filters the rich information of the outside world to ensure the efficient operation of limited cognitive resources. The underlying knowledge, i.e., tacit knowledge, hidden in the human attention allocation performances, is context-related and is hard to be expressed by experts, but it is essential for novice operator training and interaction system design. Traditional models of visual attention allocation and corresponding analysis methods seldomly involve task contextual information or present the tacit knowledge in an explicit and quantified way. Thus, it is challenging to pass on the expert's tacit knowledge to the novice or utilize it to construct an interaction system by employing traditional methods. Therefore, this paper first proposes a new model called the visual cognitive graph model based on graph theory to model the visual attention allocation associated with the task context. Then, based on this graph model, utilize the data mining method to reveal attention patterns within context to quantitatively analyze the operator's tacit knowledge during operation tasks. We introduced three physical quantities derived from graph theory to describe the tacit knowledge, which can be used directly to construct an interaction system or operator training. For example, discover the essential information within the task context, the relevant information affecting critical information, and the bridge information revealing the decision-making process. We tested the proposed method in the example of flight operation, the comparison results with the traditional eye movement graph model demonstrate that the proposed visual cognitive model can compromise the task context. The comparison results with the statistical analysis method demonstrate that our tacit knowledge mining method can reveal the underlying knowledge hidden in the visual information. Finally, we give practical applications in the examples of operator training guidance and adaptive interaction system. Our proposed method can explore more in-depth knowledge of visual information, such as the correlations of different obtained information and the way operator obtains information, most of which are even not noticed by operators themselves.

SG-FCN: A Motion and Memory-Based Deep Learning Model for Video Saliency Detection

Data-driven saliency detection has attracted strong interest as a result of applying convolutional neural networks to the detection of eye fixations. Although a number of image-based salient object and fixation detection models have been proposed, video fixation detection still requires more exploration. Different from image analysis, motion and temporal information is a crucial factor affecting human attention when viewing video sequences. Although existing models based on local contrast and low-level features have been extensively researched, they failed to simultaneously consider interframe motion and temporal information across neighboring video frames, leading to unsatisfactory performance when handling complex scenes. To this end, we propose a novel and efficient video eye fixation detection model to improve the saliency detection performance. By simulating the memory mechanism and visual attention mechanism of human beings when watching a video, we propose a step-gained fully convolutional network by combining the memory information on the time axis with the motion information on the space axis while storing the saliency information of the current frame. The model is obtained through hierarchical training, which ensures the accuracy of the detection. Extensive experiments in comparison with 11 state-of-the-art methods are carried out, and the results show that our proposed model outperforms all 11 methods across a number of publicly available datasets.

Learning to Predict Sequences of Human Visual Fixations

Most state-of-the-art visual attention models estimate the probability distribution of fixating the eyes in a location of the image, the so-called saliency maps. Yet, these models do not predict the temporal sequence of eye fixations, which may be valuable for better predicting the human eye fixations, as well as for understanding the role of the different cues during visual exploration. In this paper, we present a method for predicting the sequence of human eye fixations, which is learned from the recorded human eye-tracking data. We use least-squares policy iteration (LSPI) to learn a visual exploration policy that mimics the recorded eye-fixation examples. The model uses a different set of parameters for the different stages of visual exploration that capture the importance of the cues during the scanpath. In a series of experiments, we demonstrate the effectiveness of using LSPI for combining multiple cues at different stages of the scanpath. The learned parameters suggest that the low-level and high-level cues (semantics) are similarly important at the first eye fixation of the scanpath, and the contribution of high-level cues keeps increasing during the visual exploration. Results show that our approach obtains the state-of-the-art performances on two challenging data sets: 1) OSIE data set and 2) MIT data set.