Eye movement analysis with switching hidden Markov models

Modeling Eye Movements During Decision Making: A Review

This article reviews recent advances in the psychometric and econometric modeling of eye-movements during decision making. Eye movements offer a unique window on unobserved perceptual, cognitive, and evaluative processes of people who are engaged in decision making tasks. They provide new insights into these processes, which are not easily available otherwise, allow for explanations of fundamental search and choice phenomena, and enable predictions of future decisions. We propose a theoretical framework of the search and choice tasks that people commonly engage in and of the underlying cognitive processes involved in those tasks. We discuss how these processes drive specific eye-movement patterns. Our framework emphasizes the central role of task and strategy switching for complex goal attainment. We place the extant literature within that framework, highlight recent advances in modeling eye-movement behaviors during search and choice, discuss limitations, challenges, and open problems. An agenda for further psychometric modeling of eye movements during decision making concludes the review.

Clustering Hidden Markov Models With Variational Bayesian Hierarchical EM

The hidden Markov model (HMM) is a broadly applied generative model for representing time-series data, and clustering HMMs attract increased interest from machine learning researchers. However, the number of clusters ( K ) and the number of hidden states ( S ) for cluster centers are still difficult to determine. In this article, we propose a novel HMM-based clustering algorithm, the variational Bayesian hierarchical EM algorithm, which clusters HMMs through their densities and priors and simultaneously learns posteriors for the novel HMM cluster centers that compactly represent the structure of each cluster. The numbers K and S are automatically determined in two ways. First, we place a prior on the pair (K,S) and approximate their posterior probabilities, from which the values with the maximum posterior are selected. Second, some clusters and states are pruned out implicitly when no data samples are assigned to them, thereby leading to automatic selection of the model complexity. Experiments on synthetic and real data demonstrate that our algorithm performs better than using model selection techniques with maximum likelihood estimation.

Understanding the role of eye movement consistency in face recognition and autism through integrating deep neural networks and hidden Markov models

Greater eyes-focused eye movement pattern during face recognition is associated with better performance in adults but not in children. We test the hypothesis that higher eye movement consistency across trials, instead of a greater eyes-focused pattern, predicts better performance in children since it reflects capacity in developing visual routines. We first simulated visual routine development through combining deep neural network and hidden Markov model that jointly learn perceptual representations and eye movement strategies for face recognition. The model accounted for the advantage of eyes-focused pattern in adults, and predicted that in children (partially trained models) consistency but not pattern of eye movements predicted recognition performance. This result was then verified with data from typically developing children. In addition, lower eye movement consistency in children was associated with autism diagnosis, particularly autistic traits in social skills. Thus, children's face recognition involves visual routine development through social exposure, indexed by eye movement consistency.

Improving the Deeplabv3+ Model with Attention Mechanisms Applied to Eye Detection and Segmentation

Research on eye detection and segmentation is even more important with mask-wearing measures implemented during the COVID-19 pandemic. Thus, it is necessary to build an eye image detection and segmentation dataset (EIMDSD), including labels for detecting and segmenting. In this study, we established a dataset to reduce elaboration for chipping eye images and denoting labels. An improved DeepLabv3+ network architecture (IDLN) was also proposed for applying it to the benchmark segmentation datasets. The IDLN was modified by cascading convolutional block attention modules (CBAM) with MobileNetV2. Experiments were carried out to verify the effectiveness of the EIMDSD dataset in human eye image detection and segmentation with different deep learning models. The result shows that the IDLN model achieves the appropriate segmentation accuracy for both eye images, while the UNet and ISANet models show the best results for the left eye data and the right eye data among the tested models.

Eye movement analysis of children's attention for midline diastema

No previous studies have investigated eye-movement patterns to show children's information processing while viewing clinical images. Therefore, this study aimed to explore children and their educators' perception of a midline diastema by applying eye-movement analysis using the hidden Markov models (EMHMM). A total of 155 children between 2.5 and 5.5 years of age and their educators (n = 34) viewed pictures with and without a midline diastema while Tobii Pro Nano eye-tracker followed their eye movements. Fixation data were analysed using data-driven, and fixed regions of interest (ROIs) approaches with EMHMM. Two different eye-movement patterns were identified: explorative pattern (76%), where the children's ROIs were predominantly around the nose and mouth, and focused pattern (26%), where children's ROIs were precise, locating on the teeth with and without a diastema, and fixations transited among the ROIs with similar frequencies. Females had a significantly higher eye-movement preference for without diastema image than males. Comparisons between the different age groups showed a statistically significant difference for overall entropies. The 3.6-4.5y age groups exhibited higher entropies, indicating lower eye-movement consistency. In addition, children and their educators exhibited two specific eye-movement patterns. Children in the explorative pattern saw the midline diastema more often while their educators focussed on the image without diastema. Thus, EMHMMs are valuable in analysing eye-movement patterns in children and adults.

Understanding children’s attention to dental caries through eye-tracking

Visual attention is a significant gateway to a child's mind, and looking is one of the first behaviors young children develop. Untreated caries and the resulting poor dental aesthetics can have adverse emotional and social impacts on children's oral health-related quality of life due to its detrimental effects on self-esteem and self-concept. Therefore, we explored preschool children's eye movement patterns and visual attention to images with and without dental caries via eye movement analysis using hidden Markov models (EMHMM). We calibrated a convenience sample of 157 preschool children to the eye-tracker (Tobii Nano Pro) to ensure standardization. Consequently, each participant viewed the same standardized pictures with and without dental caries while an eye-tracking device tracked their eye movements. Subsequently, based on the sequence of viewed regions of interest (ROIs), a transition matrix was developed where the participants' previously viewed ROI informed their subsequently considered ROI. Hence, an individual's HMM was estimated from their eye movement data using a variational Bayesian approach to determine the optimal number of ROIs automatically. Consequently, this data-driven approach generated the visual task participants' most representative eye movement patterns. Preschool children exhibited two different eye movement patterns, distributed (78%) and selective (21%), which was statistically significant. Children switched between images with more similar probabilities in the distributed pattern while children remained looking at the same ROI than switching to the other ROI in the selective pattern. Nevertheless, all children exhibited an equal starting fixation on the right or left image and noticed teeth. The study findings reveal that most preschool children did not have an attentional bias to images with and without dental caries. Furthermore, only a few children selectively fixated on images with dental caries. Therefore, selective eye-movement patterns may strongly predict preschool children's sustained visual attention to dental caries. Nevertheless, future studies are essential to fully understand the developmental origins of differences in visual attention to common oral health presentations in children. Finally, EMHMM is appropriate for assessing inter-individual differences in children's visual attention.

Impact of mask use on face recognition: an eye-tracking study

We examined how mask use affects performance and eye movements in face recognition and whether strategy change reflected in eye movements is associated with performance change. Eighty-eight participants performed face recognition with masked faces either during learning only, during recognition only, or during both learning and recognition. As compared with the baseline condition where faces were unmasked during both learning and recognition, participants had impaired performance in all three scenarios, with larger impairment when mask conditions during learning and recognition did not match. When recognizing unmasked faces, whether the faces were learned with or without a mask on did not change eye movement behavior. Nevertheless, when recognizing unmasked faces that were learned with a mask on, participants who adopted more eyes-focused patterns had less performance impairment as compared with the baseline condition. When recognizing masked faces, participants had more eyes-focused patterns and more consistent gaze transition behavior than recognizing unmasked faces regardless of whether the faces were learned with or without a mask on. Nevertheless, when recognizing masked faces that were learned without a mask, participants whose gaze transition behavior was more consistent had less performance impairment as compared with the baseline condition. Thus, although eye movements during recognition were mainly driven by the mask condition during recognition but not that during learning, those who adjusted their strategy according to the mask condition difference between learning and recognition had better performance. This finding has important implications for identifying populations vulnerable to the impact of mask use and potential remedial strategies.

Sound Source Selection Based on Head Movements in Natural Group Conversation

To optimally improve signal-to-noise ratio in noisy environments, a hearing assistance device must correctly identify what is signal and what is noise. Many of the biosignal-based approaches to solving this question are themselves subject to noise, but head angle is an overt behavior that may be possible to capture in practical devices in the real world. Previous orientation studies have demonstrated that head angle is systematically related to listening target; our study aimed to examine whether this relationship is sufficiently reliable to be used in group conversations where participants may be seated in different layouts and the listener is free to turn their body as well as their head. In addition to this simple method, we developed a source-selection algorithm based on a hidden Markov model (HMM) trained on listeners’ head movement. The performance of this model and the simple head-steering method was evaluated using publicly available behavioral data. Head angle during group conversation was predictive of active talker, exhibiting an undershoot with a slope consistent with that found in simple orientation studies, but the intercept of the linear relationship was different for different talker layouts, suggesting it would be problematic to rely exclusively on this information to predict the location of auditory attention. Provided the location of all target talkers is known, the HMM source selection model implemented here, however, showed significantly lower error in identifying listeners’ auditory attention than the linear head-steering method.

Understanding the collinear masking effect in visual search through eye tracking

Recent research has reported that, while both orientation contrast and collinearity increase target salience in visual search, a combination of the two counterintuitively masks a local target. Through eye-tracking and eye-movement analysis with hidden Markov models (EMHMM), here we showed that this collinear masking effect was associated with reduced eye-fixation consistency (as measured in entropy) at the central fixation cross prior to the search display presentation. As a decreased precision of saccade landing position is shown to be related to attention shift away from the saccadic target, our result suggested that the collinear masking effect may be related to attention shift to a non-saccadic-goal location in expectation of the search display before saccading to the central fixation cross. This attention shift may consequently interfere with attention capture by the collinear distractor containing the target, resulting in the masking effect. In contrast, although older adults had longer response times, more dispersed eye-movement pattern, and lower eye-movement consistency than young adults during visual search, the two age groups did not differ in the masking effect, suggesting limited contribution from ageing-related cognitive decline. Thus, participants' pre-saccadic attention shift prior to search may be an important factor influencing their search behavior.

Eye movement analysis with hidden Markov models (EMHMM) with co-clustering

The eye movement analysis with hidden Markov models (EMHMM) method provides quantitative measures of individual differences in eye-movement pattern. However, it is limited to tasks where stimuli have the same feature layout (e.g., faces). Here we proposed to combine EMHMM with the data mining technique co-clustering to discover participant groups with consistent eye-movement patterns across stimuli for tasks involving stimuli with different feature layouts. Through applying this method to eye movements in scene perception, we discovered explorative (switching between the foreground and background information or different regions of interest) and focused (mainly looking at the foreground with less switching) eye-movement patterns among Asian participants. Higher similarity to the explorative pattern predicted better foreground object recognition performance, whereas higher similarity to the focused pattern was associated with better feature integration in the flanker task. These results have important implications for using eye tracking as a window into individual differences in cognitive abilities and styles. Thus, EMHMM with co-clustering provides quantitative assessments on eye-movement patterns across stimuli and tasks. It can be applied to many other real-life visual tasks, making a significant impact on the use of eye tracking to study cognitive behavior across disciplines.

How Do Art Skills Influence Visual Search? - Eye Movements Analyzed With Hidden Markov Models

The results of two experiments are analyzed to find out how artistic expertise influences visual search. Experiment I comprised survey data of 1,065 students on self-reported visual memory skills and their ability to find three targets in four images of artwork. Experiment II comprised eye movement data of 50 Visual Literacy (VL) experts and non-experts whose eye movements during visual search were analyzed for nine images of artwork as an external validation of the assessment tasks performed in Sample I. No time constraint was set for completion of the visual search task. A latent profile analysis revealed four typical solution patterns for the students in Sample I, including a mainstream group, a group that completes easy images fast and difficult images slowly, a fast and erroneous group, and a slow working student group, depending on task completion time and on the probability of finding all three targets. Eidetic memory, performance in art education and visual imagination as self-reported visual skills have significant impact on latent class membership probability. We present a hidden Markov model (HMM) approach to uncover underlying regions of attraction that result from visual search eye-movement behavior in Experiment II. VL experts and non-experts did not significantly differ in task time and number of targets found but they did differ in their visual search process: compared to non-experts, experts showed greater precision in fixating specific prime and target regions, assessed through hidden state fixation overlap. Exploratory analysis of HMMs revealed differences between experts and non-experts in image locations of attraction (HMM states). Experts seem to focus their attention on smaller image parts whereas non-experts used wider parts of the image during their search. Differences between experts and non-experts depend on the relative saliency of targets embedded in images. HMMs can determine the effect of expertise on exploratory eye movements executed during visual search tasks. Further research on HMMs and art expertise is required to confirm exploratory results.

Detecting Changes and Avoiding Catastrophic Forgetting in Dynamic Partially Observable Environments

The ability of an agent to detect changes in an environment is key to successful adaptation. This ability involves at least two phases: learning a model of an environment, and detecting that a change is likely to have occurred when this model is no longer accurate. This task is particularly challenging in partially observable environments, such as those modeled with partially observable Markov decision processes (POMDPs). Some predictive learners are able to infer the state from observations and thus perform better with partial observability. Predictive state representations (PSRs) and neural networks are two such tools that can be trained to predict the probabilities of future observations. However, most such existing methods focus primarily on static problems in which only one environment is learned. In this paper, we propose an algorithm that uses statistical tests to estimate the probability of different predictive models to fit the current environment. We exploit the underlying probability distributions of predictive models to provide a fast and explainable method to assess and justify the model's beliefs about the current environment. Crucially, by doing so, the method can label incoming data as fitting different models, and thus can continuously train separate models in different environments. This new method is shown to prevent catastrophic forgetting when new environments, or tasks, are encountered. The method can also be of use when AI-informed decisions require justifications because its beliefs are based on statistical evidence from observations. We empirically demonstrate the benefit of the novel method with simulations in a set of POMDP environments.

An Analysis of Eye-Tracking Features and Modelling Methods for Free-Viewed Standard Stimulus: Application for Schizophrenia Detection

Currently psychiatry is a medical field lacking an automated diagnostic process. The presence of a mental disorder is established by observing its typical symptoms. Eye-movement specifics have already been established as an "endophenotype" for schizophrenia, but an automated diagnostic process of eye-movement analysis is still lacking. This article presents several novel approaches for the automatic detection of a schizophrenic disorder based on a free-view image test using a Rorschach inkblot and an eye tracker. Several features that enabled us to analyse the eye-tracker signal as a whole as well as its specific parts were tested. The variety of features spans global (heat maps, gaze plots), sequences of features (means, variances, and spectra), static (x and y signals as 2D images), dynamic (velocities), and model-based (limiting probabilities and transition matrices) categories. For each set of features, a proper modelling and classification method was designed (convolutional, recurrent, fully connected and combined neural networks; Hidden Markov models). By doing so, it was possible to find the importance of each feature and its physical representation using k-fold cross validation and a paired t-test. The dataset was sampled on 22 people with schizophrenia and 22 healthy individuals. The most successful approach was based on heat maps using all data and convolutional networks, reaching a 78.8% accuracy, which is a 10.5% improvement over the reference method. From all tested methods, there are two in an 85% accuracy range and over fifteen others in a 75% accuracy range at a 10% significance level.