Eye movement analysis with hidden Markov models (EMHMM) with co-clustering

Spatiotemporal Eye Movement Dynamics Reveal Altered Face Prioritization in Early Visual Processing Among Autistic Children

BACKGROUND

Reduced social attention - looking at faces - is one of the most common manifestations of social difficulty in autism central to social development. Although reduced social attention is well-characterized in autism, qualitative differences in how social attention unfolds across time remains unknown.

METHODS

We used a computational modeling (i.e., hidden Markov modeling) approach to assess and compare the spatiotemporal dynamics of social attention in a large, well-characterized sample of autistic (n = 280) and neurotypical (n = 120) children (ages 6-11) that completed three social eye-tracking assays across three longitudinal time points (Baseline, 6 weeks, 24 weeks).

RESULTS

Our analysis supported the existence of two common eye movement patterns that emerged across three ET assays. A focused pattern was characterized by small face regions of interest, which had high probability of capturing fixations early in visual processing. In contrast, an exploratory pattern was characterized by larger face regions of interest, with lower initial probability of fixation, and more non-social regions of interest. In the context of social perception, autistic children showed significantly more exploratory eye movement patterns than neurotypical children across all social perception assays and all three longitudinal time points. Eye movement patterns were associated with clinical features of autism, including adaptive function, face recognition, and autism symptom severity.

CONCLUSIONS

Decreased likelihood of precisely looking to faces early in social visual processing may be an important feature of autism that was associated with autism-related symptomology and may reflect less visual sensitivity to face information.

Human attention guided explainable artificial intelligence for computer vision models

Explainable artificial intelligence (XAI) has been increasingly investigated to enhance the transparency of black-box artificial intelligence models, promoting better user understanding and trust. Developing an XAI that is faithful to models and plausible to users is both a necessity and a challenge. This work examines whether embedding human attention knowledge into saliency-based XAI methods for computer vision models could enhance their plausibility and faithfulness. Two novel XAI methods for object detection models, namely FullGrad-CAM and FullGrad-CAM++, were first developed to generate object-specific explanations by extending the current gradient-based XAI methods for image classification models. Using human attention as the objective plausibility measure, these methods achieve higher explanation plausibility. Interestingly, all current XAI methods when applied to object detection models generally produce saliency maps that are less faithful to the model than human attention maps from the same object detection task. Accordingly, human attention-guided XAI (HAG-XAI) was proposed to learn from human attention how to best combine explanatory information from the models to enhance explanation plausibility by using trainable activation functions and smoothing kernels to maximize the similarity between XAI saliency map and human attention map. The proposed XAI methods were evaluated on widely used BDD-100K, MS-COCO, and ImageNet datasets and compared with typical gradient-based and perturbation-based XAI methods. Results suggest that HAG-XAI enhanced explanation plausibility and user trust at the expense of faithfulness for image classification models, and it enhanced plausibility, faithfulness, and user trust simultaneously and outperformed existing state-of-the-art XAI methods for object detection models.

Understanding the Role of Eye Movement Pattern and Consistency in Isolated English Word Reading Through Hidden Markov Modeling

In isolated English word reading, readers have the optimal performance when their initial eye fixation is directed to the area between the beginning and word center, that is, the optimal viewing position (OVP). Thus, how well readers voluntarily direct eye gaze to this OVP during isolated word reading may be associated with reading performance. Using Eye Movement analysis with Hidden Markov Models, we discovered two representative eye movement patterns during lexical decisions through clustering, which focused at the OVP and the word center, respectively. Higher eye movement similarity to the OVP-focusing pattern predicted faster lexical decision time in addition to cognitive abilities and lexical knowledge. However, the OVP-focusing pattern was associated with longer isolated single letter naming time, suggesting conflicting visual abilities required for identifying isolated letters and multi-letter words. In contrast, in both word and pseudoword naming, although clustering did not reveal an OVP-focused pattern, higher consistency of the first fixation as measured in entropy predicted faster naming time in addition to cognitive abilities and lexical knowledge. Thus, developing a consistent eye movement pattern focusing on the OVP is essential for word orthographic processing and reading fluency. This finding has important implications for interventions for reading difficulties.

Idiosyncratic fixation patterns generalize across dynamic and static facial expression recognition

Facial expression recognition (FER) is crucial for understanding the emotional state of others during human social interactions. It has been assumed that humans share universal visual sampling strategies to achieve this task. However, recent studies in face identification have revealed striking idiosyncratic fixation patterns, questioning the universality of face processing. More importantly, very little is known about whether such idiosyncrasies extend to the biological relevant recognition of static and dynamic facial expressions of emotion (FEEs). To clarify this issue, we tracked observers' eye movements categorizing static and ecologically valid dynamic faces displaying the six basic FEEs, all normalized for time presentation (1 s), contrast and global luminance across exposure time. We then used robust data-driven analyses combining statistical fixation maps with hidden Markov Models to explore eye-movements across FEEs and stimulus modalities. Our data revealed three spatially and temporally distinct equally occurring face scanning strategies during FER. Crucially, such visual sampling strategies were mostly comparably effective in FER and highly consistent across FEEs and modalities. Our findings show that spatiotemporal idiosyncratic gaze strategies also occur for the biologically relevant recognition of FEEs, further questioning the universality of FER and, more generally, face processing.

Explanation strategies in humans versus current explainable artificial intelligence: Insights from image classification

Explainable AI (XAI) methods provide explanations of AI models, but our understanding of how they compare with human explanations remains limited. Here, we examined human participants' attention strategies when classifying images and when explaining how they classified the images through eye-tracking and compared their attention strategies with saliency-based explanations from current XAI methods. We found that humans adopted more explorative attention strategies for the explanation task than the classification task itself. Two representative explanation strategies were identified through clustering: One involved focused visual scanning on foreground objects with more conceptual explanations, which contained more specific information for inferring class labels, whereas the other involved explorative scanning with more visual explanations, which were rated higher in effectiveness for early category learning. Interestingly, XAI saliency map explanations had the highest similarity to the explorative attention strategy in humans, and explanations highlighting discriminative features from invoking observable causality through perturbation had higher similarity to human strategies than those highlighting internal features associated with higher class score. Thus, humans use both visual and conceptual information during explanation, which serve different purposes, and XAI methods that highlight features informing observable causality match better with human explanations, potentially more accessible to users.

Autistic Individuals Do Not Alter Visual Processing Strategy During Encoding Versus Recognition of Faces: A Hidden Markov Modeling Approach

PURPOSE

Visual face recognition-the ability to encode, discriminate, and recognize the faces of others-is fundamentally supported by eye movements and is a common source of difficulty for autistic individuals. We aimed to evaluate how visual processing strategies (i.e., eye movement patterns) directly support encoding and recognition of faces in autistic and neurotypical (NT) individuals.

METHODS

We used a hidden Markov modeling approach to evaluate the spatiotemporal dynamics of eye movements in autistic (n = 15) and neurotypical (NT) adolescents (n = 17) during a face identity recognition task.

RESULTS

We discovered distinct eye movement patterns among all participants, which included a focused and exploratory strategy. When evaluating change in visual processing strategy across encoding and recognition phases, autistic individuals did not shift their eye movement patterns like their NT peers, who shifted to a more exploratory visual processing strategy during recognition.

CONCLUSION

These findings suggest that autistic individuals do not modulate their visual processing strategy across encoding and recognition of faces, which may be an indicator of less efficient face processing.

Attentional heterogeneity in social anxiety disorder: Evidence from Hidden Markov Models

There is some evidence for heterogeneity in attentional processes among individuals with social anxiety. However, there is limited work considering how attentional processes may differ as a mechanism in a naturalistic task-based context (e.g., public speaking). In this secondary analysis we tested attentional heterogeneity among individuals diagnosed with social anxiety disorder (N = 21) in the context of a virtual reality exposure treatment study. Participants completed a public speaking challenge in an immersive 360°-video virtual reality environment with eye tracking at pre-treatment, post-treatment, and at 1-week follow-up. Using a Hidden Markov Model (HMM) approach with clustering we tested whether there were distinct profiles of attention pre-treatment and whether there were changes following the intervention. As a secondary aim we tested whether the distinct attentional profiles at pre-treatment predicted differential treatment outcomes. We found two distinct attentional profiles pre-treatment that we characterized as audience-focused and audience-avoidant. However, by the 1-week follow-up the two profiles were no longer meaningfully different. We found a meaningful difference between HMM groups for fear of public speaking at post-treatment b = -8.54, 95% Highest Density Interval (HDI) [-16.00, -0.90], Bayes Factor (BF) = 8.31 but not at one-week follow-up b = -5.83, 95% HDI [-13.25, 1.81], BF = 2.28. These findings provide support for heterogeneity in attentional processes among socially anxious individuals, but our findings indicate that this may change following treatment. Moreover, our results offer preliminary mechanistic evidence that patterns of avoidance may be specifically related to poorer treatment outcomes for virtual reality exposure therapy.

Refixation behavior in naturalistic viewing: Methods, mechanisms, and neural correlates

When freely viewing a scene, the eyes often return to previously visited locations. By tracking eye movements and coregistering eye movements and EEG, such refixations are shown to have multiple roles: repairing insufficient encoding from precursor fixations, supporting ongoing viewing by resampling relevant locations prioritized by precursor fixations, and aiding the construction of memory representations. All these functions of refixation behavior are understood to be underpinned by three oculomotor and cognitive systems and their associated brain structures. First, immediate saccade planning prior to refixations involves attentional selection of candidate locations to revisit. This process is likely supported by the dorsal attentional network. Second, visual working memory, involved in maintaining task-related information, is likely supported by the visual cortex. Third, higher-order relevance of scene locations, which depends on general knowledge and understanding of scene meaning, is likely supported by the hippocampal memory system. Working together, these structures bring about viewing behavior that balances exploring previously unvisited areas of a scene with exploiting visited areas through refixations.

The effects of object size on spatial orientation: an eye movement study

Introduction

The processing of visual information in the human brain is divided into two streams, namely, the dorsal and ventral streams, object identification is related to the ventral stream and motion processing is related to the dorsal stream. Object identification is interconnected with motion processing, object size was found to affect the information processing of motion characteristics in uniform linear motion. However, whether the object size affects the spatial orientation is still unknown.

Methods

Thirty-eight college students were recruited to participate in an experiment based on the spatial visualization dynamic test. Eyelink 1,000 Plus was used to collect eye movement data. The final direction difference (the difference between the final moving direction of the target and the final direction of the moving target pointing to the destination point), rotation angle (the rotation angle of the knob from the start of the target movement to the moment of key pressing) and eye movement indices under conditions of different object sizes and motion velocities were compared.

Results

The final direction difference and rotation angle under the condition of a 2.29°-diameter moving target and a 0.76°-diameter destination point were significantly smaller than those under the other conditions (a 0.76°-diameter moving target and a 0.76°-diameter destination point; a 0.76°-diameter moving target and a 2.29°-diameter destination point). The average pupil size under the condition of a 2.29°-diameter moving target and a 0.76°-diameter destination point was significantly larger than the average pupil size under other conditions (a 0.76°-diameter moving target and a 0.76°-diameter destination point; a 0.76°-diameter moving target and a 2.29°-diameter destination point).

Discussion

A relatively large moving target can resist the landmark attraction effect in spatial orientation, and the influence of object size on spatial orientation may originate from differences in cognitive resource consumption. The present study enriches the interaction theory of the processing of object characteristics and motion characteristics and provides new ideas for the application of eye movement technology in the examination of spatial orientation ability.

Influence of prior knowledge on eye movements to scenes as revealed by hidden Markov models

Human visual experience usually provides ample opportunity to accumulate knowledge about events unfolding in the environment. In typical scene perception experiments, however, participants view images that are unrelated to each other and, therefore, they cannot accumulate knowledge relevant to the upcoming visual input. Consequently, the influence of such knowledge on how this input is processed remains underexplored. Here, we investigated this influence in the context of gaze control. We used sequences of static film frames arranged in a way that allowed us to compare eye movements to identical frames between two groups: a group that accumulated prior knowledge relevant to the situations depicted in these frames and a group that did not. We used a machine learning approach based on hidden Markov models fitted to individual scanpaths to demonstrate that the gaze patterns from the two groups differed systematically and, thereby, showed that recently accumulated prior knowledge contributes to gaze control. Next, we leveraged the interpretability of hidden Markov models to characterize these differences. Additionally, we report two unexpected and interesting caveats of our approach. Overall, our results highlight the importance of recently acquired prior knowledge for oculomotor control and the potential of hidden Markov models as a tool for investigating it.

Clustering Hidden Markov Models With Variational Bayesian Hierarchical EM

The hidden Markov model (HMM) is a broadly applied generative model for representing time-series data, and clustering HMMs attract increased interest from machine learning researchers. However, the number of clusters ( K ) and the number of hidden states ( S ) for cluster centers are still difficult to determine. In this article, we propose a novel HMM-based clustering algorithm, the variational Bayesian hierarchical EM algorithm, which clusters HMMs through their densities and priors and simultaneously learns posteriors for the novel HMM cluster centers that compactly represent the structure of each cluster. The numbers K and S are automatically determined in two ways. First, we place a prior on the pair (K,S) and approximate their posterior probabilities, from which the values with the maximum posterior are selected. Second, some clusters and states are pruned out implicitly when no data samples are assigned to them, thereby leading to automatic selection of the model complexity. Experiments on synthetic and real data demonstrate that our algorithm performs better than using model selection techniques with maximum likelihood estimation.

Understanding the role of eye movement consistency in face recognition and autism through integrating deep neural networks and hidden Markov models

Greater eyes-focused eye movement pattern during face recognition is associated with better performance in adults but not in children. We test the hypothesis that higher eye movement consistency across trials, instead of a greater eyes-focused pattern, predicts better performance in children since it reflects capacity in developing visual routines. We first simulated visual routine development through combining deep neural network and hidden Markov model that jointly learn perceptual representations and eye movement strategies for face recognition. The model accounted for the advantage of eyes-focused pattern in adults, and predicted that in children (partially trained models) consistency but not pattern of eye movements predicted recognition performance. This result was then verified with data from typically developing children. In addition, lower eye movement consistency in children was associated with autism diagnosis, particularly autistic traits in social skills. Thus, children's face recognition involves visual routine development through social exposure, indexed by eye movement consistency.

Music reading experience modulates eye movement pattern in English reading but not in Chinese reading

Here we tested the hypothesis that in Chinese-English bilinguals, music reading experience may modulate eye movement planning in reading English but not Chinese sentences due to the similarity in perceptual demands on processing sequential symbol strings separated by spaces between music notation and English sentence reading. Chinese–English bilingual musicians and non-musicians read legal, semantically incorrect, and syntactically (and semantically) incorrect sentences in both English and Chinese. In English reading, musicians showed more dispersed eye movement patterns in reading syntactically incorrect sentences than legal sentences, whereas non-musicians did not. This effect was not observed in Chinese reading. Musicians also had shorter saccade lengths when viewing syntactically incorrect than correct musical notations and sentences in an unfamiliar alphabetic language (Tibetan), whereas non-musicians did not. Thus, musicians’ eye movement planning was disturbed by syntactic violations in both music and English reading but not in Chinese reading, and this effect was generalized to an unfamiliar alphabetic language. These results suggested that music reading experience may modulate perceptual processes in reading differentially in bilinguals’ two languages, depending on their processing similarities.

Impact of mask use on face recognition: an eye-tracking study

We examined how mask use affects performance and eye movements in face recognition and whether strategy change reflected in eye movements is associated with performance change. Eighty-eight participants performed face recognition with masked faces either during learning only, during recognition only, or during both learning and recognition. As compared with the baseline condition where faces were unmasked during both learning and recognition, participants had impaired performance in all three scenarios, with larger impairment when mask conditions during learning and recognition did not match. When recognizing unmasked faces, whether the faces were learned with or without a mask on did not change eye movement behavior. Nevertheless, when recognizing unmasked faces that were learned with a mask on, participants who adopted more eyes-focused patterns had less performance impairment as compared with the baseline condition. When recognizing masked faces, participants had more eyes-focused patterns and more consistent gaze transition behavior than recognizing unmasked faces regardless of whether the faces were learned with or without a mask on. Nevertheless, when recognizing masked faces that were learned without a mask, participants whose gaze transition behavior was more consistent had less performance impairment as compared with the baseline condition. Thus, although eye movements during recognition were mainly driven by the mask condition during recognition but not that during learning, those who adjusted their strategy according to the mask condition difference between learning and recognition had better performance. This finding has important implications for identifying populations vulnerable to the impact of mask use and potential remedial strategies.

When Eyes Wander Around: Mind-Wandering as Revealed by Eye Movement Analysis with Hidden Markov Models

Mind-wandering has been shown to largely influence our learning efficiency, especially in the digital and distracting era nowadays. Detecting mind-wandering thus becomes imperative in educational scenarios. Here, we used a wearable eye-tracker to record eye movements during the sustained attention to response task. Eye movement analysis with hidden Markov models (EMHMM), which takes both spatial and temporal eye-movement information into account, was used to examine if participants' eye movement patterns can differentiate between the states of focused attention and mind-wandering. Two representative eye movement patterns were discovered through clustering using EMHMM: centralized and distributed patterns. Results showed that participants with the centralized pattern had better performance on detecting targets and rated themselves as more focused than those with the distributed pattern. This study indicates that distinct eye movement patterns are associated with different attentional states (focused attention vs. mind-wandering) and demonstrates a novel approach in using EMHMM to study attention. Moreover, this study provides a potential approach to capture the mind-wandering state in the classroom without interrupting the ongoing learning behavior.

Cracking the Code of Live Human Social Interactions in Autism: A Review of the Eye-Tracking Literature

Human social interaction involves a complex, dynamic exchange of verbal and non-verbal information. Over the last decade, eye-tracking technology has afforded unique insight into the way eye gaze information, including both holding gaze and shifting gaze, organizes live human interactions. For example, while playing a social game together, speakers end their turn by directing gaze at the listener, who begins to speak with averted gaze (Ho et al., 2015). These findings reflect how eye gaze can be used to signal important turn-taking transitions in social interactions. Deficits in conversational turn-taking is a core feature of autism spectrum disorders. Individuals on the autism spectrum also have notable difficulties processing eye gaze information (Griffin & Scherf, 2020). A central hypothesis in the literature is that the difficulties in processing eye gaze information are foundational to the social communication deficits that make social interactions so challenging for individuals on the autism spectrum. Although eye-tracking technology has been used extensively to assess the way individuals on the spectrum attend to stimuli presented on computer screens (for review see Papagiannopoulou et al., 2014), it has rarely been used to evaluate the critical question regarding whether and how autistic individuals process non-verbal social cues from their partners during live social interactions. Here, we review this emerging literature with a focus on characterizing the experimental paradigms and eye-tracking procedures to understand the scope (and limitations) of research questions and findings. We discuss the theoretical implications of the findings from this review and provide recommendations for future work that will be essential to understand whether and how fundamental difficulties in perceiving and processing information about eye gaze cues interfere with social communication skills in autism.

The effects of attentional and interpretation biases on later pain outcomes among younger and older adults: A prospective study

BACKGROUND

Studies examining the effect of biased cognitions on later pain outcomes have primarily focused on attentional biases, leaving the role of interpretation biases largely unexplored. Also, few studies have examined pain-related cognitive biases in elderly persons. The current study aims to fill these research gaps.

METHODS

Younger and older adults with and without chronic pain (N = 126) completed an interpretation bias task and a free-viewing task of injury and neutral scenes at baseline. Participants' pain intensity and disability were assessed at baseline and at a 6-month follow-up. A machine-learning data-driven approach to analysing eye movement data was adopted.

RESULTS

Eye movement analyses revealed two common attentional pattern subgroups for scene-viewing: an "explorative" group and a "focused" group. At baseline, participants with chronic pain endorsed more injury-/illness-related interpretations compared to pain-free controls, but they did not differ in eye movements on scene images. Older adults interpreted illness-related scenarios more negatively compared to younger adults, but there was also no difference in eye movements between age groups. Moreover, negative interpretation biases were associated with baseline but not follow-up pain disability, whereas a focused gaze tendency for injury scenes was associated with follow-up but not baseline pain disability. Additionally, there was an indirect effect of interpretation biases on pain disability 6 months later through attentional bias for pain-related images.

CONCLUSIONS

The present study provided evidence for pain status and age group differences in injury-/illness-related interpretation biases. Results also revealed distinct roles of interpretation and attentional biases in pain chronicity.

SIGNIFICANCE

Adults with chronic pain endorsed more injury-/illness-related interpretations than pain-free controls. Older adults endorsed more illness interpretations than younger adults. A more negative interpretation bias indirectly predicted pain disability 6 months later through hypervigilance towards pain.

Understanding the collinear masking effect in visual search through eye tracking

Recent research has reported that, while both orientation contrast and collinearity increase target salience in visual search, a combination of the two counterintuitively masks a local target. Through eye-tracking and eye-movement analysis with hidden Markov models (EMHMM), here we showed that this collinear masking effect was associated with reduced eye-fixation consistency (as measured in entropy) at the central fixation cross prior to the search display presentation. As a decreased precision of saccade landing position is shown to be related to attention shift away from the saccadic target, our result suggested that the collinear masking effect may be related to attention shift to a non-saccadic-goal location in expectation of the search display before saccading to the central fixation cross. This attention shift may consequently interfere with attention capture by the collinear distractor containing the target, resulting in the masking effect. In contrast, although older adults had longer response times, more dispersed eye-movement pattern, and lower eye-movement consistency than young adults during visual search, the two age groups did not differ in the masking effect, suggesting limited contribution from ageing-related cognitive decline. Thus, participants' pre-saccadic attention shift prior to search may be an important factor influencing their search behavior.