Assessment of eye-tracking scanpath outliers using fractal geometry

Outlier scanpaths identification is a crucial preliminary step in designing visual software, digital media analysis, radiology training and clustering participants in eye-tracking experiments. However, the task is challenging due to the visual irregularity of the scanpath shapes and the difficulty in dimensionality reduction due to geometric complexity. Conventional approaches have used heat maps to exclude scanpaths that lack a similarity pattern. However, the typically-used packages, such as ScanMatch and MultiMatch often generate discordant results when outlier identification is done empirically. This paper introduces a novel outlier evaluation approach by integrating the fractal dimension (FD), capturing the geometrical complexity of patterns, as an additional parameter with the heat map. This additional parameter is used to evaluate the degree of influence of a scanpath within a dataset. More specifically, the 2D Cartesian coordinates of a scanpath are fitted to a space filling 1D fractal curve to characterise its temporal FD. The FDs of the scanpaths are then compared to match their geometric complexity to one another. The findings indicate that the FD can be a beneficial additional parameter when evaluating the candidacy of poorly matching scanpaths as outliers and performs better at identifying unusual scanpaths than using other methods, including scanpath matching, Jaccard, or bounding box methods alone.

The attenuated visual scanpaths of patients with schizophrenia whilst recognizing emotional facial expressions are worsened in natural social scenes

Alteration of visual scanpaths under emotional facial expression in schizophrenia patients has been described in recent years; however, it is not clear whether such results are different when they transfer to faces in natural social scenes. The present study was designed to investigate the effects of emotional faces in natural social scenes on the gaze patterns of patients with schizophrenia, compared to gaze at isolated faces. A novel theme identification task was used where participants selected a positive, neutral or negative word to describe an emotional picture. Participants were 29 patients with schizophrenia and 31 healthy controls. The Positive and Negative Syndrome Scale (PANSS) and the Scale of Social Function in Psychosis Inpatients (SSPI) were used to assess symptoms and social functioning. In total, patients with schizophrenia showed significantly fewer fixations, saccades numbers and decreased fixations in areas of interest. As expected, patients showed shorter scanpath length, but only in the pictures with social settings. Furthermore, the effect size of scanpaths parameters under social scene was all greater than isolated face. In addition, patients compared to controls showed more abnormal scanpath parameters processing negative and neutral faces than positive faces, especially in social scene. The present study suggests that scanpath length for social scene faces may be more sensitive than for isolated face pictures. Our findings further support restricted scanpath whilst recognizing emotional facial expressions in natural social scenes as a favorable topic for further investigation as a trait marker.

Reduced scanning of salient facial features mediates the association between paranoia and emotion recognition

The vigilance-avoidance hypothesis of paranoia states that in paranoia visual attention is shifted away from threat-related stimuli. This may be an explanation for reduced scanning of salient facial features in psychosis and subsequently impaired emotion recognition. Here, we explored whether higher levels of paranoia would predict reduced visual attention to salient facial features and impaired emotion recognition and whether reduced visual attention to salient facial features mediates the association between paranoia and errors in emotion recognition. Participants with schizophrenia (SZ, n = 22) and healthy controls (HC, n = 19) completed questionnaire assessments of paranoia and negative symptoms and conducted an emotion recognition task comprised of dynamic facial stimuli. Additionally, visual attention (number of fixations) to salient facial features was assessed using eye-tracking. SZ made more errors in affect recognition than HC. Visual attention to salient facial features did not differ between SZ and HC but significantly mediated the significant association between paranoia and errors in the emotion recognition task in the complete sample. Negative symptoms also predicted errors in emotion recognition but this association was not mediated by visual attention. Our findings are in line with the avoidance-assumption of a vigilance-avoidance hypothesis of paranoia, in which correct facial emotion recognition is prevented due to an avoidance of salient facial features.

Influence of initial fixation position in scene viewing

During scene perception our eyes generate complex sequences of fixations. Predictors of fixation locations are bottom-up factors such as luminance contrast, top-down factors like viewing instruction, and systematic biases, e.g., the tendency to place fixations near the center of an image. However, comparatively little is known about the dynamics of scanpaths after experimental manipulation of specific fixation locations. Here we investigate the influence of initial fixation position on subsequent eye-movement behavior on an image. We presented 64 colored photographs to participants who started their scanpaths from one of two experimentally controlled positions in the right or left part of an image. Additionally, we used computational models to predict the images' fixation locations and classified them as balanced images or images with high conspicuity on either the left or right side of a picture. The manipulation of the starting position influenced viewing behavior for several seconds and produced a tendency to overshoot to the image side opposite to the starting position. Possible mechanisms for the generation of this overshoot were investigated using numerical simulations of statistical and dynamical models. Our model comparisons show that inhibitory tagging is a viable mechanism for dynamical planning of scanpaths.

Restricted attention to social cues in schizophrenia patients

Deficits of psychosocial functioning are a robust finding in schizophrenia. Research on social cognition may open a new avenue for the development of effective interventions. As a correlate of social perceptive information processing deficits, schizophrenia patients (SZP) show deviant gaze behavior (GB) while viewing emotional faces. As understanding of a social environment requires gathering complex social information, our study aimed at investigating the gaze behavior of SZP related to social interactions and its impact on the level of social and role functioning. GB of 32 SZP and 37 healthy control individuals (HCI) was investigated with a high-resolution eye tracker during an unguided viewing of 12 complex pictures of social interaction scenes. Regarding whole pictures, SZP showed a shorter scanpath length, fewer fixations and a shorter mean distance between fixations. Furthermore, SZP exhibited fewer and shorter fixations on faces, but not on the socially informative bodies nor on the background, suggesting a cue-specific abnormality. Logistic regression with bootstrapping yielded a model including two GB parameters; a subsequent ROC curve analysis indicated an excellent ability of group discrimination (AUC .85). Face-related GB aberrations correlated with lower social and role functioning and with delusional thinking, but not with negative symptoms. Training of spontaneous integration of face-related social information seems promising to enable a holistic perception of social information, which may in turn improve social and role functioning. The observed ability to discriminate SZP from HCI warrants further research on the predictive validity of GB in psychosis risk prediction.

Introducing context-dependent and spatially-variant viewing biases in saccadic models

Previous research showed the existence of systematic tendencies in viewing behavior during scene exploration. For instance, saccades are known to follow a positively skewed, long-tailed distribution, and to be more frequently initiated in the horizontal or vertical directions. In this study, we hypothesize that these viewing biases are not universal, but are modulated by the semantic visual category of the stimulus. We show that the joint distribution of saccade amplitudes and orientations significantly varies from one visual category to another. These joint distributions are in addition spatially variant within the scene frame. We demonstrate that a saliency model based on this better understanding of viewing behavioral biases and blind to any visual information outperforms well-established saliency models. We also propose a saccadic model that takes into account classical low-level features and spatially-variant and context-dependent viewing biases. This model outperforms state-of-the-art saliency models, and provides scanpaths in close agreement with human behavior. The better description of viewing biases will not only improve current models of visual attention but could also influence many other applications such as the design of human-computer interfaces, patient diagnosis or image/video processing applications.

Saccadic model of eye movements for free-viewing condition

In this paper, we propose a new framework to predict visual scanpaths of observers while they freely watch a visual scene. The visual fixations are inferred from bottom-up saliency and several oculomotor biases. Bottom-up saliency is represented by a saliency map whereas the oculomotor biases (saccade amplitudes and saccade orientations) are modeled using public eye tracking datasets. Our experiments show that the simulated scanpaths exhibit similar trends of human eye movements in a free-viewing condition. The generated scanpaths are more similar to human scanpaths than those generated by two existing methods. In addition, we show that computing saliency maps from simulated visual scanpaths allows to outperform existing saliency models.

Effects of Training of Affect Recognition on the recognition and visual exploration of emotional faces in schizophrenia

BACKGROUND

Schizophrenia patients have impairments in facial affect recognition and display scanpath abnormalities during the visual exploration of faces. These abnormalities are characterized by fewer fixations on salient feature areas and longer fixation durations. The present study investigated whether social-cognitive remediation not only improves performance in facial affect recognition but also normalizes patients' gaze behavior while looking at faces.

METHODS

Within a 2 × 2-design (group × time), 16 schizophrenia patients and 16 healthy controls performed a facial affect recognition task with concomitant infrared oculography at baseline (T0) and after six weeks (T1). Between the measurements, patients completed the Training of Affect Recognition (TAR) program. The influence of the training on facial affect recognition (percent of correct answers) and gaze behavior (number and mean duration of fixations into salient or non-salient facial areas) was assessed.

RESULTS

In line with former studies, at baseline patients showed poorer facial affect recognition than controls and aberrant scanpaths, and after TAR facial affect recognition was improved. Concomitant with improvements in performance, the number of fixations in feature areas ('mouth') increased while fixations in non-feature areas ('white space') decreased. However, the change in fixation behavior did not correlate with the improvement in performance.

CONCLUSIONS

After TAR, patients pay more attention to facial areas that contain information about a displayed emotion. Although this may contribute to the improved performance, the lack of a statistical correlation implies that this factor is not sufficient to explain the underlying mechanism of the treatment effect.