Imaging Time Series of Eye Tracking Data to Classify Attentional States

Features of attention network impairment in patients with temporal lobe epilepsy: Evidence from eye-tracking and electroencephalogram

AIM

To explore multiple features of attention impairments in patients with temporal lobe epilepsy (TLE).

METHODS

A total of 93 patients diagnosed with TLE at Xiangya Hospital during May 2022 and December 2022 and 85 healthy controls were included in this study. Participants were asked to complete neuropsychological scales and attention network test (ANT) with recording of eye-tracking and electroencephalogram.

RESULTS

All means of evaluation showed impaired attention functions in TLE patients. ANT results showed impaired orienting (p < 0.001) and executive control (p = 0.041) networks. Longer mean first saccade time (p = 0.046) and more total saccadic counts (p = 0.035) were found in eye-tracking results, indicating abnormal alerting and orienting networks. Both alerting, orienting and executive control networks were abnormal, manifesting as decreased amplitudes (N1 & P3, p < 0.001) and extended latency (P3, p = 0.002). The energy of theta, alpha and beta were all sensitive to the changes of alerting and executive control network with time, but only beta power was sensitive to the changes of orienting network.

CONCLUSION

Our findings are helpful for early identification of patients with TLE combined with attention impairments, which have strong clinical guiding significance for long-term monitoring and intervention.

Decoupling of the pupillary light response during internal attention: The modulating effect of luminance intensity

In a world full of sensory stimuli, attention guides us between the external environment and our internal thoughts. While external attention involves processing sensory stimuli, internal attention is devoted to self-generated representations such as planning or spontaneous mind wandering. They both draw from common cognitive resources, thus simultaneous engagement in both often leads to interference between processes. In order to maintain internal focus, an attentional mechanism known as perceptual decoupling takes effect. This mechanism supports internal cognition by decoupling attention from the perception of sensory information. Two previous studies of our lab investigated to what extent perceptual decoupling is evident in voluntary eye movements. Findings showed that the effect is mediated by the internal task modality and workload (visuospatial > arithmetic and high > low, respectively). However, it remains unclear whether it extends to involuntary eye behavior, which may not share cognitive resources with internal activities. Therefore, the present experiment aimed to further elucidate attentional dynamics by examining whether internal attention affects the pupillary light response (PLR). Specifically, we consistently observed that workload and task modality of the internal task reduced the PLR to luminance changes of medium intensity. However, the PLR to strong luminance changes was less or not at all affected by the internal task. These results suggest that perceptual decoupling effects may be less consistent in involuntary eye behavior, particularly in the context of a salient visual stimulus.

AtAwAR Translate: Attention-Aware Language Translation Application in Augmented Reality for Mobile Phones

As lightweight, low-cost EEG headsets emerge, the feasibility of consumer-oriented brain-computer interfaces (BCI) increases. The combination of portable smartphones and easy-to-use EEG dry electrode headbands offers intriguing new applications and methods of human-computer interaction. In previous research, augmented reality (AR) scenarios have been identified to profit from additional user state information-such as that provided by a BCI. In this work, we implemented a system that integrates user attentional state awareness into a smartphone application for an AR written language translator. The attentional state of the user is classified in terms of internally and externally directed attention by using the Muse 2 electroencephalography headband with four frontal electrodes. The classification results are used to adapt the behavior of the translation app, which uses the smartphone's camera to display translated text as augmented reality elements. We present the first mobile BCI system that uses a smartphone and a low-cost EEG device with few electrodes to provide attention awareness to an AR application. Our case study with 12 participants did not fully support the assumption that the BCI improves usability. However, we are able to show that the classification accuracy and ease of setup are promising paths toward mobile consumer-oriented BCI usage. For future studies, other use cases, applications, and adaptations will be tested for this setup to explore the usability.

The development of an eye movement-based deep learning system for laparoscopic surgical skills assessment

The development of valid, reliable, and objective methods of skills assessment is central to modern surgical training. Numerous rating scales have been developed and validated for quantifying surgical performance. However, many of these scoring systems are potentially flawed in their design in terms of reliability. Eye-tracking techniques, which provide a more objective investigation of the visual-cognitive aspects of the decision-making process, recently have been utilized in surgery domains for skill assessment and training, and their use has been focused on investigating differences between expert and novice surgeons to understand task performance, identify experienced surgeons, and establish training approaches. Ten graduate students at the National Taiwan University of Science and Technology with no prior laparoscopic surgical skills were recruited to perform the FLS peg transfer task. Then k-means clustering algorithm was used to split 500 trials into three dissimilar clusters, grouped as novice, intermediate, and expert levels, by an objective performance assessment parameter incorporating task duration with error score. Two types of data sets, namely, time series data extracted from coordinates of eye fixation and image data from videos, were used to implement and test our proposed skill level detection system with ensemble learning and a CNN algorithm. Results indicated that ensemble learning and the CNN were able to correctly classify skill levels with accuracies of 76.0% and 81.2%, respectively. Furthermore, the incorporation of coordinates of eye fixation and image data allowed the discrimination of skill levels with a classification accuracy of 82.5%. We examined more levels of training experience and further integrated an eye tracking technique and deep learning algorithms to develop a tool for objective assessment of laparoscopic surgical skill. With a relatively unbalanced sample, our results have demonstrated that the approach combining the features of visual fixation coordinates and images achieved a very promising level of performance for classifying skill levels of trainees.

Multimodal EEG and Eye Tracking Feature Fusion Approaches for Attention Classification in Hybrid BCIs

Often, various modalities capture distinct aspects of particular mental states or activities. While machine learning algorithms can reliably predict numerous aspects of human cognition and behavior using a single modality, they can benefit from the combination of multiple modalities. This is why hybrid BCIs are gaining popularity. However, it is not always straightforward to combine features from a multimodal dataset. Along with the method for generating the features, one must decide when the modalities should be combined during the classification process. We compare unimodal EEG and eye tracking classification of internally and externally directed attention to multimodal approaches for early, middle, and late fusion in this study. On a binary dataset with a chance level of 0.5, late fusion of the data achieves the highest classification accuracy of 0.609–0.675 (95%-confidence interval). In general, the results indicate that for these modalities, middle or late fusion approaches are better suited than early fusion approaches. Additional validation of the observed trend will require the use of additional datasets, alternative feature generation mechanisms, decision rules, and neural network designs. We conclude with a set of premises that need to be considered when deciding on a multimodal attentional state classification approach.

Combining Implicit and Explicit Feature Extraction for Eye Tracking: Attention Classification Using a Heterogeneous Input

Statistical measurements of eye movement-specific properties, such as fixations, saccades, blinks, or pupil dilation, are frequently utilized as input features for machine learning algorithms applied to eye tracking recordings. These characteristics are intended to be interpretable aspects of eye gazing behavior. However, prior research has demonstrated that when trained on implicit representations of raw eye tracking data, neural networks outperform these traditional techniques. To leverage the strengths and information of both feature sets, we integrated implicit and explicit eye tracking features in one classification approach in this work. A neural network was adapted to process the heterogeneous input and predict the internally and externally directed attention of 154 participants. We compared the accuracies reached by the implicit and combined features for different window lengths and evaluated the approaches in terms of person- and task-independence. The results indicate that combining implicit and explicit feature extraction techniques for eye tracking data improves classification results for attentional state detection significantly. The attentional state was correctly classified during new tasks with an accuracy better than chance, and person-independent classification even outperformed person-dependently trained classifiers for some settings. For future experiments and applications that require eye tracking data classification, we suggest to consider implicit data representation in addition to interpretable explicit features.