A Generative Model of Cognitive State from Task and Eye Movements

Low-Quality Video Target Detection Based on EEG Signal Using Eye Movement Alignment

The target detection based on electroencephalogram (EEG) signals is a new target detection method. This method recognizes the target by decoding the specific neural response when an operator observes the target, which has important theoretical and application values. This paper focuses on the EEG detection of low-quality video targets, which breaks through the limitation of previous target detection based on EEG signals only for high-quality video targets. We first design an experimental paradigm for EEG-based low-quality video target detection and propose an epoch extraction method based on eye movement signals to solve the asynchronous problem faced by low-quality video target detection. Then, the neural representation in the process of operator recognition is analyzed based on the time domain, frequency domain, and source space domain, respectively. We design the time-frequency features based on continuous wavelet transform according to the neural representation and obtain an average decoding test accuracy of 84.56%. The research results of this paper lay the foundation for the development of a video target detection system based on EEG signals in the future.

Internal states as a source of subject-dependent movement variability are represented by large-scale brain networks

Humans' ability to adapt and learn relies on reflecting on past performance. These experiences form latent representations called internal states that induce movement variability that improves how we interact with our environment. Our study uncovered temporal dynamics and neural substrates of two states from ten subjects implanted with intracranial depth electrodes while they performed a goal-directed motor task with physical perturbations. We identified two internal states using state-space models: one tracking past errors and the other past perturbations. These states influenced reaction times and speed errors, revealing how subjects strategize from trial history. Using local field potentials from over 100 brain regions, we found large-scale brain networks such as the dorsal attention and default mode network modulate visuospatial attention based on recent performance and environmental feedback. Notably, these networks were more prominent in higher-performing subjects, emphasizing their role in improving motor performance by regulating movement variability through internal states.

Beyond simple laboratory studies: Developing sophisticated models to study rich behavior

Psychology and neuroscience are concerned with the study of behavior, of internal cognitive processes, and their neural foundations. However, most laboratory studies use constrained experimental settings that greatly limit the range of behaviors that can be expressed. While focusing on restricted settings ensures methodological control, it risks impoverishing the object of study: by restricting behavior, we might miss key aspects of cognitive and neural functions. In this article, we argue that psychology and neuroscience should increasingly adopt innovative experimental designs, measurement methods, analysis techniques and sophisticated computational models to probe rich, ecologically valid forms of behavior, including social behavior. We discuss the challenges of studying rich forms of behavior as well as the novel opportunities offered by state-of-the-art methodologies and new sensing technologies, and we highlight the importance of developing sophisticated formal models. We exemplify our arguments by reviewing some recent streams of research in psychology, neuroscience and other fields (e.g., sports analytics, ethology and robotics) that have addressed rich forms of behavior in a model-based manner. We hope that these "success cases" will encourage psychologists and neuroscientists to extend their toolbox of techniques with sophisticated behavioral models - and to use them to study rich forms of behavior as well as the cognitive and neural processes that they engage.

Modeling eye movement in dynamic interactive tasks for maximizing situation awareness based on Markov decision process

For complex dynamic interactive tasks (such as aviating), operators need to continuously extract information from areas of interest (AOIs) through eye movement to maintain high level of situation awareness (SA), as failures of SA may cause task performance degradation, even system accident. Most of the current eye movement models focus on either static tasks (such as image viewing) or simple dynamic tasks (such as video watching), without considering SA. In this study, an eye movement model with the goal of maximizing SA is proposed based on Markov decision process (MDP), which is designed to describe the dynamic eye movement of experienced operators in dynamic interactive tasks. Two top-down factors, expectancy and value, are introduced into this model to represent the update probability and the importance of information in AOIs, respectively. In particular, the model regards sequence of eye fixations to different AOIs as sequential decisions to maximize the SA-related reward (value) in the context of uncertain information update (expectancy). Further, this model was validated with a flight simulation experiment. Results show that the predicted probabilities of fixation on and shift between AOIs are highly correlated (\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R = 0.928$$\end{document}R=0.928 and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$R = 0.951$$\end{document}R=0.951, respectively) with those of the experiment data.

Between the Scenes

We constantly move our eyes to new information while inspecting a scene, but these patterns of eye movements change based on the task and goals of the observer. Inhibition of return (IOR) may facilitate visual search by reducing the likelihood of revisiting previously attended locations. However, IOR may present in any visual task, or it may be search-specific. We investigated the presence of IOR in foraging, memorization, change detection, and two versions of visual search. One version of search used a static search array that remained stable throughout the trial, but the second used a scene flickering paradigm similar to the change detection task. IOR was observed in both versions of visual search, memorization, and foraging, but not in change detection. Visual search and change detection both had temporal nonscene components, and we observed that IOR could be maintained despite the scene removal but only for search. Although IOR is maintained in scene coordinates, short disruptions to this scene are insufficient to completely remove the inhibitory tags. Finally, we compare return saccades in trials without a probe and observe fewer return saccades in tasks for which IOR was observed, providing further evidence that IOR might serve as a novelty drive.

Convolutional neural networks can decode eye movement data: A black box approach to predicting task from eye movements

Previous attempts to classify task from eye movement data have relied on model architectures designed to emulate theoretically defined cognitive processes and/or data that have been processed into aggregate (e.g., fixations, saccades) or statistical (e.g., fixation density) features. Black box convolutional neural networks (CNNs) are capable of identifying relevant features in raw and minimally processed data and images, but difficulty interpreting these model architectures has contributed to challenges in generalizing lab-trained CNNs to applied contexts. In the current study, a CNN classifier was used to classify task from two eye movement datasets (Exploratory and Confirmatory) in which participants searched, memorized, or rated indoor and outdoor scene images. The Exploratory dataset was used to tune the hyperparameters of the model, and the resulting model architecture was retrained, validated, and tested on the Confirmatory dataset. The data were formatted into timelines (i.e., x-coordinate, y-coordinate, pupil size) and minimally processed images. To further understand the informational value of each component of the eye movement data, the timeline and image datasets were broken down into subsets with one or more components systematically removed. Classification of the timeline data consistently outperformed the image data. The Memorize condition was most often confused with Search and Rate. Pupil size was the least uniquely informative component when compared with the x- and y-coordinates. The general pattern of results for the Exploratory dataset was replicated in the Confirmatory dataset. Overall, the present study provides a practical and reliable black box solution to classifying task from eye movement data.

The effectiveness of eye tracking in the diagnosis of cognitive disorders: A systematic review and meta-analysis

Background

Eye tracking (ET) is a viable marker for the recognition of cognitive disorders. We assessed the accuracy and clinical value of ET for the diagnosis of cognitive disorders in patients.

Methods

We searched the Medline, Embase, Web of Science, Cochrane Library, and Pubmed databases from inception to March 2, 2021, as well as the reference lists of identified primary studies. We included articles written in English that investigated ET for cognitive disorder patients—Mild cognitive impairment (MCI), Alzheimer’s disease (AD), Amyotrophic lateral sclerosis (ALS), and dementia. Two independent researchers extracted the data and the characteristics of each study; We calculated pooled sensitivities and specificities. A hierarchical summary of receiver performance characteristics (HSROC) model was used to test the diagnostic accuracy of ET for cognitive impairment (CI).

Findings

11 studies met the inclusion criteria and were included in qualitative comprehensive analysis. Meta-analysis was performed on 9 trials using Neuropsychological Cognitive Testing (NCT) as the reference standard. The comprehensive sensitivity and specificity of ET for detecting cognitive disorders were 0.75 (95% CI 0.72–0.79) and 0.73 (95% CI 0.70 to 0.76), respectively. The combined positive likelihood ratio (LR+) was 2.74 (95%CI 2.32–3.24) and the negative likelihood ratio (LR−) was 0.27 (95%CI 0.18–0.42).

Conclusions

This review showed that ET technology could be used to detect the decline in CI, clinical use of ET techniques in combination with other tools to assess CI can be encouraged.

Machine learning-based classification of viewing behavior using a wide range of statistical oculomotor features

Since the seminal work of Yarbus, multiple studies have demonstrated the influence of task-set on oculomotor behavior and the current cognitive state. In more recent years, this field of research has expanded by evaluating the costs of abruptly switching between such different tasks. At the same time, the field of classifying oculomotor behavior has been moving toward more advanced, data-driven methods of decoding data. For the current study, we used a large dataset compiled over multiple experiments and implemented separate state-of-the-art machine learning methods for decoding both cognitive state and task-switching. We found that, by extracting a wide range of oculomotor features, we were able to implement robust classifier models for decoding both cognitive state and task-switching. Our decoding performance highlights the feasibility of this approach, even invariant of image statistics. Additionally, we present a feature ranking for both models, indicating the relative magnitude of different oculomotor features for both classifiers. These rankings indicate a separate set of important predictors for decoding each task, respectively. Finally, we discuss the implications of the current approach related to interpreting the decoding results.

No Advantage for Separating Overt and Covert Attention in Visual Search

We move our eyes roughly three times every second while searching complex scenes, but covert attention helps to guide where we allocate those overt fixations. Covert attention may be allocated reflexively or voluntarily, and speeds the rate of information processing at the attended location. Reducing access to covert attention hinders performance, but it is not known to what degree the locus of covert attention is tied to the current gaze position. We compared visual search performance in a traditional gaze-contingent display, with a second task where a similarly sized contingent window is controlled with a mouse, allowing a covert aperture to be controlled independently by overt gaze. Larger apertures improved performance for both the mouse- and gaze-contingent trials, suggesting that covert attention was beneficial regardless of control type. We also found evidence that participants used the mouse-controlled aperture somewhat independently of gaze position, suggesting that participants attempted to untether their covert and overt attention when possible. This untethering manipulation, however, resulted in an overall cost to search performance, a result at odds with previous results in a change blindness paradigm. Untethering covert and overt attention may therefore have costs or benefits depending on the task demands in each case.

Salience Models: A Computational Cognitive Neuroscience Review

The seminal model by Laurent Itti and Cristoph Koch demonstrated that we can compute the entire flow of visual processing from input to resulting fixations. Despite many replications and follow-ups, few have matched the impact of the original model-so what made this model so groundbreaking? We have selected five key contributions that distinguish the original salience model by Itti and Koch; namely, its contribution to our theoretical, neural, and computational understanding of visual processing, as well as the spatial and temporal predictions for fixation distributions. During the last 20 years, advances in the field have brought up various techniques and approaches to salience modelling, many of which tried to improve or add to the initial Itti and Koch model. One of the most recent trends has been to adopt the computational power of deep learning neural networks; however, this has also shifted their primary focus to spatial classification. We present a review of recent approaches to modelling salience, starting from direct variations of the Itti and Koch salience model to sophisticated deep-learning architectures, and discuss the models from the point of view of their contribution to computational cognitive neuroscience.