1
|
Shi J, Bi L, Xu X, Feleke AG, Fei W. Low-Quality Video Target Detection Based on EEG Signal Using Eye Movement Alignment. CYBORG AND BIONIC SYSTEMS 2024; 5:0121. [PMID: 38966125 PMCID: PMC11222288 DOI: 10.34133/cbsystems.0121] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/15/2024] [Accepted: 04/01/2024] [Indexed: 07/06/2024] Open
Abstract
The target detection based on electroencephalogram (EEG) signals is a new target detection method. This method recognizes the target by decoding the specific neural response when an operator observes the target, which has important theoretical and application values. This paper focuses on the EEG detection of low-quality video targets, which breaks through the limitation of previous target detection based on EEG signals only for high-quality video targets. We first design an experimental paradigm for EEG-based low-quality video target detection and propose an epoch extraction method based on eye movement signals to solve the asynchronous problem faced by low-quality video target detection. Then, the neural representation in the process of operator recognition is analyzed based on the time domain, frequency domain, and source space domain, respectively. We design the time-frequency features based on continuous wavelet transform according to the neural representation and obtain an average decoding test accuracy of 84.56%. The research results of this paper lay the foundation for the development of a video target detection system based on EEG signals in the future.
Collapse
Affiliation(s)
- Jianting Shi
- School of Mechanical Engineering,
Beijing Institute of Technology, Beijing 100081, China
| | - Luzheng Bi
- School of Mechanical Engineering,
Beijing Institute of Technology, Beijing 100081, China
| | - Xinbo Xu
- School of Mechanical Engineering,
Beijing Institute of Technology, Beijing 100081, China
| | - Aberham Genetu Feleke
- School of Mechanical Engineering,
Beijing Institute of Technology, Beijing 100081, China
| | - Weijie Fei
- School of Mechanical Engineering,
Beijing Institute of Technology, Beijing 100081, China
| |
Collapse
|
2
|
Breault MS, Sacré P, Fitzgerald ZB, Gale JT, Cullen KE, González-Martínez JA, Sarma SV. Internal states as a source of subject-dependent movement variability are represented by large-scale brain networks. Nat Commun 2023; 14:7837. [PMID: 38030611 PMCID: PMC10687170 DOI: 10.1038/s41467-023-43257-4] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2022] [Accepted: 11/03/2023] [Indexed: 12/01/2023] Open
Abstract
Humans' ability to adapt and learn relies on reflecting on past performance. These experiences form latent representations called internal states that induce movement variability that improves how we interact with our environment. Our study uncovered temporal dynamics and neural substrates of two states from ten subjects implanted with intracranial depth electrodes while they performed a goal-directed motor task with physical perturbations. We identified two internal states using state-space models: one tracking past errors and the other past perturbations. These states influenced reaction times and speed errors, revealing how subjects strategize from trial history. Using local field potentials from over 100 brain regions, we found large-scale brain networks such as the dorsal attention and default mode network modulate visuospatial attention based on recent performance and environmental feedback. Notably, these networks were more prominent in higher-performing subjects, emphasizing their role in improving motor performance by regulating movement variability through internal states.
Collapse
Affiliation(s)
- Macauley Smith Breault
- Picower Institute for Learning and Memory, Massachusetts Institute of Technology, Cambridge, MA, USA.
- Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA.
| | - Pierre Sacré
- Department of Electrical Engineering and Computer Science, School of Engineering, University of Liège, Liège, Belgium
| | - Zachary B Fitzgerald
- Department of Neurology, Feinberg School of Medicine, Northwestern University, Chicago, IL, USA
| | | | - Kathleen E Cullen
- Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA
| | | | - Sridevi V Sarma
- Department of Biomedical Engineering, Johns Hopkins University, Baltimore, MD, USA
| |
Collapse
|
3
|
Maselli A, Gordon J, Eluchans M, Lancia GL, Thiery T, Moretti R, Cisek P, Pezzulo G. Beyond simple laboratory studies: Developing sophisticated models to study rich behavior. Phys Life Rev 2023; 46:220-244. [PMID: 37499620 DOI: 10.1016/j.plrev.2023.07.006] [Citation(s) in RCA: 12] [Impact Index Per Article: 12.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/05/2023] [Accepted: 07/06/2023] [Indexed: 07/29/2023]
Abstract
Psychology and neuroscience are concerned with the study of behavior, of internal cognitive processes, and their neural foundations. However, most laboratory studies use constrained experimental settings that greatly limit the range of behaviors that can be expressed. While focusing on restricted settings ensures methodological control, it risks impoverishing the object of study: by restricting behavior, we might miss key aspects of cognitive and neural functions. In this article, we argue that psychology and neuroscience should increasingly adopt innovative experimental designs, measurement methods, analysis techniques and sophisticated computational models to probe rich, ecologically valid forms of behavior, including social behavior. We discuss the challenges of studying rich forms of behavior as well as the novel opportunities offered by state-of-the-art methodologies and new sensing technologies, and we highlight the importance of developing sophisticated formal models. We exemplify our arguments by reviewing some recent streams of research in psychology, neuroscience and other fields (e.g., sports analytics, ethology and robotics) that have addressed rich forms of behavior in a model-based manner. We hope that these "success cases" will encourage psychologists and neuroscientists to extend their toolbox of techniques with sophisticated behavioral models - and to use them to study rich forms of behavior as well as the cognitive and neural processes that they engage.
Collapse
Affiliation(s)
- Antonella Maselli
- Institute of Cognitive Sciences and Technologies, National Research Council, Rome, Italy
| | - Jeremy Gordon
- University of California, Berkeley, Berkeley, CA, 94704, United States
| | - Mattia Eluchans
- Institute of Cognitive Sciences and Technologies, National Research Council, Rome, Italy; University of Rome "La Sapienza", Rome, Italy
| | - Gian Luca Lancia
- Institute of Cognitive Sciences and Technologies, National Research Council, Rome, Italy; University of Rome "La Sapienza", Rome, Italy
| | - Thomas Thiery
- Department of Psychology, University of Montréal, Montréal, Québec, Canada
| | - Riccardo Moretti
- Institute of Cognitive Sciences and Technologies, National Research Council, Rome, Italy; University of Rome "La Sapienza", Rome, Italy
| | - Paul Cisek
- Department of Neuroscience, University of Montréal, Montréal, Québec, Canada
| | - Giovanni Pezzulo
- Institute of Cognitive Sciences and Technologies, National Research Council, Rome, Italy.
| |
Collapse
|
4
|
Modeling eye movement in dynamic interactive tasks for maximizing situation awareness based on Markov decision process. Sci Rep 2022; 12:13298. [PMID: 35918377 PMCID: PMC9346140 DOI: 10.1038/s41598-022-17433-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2022] [Accepted: 07/25/2022] [Indexed: 11/09/2022] Open
Abstract
For complex dynamic interactive tasks (such as aviating), operators need to continuously extract information from areas of interest (AOIs) through eye movement to maintain high level of situation awareness (SA), as failures of SA may cause task performance degradation, even system accident. Most of the current eye movement models focus on either static tasks (such as image viewing) or simple dynamic tasks (such as video watching), without considering SA. In this study, an eye movement model with the goal of maximizing SA is proposed based on Markov decision process (MDP), which is designed to describe the dynamic eye movement of experienced operators in dynamic interactive tasks. Two top-down factors, expectancy and value, are introduced into this model to represent the update probability and the importance of information in AOIs, respectively. In particular, the model regards sequence of eye fixations to different AOIs as sequential decisions to maximize the SA-related reward (value) in the context of uncertain information update (expectancy). Further, this model was validated with a flight simulation experiment. Results show that the predicted probabilities of fixation on and shift between AOIs are highly correlated (\documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$R = 0.928$$\end{document}R=0.928 and \documentclass[12pt]{minimal}
\usepackage{amsmath}
\usepackage{wasysym}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{amsbsy}
\usepackage{mathrsfs}
\usepackage{upgreek}
\setlength{\oddsidemargin}{-69pt}
\begin{document}$$R = 0.951$$\end{document}R=0.951, respectively) with those of the experiment data.
Collapse
|
5
|
Nadezhda M, Dovbnyuk K, Merzon L, MacInnes WJ. Between the Scenes. Exp Psychol 2022; 69:185-195. [PMID: 36305454 PMCID: PMC9730397 DOI: 10.1027/1618-3169/a000556] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022]
Abstract
We constantly move our eyes to new information while inspecting a scene, but these patterns of eye movements change based on the task and goals of the observer. Inhibition of return (IOR) may facilitate visual search by reducing the likelihood of revisiting previously attended locations. However, IOR may present in any visual task, or it may be search-specific. We investigated the presence of IOR in foraging, memorization, change detection, and two versions of visual search. One version of search used a static search array that remained stable throughout the trial, but the second used a scene flickering paradigm similar to the change detection task. IOR was observed in both versions of visual search, memorization, and foraging, but not in change detection. Visual search and change detection both had temporal nonscene components, and we observed that IOR could be maintained despite the scene removal but only for search. Although IOR is maintained in scene coordinates, short disruptions to this scene are insufficient to completely remove the inhibitory tags. Finally, we compare return saccades in trials without a probe and observe fewer return saccades in tasks for which IOR was observed, providing further evidence that IOR might serve as a novelty drive.
Collapse
Affiliation(s)
| | | | - Liya Merzon
- Department of Neuroscience and Biomedical Engineering, Aalto University, Aalto, Finland
| | - W. Joseph MacInnes
- Department of Psychology, Vision Modelling Laboratory, HSE University, Moscow, Russian Federation,Department of Computer Science, Swansea University, Swansea, UK,W. Joseph MacInnes, Department of Psychology, Vision Modelling Laboratory, HSE University, 20 Myasnitskaya, 10100 Moscow, Russian Federation,
| |
Collapse
|
6
|
Cole ZJ, Kuntzelman KM, Dodd MD, Johnson MR. Convolutional neural networks can decode eye movement data: A black box approach to predicting task from eye movements. J Vis 2021; 21:9. [PMID: 34264288 PMCID: PMC8288051 DOI: 10.1167/jov.21.7.9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Abstract
Previous attempts to classify task from eye movement data have relied on model architectures designed to emulate theoretically defined cognitive processes and/or data that have been processed into aggregate (e.g., fixations, saccades) or statistical (e.g., fixation density) features. Black box convolutional neural networks (CNNs) are capable of identifying relevant features in raw and minimally processed data and images, but difficulty interpreting these model architectures has contributed to challenges in generalizing lab-trained CNNs to applied contexts. In the current study, a CNN classifier was used to classify task from two eye movement datasets (Exploratory and Confirmatory) in which participants searched, memorized, or rated indoor and outdoor scene images. The Exploratory dataset was used to tune the hyperparameters of the model, and the resulting model architecture was retrained, validated, and tested on the Confirmatory dataset. The data were formatted into timelines (i.e., x-coordinate, y-coordinate, pupil size) and minimally processed images. To further understand the informational value of each component of the eye movement data, the timeline and image datasets were broken down into subsets with one or more components systematically removed. Classification of the timeline data consistently outperformed the image data. The Memorize condition was most often confused with Search and Rate. Pupil size was the least uniquely informative component when compared with the x- and y-coordinates. The general pattern of results for the Exploratory dataset was replicated in the Confirmatory dataset. Overall, the present study provides a practical and reliable black box solution to classifying task from eye movement data.
Collapse
|
7
|
Liu Z, Yang Z, Gu Y, Liu H, Wang P. The effectiveness of eye tracking in the diagnosis of cognitive disorders: A systematic review and meta-analysis. PLoS One 2021; 16:e0254059. [PMID: 34252113 PMCID: PMC8274929 DOI: 10.1371/journal.pone.0254059] [Citation(s) in RCA: 22] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/15/2021] [Accepted: 06/18/2021] [Indexed: 02/05/2023] Open
Abstract
Background Eye tracking (ET) is a viable marker for the recognition of cognitive disorders. We assessed the accuracy and clinical value of ET for the diagnosis of cognitive disorders in patients. Methods We searched the Medline, Embase, Web of Science, Cochrane Library, and Pubmed databases from inception to March 2, 2021, as well as the reference lists of identified primary studies. We included articles written in English that investigated ET for cognitive disorder patients—Mild cognitive impairment (MCI), Alzheimer’s disease (AD), Amyotrophic lateral sclerosis (ALS), and dementia. Two independent researchers extracted the data and the characteristics of each study; We calculated pooled sensitivities and specificities. A hierarchical summary of receiver performance characteristics (HSROC) model was used to test the diagnostic accuracy of ET for cognitive impairment (CI). Findings 11 studies met the inclusion criteria and were included in qualitative comprehensive analysis. Meta-analysis was performed on 9 trials using Neuropsychological Cognitive Testing (NCT) as the reference standard. The comprehensive sensitivity and specificity of ET for detecting cognitive disorders were 0.75 (95% CI 0.72–0.79) and 0.73 (95% CI 0.70 to 0.76), respectively. The combined positive likelihood ratio (LR+) was 2.74 (95%CI 2.32–3.24) and the negative likelihood ratio (LR−) was 0.27 (95%CI 0.18–0.42). Conclusions This review showed that ET technology could be used to detect the decline in CI, clinical use of ET techniques in combination with other tools to assess CI can be encouraged.
Collapse
Affiliation(s)
- Zicai Liu
- Department of Rehabilitation Medicine, Yue Bei People's Hospital, Shaoguan, Guangdong, China
| | - Zhen Yang
- Histology and Imaging platform, Core Facilities of West China Hospital, Sichuan University, China
| | - Yueming Gu
- Rehabilitation College of Gannan Medical University, Ganzhou, Jiangxi, China
| | - Huiyu Liu
- Department of Rehabilitation Medicine, Yue Bei People's Hospital, Shaoguan, Guangdong, China
| | - Pu Wang
- Department of Rehabilitation Medicine, The 7th Affiliated Hospital of Sun Yat-Sen University (Shenzhen), Shenzhen, Guangdong, China.,Guangdong Engineering and Technology Research Center for Rehabilitation Medicine and Translation, Guangzhou, China
| |
Collapse
|
8
|
Kootstra T, Teuwen J, Goudsmit J, Nijboer T, Dodd M, Van der Stigchel S. Machine learning-based classification of viewing behavior using a wide range of statistical oculomotor features. J Vis 2021; 20:1. [PMID: 32876676 PMCID: PMC7476673 DOI: 10.1167/jov.20.9.1] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Abstract
Since the seminal work of Yarbus, multiple studies have demonstrated the influence of task-set on oculomotor behavior and the current cognitive state. In more recent years, this field of research has expanded by evaluating the costs of abruptly switching between such different tasks. At the same time, the field of classifying oculomotor behavior has been moving toward more advanced, data-driven methods of decoding data. For the current study, we used a large dataset compiled over multiple experiments and implemented separate state-of-the-art machine learning methods for decoding both cognitive state and task-switching. We found that, by extracting a wide range of oculomotor features, we were able to implement robust classifier models for decoding both cognitive state and task-switching. Our decoding performance highlights the feasibility of this approach, even invariant of image statistics. Additionally, we present a feature ranking for both models, indicating the relative magnitude of different oculomotor features for both classifiers. These rankings indicate a separate set of important predictors for decoding each task, respectively. Finally, we discuss the implications of the current approach related to interpreting the decoding results.
Collapse
Affiliation(s)
- Timo Kootstra
- Experimental Psychology, Helmholtz Institute, Utrecht University, The Netherlands
| | - Jonas Teuwen
- Radboud University Medical Center/Netherlands Cancer Institute, The Netherlands
| | | | - Tanja Nijboer
- Experimental Psychology, Helmholtz Institute, Utrecht University, The Netherlands.,Center of Excellence for Rehabilitation Medicine, Brain Center Rudolf Magnus, University Medical Center Utrecht, Utrecht University and De Hoogstraat Rehabilitation, 3583 TM Utrecht, The Netherlands
| | | | | |
Collapse
|
9
|
MacInnes WJ, Jóhannesson ÓI, Chetverikov A, Kristjánsson Á. No Advantage for Separating Overt and Covert Attention in Visual Search. Vision (Basel) 2020; 4:E28. [PMID: 32443506 PMCID: PMC7356832 DOI: 10.3390/vision4020028] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2020] [Revised: 04/02/2020] [Accepted: 05/10/2020] [Indexed: 11/22/2022] Open
Abstract
We move our eyes roughly three times every second while searching complex scenes, but covert attention helps to guide where we allocate those overt fixations. Covert attention may be allocated reflexively or voluntarily, and speeds the rate of information processing at the attended location. Reducing access to covert attention hinders performance, but it is not known to what degree the locus of covert attention is tied to the current gaze position. We compared visual search performance in a traditional gaze-contingent display, with a second task where a similarly sized contingent window is controlled with a mouse, allowing a covert aperture to be controlled independently by overt gaze. Larger apertures improved performance for both the mouse- and gaze-contingent trials, suggesting that covert attention was beneficial regardless of control type. We also found evidence that participants used the mouse-controlled aperture somewhat independently of gaze position, suggesting that participants attempted to untether their covert and overt attention when possible. This untethering manipulation, however, resulted in an overall cost to search performance, a result at odds with previous results in a change blindness paradigm. Untethering covert and overt attention may therefore have costs or benefits depending on the task demands in each case.
Collapse
Affiliation(s)
- W. Joseph MacInnes
- School of Psychology, National Research University Higher School of Economics, Moscow 101000, Russia;
- Vision Modelling Lab, Faculty of Social Sciences, National Research University Higher School of Economics, Moscow 101000, Russia
| | - Ómar I. Jóhannesson
- Icelandic Vision Laboratory, Department of Psychology, University of Iceland, 102 Reykjavik, Iceland;
| | - Andrey Chetverikov
- Donders Institute for Brain, Cognition, and Behaviour, Radboud University, 6525 EN Nijmegen, The Netherlands;
| | - Árni Kristjánsson
- School of Psychology, National Research University Higher School of Economics, Moscow 101000, Russia;
- Icelandic Vision Laboratory, Department of Psychology, University of Iceland, 102 Reykjavik, Iceland;
| |
Collapse
|
10
|
Krasovskaya S, MacInnes WJ. Salience Models: A Computational Cognitive Neuroscience Review. Vision (Basel) 2019; 3:E56. [PMID: 31735857 PMCID: PMC6969943 DOI: 10.3390/vision3040056] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2019] [Revised: 10/12/2019] [Accepted: 10/22/2019] [Indexed: 11/21/2022] Open
Abstract
The seminal model by Laurent Itti and Cristoph Koch demonstrated that we can compute the entire flow of visual processing from input to resulting fixations. Despite many replications and follow-ups, few have matched the impact of the original model-so what made this model so groundbreaking? We have selected five key contributions that distinguish the original salience model by Itti and Koch; namely, its contribution to our theoretical, neural, and computational understanding of visual processing, as well as the spatial and temporal predictions for fixation distributions. During the last 20 years, advances in the field have brought up various techniques and approaches to salience modelling, many of which tried to improve or add to the initial Itti and Koch model. One of the most recent trends has been to adopt the computational power of deep learning neural networks; however, this has also shifted their primary focus to spatial classification. We present a review of recent approaches to modelling salience, starting from direct variations of the Itti and Koch salience model to sophisticated deep-learning architectures, and discuss the models from the point of view of their contribution to computational cognitive neuroscience.
Collapse
Affiliation(s)
- Sofia Krasovskaya
- Vision Modelling Laboratory, Faculty of Social Science, National Research University Higher School of Economics, 101000 Moscow, Russia
- School of Psychology, National Research University Higher School of Economics, 101000 Moscow, Russia
| | - W. Joseph MacInnes
- Vision Modelling Laboratory, Faculty of Social Science, National Research University Higher School of Economics, 101000 Moscow, Russia
- School of Psychology, National Research University Higher School of Economics, 101000 Moscow, Russia
| |
Collapse
|