1
|
Nolte D, Vidal De Palol M, Keshava A, Madrid-Carvajal J, Gert AL, von Butler EM, Kömürlüoğlu P, König P. Combining EEG and eye-tracking in virtual reality: Obtaining fixation-onset event-related potentials and event-related spectral perturbations. Atten Percept Psychophys 2025; 87:207-227. [PMID: 38977612 PMCID: PMC11845549 DOI: 10.3758/s13414-024-02917-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 06/03/2024] [Indexed: 07/10/2024]
Abstract
Extensive research conducted in controlled laboratory settings has prompted an inquiry into how results can be generalized to real-world situations influenced by the subjects' actions. Virtual reality lends itself ideally to investigating complex situations but requires accurate classification of eye movements, especially when combining it with time-sensitive data such as EEG. We recorded eye-tracking data in virtual reality and classified it into gazes and saccades using a velocity-based classification algorithm, and we cut the continuous data into smaller segments to deal with varying noise levels, as introduced in the REMoDNav algorithm. Furthermore, we corrected for participants' translational movement in virtual reality. Various measures, including visual inspection, event durations, and the velocity and dispersion distributions before and after gaze onset, indicate that we can accurately classify the continuous, free-exploration data. Combining the classified eye-tracking with the EEG data, we generated fixation-onset event-related potentials (ERPs) and event-related spectral perturbations (ERSPs), providing further evidence for the quality of the eye-movement classification and timing of the onset of events. Finally, investigating the correlation between single trials and the average ERP and ERSP identified that fixation-onset ERSPs are less time sensitive, require fewer repetitions of the same behavior, and are potentially better suited to study EEG signatures in naturalistic settings. We modified, designed, and tested an algorithm that allows the combination of EEG and eye-tracking data recorded in virtual reality.
Collapse
Affiliation(s)
- Debora Nolte
- Institute of Cognitive Science, University of Osnabrück, Wachsbleiche 27, 49090, Osnabrueck, Germany.
| | - Marc Vidal De Palol
- Institute of Cognitive Science, University of Osnabrück, Wachsbleiche 27, 49090, Osnabrueck, Germany
| | - Ashima Keshava
- Institute of Cognitive Science, University of Osnabrück, Wachsbleiche 27, 49090, Osnabrueck, Germany
| | - John Madrid-Carvajal
- Institute of Cognitive Science, University of Osnabrück, Wachsbleiche 27, 49090, Osnabrueck, Germany
| | - Anna L Gert
- Institute of Cognitive Science, University of Osnabrück, Wachsbleiche 27, 49090, Osnabrueck, Germany
| | - Eva-Marie von Butler
- Institute of Cognitive Science, University of Osnabrück, Wachsbleiche 27, 49090, Osnabrueck, Germany
| | - Pelin Kömürlüoğlu
- Institute of Cognitive Science, University of Osnabrück, Wachsbleiche 27, 49090, Osnabrueck, Germany
| | - Peter König
- Institute of Cognitive Science, University of Osnabrück, Wachsbleiche 27, 49090, Osnabrueck, Germany
- Department of Neurophysiology and Pathophysiology, University Medical Center Hamburg-Eppendorf, Hamburg, Germany
| |
Collapse
|
2
|
Keshava A, Nezami FN, Neumann H, Izdebski K, Schüler T, König P. Just-in-time: Gaze guidance in natural behavior. PLoS Comput Biol 2024; 20:e1012529. [PMID: 39446872 PMCID: PMC11537419 DOI: 10.1371/journal.pcbi.1012529] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/11/2022] [Revised: 11/05/2024] [Accepted: 10/01/2024] [Indexed: 10/26/2024] Open
Abstract
Natural eye movements have primarily been studied for over-learned activities such as tea-making, sandwich-making, and hand-washing, which have a fixed sequence of associated actions. These studies demonstrate a sequential activation of low-level cognitive schemas facilitating task completion. However, whether these action schemas are activated in the same pattern when a task is novel and a sequence of actions must be planned in the moment is unclear. Here, we recorded gaze and body movements in a naturalistic task to study action-oriented gaze behavior. In a virtual environment, subjects moved objects on a life-size shelf to achieve a given order. To compel cognitive planning, we added complexity to the sorting tasks. Fixations aligned with the action onset showed gaze as tightly coupled with the action sequence, and task complexity moderately affected the proportion of fixations on the task-relevant regions. Our analysis revealed that gaze fixations were allocated to action-relevant targets just in time. Planning behavior predominantly corresponded to a greater visual search for task-relevant objects before the action onset. The results support the idea that natural behavior relies on the frugal use of working memory, and humans refrain from encoding objects in the environment to plan long-term actions. Instead, they prefer just-in-time planning by searching for action-relevant items at the moment, directing their body and hand to it, monitoring the action until it is terminated, and moving on to the following action.
Collapse
Affiliation(s)
- Ashima Keshava
- Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany
| | | | | | | | | | - Peter König
- Institute of Cognitive Science, University of Osnabrück, Osnabrück, Germany
- University Medical Center, Hamburg-Eppendorf, Germany
| |
Collapse
|
3
|
Treuting RL, Boroujeni KB, Gerrity CG, Tiesinga P, Womelsdorf T. Anterior Cingulate Cortex Causally Supports Meta-Learning. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2024:2024.06.12.598723. [PMID: 38915609 PMCID: PMC11195175 DOI: 10.1101/2024.06.12.598723] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/26/2024]
Abstract
In dynamic environments with volatile rewards the anterior cingulate cortex (ACC) is believed to determine whether a visual object is relevant and should be chosen. The ACC may achieve this by integrating reward information over time to estimate which objects are worth to explore and which objects should be avoided. Such a higher-order meta-awareness about which objects should be explored predicts that the ACC causally contributes to choices when the reward values of objects are unknown and must be inferred from ongoing exploration. We tested this suggestion in nonhuman primates using a learning task that varied the number of object features that could be relevant, and by controlling the motivational value of choosing objects. During learning the ACC was transiently micro-stimulated when subjects foveated the to-be-chosen stimulus. We found that stimulation selectively impaired learning when feature uncertainty and motivational value of choices were high, which was linked to a deficit in using reward outcomes for feature-specific credit assignment. Application of an adaptive reinforcement learning model confirmed a primary deficit in weighting prediction errors that led to a meta-learning impairment to adaptively increase exploration during learning and to an impaired use of working memory to support learning. These findings provide causal evidence that the reward history traces in ACC are essential for meta-adjusting the exploration-exploitation balance and the strength of working memory of object values during adaptive behavior.
Collapse
Affiliation(s)
- Robert Louis Treuting
- Department of Biomedical Engineering, Vanderbilt University, Nashville, TN 37240
- Vanderbilt Brain Institute, Nashville, TN 372404
| | - Kianoush Banaie Boroujeni
- Department of Psychology, Vanderbilt University, Nashville, TN 37240
- Princeton Neuroscience Institute, Princeton University, Princeton, NJ 08540
| | - Charles Grimes Gerrity
- Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN 37240
| | - Paul Tiesinga
- Donders Institute for Brain, Cognition and Behaviour, Radboud University Nijmegen, Nijmegen 6525 EN, Netherlands
| | - Thilo Womelsdorf
- Department of Biomedical Engineering, Vanderbilt University, Nashville, TN 37240
- Vanderbilt Brain Institute, Nashville, TN 372404
- Department of Psychology, Vanderbilt University, Nashville, TN 37240
- Department of Electrical and Computer Engineering, Vanderbilt University, Nashville, TN 37240
| |
Collapse
|
4
|
He X, Selesnick I, Zhu M. Research Progress of Eye Movement Analyses and its Detection Algorithms in Alzheimer's Disease. Curr Alzheimer Res 2024; 21:91-100. [PMID: 38661033 DOI: 10.2174/0115672050300564240416074025] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/04/2024] [Revised: 03/29/2024] [Accepted: 03/29/2024] [Indexed: 04/26/2024]
Abstract
Alzheimer's disease (AD) has been considered one of the most challenging forms of dementia. The earlier the people are diagnosed with AD, the easier it is for doctors to find a treatment. Based on the previous literature summarizing the research results on the relationship between eye movement and AD before 2013, this paper reviewed 34 original eye movements research papers only closely related to AD published in the past ten years and pointed out that the prosaccade (4 papers) and antisaccade (5 papers) tasks, reading tasks (3 papers), visual search tasks (3 papers) are still the research objects of many researchers, Some researchers have looked at King-Devick tasks (2 papers), reading tasks (3 papers) and special tasks (8 papers), and began to use combinations of different saccade tasks to detect the relationship between eye movement and AD, which had not been done before. These reflect the diversity of eye movement tasks and the complexity and difficulty of the relationship between eye movement and AD. On this basis, the current processing and analysis methods of eye movement datasets are analyzed and discussed in detail, and we note that certain key data that may be especially important for the early diagnosis of AD by using eye movement studies cannot be miss-classified as noise and removed. Finally, we note that the development of methods that can accurately denoise and classify and quickly process massive eye movement data is quite significant for detecting eye movements in early diagnosis of AD.
Collapse
Affiliation(s)
- Xueying He
- School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan, Hubei, CN, USA
- Tandon School of Engineering, New York University, Brooklyn, NY, USA
| | - Ivan Selesnick
- Tandon School of Engineering, New York University, Brooklyn, NY, USA
| | - Ming Zhu
- School of Electronic Information and Communications, Huazhong University of Science and Technology, Wuhan, Hubei, CN, USA
| |
Collapse
|
5
|
Wiebe A, Aslan B, Brockmann C, Lepartz A, Dudek D, Kannen K, Selaskowski B, Lux S, Ettinger U, Philipsen A, Braun N. Multimodal assessment of adult attention-deficit hyperactivity disorder: A controlled virtual seminar room study. Clin Psychol Psychother 2023; 30:1111-1129. [PMID: 37209018 DOI: 10.1002/cpp.2863] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/22/2023] [Accepted: 04/29/2023] [Indexed: 05/21/2023]
Abstract
In the assessment of adult attention-deficit hyperactivity disorder (ADHD) symptoms, the diagnostic value of neuropsychological testing is limited. Partly, this is due to the rather low ecological validity of traditional neuropsychological tests, which usually present abstract stimuli on a computer screen. A potential remedy for this shortcoming might be the use of virtual reality (VR), which enables a more realistic and complex, yet still standardized test environment. The present study investigates a new VR-based multimodal assessment tool for adult ADHD, the virtual seminar room (VSR). Twenty-five unmedicated ADHD patients, 25 medicated ADHD patients, and 25 healthy controls underwent a virtual continuous performance task (CPT) in the VSR with concurrent visual, auditive, and audiovisual distractions. Simultaneously, head movements (actigraphy), gaze behaviour (eye tracking), subjective experience, electroencephalography (EEG), and functional near-infrared spectroscopy (fNIRS) were recorded. Significant differences between unmedicated patients with ADHD and healthy controls were found in CPT performance, head actigraphy, distractor gaze behaviour, and subjective experience. Moreover, CPT performance parameters demonstrated potential utility for assessing medication effects within the ADHD population. No group differences were found in the Theta-Beta-Ratio (EEG) or dorsolateral-prefrontal oxy-haemoglobin (fNIRS). Overall, the results are very promising regarding the potential of the VSR as an assessment tool for adult ADHD. In particular, the combined assessment of CPT, actigraphy, and eye tracking parameters appears to be a valid approach to more accurately capture the heterogeneous symptom presentation of the disorder.
Collapse
Affiliation(s)
- Annika Wiebe
- Department of Psychiatry and Psychotherapy, University Hospital Bonn, Bonn, Germany
| | - Behrem Aslan
- Department of Psychiatry and Psychotherapy, University Hospital Bonn, Bonn, Germany
| | - Charlotte Brockmann
- Department of Psychiatry and Psychotherapy, University Hospital Bonn, Bonn, Germany
| | - Alexandra Lepartz
- Department of Psychiatry and Psychotherapy, University Hospital Bonn, Bonn, Germany
| | - Dominika Dudek
- Department of Psychiatry and Psychotherapy, University Hospital Bonn, Bonn, Germany
| | - Kyra Kannen
- Department of Psychiatry and Psychotherapy, University Hospital Bonn, Bonn, Germany
| | - Benjamin Selaskowski
- Department of Psychiatry and Psychotherapy, University Hospital Bonn, Bonn, Germany
| | - Silke Lux
- Department of Psychiatry and Psychotherapy, University Hospital Bonn, Bonn, Germany
| | | | - Alexandra Philipsen
- Department of Psychiatry and Psychotherapy, University Hospital Bonn, Bonn, Germany
| | - Niclas Braun
- Department of Psychiatry and Psychotherapy, University Hospital Bonn, Bonn, Germany
| |
Collapse
|
6
|
Zhang J, Liu L, Wei J, Wu X, Luo J, Wei H, Ning L, He Y. High expression level of the FTH1 gene is associated with poor prognosis in children with non-M3 acute myeloid leukemia. Front Oncol 2023; 12:1068094. [PMID: 36818670 PMCID: PMC9928996 DOI: 10.3389/fonc.2022.1068094] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2022] [Accepted: 12/29/2022] [Indexed: 02/04/2023] Open
Abstract
Acute myelogenous leukemia (AML) is a disease that severely affects the physical health of children. Thus, we aimed to identify biomarkers associated with AML prognosis in children. Using transcriptomics on an mRNA dataset from 27 children with non-M3 AML, we selected genes from among those with the top 5000 median absolute deviation (MAD) values for subsequent analysis which showed that two modules were associated with AML risk groups. Thus, enrichment analysis was performed using genes from these modules. A one-way Cox analysis was performed on a dataset of 149 non-M3 AML patients downloaded from the TCGA. This identified four genes as significant: FTH1, RCC2, ABHD17B, and IRAK1. Through survival analysis, FTH1 was identified as a key gene associated with AML prognosis. We verified the proliferative and regulatory effects of ferroptosis on MOLM-13 and THP-1 cells using Liproxstatin-1 and Erastin respectively by CCK-8 and flow cytometry assays. Furthermore, we assayed expression levels of FTH1 in MOLM-13 and THP-1 cells after induction and inhibition of ferroptosis by real-time quantitative PCR, which showed that upregulated FTH1 expression promoted proliferation and inhibited apoptosis in leukemia cells. In conclusion, high expression of FTH1 promoted proliferation and inhibited apoptosis of leukemic cells through the ferroptosis pathway and is thus a potential risk factor that affects the prognosis of non-M3 AML in children.
Collapse
Affiliation(s)
- Junlin Zhang
- First Affiliated Hospital of Guangxi Medical University, Nanning, China
| | - Liying Liu
- First Affiliated Hospital of Guangxi Medical University, Nanning, China
| | - Jinshuang Wei
- First Affiliated Hospital of Guangxi Medical University, Nanning, China
| | - Xiaojing Wu
- First Affiliated Hospital of Guangxi Medical University, Nanning, China
| | - Jianming Luo
- First Affiliated Hospital of Guangxi Medical University, Nanning, China
- The Key Laboratory of Children’s Disease Research in Guangxi’s Colleges and Universities, Education Department of Guangxi Zhuang Autonomous Region, Nanning, China
| | - Hongying Wei
- First Affiliated Hospital of Guangxi Medical University, Nanning, China
| | - Liao Ning
- First Affiliated Hospital of Guangxi Medical University, Nanning, China
- The Key Laboratory of Children’s Disease Research in Guangxi’s Colleges and Universities, Education Department of Guangxi Zhuang Autonomous Region, Nanning, China
| | - Yunyan He
- First Affiliated Hospital of Guangxi Medical University, Nanning, China
- The Key Laboratory of Children’s Disease Research in Guangxi’s Colleges and Universities, Education Department of Guangxi Zhuang Autonomous Region, Nanning, China
| |
Collapse
|
7
|
Wu C, Hou G, Lin Y, Sa Z, Yan J, Zhang X, Liang Y, Yang K, Zhang Y, Lang H. Exploring links between Chinese military recruits' psychological stress and coping style from the person-environment fit perspective: The chain mediating effect of self-efficacy and social support. Front Psychol 2022; 13:996865. [PMID: 36405197 PMCID: PMC9673819 DOI: 10.3389/fpsyg.2022.996865] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/18/2022] [Accepted: 10/17/2022] [Indexed: 11/06/2022] Open
Abstract
The choice of coping style of recruits under psychological stress in the process of military task execution has been an important topic in the promotion of military operations and cohesion of military forces. Taking a positive coping style under psychological stress can help recruits overcome the negative effects of stress and improve military morale and group combat effectiveness. Although soldiers' psychological stress in the process of military mission execution having an impact on coping style has been studied by a large body of literature, very little literature has focused on the mechanism of self-efficacy and social support between recruits' psychological stress and coping style from the person-environment fit perspective. Therefore, this study was conducted to analyze the impact of recruits' psychological stress on coping style through a chain mediation model and to discuss the role of self-efficacy and social support in this relationship. Two waves of survey data were utilized to test the research hypotheses on a sample of 1028 Chinese recruits performing military tasks. The results indicated that recruits' psychological stress negatively impacted positive coping styles and positively correlated with negative ones. In addition, self-efficacy and social support mediated the relationship between psychological stress and positive coping style, and self-efficacy mediated the relationship between psychological stress and negative coping style. More importantly, self-efficacy and social support play the chain mediating effect between psychological stress and positive coping style.
Collapse
Affiliation(s)
- Chao Wu
- Department of Nursing, Fourth Military Medical University, Xi'an, China
| | - Guangdong Hou
- Department of Urology, Xijing Hospital, Fourth Military Medical University, Xi'an, China
| | - Yawei Lin
- Department of Nursing, Fourth Military Medical University, Xi'an, China
| | - Zhen Sa
- 69245 Troops of the Chinese People's Liberation Army, Xinjiang, China
| | - Jiaran Yan
- Department of Nursing, Fourth Military Medical University, Xi'an, China
| | - Xinyan Zhang
- Department of Engineer, Army 75 Group Military Hospital, Kunming, China
| | - Ying Liang
- 69243 Troops of the Chinese People's Liberation Army, Xinjiang, China
| | - Kejian Yang
- The 960th Hospital of the PLA Joint Logistics Support Force, Jinan, China
- *Correspondence: Kejian Yang
| | - Yuhai Zhang
- Department of Health Statistics, School of Preventive Medicine, Fourth Military Medical University, Xi'an, China
- Yuhai Zhang
| | - Hongjuan Lang
- Department of Nursing, Fourth Military Medical University, Xi'an, China
- Hongjuan Lang
| |
Collapse
|
8
|
Evaluating Eye Movement Event Detection: A Review of the State of the Art. Behav Res Methods 2022:10.3758/s13428-021-01763-7. [PMID: 35715615 DOI: 10.3758/s13428-021-01763-7] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 11/27/2021] [Indexed: 11/08/2022]
Abstract
Detecting eye movements in raw eye tracking data is a well-established research area by itself, as well as a common pre-processing step before any subsequent analysis. As in any field, however, progress and successful collaboration can only be achieved provided a shared understanding of the pursued goal. This is often formalised via defining metrics that express the quality of an approach to solving the posed problem. Both the big-picture intuition behind the evaluation strategies and seemingly small implementation details influence the resulting measures, making even studies with outwardly similar procedures essentially incomparable, impeding a common understanding. In this review, we systematically describe and analyse evaluation methods and measures employed in the eye movement event detection field to date. While recently developed evaluation strategies tend to quantify the detector's mistakes at the level of whole eye movement events rather than individual gaze samples, they typically do not separate establishing correspondences between true and predicted events from the quantification of the discovered errors. In our analysis we separate these two steps where possible, enabling their almost arbitrary combinations in an evaluation pipeline. We also present the first large-scale empirical analysis of event matching strategies in the literature, examining these various combinations both in practice and theoretically. We examine the particular benefits and downsides of the evaluation methods, providing recommendations towards more intuitive and informative assessment. We implemented the evaluation strategies on which this work focuses in a single publicly available library: https://github.com/r-zemblys/EM-event-detection-evaluation .
Collapse
|
9
|
Womelsdorf T, Watson MR, Tiesinga P. Learning at Variable Attentional Load Requires Cooperation of Working Memory, Meta-learning, and Attention-augmented Reinforcement Learning. J Cogn Neurosci 2021; 34:79-107. [PMID: 34813644 PMCID: PMC9830786 DOI: 10.1162/jocn_a_01780] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/12/2023]
Abstract
Flexible learning of changing reward contingencies can be realized with different strategies. A fast learning strategy involves using working memory of recently rewarded objects to guide choices. A slower learning strategy uses prediction errors to gradually update value expectations to improve choices. How the fast and slow strategies work together in scenarios with real-world stimulus complexity is not well known. Here, we aim to disentangle their relative contributions in rhesus monkeys while they learned the relevance of object features at variable attentional load. We found that learning behavior across six monkeys is consistently best predicted with a model combining (i) fast working memory and (ii) slower reinforcement learning from differently weighted positive and negative prediction errors as well as (iii) selective suppression of nonchosen feature values and (iv) a meta-learning mechanism that enhances exploration rates based on a memory trace of recent errors. The optimal model parameter settings suggest that these mechanisms cooperate differently at low and high attentional loads. Whereas working memory was essential for efficient learning at lower attentional loads, enhanced weighting of negative prediction errors and meta-learning were essential for efficient learning at higher attentional loads. Together, these findings pinpoint a canonical set of learning mechanisms and suggest how they may cooperate when subjects flexibly adjust to environments with variable real-world attentional demands.
Collapse
Affiliation(s)
- Thilo Womelsdorf
- Department of Psychology, Vanderbilt University, Nashville, TN 37240
| | - Marcus R. Watson
- School of Kinesiology and Health Science, Centre for Vision Research, York University, 4700 Keele Street, Toronto, Ontario M6J 1P3, Canada
| | - Paul Tiesinga
- Donders Institute for Brain, Cognition and Behaviour, Radboud University Nijmegen, Nijmegen 6525 EN, Netherlands
| |
Collapse
|