1
|
Mayol-Troncoso R, Gaspar PA, Verdugo R, Mariman JJ, Maldonado PE. Fixational eye movements and their associated evoked potentials during natural vision are altered in schizophrenia. Schizophr Res Cogn 2024; 38:100324. [PMID: 39238484 PMCID: PMC11375315 DOI: 10.1016/j.scog.2024.100324] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/25/2023] [Revised: 07/17/2024] [Accepted: 08/11/2024] [Indexed: 09/07/2024]
Abstract
Background Visual exploration is abnormal in schizophrenia; however, few studies have investigated the physiological responses during selecting objectives in more ecological scenarios. This study aimed to demonstrate that people with schizophrenia have difficulties observing the prominent elements of an image due to a deficit mechanism of sensory modulation (active sensing) during natural vision. Methods An electroencephalogram recording with eye tracking data was collected on 18 healthy individuals and 18 people affected by schizophrenia while looking at natural images. These had a prominent color element and blinking produced by changes in image luminance. Results We found fewer fixations when all images were scanned, late focus on prominent image areas, decreased amplitude in the eye-fixation-related potential, and decreased intertrial coherence in the SCZ group. Conclusions The decrease in the visual attention response evoked by the prominence of visual stimuli in patients affected by schizophrenia is generated by a reduction in endogenous attention mechanisms to initiate and maintain visual exploration. Further work is required to explain the relationship of this decrease with clinical indicators.
Collapse
Affiliation(s)
- Rocío Mayol-Troncoso
- Departamento de Psiquiatría y Salud Mental, Facultad de Medicina, Universidad de Chile
- Millennium Nucleus to Improve the Mental Health of Adolescents and Youths, Imhay, Chile
- Facultad de Psicología, Universidad Alberto Hurtado, Chile
- Clínica Psiquiátrica Universitaria, Hospital Clínico de la Universidad de Chile, Laboratorio Psiquiatría Traslacional
| | - Pablo A Gaspar
- Departamento de Psiquiatría y Salud Mental, Facultad de Medicina, Universidad de Chile
- Millennium Nucleus to Improve the Mental Health of Adolescents and Youths, Imhay, Chile
- Clínica Alemana, Santiago, Chile
- Department of Neuroscience, Faculty of Medicine, Universidad de Chile
- Clínica Psiquiátrica Universitaria, Hospital Clínico de la Universidad de Chile, Laboratorio Psiquiatría Traslacional
| | - Roberto Verdugo
- Biomedical Neuroscience Institute (BNI)
- Department of Neuroscience, Faculty of Medicine, Universidad de Chile
- Instituto Psiquiátrico Dr. José Horwitz Barak, Chile
| | - Juan J Mariman
- Department of Physical Therapy, Faculty of Arts and Physical Education, Universidad Metropolitana de Ciencias de la Educación, Santiago, Chile
- Department of Physical Therapy, Faculty of Medicine, Universidad de Chile
- Nucleus of wellbeing and human development, education research center (CIE-UMCE), Universidad Metropolitana de Ciencias de la educación
| | - Pedro E Maldonado
- Biomedical Neuroscience Institute (BNI)
- Department of Neuroscience, Faculty of Medicine, Universidad de Chile
- Nacional Center for Artificial Intelligence (CENIA), Chile
| |
Collapse
|
2
|
Gordon SM, Dalangin B, Touryan J. Saccade size predicts onset time of object processing during visual search of an open world virtual environment. Neuroimage 2024; 298:120781. [PMID: 39127183 DOI: 10.1016/j.neuroimage.2024.120781] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2024] [Revised: 08/02/2024] [Accepted: 08/08/2024] [Indexed: 08/12/2024] Open
Abstract
OBJECTIVE To date the vast majority of research in the visual neurosciences have been forced to adopt a highly constrained perspective of the vision system in which stimuli are processed in an open-loop reactive fashion (i.e., abrupt stimulus presentation followed by an evoked neural response). While such constraints enable high construct validity for neuroscientific investigation, the primary outcomes have been a reductionistic approach to isolate the component processes of visual perception. In electrophysiology, of the many neural processes studied under this rubric, the most well-known is, arguably, the P300 evoked response. There is, however, relatively little known about the real-world corollary of this component in free-viewing paradigms where visual stimuli are connected to neural function in a closed-loop. While growing evidence suggests that neural activity analogous to the P300 does occur in such paradigms, it is an open question when this response occurs and what behavioral or environmental factors could be used to isolate this component. APPROACH The current work uses convolutional networks to decode neural signals during a free-viewing visual search task in a closed-loop paradigm within an open-world virtual environment. From the decoded activity we construct fixation-locked response profiles that enable estimations of the variable latency of any P300 analogue around the moment of fixation. We then use these estimates to investigate which factors best reduce variable latency and, thus, predict the onset time of the response. We consider measurable, search-related factors encompassing top-down (i.e., goal driven) and bottom-up (i.e., stimulus driven) processes, such as fixation duration and salience. We also consider saccade size as an intermediate factor reflecting the integration of these two systems. MAIN RESULTS The results show that of these factors only saccade size reliably determines the onset time of P300 analogous activity for this task. Specifically, we find that for large saccades the variability in response onset is small enough to enable analysis using traditional ensemble averaging methods. SIGNIFICANCE The results show that P300 analogous activity does occur during closed-loop, free-viewing visual search while highlighting distinct differences between the open-loop version of this response and its real-world analogue. The results also further establish saccades, and saccade size, as a key factor in real-world visual processing.
Collapse
Affiliation(s)
| | | | - Jonathan Touryan
- DEVCOM Army Research Laboratory, Aberdeen Proving Ground, MD, USA
| |
Collapse
|
3
|
Hilton C, Kapaj A, Fabrikant SI. Fixation-related potentials during mobile map assisted navigation in the real world: The effect of landmark visualization style. Atten Percept Psychophys 2024:10.3758/s13414-024-02864-z. [PMID: 38468023 DOI: 10.3758/s13414-024-02864-z] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/13/2024] [Indexed: 03/13/2024]
Abstract
An often-proposed enhancement for mobile maps to aid assisted navigation is the presentation of landmark information, yet understanding of the manner in which they should be displayed is limited. In this study, we investigated whether the visualization of landmarks as 3D map symbols with either an abstract or realistic style influenced the subsequent processing of those landmarks during route navigation. We utilized a real-world mobile electroencephalography approach to this question by combining several tools developed to overcome the challenges typically encountered in real-world neuroscience research. We coregistered eye-movement and EEG recordings from 45 participants as they navigated through a real-world environment using a mobile map. Analyses of fixation event-related potentials revealed that the amplitude of the parietal P200 component was enhanced when participants fixated landmarks in the real world that were visualized on the mobile map in a realistic style, and that frontal P200 latencies were prolonged for landmarks depicted in either a realistic or abstract style compared with features of the environment that were not presented on the map, but only for the male participants. In contrast, we did not observe any significant effects of landmark visualization style on visual P1-N1 peaks or the parietal late positive component. Overall, the findings indicate that the cognitive matching process between landmarks seen in the environment and those previously seen on a map is facilitated by more realistic map display, while low-level perceptual processing of landmarks and recall of associated information are unaffected by map visualization style.
Collapse
Affiliation(s)
- Christopher Hilton
- Geographic Information Visualization & Analysis (GIVA), Department of Geography, University of Zurich- Irchel, Winterthurerstr. 190, CH-8057, Zurich, Switzerland.
- Institute of Psychology and Ergonomics, Technische Universität Berlin, Berlin, Germany.
| | - Armand Kapaj
- Geographic Information Visualization & Analysis (GIVA), Department of Geography, University of Zurich- Irchel, Winterthurerstr. 190, CH-8057, Zurich, Switzerland
| | - Sara Irina Fabrikant
- Geographic Information Visualization & Analysis (GIVA), Department of Geography, University of Zurich- Irchel, Winterthurerstr. 190, CH-8057, Zurich, Switzerland
| |
Collapse
|
4
|
Spiering L, Dimigen O. (Micro)saccade-related potentials during face recognition: A study combining EEG, eye-tracking, and deconvolution modeling. Atten Percept Psychophys 2024:10.3758/s13414-024-02846-1. [PMID: 38296873 DOI: 10.3758/s13414-024-02846-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 01/02/2024] [Indexed: 02/02/2024]
Abstract
Under natural viewing conditions, complex stimuli such as human faces are typically looked at several times in succession, implying that their recognition may unfold across multiple eye fixations. Although electrophysiological (EEG) experiments on face recognition typically prohibit eye movements, participants still execute frequent (micro)saccades on the face, each of which generates its own visuocortical response. This finding raises the question of whether the fixation-related potentials (FRPs) evoked by these tiny gaze shifts also contain psychologically valuable information about face processing. Here, we investigated this question by corecording EEG and eye movements in an experiment with emotional faces (happy, angry, neutral). Deconvolution modeling was used to separate the stimulus ERPs to face onset from the FRPs generated by subsequent microsaccades-induced refixations on the face. As expected, stimulus ERPs exhibited typical emotion effects, with a larger early posterior negativity (EPN) for happy/angry compared with neutral faces. Eye tracking confirmed that participants made small saccades in 98% of the trials, which were often aimed at the left eye of the stimulus face. However, while each saccade produced a strong response over visual areas, this response was unaffected by the face's emotional expression, both for the first and for subsequent (micro)saccades. This finding suggests that the face's affective content is rapidly evaluated after stimulus onset, leading to only a short-lived sensory enhancement by arousing stimuli that does not repeat itself during immediate refixations. Methodologically, our work demonstrates how eye tracking and deconvolution modeling can be used to extract several brain responses from each EEG trial, providing insights into neural processing at different latencies after stimulus onset.
Collapse
Affiliation(s)
- Lisa Spiering
- Department of Psychology, Humboldt-Universität zu Berlin, Berlin, Germany
- Department of Experimental Psychology, University of Oxford, Oxford, UK
| | - Olaf Dimigen
- Department of Psychology, Humboldt-Universität zu Berlin, Berlin, Germany.
- Department of Psychology, University of Groningen, Grote Kruisstraat 2/1, 9712 TS, Groningen, The Netherlands.
| |
Collapse
|
5
|
Nikolaev AR, Meghanathan RN, van Leeuwen C. Refixation behavior in naturalistic viewing: Methods, mechanisms, and neural correlates. Atten Percept Psychophys 2024:10.3758/s13414-023-02836-9. [PMID: 38169029 DOI: 10.3758/s13414-023-02836-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 12/17/2023] [Indexed: 01/05/2024]
Abstract
When freely viewing a scene, the eyes often return to previously visited locations. By tracking eye movements and coregistering eye movements and EEG, such refixations are shown to have multiple roles: repairing insufficient encoding from precursor fixations, supporting ongoing viewing by resampling relevant locations prioritized by precursor fixations, and aiding the construction of memory representations. All these functions of refixation behavior are understood to be underpinned by three oculomotor and cognitive systems and their associated brain structures. First, immediate saccade planning prior to refixations involves attentional selection of candidate locations to revisit. This process is likely supported by the dorsal attentional network. Second, visual working memory, involved in maintaining task-related information, is likely supported by the visual cortex. Third, higher-order relevance of scene locations, which depends on general knowledge and understanding of scene meaning, is likely supported by the hippocampal memory system. Working together, these structures bring about viewing behavior that balances exploring previously unvisited areas of a scene with exploiting visited areas through refixations.
Collapse
Affiliation(s)
- Andrey R Nikolaev
- Department of Psychology, Lund University, Box 213, 22100, Lund, Sweden.
- Brain & Cognition Research Unit, KU Leuven-University of Leuven, Leuven, Belgium.
| | | | - Cees van Leeuwen
- Brain & Cognition Research Unit, KU Leuven-University of Leuven, Leuven, Belgium
- Center for Cognitive Science, Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau, Kaiserslautern, Germany
| |
Collapse
|
6
|
Huber-Huber C, Melcher D. Saccade execution increases the preview effect with faces: An EEG and eye-tracking coregistration study. Atten Percept Psychophys 2023:10.3758/s13414-023-02802-5. [PMID: 37917292 DOI: 10.3758/s13414-023-02802-5] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 09/27/2023] [Indexed: 11/04/2023]
Abstract
Under naturalistic viewing conditions, humans conduct about three to four saccadic eye movements per second. These dynamics imply that in real life, humans rarely see something completely new; there is usually a preview of the upcoming foveal input from extrafoveal regions of the visual field. In line with results from the field of reading research, we have shown with EEG and eye-tracking coregistration that an extrafoveal preview also affects postsaccadic visual object processing and facilitates discrimination. Here, we ask whether this preview effect in the fixation-locked N170, and in manual responses to the postsaccadic target face (tilt discrimination), requires saccade execution. Participants performed a gaze-contingent experiment in which extrafoveal face images could change their orientation during a saccade directed to them. In a control block, participants maintained stable gaze throughout the experiment and the extrafoveal face reappeared foveally after a simulated saccade latency. Compared with this no-saccade condition, the neural and the behavioral preview effects were much larger in the saccade condition. We also found shorter first fixation durations after an invalid preview, which is in contrast to reading studies. We interpret the increased preview effect under saccade execution as the result of the additional sensorimotor processes that come with gaze behavior compared with visual perception under stable fixation. In addition, our findings call into question whether EEG studies with fixed gaze capture key properties and dynamics of active, natural vision.
Collapse
Affiliation(s)
- Christoph Huber-Huber
- Center for Mind/Brain Sciences (CIMeC), University of Trento, Corso Bettini 31, 38068, Rovereto, Italy.
| | - David Melcher
- Center for Brain & Health, New York University Abu Dhabi, Abu Dhabi, United Arab Emirates
- Psychology Program, Division of Science, New York University Abu Dhabi, Abu Dhabi, United Arab Emirates
| |
Collapse
|
7
|
Face familiarity revealed by fixational eye movements and fixation-related potentials in free viewing. Sci Rep 2022; 12:20178. [PMID: 36418497 PMCID: PMC9684544 DOI: 10.1038/s41598-022-24603-w] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2022] [Accepted: 11/17/2022] [Indexed: 11/25/2022] Open
Abstract
Event-related potentials (ERPs) and the oculomotor inhibition (OMI) in response to visual transients are known to be sensitive to stimulus properties, attention, and expectation. We have recently found that the OMI is also sensitive to face familiarity. In natural vision, stimulation of the visual cortex is generated primarily by saccades, and it has been recently suggested that fixation-related potentials (FRPs) share similar components with the ERPs. Here, we investigated whether FRPs and microsaccade inhibition (OMI) in free viewing are sensitive to face familiarity. Observers freely watched a slideshow of seven unfamiliar and one familiar facial images presented randomly for 4-s periods, with multiple images per identity. We measured the occipital fixation-related N1 relative to the P1 magnitude as well as the associated fixation-triggered OMI. We found that the average N1-P1 was significantly smaller and the OMI was shorter for the familiar face, compared with any of the seven unfamiliar faces. Moreover, the P1 was suppressed across saccades for the familiar but not for the unfamiliar faces. Our results highlight the sensitivity of the occipital FRPs to stimulus properties such as face familiarity and advance our understanding of the integration process across successive saccades in natural vision.
Collapse
|
8
|
Fixation-related saccadic inhibition in free viewing in response to stimulus saliency. Sci Rep 2022; 12:6619. [PMID: 35459790 PMCID: PMC9033846 DOI: 10.1038/s41598-022-10605-1] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/26/2021] [Accepted: 04/11/2022] [Indexed: 01/04/2023] Open
Abstract
Microsaccades that occur during fixation were studied extensively in response to transient stimuli, showing a typical inhibition (Oculomotor Inhibition, OMI), and a later release with a latency that depends on stimulus saliency, attention, and expectations. Here, we investigated the hypothesis that in free viewing every saccade provides a new transient stimulation that should result in a stimulus-dependent OMI like a flashed presentation during fixation. Participants (N = 16) freely inspected static displays of randomly oriented Gabor texture images, with varied contrast and spatial frequency (SF) for periods of 10 s each. Eye tracking recordings were divided into epochs triggered by saccade landing (> 1 dva), and microsaccade latency relative to fixation onset was computed (msRT). We found that the msRT in free viewing was shorter for more salient stimuli (higher contrast or lower SF), as previously found for flashed stimuli. It increased with saccade size and decreased across successive saccades, but only for higher contrast, suggesting contrast-dependent repetition enhancement in free viewing. Our results indicate that visual stimulus-dependent inhibition of microsaccades also applies to free viewing. These findings are in agreement with the similarity found between event-related and fixation-related potentials and open the way for studies combining both approaches to study natural vision.
Collapse
|
9
|
Callahan-Flintoft C, Barentine C, Touryan J, Ries AJ. A Case for Studying Naturalistic Eye and Head Movements in Virtual Environments. Front Psychol 2022; 12:650693. [PMID: 35035362 PMCID: PMC8759101 DOI: 10.3389/fpsyg.2021.650693] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/08/2021] [Accepted: 11/10/2021] [Indexed: 12/03/2022] Open
Abstract
Using head mounted displays (HMDs) in conjunction with virtual reality (VR), vision researchers are able to capture more naturalistic vision in an experimentally controlled setting. Namely, eye movements can be accurately tracked as they occur in concert with head movements as subjects navigate virtual environments. A benefit of this approach is that, unlike other mobile eye tracking (ET) set-ups in unconstrained settings, the experimenter has precise control over the location and timing of stimulus presentation, making it easier to compare findings between HMD studies and those that use monitor displays, which account for the bulk of previous work in eye movement research and vision sciences more generally. Here, a visual discrimination paradigm is presented as a proof of concept to demonstrate the applicability of collecting eye and head tracking data from an HMD in VR for vision research. The current work’s contribution is 3-fold: firstly, results demonstrating both the strengths and the weaknesses of recording and classifying eye and head tracking data in VR, secondly, a highly flexible graphical user interface (GUI) used to generate the current experiment, is offered to lower the software development start-up cost of future researchers transitioning to a VR space, and finally, the dataset analyzed here of behavioral, eye and head tracking data synchronized with environmental variables from a task specifically designed to elicit a variety of eye and head movements could be an asset in testing future eye movement classification algorithms.
Collapse
Affiliation(s)
- Chloe Callahan-Flintoft
- Humans in Complex System Directorate, United States Army Research Laboratory, Adelphi, MD, United States
| | - Christian Barentine
- Warfighter Effectiveness Research Center, United States Air Force Academy, Colorado Springs, CO, United States
| | - Jonathan Touryan
- Humans in Complex System Directorate, United States Army Research Laboratory, Adelphi, MD, United States
| | - Anthony J Ries
- Humans in Complex System Directorate, United States Army Research Laboratory, Adelphi, MD, United States.,Warfighter Effectiveness Research Center, United States Air Force Academy, Colorado Springs, CO, United States
| |
Collapse
|
10
|
Nicolas G, Castet E, Rabier A, Kristensen E, Dojat M, Guérin-Dugué A. Neural correlates of intra-saccadic motion perception. J Vis 2021; 21:19. [PMID: 34698810 PMCID: PMC8556557 DOI: 10.1167/jov.21.11.19] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022] Open
Abstract
Retinal motion of the visual scene is not consciously perceived during ocular saccades in normal everyday conditions. It has been suggested that extra-retinal signals actively suppress intra-saccadic motion perception to preserve stable perception of the visual world. However, using stimuli optimized to preferentially activate the M-pathway, Castet and Masson (2000) demonstrated that motion can be perceived during a saccade. Based on this psychophysical paradigm, we used electroencephalography and eye-tracking recordings to investigate the neural correlates related to the conscious perception of intra-saccadic motion. We demonstrated the effective involvement during saccades of the cortical areas V1-V2 and MT-V5, which convey motion information along the M-pathway. We also showed that individual motion perception was related to retinal temporal frequency.
Collapse
Affiliation(s)
- Gaëlle Nicolas
- Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-Lab, 38000 Grenoble, France.,
| | - Eric Castet
- LPC, Laboratoire de Psychologie Cognitive (UMR 7290), Aix-Marseille Univ, CNRS, LPC, Marseille, France.,
| | - Adrien Rabier
- Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-Lab, 38000 Grenoble, France.,
| | | | - Michel Dojat
- Univ. Grenoble Alpes, Inserm, U1216, Grenoble Institut Neurosciences, GIN, 38000 Grenoble, France.,
| | - Anne Guérin-Dugué
- Univ. Grenoble Alpes, CNRS, Grenoble INP, GIPSA-Lab, 38000 Grenoble, France.,
| |
Collapse
|
11
|
Stankov AD, Touryan J, Gordon S, Ries AJ, Ki J, Parra LC. During natural viewing, neural processing of visual targets continues throughout saccades. J Vis 2021; 21:7. [PMID: 34491271 PMCID: PMC8431980 DOI: 10.1167/jov.21.10.7] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022] Open
Abstract
Relatively little is known about visual processing during free-viewing visual search in realistic dynamic environments. Free-viewing is characterized by frequent saccades. During saccades, visual processing is thought to be suppressed, yet we know that the presaccadic visual content can modulate postsaccadic processing. To better understand these processes in a realistic setting, we study here saccades and neural responses elicited by the appearance of visual targets in a realistic virtual environment. While subjects were being driven through a 3D virtual town, they were asked to discriminate between targets that appear on the road. Using a system identification approach, we separated overlapping and correlated activity evoked by visual targets, saccades, and button presses. We found that the presence of a target enhances early occipital as well as late frontocentral saccade-related responses. The earlier potential, shortly after 125 ms post-saccade onset, was enhanced for targets that appeared in the peripheral vision as compared to the central vision, suggesting that fast peripheral processing initiated before saccade onset. The later potential, at 195 ms post-saccade onset, was strongly modulated by the visibility of the target. Together these results suggest that, during natural viewing, neural processing of the presaccadic visual stimulus continues throughout the saccade, apparently unencumbered by saccadic suppression.
Collapse
Affiliation(s)
- Atanas D Stankov
- Department of Biomedical Engineering, City College of New York, New York, NY, USA.,
| | - Jonathan Touryan
- U.S. Army Research Laboratory, Aberdeen Proving Ground, MD, USA.,
| | | | - Anthony J Ries
- U.S. Army Research Laboratory, Aberdeen Proving Ground, MD, USA.,
| | - Jason Ki
- Department of Biomedical Engineering, City College of New York, New York, NY, USA.,
| | - Lucas C Parra
- Department of Biomedical Engineering, City College of New York, New York, NY, USA.,
| |
Collapse
|
12
|
Ranjan R, Chandra Sahana B, Kumar Bhandari A. Ocular artifact elimination from electroencephalography signals: A systematic review. Biocybern Biomed Eng 2021. [DOI: 10.1016/j.bbe.2021.06.007] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 02/02/2023]
|
13
|
Huber-Huber C, Steininger J, Grüner M, Ansorge U. Psychophysical dual-task setups do not measure pre-saccadic attention but saccade-related strengthening of sensory representations. Psychophysiology 2021; 58:e13787. [PMID: 33615491 PMCID: PMC8244053 DOI: 10.1111/psyp.13787] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/14/2020] [Revised: 01/22/2021] [Accepted: 01/25/2021] [Indexed: 11/29/2022]
Abstract
Visual attention and saccadic eye movements are linked in a tight, yet flexible fashion. In humans, this link is typically studied with dual‐task setups. Participants are instructed to execute a saccade to some target location, while a discrimination target is flashed on a screen before the saccade can be made. Participants are also instructed to report a specific feature of this discrimination target at the trial end. Discrimination performance is usually better if the discrimination target occurred at the same location as the saccade target compared to when it occurred at a different location, which is explained by the mandatory shift of attention to the saccade target location before saccade onset. This pre‐saccadic shift of attention presumably enhances the perception of the discrimination target if it occurred at the same, but not if it occurred at a different location. It is, however, known that a dual‐task setup can alter the primary process under investigation. Here, we directly compared pre‐saccadic attention in single‐task versus dual‐task setups using concurrent electroencephalography (EEG) and eye‐tracking. Our results corroborate the idea of a pre‐saccadic shift of attention. They, however, question that this shift leads to the same‐position discrimination advantage. The relation of saccade and discrimination target position affected the EEG signal only after saccade onset. Our results, thus, favor an alternative explanation based on the role of saccades for the consolidation of sensory and short‐term memory. We conclude that studies with dual‐task setups arrived at a valid conclusion despite not measuring exactly what they intended to measure. In humans, the relation between visual attention and saccadic eye movements is usually studied with psychophysical dual‐task setups. Here, we employ concurrent EEG and eye‐tracking to directly compare dual‐task to single‐task setups and conclude in line with previous research that attention precedes saccades. However, our results suggest that dual‐task setups do not measure what they are supposed to measure, that is, the pre‐saccadic shift of attention.
Collapse
Affiliation(s)
- Christoph Huber-Huber
- Donders Institute for Brain, Cognition, and Behaviour, Radboud University, Nijmegen, Netherlands.,Center for Mind/Brain Sciences (CIMeC), University of Trento, Rovereto, Italy
| | - Julia Steininger
- Department of Cognition, Emotion, and Methods in Psychology, University of Vienna, Vienna, Austria
| | - Markus Grüner
- Department of Cognition, Emotion, and Methods in Psychology, University of Vienna, Vienna, Austria
| | - Ulrich Ansorge
- Department of Cognition, Emotion, and Methods in Psychology, University of Vienna, Vienna, Austria
| |
Collapse
|
14
|
Zhao DG, Vasilyev AN, Kozyrskiy BL, Melnichuk EV, Isachenko AV, Velichkovsky BM, Shishkin SL. A passive BCI for monitoring the intentionality of the gaze-based moving object selection. J Neural Eng 2021; 18. [PMID: 33418554 DOI: 10.1088/1741-2552/abda09] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/22/2020] [Accepted: 01/08/2021] [Indexed: 11/11/2022]
Abstract
OBJECTIVE The use of an electroencephalogram (EEG) anticipation-related component, the expectancy wave (E-wave), in brain-machine interaction was proposed more than 50 years ago. This possibility was not explored for decades, but recently it was shown that voluntary attempts to select items using eye fixations, but not spontaneous eye fixations, are accompanied by the E-wave. Thus, the use of the E-wave detection was proposed for the enhancement of gaze interaction technology, which has a strong need for a mean to decide if a gaze behaviour is voluntary or not. Here, we attempted at estimating whether this approach can be used in the context of moving object selection through smooth pursuit eye movements. APPROACH 18 participants selected, one by one, items which moved on a computer screen, by gazing at them. In separate runs, the participants performed tasks not related to voluntary selection but also provoking smooth pursuit. A low-cost consumer-grade eye tracker was used for item selection. MAIN RESULTS A component resembling the E-wave was found in the averaged EEG segments time-locked to voluntary selection events of every participant. Linear discriminant analysis with shrinkage regularization (sLDA) classified the intentional and spontaneous smooth pursuit eye movements, using single-trial 300 ms long EEG segments, significantly above chance in eight participants. When the classifier output was averaged over ten subsequent data segments, median group ROC AUC of 0.75 was achieved. SIGNIFICANCE The results suggest the possible usefulness of the E-wave detection in the gaze-based selection of moving items, e.g., in video games. This technique might be more effective when trial data can be averaged, thus it could be considered for use in passive interfaces, for example, in estimating the degree of the user's involvement during gaze-based interaction.
Collapse
Affiliation(s)
- Darisy Guanlinovich Zhao
- Laboratory for Neurocognitive Technology, NRC Kurchatov Institute, 1, Akademika Kurchatova pl., Moscow, 123182, RUSSIAN FEDERATION
| | - Anatoly N Vasilyev
- Laboratory for Neurophysiology and Neuro-Computer Interfaces, Lomonosov Moscow State University, 1, Akademika Kurchatova pl., Moscow, 123182, RUSSIAN FEDERATION
| | - Bogdan L Kozyrskiy
- Department of Data Science, EURECOM, 450 Route des Chappes, Sophia Antipolis, Provence-Alpes-Côte d'Azu, CS 50193 - 0690, FRANCE
| | - Eugeny V Melnichuk
- Laboratory for Neurocognitive Technologies, NRC Kurchatov Institute, 1, Akademika Kurchatova pl., Moscow, 123182, RUSSIAN FEDERATION
| | - Andrey V Isachenko
- Laboratory for Neurocognitive Technologies, NRC Kurchatov Institute, 1, Akademika Kurchatova pl., Moscow, 123182, RUSSIAN FEDERATION
| | - Boris M Velichkovsky
- Laboratory for Neurocognitive Technologies, NRC Kurchatov Institute, 1, Akademika Kurchatova pl., Moscow, 123182, RUSSIAN FEDERATION
| | - Sergei L Shishkin
- MEG Center, Moscow State University of Psychology and Education, 2А-2, Shelepikhinskaya Naberezhnaya, Moscow, 123290, RUSSIAN FEDERATION
| |
Collapse
|
15
|
Dimigen O, Ehinger BV. Regression-based analysis of combined EEG and eye-tracking data: Theory and applications. J Vis 2021; 21:3. [PMID: 33410892 PMCID: PMC7804566 DOI: 10.1167/jov.21.1.3] [Citation(s) in RCA: 33] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2019] [Accepted: 08/14/2020] [Indexed: 12/27/2022] Open
Abstract
Fixation-related potentials (FRPs), neural responses aligned to the end of saccades, are a promising tool for studying the dynamics of attention and cognition under natural viewing conditions. In the past, four methodological problems have complicated the analysis of such combined eye-tracking/electroencephalogram experiments: (1) the synchronization of data streams, (2) the removal of ocular artifacts, (3) the condition-specific temporal overlap between the brain responses evoked by consecutive fixations, and (4) the fact that numerous low-level stimulus and saccade properties also influence the postsaccadic neural responses. Although effective solutions exist for the first two problems, the latter two are only beginning to be addressed. In the current paper, we present and review a unified regression-based framework for FRP analysis that allows us to deconvolve overlapping potentials while also controlling for both linear and nonlinear confounds on the FRP waveform. An open software implementation is provided for all procedures. We then demonstrate the advantages of this proposed (non)linear deconvolution modeling approach for data from three commonly studied paradigms: face perception, scene viewing, and reading. First, for a traditional event-related potential (ERP) face recognition experiment, we show how this technique can separate stimulus ERPs from overlapping muscle and brain potentials produced by small (micro)saccades on the face. Second, in natural scene viewing, we model and isolate multiple nonlinear effects of saccade parameters on the FRP. Finally, for a natural sentence reading experiment using the boundary paradigm, we show how it is possible to study the neural correlates of parafoveal preview after removing spurious overlap effects caused by the associated difference in average fixation time. Our results suggest a principal way of measuring reliable eye movement-related brain activity during natural vision.
Collapse
Affiliation(s)
- Olaf Dimigen
- Department of Psychology, Humboldt-Universität zu Berlin, Berlin, Germany
| | - Benedikt V Ehinger
- Institute of Cognitive Science, Universität Osnabrück, Osnabrück, Germany
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
| |
Collapse
|
16
|
Savage SW, Potter DD, Tatler BW. The effects of cognitive distraction on behavioural, oculomotor and electrophysiological metrics during a driving hazard perception task. ACCIDENT; ANALYSIS AND PREVENTION 2020; 138:105469. [PMID: 32113007 DOI: 10.1016/j.aap.2020.105469] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/19/2019] [Revised: 01/10/2020] [Accepted: 02/10/2020] [Indexed: 06/10/2023]
Abstract
Previous research has demonstrated that the distraction caused by holding a mobile telephone conversation is not limited to the period of the actual conversation (Haigney, 1995; Redelmeier & Tibshirani, 1997; Savage et al., 2013). In a prior study we identified potential eye movement and EEG markers of cognitive distraction during driving hazard perception. However the extent to which these markers are affected by the demands of the hazard perception task are unclear. Therefore in the current study we assessed the effects of secondary cognitive task demand on eye movement and EEG metrics separately for periods prior to, during and after the hazard was visible. We found that when no hazard was present (prior and post hazard windows), distraction resulted in changes to various elements of saccadic eye movements. However, when the target was present, distraction did not affect eye movements. We have previously found evidence that distraction resulted in an overall decrease in theta band output at occipital sites of the brain. This was interpreted as evidence that distraction results in a reduction in visual processing. The current study confirmed this by examining the effects of distraction on the lambda response component of subjects eye fixation related potentials (EFRPs). Furthermore, we demonstrated that although detections of hazards were not affected by distraction, both eye movement and EEG metrics prior to the onset of the hazard were sensitive to changes in cognitive workload. This suggests that changes to specific aspects of the saccadic eye movement system could act as unobtrusive markers of distraction even prior to a breakdown in driving performance.
Collapse
Affiliation(s)
- Steven W Savage
- Schepens Eye Research Institute, Massachusetts Eye and Ear, Harvard Medical School, 20 Staniford Street, 02114, Boston, MA, USA.
| | | | | |
Collapse
|
17
|
Auerbach-Asch CR, Bein O, Deouell LY. Face Selective Neural Activity: Comparisons Between Fixed and Free Viewing. Brain Topogr 2020; 33:336-354. [PMID: 32236786 DOI: 10.1007/s10548-020-00764-7] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/16/2019] [Accepted: 03/14/2020] [Indexed: 10/24/2022]
Abstract
Event Related Potentials (ERPs) are widely used to study category-selective EEG responses to visual stimuli, such as the face-selective N170 component. Typically, this is done by flashing stimuli at the point of static gaze fixation. While allowing for good experimental control, these paradigms ignore the dynamic role of eye-movements in natural vision. Fixation-related potentials (FRPs), obtained using simultaneous EEG and eye-tracking, overcome this limitation. Various studies have used FRPs to study processes such as lexical processing, target detection and attention allocation. The goal of this study was to carefully compare face-sensitive activity time-locked to an abrupt stimulus onset at fixation, with that time-locked to a self-generated fixation on a stimulus. Twelve participants participated in three experimental conditions: Free-viewing (FRPs), Cued-viewing (FRPs) and Control (ERPs). We used a multiple regression approach to disentangle overlapping activity components. Our results show that the N170 face-effect is evident for the first fixation on a stimulus, whether it follows a self-generated saccade or stimulus appearance at fixation point. The N170 face-effect has similar topography across viewing conditions, but there were major differences within each stimulus category. We ascribe these differences to an overlap of the fixation-related lambda response and the N170. We tested the plausibility of this account using dipole simulations. Finally, the N170 exhibits category-specific adaptation in free viewing. This study establishes the comparability of the free-viewing N170 face-effect with the classic event-related effect, while highlighting the importance of accounting for eye-movement related effects.
Collapse
Affiliation(s)
- Carmel R Auerbach-Asch
- The Edmond and Lily Safra Center for Brain Sciences, The Hebrew University of Jerusalem, Edmond J. Safra Campus, Givat-Ram, 91904, Jerusalem, Israel.
| | - Oded Bein
- The Department of Psychology, New York University, 6 Washington Pl, New York, NY, 10003, USA
| | - Leon Y Deouell
- The Edmond and Lily Safra Center for Brain Sciences, The Hebrew University of Jerusalem, Edmond J. Safra Campus, Givat-Ram, 91904, Jerusalem, Israel
- The Department of Psychology, The Hebrew University of Jerusalem, Mount Scopus, 91905, Jerusalem, Israel
| |
Collapse
|
18
|
Ahlström C, Solis-Marcos I, Nilsson E, Åkerstedt T. The impact of driver sleepiness on fixation-related brain potentials. J Sleep Res 2019; 29:e12962. [PMID: 31828862 DOI: 10.1111/jsr.12962] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/12/2019] [Revised: 11/14/2019] [Accepted: 11/18/2019] [Indexed: 11/30/2022]
Abstract
The effects of driver sleepiness are often quantified as deteriorated driving performance, increased blink durations and high levels of subjective sleepiness. Driver sleepiness has also been associated with increasing levels of electroencephalogram (EEG) power, especially in the alpha range. The present exploratory study investigated a new measure of driver sleepiness, the EEG fixation-related lambda response. Thirty young male drivers (23.6 ± 1.7 years old) participated in a driving simulator experiment in which they drove on rural and suburban roads in simulated daylight versus darkness during both the daytime (full sleep) and night-time (sleep deprived). The results show lower lambda responses during night driving and with longer time on task, indicating that sleep deprivation and time on task cause a general decrement in cortical responsiveness to incoming visual stimuli. Levels of subjective sleepiness and line crossings were higher under the same conditions. Furthermore, results of a linear mixed-effects model showed that low lambda responses are associated with high subjective sleepiness and more line crossings. We suggest that the fixation-related lambda response can be used to investigate driving impairment induced by sleep deprivation while driving and that, after further refinement, it may be useful as an objective measure of driver sleepiness.
Collapse
Affiliation(s)
- Christer Ahlström
- Swedish National Road and Transport Research Institute (VTI), Linköping, Sweden.,Department of Biomedical Engineering, Linköping University, Linköping, Sweden
| | | | - Emma Nilsson
- Volvo Cars Safety Centre, Volvo Car Corporation, Göteborg, Sweden.,Department of Mechanics and Maritime Sciences, Chalmers University of Technology, Göteborg, Sweden
| | - Torbjörn Åkerstedt
- Stress Research Institute, Stockholm University, Stockholm, Sweden.,Clinical Neuroscience, Karolinska Institute, Stockholm, Sweden
| |
Collapse
|
19
|
Loberg O, Hautala J, Hämäläinen JA, Leppänen PHT. Influence of reading skill and word length on fixation-related brain activity in school-aged children during natural reading. Vision Res 2019; 165:109-122. [PMID: 31710840 DOI: 10.1016/j.visres.2019.07.008] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2019] [Revised: 07/25/2019] [Accepted: 07/28/2019] [Indexed: 10/25/2022]
Abstract
Word length is one of the main determinants of eye movements during reading and has been shown to influence slow readers more strongly than typical readers. The influence of word length on reading in individuals with different reading skill levels has been shown in separate eye-tracking and electroencephalography studies. However, the influence of reading difficulty on cortical correlates of word length effect during natural reading is unknown. To investigate how reading skill is related to brain activity during natural reading, we performed an exploratory analysis on our data set from a previous study, where slow reading (N = 27) and typically reading (N = 65) 12-to-13.5-year-old children read sentences while co-registered ET-EEG was recorded. We extracted fixation-related potentials (FRPs) from the sentences using the linear deconvolution approach. We examined standard eye-movement variables and deconvoluted FRP estimates: intercept of the response, categorical effect of first fixation versus additional fixation and continuous effect of word length. We replicated the pattern of stronger word length effect in eye movements for slow readers. We found a difference between typical readers and slow readers in the FRP intercept, which contains activity that is common to all fixations, within a fixation time-window of 50-300 ms. For both groups, the word length effect was present in brain activity during additional fixations; however, this effect was not different between groups. This suggests that stronger word length effect in the eye movements of slow readers might be mainly due re-fixations, which are more probable due to the lower efficiency of visual processing.
Collapse
Affiliation(s)
- Otto Loberg
- Department of Psychology, University of Jyväskylä, Finland.
| | | | | | | |
Collapse
|
20
|
Dimigen O. Optimizing the ICA-based removal of ocular EEG artifacts from free viewing experiments. Neuroimage 2019; 207:116117. [PMID: 31689537 DOI: 10.1016/j.neuroimage.2019.116117] [Citation(s) in RCA: 87] [Impact Index Per Article: 14.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/18/2019] [Revised: 07/01/2019] [Accepted: 08/20/2019] [Indexed: 11/30/2022] Open
Abstract
Combining EEG with eye-tracking is a promising approach to study neural correlates of natural vision, but the resulting recordings are also heavily contaminated by activity of the eye balls, eye lids, and extraocular muscles. While Independent Component Analysis (ICA) is commonly used to suppress these ocular artifacts, its performance under free viewing conditions has not been systematically evaluated and many published reports contain residual artifacts. Here I evaluated and optimized ICA-based correction for two tasks with unconstrained eye movements: visual search in images and sentence reading. In a first step, four parameters of the ICA pipeline were varied orthogonally: the (1) high-pass and (2) low-pass filter applied to the training data, (3) the proportion of training data containing myogenic saccadic spike potentials (SP), and (4) the threshold for eye tracker-based component rejection. In a second step, the eye-tracker was used to objectively quantify the correction quality of each ICA solution, both in terms of undercorrection (residual artifacts) and overcorrection (removal of neurogenic activity). As a benchmark, results were compared to those obtained with an alternative spatial filter, Multiple Source Eye Correction (MSEC). With commonly used settings, Infomax ICA not only left artifacts in the data, but also distorted neurogenic activity during eye movement-free intervals. However, correction results could be strongly improved by training the ICA on optimally filtered data in which SPs were massively overweighted. With optimized procedures, ICA removed virtually all artifacts, including the SP and its associated spectral broadband artifact from both viewing paradigms, with little distortion of neural activity. It also outperformed MSEC in terms of SP correction. Matlab code is provided.
Collapse
Affiliation(s)
- Olaf Dimigen
- Humboldt-Universität zu Berlin, Unter den Linden 6, 10099, Berlin, Germany.
| |
Collapse
|
21
|
Huber-Huber C, Buonocore A, Dimigen O, Hickey C, Melcher D. The peripheral preview effect with faces: Combined EEG and eye-tracking suggests multiple stages of trans-saccadic predictive and non-predictive processing. Neuroimage 2019; 200:344-362. [PMID: 31260837 DOI: 10.1016/j.neuroimage.2019.06.059] [Citation(s) in RCA: 22] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/12/2018] [Revised: 05/23/2019] [Accepted: 06/25/2019] [Indexed: 02/06/2023] Open
Abstract
The world appears stable despite saccadic eye-movements. One possible explanation for this phenomenon is that the visual system predicts upcoming input across saccadic eye-movements based on peripheral preview of the saccadic target. We tested this idea using concurrent electroencephalography (EEG) and eye-tracking. Participants made cued saccades to peripheral upright or inverted face stimuli that changed orientation (invalid preview) or maintained orientation (valid preview) while the saccade was completed. Experiment 1 demonstrated better discrimination performance and a reduced fixation-locked N170 component (fN170) with valid than with invalid preview, demonstrating integration of pre- and post-saccadic information. Moreover, the early fixation-related potentials (FRP) showed a preview face inversion effect suggesting that some pre-saccadic input was represented in the brain until around 170 ms post fixation-onset. Experiment 2 replicated Experiment 1 and manipulated the proportion of valid and invalid trials to test whether the preview effect reflects context-based prediction across trials. A whole-scalp Bayes factor analysis showed that this manipulation did not alter the fN170 preview effect but did influence the face inversion effect before the saccade. The pre-saccadic inversion effect declined earlier in the mostly invalid block than in the mostly valid block, which is consistent with the notion of pre-saccadic expectations. In addition, in both studies, we found strong evidence for an interaction between the pre-saccadic preview stimulus and the post-saccadic target as early as 50 ms (Experiment 2) or 90 ms (Experiment 1) into the new fixation. These findings suggest that visual stability may involve three temporal stages: prediction about the saccadic target, integration of pre-saccadic and post-saccadic information at around 50-90 ms post fixation onset, and post-saccadic facilitation of rapid categorization.
Collapse
Affiliation(s)
- Christoph Huber-Huber
- Center for Mind/Brain Sciences (CIMeC), University of Trento, Corso Bettini 31, Rovereto, TN, 38068, Italy.
| | - Antimo Buonocore
- Werner Reichardt Centre for Integrative Neuroscience, Tuebingen University, Otfried-Müller-Straße 25, Tuebingen, 72076, Germany; Hertie Institute for Clinical Brain Research, Tuebingen University, Tuebingen, 72076, Germany
| | - Olaf Dimigen
- Department of Psychology, Humboldt-Universität zu Berlin, Unter Den Linden 6, 10099, Berlin, Germany
| | - Clayton Hickey
- Center for Mind/Brain Sciences (CIMeC), University of Trento, Corso Bettini 31, Rovereto, TN, 38068, Italy
| | - David Melcher
- Center for Mind/Brain Sciences (CIMeC), University of Trento, Corso Bettini 31, Rovereto, TN, 38068, Italy
| |
Collapse
|