1
|
Hoogerbrugge AJ, Strauch C, Böing S, Nijboer TCW, Van der Stigchel S. Just-in-Time Encoding Into Visual Working Memory Is Contingent Upon Constant Availability of External Information. J Cogn 2024; 7:39. [PMID: 38706717 PMCID: PMC11067970 DOI: 10.5334/joc.364] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/08/2023] [Accepted: 04/16/2024] [Indexed: 05/07/2024] Open
Abstract
Humans maintain an intricate balance between storing information in visual working memory (VWM) and just-in-time sampling of the external world, rooted in a trade-off between the cost of maintaining items in VWM versus retrieving information as it is needed. Previous studies have consistently shown that one prerequisite of just-in-time sampling is a high degree of availability of external information, and that introducing a delay before being able to access information led participants to rely less on the external world and more on VWM. However, these studies manipulated availability in such a manner that the cost of sampling was stable and predictable. It is yet unclear whether participants become less reliant on external information when it is more difficult to factor in the cost of sampling that information. In two experiments, participants copied an example layout from the left to the right side of the screen. In Experiment 1, intermittent occlusion of the example layout led participants to attempt to encode more items per inspection than when the layout was constantly available, but this did not consistently result in more correct placements. However, these findings could potentially be explained by inherent differences in how long the example layout could be viewed. Therefore in Experiment 2, the example layout only became available after a gaze-contingent delay, which could be constant or variable. Here, the introduction of any delay led to increased VWM load compared to no delay, although the degree of variability in the delay did not alter behaviour. These results reaffirm that the nature of when we engage VWM is dynamical, and suggest that any disruption to the continuous availability of external information is the main driver of increased VWM usage relative to whether availability is predictable or not.
Collapse
Affiliation(s)
- Alex J. Hoogerbrugge
- Experimental Psychology, Helmholtz Institute, Utrecht University, Utrecht, The Netherlands
| | - Christoph Strauch
- Experimental Psychology, Helmholtz Institute, Utrecht University, Utrecht, The Netherlands
| | - Sanne Böing
- Experimental Psychology, Helmholtz Institute, Utrecht University, Utrecht, The Netherlands
| | - Tanja C. W. Nijboer
- Experimental Psychology, Helmholtz Institute, Utrecht University, Utrecht, The Netherlands
| | | |
Collapse
|
2
|
Nikolaev AR, Meghanathan RN, van Leeuwen C. Refixation behavior in naturalistic viewing: Methods, mechanisms, and neural correlates. Atten Percept Psychophys 2024:10.3758/s13414-023-02836-9. [PMID: 38169029 DOI: 10.3758/s13414-023-02836-9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 12/17/2023] [Indexed: 01/05/2024]
Abstract
When freely viewing a scene, the eyes often return to previously visited locations. By tracking eye movements and coregistering eye movements and EEG, such refixations are shown to have multiple roles: repairing insufficient encoding from precursor fixations, supporting ongoing viewing by resampling relevant locations prioritized by precursor fixations, and aiding the construction of memory representations. All these functions of refixation behavior are understood to be underpinned by three oculomotor and cognitive systems and their associated brain structures. First, immediate saccade planning prior to refixations involves attentional selection of candidate locations to revisit. This process is likely supported by the dorsal attentional network. Second, visual working memory, involved in maintaining task-related information, is likely supported by the visual cortex. Third, higher-order relevance of scene locations, which depends on general knowledge and understanding of scene meaning, is likely supported by the hippocampal memory system. Working together, these structures bring about viewing behavior that balances exploring previously unvisited areas of a scene with exploiting visited areas through refixations.
Collapse
Affiliation(s)
- Andrey R Nikolaev
- Department of Psychology, Lund University, Box 213, 22100, Lund, Sweden.
- Brain & Cognition Research Unit, KU Leuven-University of Leuven, Leuven, Belgium.
| | | | - Cees van Leeuwen
- Brain & Cognition Research Unit, KU Leuven-University of Leuven, Leuven, Belgium
- Center for Cognitive Science, Rheinland-Pfälzische Technische Universität Kaiserslautern-Landau, Kaiserslautern, Germany
| |
Collapse
|
3
|
Han NX, Eckstein MP. Inferential eye movement control while following dynamic gaze. eLife 2023; 12:e83187. [PMID: 37615158 PMCID: PMC10473837 DOI: 10.7554/elife.83187] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/02/2022] [Accepted: 07/31/2023] [Indexed: 08/25/2023] Open
Abstract
Attending to other people's gaze is evolutionary important to make inferences about intentions and actions. Gaze influences covert attention and triggers eye movements. However, we know little about how the brain controls the fine-grain dynamics of eye movements during gaze following. Observers followed people's gaze shifts in videos during search and we related the observer eye movement dynamics to the time course of gazer head movements extracted by a deep neural network. We show that the observers' brains use information in the visual periphery to execute predictive saccades that anticipate the information in the gazer's head direction by 190-350ms. The brain simultaneously monitors moment-to-moment changes in the gazer's head velocity to dynamically alter eye movements and re-fixate the gazer (reverse saccades) when the head accelerates before the initiation of the first forward gaze-following saccade. Using saccade-contingent manipulations of the videos, we experimentally show that the reverse saccades are planned concurrently with the first forward gaze-following saccade and have a functional role in reducing subsequent errors fixating on the gaze goal. Together, our findings characterize the inferential and functional nature of social attention's fine-grain eye movement dynamics.
Collapse
Affiliation(s)
- Nicole Xiao Han
- Department of Psychological and Brain Sciences, Institute for Collaborative Biotechnologies, University of California, Santa BarbaraSanta BarbaraUnited States
| | - Miguel Patricio Eckstein
- Department of Psychological and Brain Sciences, Department of Electrical and Computer Engineering, Department of Computer Science, Institute for Collaborative Biotechnologies, University of California, Santa BarbaraSanta BarbaraUnited States
| |
Collapse
|
4
|
Doidy F, Desaunay P, Rebillard C, Clochon P, Lambrechts A, Wantzen P, Guénolé F, Baleyte JM, Eustache F, Bowler DM, Lebreton K, Guillery-Girard B. How scene encoding affects memory discrimination: Analysing eye movements data using data driven methods. VISUAL COGNITION 2023. [DOI: 10.1080/13506285.2023.2188335] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/08/2023]
Affiliation(s)
- F. Doidy
- Normandie Université, UNICAEN, PSL Université Paris, EPHE, INSERM, U1077, CHU de Caen, GIP Cyceron, Neuropsychologie et Imagerie de la Mémoire Humaine, Caen, France
| | - P. Desaunay
- Normandie Université, UNICAEN, PSL Université Paris, EPHE, INSERM, U1077, CHU de Caen, GIP Cyceron, Neuropsychologie et Imagerie de la Mémoire Humaine, Caen, France
- Service de Psychiatrie de l’enfant et de l’adolescent, CHU de Caen, Caen, France
| | - C. Rebillard
- Normandie Université, UNICAEN, PSL Université Paris, EPHE, INSERM, U1077, CHU de Caen, GIP Cyceron, Neuropsychologie et Imagerie de la Mémoire Humaine, Caen, France
| | - P. Clochon
- Normandie Université, UNICAEN, PSL Université Paris, EPHE, INSERM, U1077, CHU de Caen, GIP Cyceron, Neuropsychologie et Imagerie de la Mémoire Humaine, Caen, France
| | - A. Lambrechts
- Autism Research Group, Department of Psychology, City, University of London, London, UK
| | - P. Wantzen
- Normandie Université, UNICAEN, PSL Université Paris, EPHE, INSERM, U1077, CHU de Caen, GIP Cyceron, Neuropsychologie et Imagerie de la Mémoire Humaine, Caen, France
| | - F. Guénolé
- Normandie Université, UNICAEN, PSL Université Paris, EPHE, INSERM, U1077, CHU de Caen, GIP Cyceron, Neuropsychologie et Imagerie de la Mémoire Humaine, Caen, France
- Service de Psychiatrie de l’enfant et de l’adolescent, CHU de Caen, Caen, France
| | - J. M. Baleyte
- Normandie Université, UNICAEN, PSL Université Paris, EPHE, INSERM, U1077, CHU de Caen, GIP Cyceron, Neuropsychologie et Imagerie de la Mémoire Humaine, Caen, France
- Service de Psychiatrie de l’enfant et de l’adolescent, Centre Hospitalier Interuniversitaire de Créteil, Créteil, France
| | - F. Eustache
- Normandie Université, UNICAEN, PSL Université Paris, EPHE, INSERM, U1077, CHU de Caen, GIP Cyceron, Neuropsychologie et Imagerie de la Mémoire Humaine, Caen, France
| | - D. M. Bowler
- Autism Research Group, Department of Psychology, City, University of London, London, UK
| | - K. Lebreton
- Normandie Université, UNICAEN, PSL Université Paris, EPHE, INSERM, U1077, CHU de Caen, GIP Cyceron, Neuropsychologie et Imagerie de la Mémoire Humaine, Caen, France
| | - B. Guillery-Girard
- Normandie Université, UNICAEN, PSL Université Paris, EPHE, INSERM, U1077, CHU de Caen, GIP Cyceron, Neuropsychologie et Imagerie de la Mémoire Humaine, Caen, France
| |
Collapse
|
5
|
Nikolaev AR, Bramão I, Johansson R, Johansson M. Episodic memory formation in unrestricted viewing. Neuroimage 2023; 266:119821. [PMID: 36535321 DOI: 10.1016/j.neuroimage.2022.119821] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/22/2022] [Revised: 11/16/2022] [Accepted: 12/15/2022] [Indexed: 12/23/2022] Open
Abstract
The brain systems of episodic memory and oculomotor control are tightly linked, suggesting a crucial role of eye movements in memory. But little is known about the neural mechanisms of memory formation across eye movements in unrestricted viewing behavior. Here, we leverage simultaneous eye tracking and EEG recording to examine episodic memory formation in free viewing. Participants memorized multi-element events while their EEG and eye movements were concurrently recorded. Each event comprised elements from three categories (face, object, place), with two exemplars from each category, in different locations on the screen. A subsequent associative memory test assessed participants' memory for the between-category associations that specified each event. We used a deconvolution approach to overcome the problem of overlapping EEG responses to sequential saccades in free viewing. Brain activity was time-locked to the fixation onsets, and we examined EEG power in the theta and alpha frequency bands, the putative oscillatory correlates of episodic encoding mechanisms. Three modulations of fixation-related EEG predicted high subsequent memory performance: (1) theta increase at fixations after between-category gaze transitions, (2) theta and alpha increase at fixations after within-element gaze transitions, (3) alpha decrease at fixations after between-exemplar gaze transitions. Thus, event encoding with unrestricted viewing behavior was characterized by three neural mechanisms, manifested in fixation-locked theta and alpha EEG activity that rapidly turned on and off during the unfolding eye movement sequences. These three distinct neural mechanisms may be the essential building blocks that subserve the buildup of coherent episodic memories during unrestricted viewing behavior.
Collapse
Affiliation(s)
- Andrey R Nikolaev
- Department of Psychology, Lund Memory Lab, Lund University, Lund, Sweden; Brain and Cognition Research Unit, KU Leuven, Leuven, Belgium.
| | - Inês Bramão
- Department of Psychology, Lund Memory Lab, Lund University, Lund, Sweden
| | - Roger Johansson
- Department of Psychology, Lund Memory Lab, Lund University, Lund, Sweden
| | - Mikael Johansson
- Department of Psychology, Lund Memory Lab, Lund University, Lund, Sweden
| |
Collapse
|
6
|
Stapel J, El Hassnaoui M, Happee R. Measuring Driver Perception: Combining Eye-Tracking and Automated Road Scene Perception. HUMAN FACTORS 2022; 64:714-731. [PMID: 32993382 PMCID: PMC9136390 DOI: 10.1177/0018720820959958] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 05/14/2023]
Abstract
OBJECTIVE To investigate how well gaze behavior can indicate driver awareness of individual road users when related to the vehicle's road scene perception. BACKGROUND An appropriate method is required to identify how driver gaze reveals awareness of other road users. METHOD We developed a recognition-based method for labeling of driver situation awareness (SA) in a vehicle with road-scene perception and eye tracking. Thirteen drivers performed 91 left turns on complex urban intersections and identified images of encountered road users among distractor images. RESULTS Drivers fixated within 2° for 72.8% of relevant and 27.8% of irrelevant road users and were able to recognize 36.1% of the relevant and 19.4% of irrelevant road users one min after leaving the intersection. Gaze behavior could predict road user relevance but not the outcome of the recognition task. Unexpectedly, 18% of road users observed beyond 10° were recognized. CONCLUSIONS Despite suboptimal psychometric properties leading to low recognition rates, our recognition task could identify awareness of individual road users during left turn maneuvers. Perception occurred at gaze angles well beyond 2°, which means that fixation locations are insufficient for awareness monitoring. APPLICATION Findings can be used in driver attention and awareness modelling, and design of gaze-based driver support systems.
Collapse
Affiliation(s)
- Jork Stapel
- Delft University of Technology, Netherlands
- Jork Stapel, Faculty of Mechanical Maritime and Materials Engineering, Delft University of Technology, Mekelweg 2Delft, 2628 CD, Netherlands;
| | | | | |
Collapse
|
7
|
Mahanama B, Jayawardana Y, Rengarajan S, Jayawardena G, Chukoskie L, Snider J, Jayarathna S. Eye Movement and Pupil Measures: A Review. FRONTIERS IN COMPUTER SCIENCE 2022. [DOI: 10.3389/fcomp.2021.733531] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
Abstract
Our subjective visual experiences involve complex interaction between our eyes, our brain, and the surrounding world. It gives us the sense of sight, color, stereopsis, distance, pattern recognition, motor coordination, and more. The increasing ubiquity of gaze-aware technology brings with it the ability to track gaze and pupil measures with varying degrees of fidelity. With this in mind, a review that considers the various gaze measures becomes increasingly relevant, especially considering our ability to make sense of these signals given different spatio-temporal sampling capacities. In this paper, we selectively review prior work on eye movements and pupil measures. We first describe the main oculomotor events studied in the literature, and their characteristics exploited by different measures. Next, we review various eye movement and pupil measures from prior literature. Finally, we discuss our observations based on applications of these measures, the benefits and practical challenges involving these measures, and our recommendations on future eye-tracking research directions.
Collapse
|
8
|
Udale R, Tran MT, Manohar S, Husain M. Dynamic in-flight shifts of working memory resources across saccades. J Exp Psychol Hum Percept Perform 2022; 48:21-36. [PMID: 35073141 PMCID: PMC8785606 DOI: 10.1037/xhp0000960] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/23/2020] [Revised: 08/06/2021] [Accepted: 08/13/2021] [Indexed: 11/08/2022]
Abstract
Little is known about how memory resources are allocated in natural vision across sequential eye movements and fixations, as people actively extract information from the visual environment. Here, we used gaze-contingent eye tracking to examine how such resources are dynamically reallocated from old to new information entering working memory. As participants looked sequentially at items, we interrupted the process at different times by extinguishing the display as a saccade was initiated. After a brief interval, participants were probed on one of the items that had been presented. Paradoxically, across all experiments, the final (unfixated) saccade target was recalled more precisely when more items had previously been fixated, that is, with longer rather than shorter saccade sequences. This result is difficult to explain on current models of working memory because recall error, even for the final item, is typically higher as memory load increases. The findings could however be accounted for by a model that describes how resources are dynamically reallocated on a moment-by-moment basis. During each saccade, the target is encoded by consuming a proportion of currently available resources from a limited working memory, as well as by reallocating resources away from previously encoded items. These findings reveal how working memory resources are shifted across memoranda in active vision. (PsycInfo Database Record (c) 2022 APA, all rights reserved).
Collapse
Affiliation(s)
- Rob Udale
- Department of Psychology, University of Sheffield
| | - Moc Tram Tran
- Department of Experimental Psychology, University of Oxford
| | - Sanjay Manohar
- Department of Experimental Psychology, University of Oxford
| | - Masud Husain
- Department of Experimental Psychology, University of Oxford
| |
Collapse
|
9
|
Marois A, Charbonneau B, Szolosi AM, Watson JM. The Differential Impact of Mystery in Nature on Attention: An Oculometric Study. Front Psychol 2021; 12:759616. [PMID: 34955980 PMCID: PMC8696187 DOI: 10.3389/fpsyg.2021.759616] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/16/2021] [Accepted: 11/22/2021] [Indexed: 12/16/2022] Open
Abstract
Nature exposure can provide benefits on stress, health and cognitive performance. According to Attention Restoration Theory (ART), the positive impact of nature on cognition is mainly driven by fascination. Fascinating properties of nature such as water or a winding hiking trail may capture involuntary attention, allowing the directed form of attention to rest and to recover. This claim has been supported by studies relying on eye-tracking measures of attention deployment, comparing exposure to urban and nature settings. Yet, recent studies have shown that promoting higher engagement with a nature setting can improve restorative benefits, hence challenging ART's view that voluntary attention is resting. Besides, recent evidence published by Szolosi et al. (2014) suggests that voluntary attention may be involved during exposure to high-mystery nature images which they showed as having greater potential for attention restoration. The current study explored how exposure to nature images of different scenic qualities in mystery (and restoration potential) could impact the engagement of attention. To do so, participants were shown nature images characterized by either low or high mystery properties (with allegedly low or high restoration potential, respectively) and were asked to evaluate their fascination and aesthetic levels. Concurrently, an eye tracker collected measures of pupil size, fixations and spontaneous blinks as indices of attentional engagement. Results showed that high-mystery nature images had higher engagement than low-mystery images as supported by the larger pupil dilations, the higher number of fixations and the reduced number of blinks and durations of fixations. Taken together, these results challenge ART's view that directed attention is merely resting during exposure to restorative nature and offer new hypotheses on potential mechanisms underlying attention restoration.
Collapse
Affiliation(s)
- Alexandre Marois
- École de Psychologie, Université Laval, Quebec City, QC, Canada
- Thales Research and Technology Canada, Quebec City, QC, Canada
| | - Brooke Charbonneau
- Department of Psychology, University of Colorado Denver, Denver, CO, United States
- Department of Psychology, Montana State University, Bozeman, MT, United States
| | - Andrew M. Szolosi
- Department of Recreation, Sport Pedagogy, and Consumer Sciences, Ohio University, Athens, OH, United States
| | - Jason M. Watson
- Department of Psychology, University of Colorado Denver, Denver, CO, United States
| |
Collapse
|
10
|
Meghanathan RN, van Leeuwen C, Giannini M, Nikolaev AR. Neural correlates of task-related refixation behavior. Vision Res 2020; 175:90-101. [PMID: 32795708 DOI: 10.1016/j.visres.2020.07.001] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/19/2019] [Revised: 06/28/2020] [Accepted: 07/03/2020] [Indexed: 10/23/2022]
Abstract
Eye movement research has shown that attention shifts from the currently fixated location to the next before a saccade is executed. We investigated whether the cost of the attention shift depends on higher-order processing at the time of fixation, in particular on visual working memory load differences between fixations and refixations on task-relevant items. The attention shift is reflected in EEG activity in the saccade-related potential (SRP). In a free viewing task involving visual search and memorization of multiple targets amongst distractors, we compared the SRP in first fixations versus refixations on targets and distractors. The task-relevance of targets implies that more information will be loaded in memory (e.g. both identity and location) than for distractors (e.g. location only). First fixations will involve greater memory load than refixations, since first fixations involve loading of new items, while refixations involve rehearsal of previously visited items. The SRP in the interval preceding the saccade away from a target or distractor revealed that saccade preparation is affected by task-relevance and refixation behavior. For task-relevant items only, we found longer fixation duration and higher SRP amplitudes for first fixations than for refixations over the occipital region and the opposite effect over the frontal region. Our findings provide first neurophysiological evidence that working memory loading of task-relevant information at fixation affects saccade planning.
Collapse
Affiliation(s)
- Radha Nila Meghanathan
- Laboratory for Perceptual Dynamics, Brain & Cognition Research Unit, KU Leuven - University of Leuven, Leuven, Belgium; Center for Cognitive Science, University of Kaiserslautern, Kaiserslautern, Germany.
| | - Cees van Leeuwen
- Laboratory for Perceptual Dynamics, Brain & Cognition Research Unit, KU Leuven - University of Leuven, Leuven, Belgium; Center for Cognitive Science, University of Kaiserslautern, Kaiserslautern, Germany
| | - Marcello Giannini
- Laboratory for Perceptual Dynamics, Brain & Cognition Research Unit, KU Leuven - University of Leuven, Leuven, Belgium
| | - Andrey R Nikolaev
- Laboratory for Perceptual Dynamics, Brain & Cognition Research Unit, KU Leuven - University of Leuven, Leuven, Belgium; Department of Psychology, Lund University, Lund, Sweden
| |
Collapse
|
11
|
Exploring the Cognitive Load of Expert and Novice Map Users Using EEG and Eye Tracking. ISPRS INTERNATIONAL JOURNAL OF GEO-INFORMATION 2020. [DOI: 10.3390/ijgi9070429] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/23/2022]
Abstract
The main objective of this research is to explore the cognitive processes of expert and novice map users during the retrieval of map-related information, within varying difficulty levels (i.e., easy, moderate, hard), by using eye tracking and electroencephalogram (EEG). In this context, we present a spatial memory experiment consisting of a large number of stimuli to study the effect of task difficulty on map users’ behavior through cognitive load measurements. Next to the reaction time and success rate, we used fixation and saccade related eye tracking metrics (i.e., average fixation duration, the number of fixations per second, saccade amplitude and saccade velocity), and EEG power spectrum (i.e., event-related changes in alpha and theta frequency bands) to identify the cognitive load. While fixation metrics indicated no statistically significant difference between experts and novices, saccade metrics proved the otherwise. EEG power spectral density analysis, on the other side, suggested an increase in theta (i.e., event-related synchronization) and a decrease in alpha (except moderate tasks) (i.e., event-related desynchronization) at all difficulty levels of the task for both experts and novices, which is an indicator of cognitive load. Although no significant difference emerged between two groups, we found a significant difference in their overall performances when the participants were classified as good and relatively bad learners. Triangulating EEG results with the recorded eye tracking data and the qualitative analysis of focus maps indeed provided a detailed insight on the differences of the individuals’ cognitive processes during this spatial memory task.
Collapse
|