Internal Coupling: Eye Behavior Coupled to Visual Imagery

Our eyes do not only respond to visual perception but also to internal cognition involving visual imagery, which can be referred to as internal coupling. This review synthesizes evidence on internal coupling across diverse domains including episodic memory and simulation, visuospatial memory, numerical cognition, object movement, body movement, and brightness imagery. In each domain, eye movements consistently reflect distinct aspects of mental imagery typically akin to those seen in corresponding visual experiences. Several findings further suggest that internal coupling may not only coincide with but also supports internal cognition as evidenced by improved cognitive performance. Available theoretical accounts suggest that internal coupling may serve at least two functional roles in visual imagery: facilitating memory reconstruction and indicating shifts in internal attention. Moreover, recent insights into the neurobiology of internal coupling highlight substantially shared neural pathways in externally and internally directed cognition. The review concludes by identifying open questions and promising avenues for future research such as exploring moderating roles of context and individual differences in internal coupling.

How workload and availability of spatial reference shape eye movement coupling in visuospatial working memory

Eyes are active in memory recall and visual imagination, yet our grasp of the underlying qualities and factors of these internally coupled eye movements is limited. To explore this, we studied 50 participants, examining how workload, spatial reference availability, and imagined movement direction influence internal coupling of eye movements. We designed a visuospatial working memory task in which participants mentally moved a black patch along a path within a matrix and each trial involved one step along this path (presented via speakers: up, down, left, or right). We varied workload by adjusting matrix size (3 × 3 vs. 5 × 5), manipulated availability of a spatial frame of reference by presenting either a blank screen (requiring participants to rely solely on their mental representation of the matrix) or spatial reference in the form of an empty matrix, and contrasted active task performance to two control conditions involving only active or passive listening. Our findings show that eye movements consistently matched the imagined movement of the patch in the matrix, not driven solely by auditory or semantic cues. While workload influenced pupil diameter, perceived demand, and performance, it had no observable impact on internal coupling. The availability of spatial reference enhanced coupling of eye movements, leading more frequent, precise, and resilient saccades against noise and bias. The absence of workload effects on coupled saccades in our study, in combination with the relatively high degree of coupling observed even in the invisible matrix condition, indicates that eye movements align with shifts in attention across both visually and internally represented information. This suggests that coupled eye movements are not merely strategic efforts to reduce workload, but rather a natural response to where attention is directed.

Eye movements reinstate remembered locations during episodic simulation

Imagining the future, like recalling the past, relies on the ability to retrieve and imagine a spatial context. Research suggests that eye movements support this process by reactivating spatial contextual details from memory, a process termed gaze reinstatement. While gaze reinstatement has been linked to successful memory retrieval, it remains unclear whether it supports the related process of future simulation. In the present study, we recorded both eye movements and audio while participants described familiar locations from memory and subsequently imagined future events occurring in those locations while either freely moving their eyes or maintaining central fixation. Restricting viewing during simulation significantly reduced self-reported vividness ratings, supporting a critical role for eye movements in simulation. When viewing was unrestricted, participants spontaneously reinstated gaze patterns specific to the simulated location, replicating findings of gaze reinstatement during memory retrieval. Finally, gaze-based location reinstatement was predictive of simulation success, indexed by the number of internal (episodic) details produced, with both measures peaking early and co-varying over time. Together, these findings suggest that the same oculomotor processes that support episodic memory retrieval - that is, gaze-based reinstatement of spatial context - also support episodic simulation.

Pupil Size and Eye Movements Differently Index Effort in Both Younger and Older Adults

The assessment of mental effort is increasingly relevant in neurocognitive and life span domains. Pupillometry, the measure of the pupil size, is often used to assess effort but has disadvantages. Analysis of eye movements may provide an alternative, but research has been limited to easy and difficult task demands in younger adults. An effort measure must be sensitive to the whole effort profile, including "giving up" effort investment, and capture effort in different age groups. The current study comprised three experiments in which younger (n = 66) and older (n = 44) adults listened to speech masked by background babble at different signal-to-noise ratios associated with easy, difficult, and impossible speech comprehension. We expected individuals to invest little effort for easy and impossible speech (giving up) but to exert effort for difficult speech. Indeed, pupil size was largest for difficult but lower for easy and impossible speech. In contrast, gaze dispersion decreased with increasing speech masking in both age groups. Critically, gaze dispersion during difficult speech returned to levels similar to easy speech after sentence offset, when acoustic stimulation was similar across conditions, whereas gaze dispersion during impossible speech continued to be reduced. These findings show that a reduction in eye movements is not a byproduct of acoustic factors, but instead suggest that neurocognitive processes, different from arousal-related systems regulating the pupil size, drive reduced eye movements during high task demands. The current data thus show that effort in one sensory domain (audition) differentially impacts distinct functional properties in another sensory domain (vision).

Advancing Knowledge on Situation Comprehension in Dynamic Traffic Situations by Studying Eye Movements to Empty Spatial Locations

OBJECTIVE

This study used the looking-at-nothing phenomenon to explore situation awareness (SA) and the effects of working memory (WM) load in driving situations.

BACKGROUND

While driving, people develop a mental representation of the environment. Since errors in retrieving information from this representation can have fatal consequences, it is essential for road safety to investigate this process. During retrieval, people tend to fixate spatial positions of visually encoded information, even if it is no longer available at that location. Previous research has shown that this "looking-at-nothing" behavior can be used to trace retrieval processes.

METHOD

In a video-based laboratory experiment with 2 (WM) x 3 (SA level) within-subjects design, participants (N = 33) viewed a reduced screen and evaluated auditory statements relating to different SA levels on previously seen dynamic traffic scenarios while eye movements were recorded.

RESULTS

When retrieving information, subjects more frequently fixated emptied spatial locations associated with the information relevant for the probed SA level. The retrieval of anticipations (SA level 3) in contrast to the other SA level information resulted in more frequent gaze transitions that corresponded to the spatial dynamics of future driving behavior.

CONCLUSION

The results support the idea that people build a visual-spatial mental image of a driving situation. Different gaze patterns when retrieving level-specific information indicate divergent retrieval processes.

APPLICATION

Potential applications include developing new methodologies to assess the mental representation and SA of drivers objectively.

Gaze Restriction and Reactivation of Place-bound Content Drive Eye Movements During Mental Imagery

When we imagine a picture, we move our eyes even though the picture is physically not present. These eye movements provide information about the ongoing process of mental imagery. Eye movements unfold over time, and previous research has shown that the temporal gaze dynamics of eye movements in mental imagery have unique properties, which are unrelated to those in perception. In mental imagery, refixations of previously fixated locations happen more often and in a more systematic manner than in perception. The origin of these unique properties remains unclear. We tested how the temporal structure of eye movements is influenced by the complexity of the mental image. Participants briefly saw and then maintained a pattern stimulus, consisting of one (easy condition) to four black segments (most difficult condition). When maintaining a simple pattern in imagery, participants restricted their gaze to a narrow area, and for more complex stimuli, eye movements were more spread out to distant areas. At the same time, fewer refixations were made in imagery when the stimuli were complex. The results show that refixations depend on the imagined content. While fixations of stimulus-related areas reflect the so-called 'looking at nothing' effect, gaze restriction emphasizes differences between mental imagery and perception.

Eye Movements Decrease during Effortful Speech Listening

Hearing impairment affects many older adults but is often diagnosed decades after speech comprehension in noisy situations has become effortful. Accurate assessment of listening effort may thus help diagnose hearing impairment earlier. However, pupillometry-the most used approach to assess listening effort-has limitations that hinder its use in practice. The current study explores a novel way to assess listening effort through eye movements. Building on cognitive and neurophysiological work, we examine the hypothesis that eye movements decrease when speech listening becomes challenging. In three experiments with human participants from both sexes, we demonstrate, consistent with this hypothesis, that fixation duration increases and spatial gaze dispersion decreases with increasing speech masking. Eye movements decreased during effortful speech listening for different visual scenes (free viewing, object tracking) and speech materials (simple sentences, naturalistic stories). In contrast, pupillometry was less sensitive to speech masking during story listening, suggesting pupillometric measures may not be as effective for the assessments of listening effort in naturalistic speech-listening paradigms. Our results reveal a critical link between eye movements and cognitive load, suggesting that neural activity in the brain regions that support the regulation of eye movements, such as frontal eye field and superior colliculus, are modulated when listening is effortful.SIGNIFICANCE STATEMENT Assessment of listening effort is critical for early diagnosis of age-related hearing loss. Pupillometry is most used but has several disadvantages. The current study explores a novel way to assess listening effort through eye movements. We examine the hypothesis that eye movements decrease when speech listening becomes effortful. We demonstrate, consistent with this hypothesis, that fixation duration increases and gaze dispersion decreases with increasing speech masking. Eye movements decreased during effortful speech listening for different visual scenes (free viewing, object tracking) and speech materials (sentences, naturalistic stories). Our results reveal a critical link between eye movements and cognitive load, suggesting that neural activity in brain regions that support the regulation of eye movements are modulated when listening is effortful.

Cortico-ocular Coupling in the Service of Episodic Memory Formation

Encoding of visual information is a necessary requirement for most types of episodic memories. In search for a neural signature of memory formation, amplitude modulation of neural activity has been repeatedly shown to correlate with and suggested to be functionally involved in successful memory encoding. We here report a complementary view on why and how brain activity relates to memory, indicating a functional role of cortico-ocular interactions for episodic memory formation. Recording simultaneous magnetoencephalography and eye tracking in 35 human participants, we demonstrate that gaze variability and amplitude modulations of alpha/beta oscillations (10-20Hz) in visual cortex covary and predict subsequent memory performance between and within participants. Amplitude variation during pre-stimulus baseline was associated with gaze direction variability, echoing the co-variation observed during scene encoding. We conclude that encoding of visual information engages unison coupling between oculomotor and visual areas in the service of memory formation.

A functional role for oculomotor preparation in mental arithmetic evidenced by the abducted eye paradigm

Solving subtraction and addition problems is accompanied by spontaneous leftward and rightward gaze shifts, respectively. These shifts have been related to attentional processes involved in mental arithmetic, but whether these processes induce overt attentional shifts mediated by the activation of the motor programs underlying lateral eye movements or covert shifts only is still unknown. Here, we used the abducted eye paradigm to selectively disrupt activation of the oculomotor system and prevent oculomotor preparation, which affects overt but not covert attentional shifts. Participants had to mentally solve addition and subtraction problems while fixating a screen positioned either in front of them or laterally to their left or right such that they were physically unable to programme and execute saccades further into their temporal field while they still could do so in their nasal field. In comparison to the frontal condition, rightward eye abduction impaired additions (with carrying), and leftward eye abduction impaired subtractions (with borrowing) showing that at least some arithmetic problems rely on processes dedicated to overt attentional shifts. We propose that when solving arithmetic problems requires procedures such as carrying and borrowing, oculomotor mechanisms operating on a mental space transiently built in working memory are recruited to represent one numerical magnitude in relation to another (e.g. the first operand and the result).

"Looking at nothing": An implicit ocular motor index of face recognition in developmental prosopagnosia

Subjects often look towards to previous location of a stimulus related to a task even when that stimulus is no longer visible. In this study we asked whether this effect would be preserved or reduced in subjects with developmental prosopagnosia. Participants learned faces presented in video-clips and then saw a brief montage of four faces, which was replaced by a screen with empty boxes, at which time they indicated whether the learned face had been present in the montage. Control subjects were more likely to look at the blank location where the learned face had appeared, on both hit and miss trials, though the effect was larger on hit trials. Prosopagnosic subjects showed a reduced effect, though still better on hit than on miss trials. We conclude that explicit accuracy and our implicit looking at nothing effect are parallel effects reflecting the strength of the neural activity underlying face recognition.

Imagery-related eye movements in 3D space depend on individual differences in visual object imagery

During recall of visual information people tend to move their eyes even though there is nothing to see. Previous studies indicated that such eye movements are related to the spatial location of previously seen items on 2D screens, but they also showed that eye movement behavior varies significantly across individuals. The reason for these differences remains unclear. In the present study we used immersive virtual reality to investigate how individual tendencies to process and represent visual information contribute to eye fixation patterns in visual imagery of previously inspected objects in three-dimensional (3D) space. We show that participants also look back to relevant locations when they are free to move in 3D space. Furthermore, we found that looking back to relevant locations depends on individual differences in visual object imagery abilities. We suggest that object visualizers rely less on spatial information because they tend to process and represent the visual information in terms of color and shape rather than in terms of spatial layout. This finding indicates that eye movements during imagery are subject to individual strategies, and the immersive setting in 3D space made individual differences more likely to unfold.

Unrestricted eye movements strengthen effective connectivity from hippocampal to oculomotor regions during scene construction

Scene construction is a key component of memory recall, navigation, and future imagining, and relies on the medial temporal lobes (MTL). A parallel body of work suggests that eye movements may enable the imagination and construction of scenes, even in the absence of external visual input. There are vast structural and functional connections between regions of the MTL and those of the oculomotor system. However, the directionality of connections between the MTL and oculomotor control regions, and how it relates to scene construction, has not been studied directly in human neuroimaging. In the current study, we used dynamic causal modeling (DCM) to interrogate effective connectivity between the MTL and oculomotor regions using a scene construction task in which participants' eye movements were either restricted (fixed-viewing) or unrestricted (free-viewing). By omitting external visual input, and by contrasting free- versus fixed- viewing, the directionality of neural connectivity during scene construction could be determined. As opposed to when eye movements were restricted, allowing free-viewing during construction of scenes strengthened top-down connections from the MTL to the frontal eye fields, and to lower-level cortical visual processing regions, suppressed bottom-up connections along the visual stream, and enhanced vividness of the constructed scenes. Taken together, these findings provide novel, non-invasive evidence for the underlying, directional, connectivity between the MTL memory system and oculomotor system associated with constructing vivid mental representations of scenes.

Eye-movement replay supports episodic remembering

When we bring to mind something we have seen before, our eyes spontaneously unfold in a sequential pattern strikingly similar to that made during the original encounter, even in the absence of supporting visual input. Oculomotor movements of the eye may then serve the opposite purpose of acquiring new visual information; they may serve as self-generated cues, pointing to stored memories. Over 50 years ago Donald Hebb, the forefather of cognitive neuroscience, posited that such a sequential replay of eye movements supports our ability to mentally recreate visuospatial relations during episodic remembering. However, direct evidence for this influential claim is lacking. Here we isolate the sequential properties of spontaneous eye movements during encoding and retrieval in a pure recall memory task and capture their encoding-retrieval overlap. Critically, we show that the fidelity with which a series of consecutive eye movements from initial encoding is sequentially retained during subsequent retrieval predicts the quality of the recalled memory. Our findings provide direct evidence that such scanpaths are replayed to assemble and reconstruct spatio-temporal relations as we remember and further suggest that distinct scanpath properties differentially contribute depending on the nature of the goal-relevant memory.

Schema-related eye movements support episodic simulation

Recent work indicates that eye movements support the retrieval of episodic memories by reactivating the spatiotemporal context in which they were encoded. Although similar mechanisms have been thought to support simulation of future episodes, there is currently no evidence favoring this proposal. In the present study, we investigated the role of eye movements in episodic simulation by comparing the gaze patterns of individual participants imagining future scene and event scenarios to across-participant gaze templates for those same scenarios, reflecting their shared features (i.e., schemas). Our results provide novel evidence that eye movements during episodic simulation in the face of distracting visual noise are (1) schema-specific and (2) predictive of simulation success. Together, these findings suggest that eye movements support episodic simulation via reinstatement of scene and event schemas, and more broadly, that interactions between the memory and oculomotor effector systems may underlie critical cognitive processes including constructive episodic simulation.

A consensus-based elastic matching algorithm for mapping recall fixations onto encoding fixations in the looking-at-nothing paradigm

We present an algorithmic method for aligning recall fixations with encoding fixations, to be used in looking-at-nothing paradigms that either record recall eye movements during silence or want to speed up data analysis with recordings of recall data during speech. The algorithm utilizes a novel consensus-based elastic matching algorithm to estimate which encoding fixations correspond to later recall fixations. This is not a scanpath comparison method, as fixation sequence order is ignored and only position configurations are used. The algorithm has three internal parameters and is reasonable stable over a wide range of parameter values. We then evaluate the performance of our algorithm by investigating whether the recalled objects identified by the algorithm correspond with independent assessments of what objects in the image are marked as subjectively important. Our results show that the mapped recall fixations align well with important regions of the images. This result is exemplified in four groups of use cases: to investigate the roles of low-level visual features, faces, signs and text, and people of different sizes, in recall of encoded scenes. The plots from these examples corroborate the finding that the algorithm aligns recall fixations with the most likely important regions in the images. Examples also illustrate how the algorithm can differentiate between image objects that have been fixated during silent recall vs those objects that have not been visually attended, even though they were fixated during encoding.

Pictorial low-level features in mental images: evidence from eye fixations

It is known that eye movements during object imagery reflect areas visited during encoding. But will eye movements also reflect pictorial low-level features of imagined stimuli? In this paper, three experiments are reported in which we investigate whether low-level properties of mental images elicit specific eye movements. Based on the conceptualization of mental images as depictive representations, we expected low-level visual features to influence eye fixations during mental imagery, in the absence of a visual input. In a first experiment, twenty-five participants performed a visual imagery task with high vs. low spatial frequency and high vs. low contrast gratings. We found that both during visual perception and during mental imagery, first fixations were more often allocated to the low spatial frequency-high contrast grating, thus showing that eye fixations were influenced not only by physical properties of visual stimuli but also by its imagined counterpart. In a second experiment, twenty-two participants imagined high contrast and low contrast stimuli that they had not encoded before. Again, participants allocated more fixations to the high contrast mental images than to the low contrast mental images. In a third experiment, we ruled out task difficulty as confounding variable. Our results reveal that low-level visual features are represented in the mind's eye and thus, they contribute to the characterization of mental images in terms of how much perceptual information is re-instantiated during mental imagery.

Eye movements during visual imagery and perception show spatial correspondence but have unique temporal signatures

Eye fixation patterns during mental imagery are similar to those during perception of the same picture, suggesting that oculomotor mechanisms play a role in mental imagery (i.e., the "looking at nothing" effect). Previous research has focused on the spatial similarities of eye movements during perception and mental imagery. The primary aim of this study was to assess whether the spatial similarity translates to the temporal domain. We used recurrence quantification analysis (RQA) to assess the temporal structure of eye fixations in visual perception and mental imagery and we compared the temporal as well as the spatial characteristics in mental imagery with perception by means of Bayesian hierarchical regression models. We further investigated how person and picture-specific characteristics contribute to eye movement behavior in mental imagery. Working memory capacity and mental imagery abilities were assessed to either predict gaze dynamics in visual imagery or to moderate a possible correspondence between spatial or temporal gaze dynamics in perception and mental imagery. We were able to show the spatial similarity of fixations between visual perception and imagery and we provide first evidence for its moderation by working memory capacity. Interestingly, the temporal gaze dynamics in mental imagery were unrelated to those in perception and their variance between participants was not explained by variance in visuo-spatial working memory capacity or vividness of mental images. The semantic content of the imagined pictures was the only meaningful predictor of temporal gaze dynamics. The spatial correspondence reflects shared spatial structure of mental images and perceived pictures, while the unique temporal gaze behavior could be driven by generation, maintenance and protection processes specific to visual imagery. The unique temporal gaze dynamics offer a window to new insights into the genuine process of mental imagery independent of its similarity to perception.

The Closer, The Better? Processing Relations Between Picture Elements in Historical Paintings

The present eye-tracking study investigated how audio explanations influence perception and the cognitive processing of historical paintings. Spatially close and distant pairs of picture elements and their semantic relations were named in an audio text either immediately after each other or with descriptions of other elements in between. It was assumed that the number of backward fixation counts on the first of the two mentioned related picture elements should be higher if they are spatially close rather than spatially distant. There should also be more backward fixation counts if the elements are named temporally close rather than temporally distant. Similar predictions were made for the retention of these picture elements and their relations. A 2x2x2 within-subject design (n=36) with spatial distance (close vs. distant), temporal distance (close vs. distant) and painting (Leutze vs. West) revealed more background fixation counts for spatially close compared to spatially distant elements but just for the Leutze painting. Accordingly, the relations between the spatially close pairs were retained better than between the spatially distant pairs in the Leutze painting but vice versa for the West painting. The results are discussed with regard to the spatial contiguity principle of multimedia learning and research on text coherence.

The link between detail generation and eye movements when encoding and retrieving complex images

Examining eye movement patterns when encoding and retrieving visually complex memories is useful to understand the link between visuo-perceptual processes and how associated details are represented within these memories. Here, we used images of real-world scenes (e.g., a couple grocery shopping) to examine how encoding and retrieval eye movements are linked to the details used to describe complex images during these two phases of memory. Given that memories are often elaborated upon during retrieval, we also examined whether eye-movements at retrieval related to details that were the same as those described when encoding the image (reinstated details) as well as details about the image event that were not initially described at encoding (newly generated details). Testing young healthy participants, we found that retrieval eye movements, specifically eye fixation rate, predicted reinstated details, but not newly generated details. This suggests that visuo-perceptual processes are preferentially engaged at retrieval to reactivate perceived information. At encoding, we found a relationship between eye movements and detail generation that changed over time. This relationship was positive early on in the encoding phase but changed to a negative relationship later in the phase, indicating that a unique relationship exists between activating visuo-perceptual processes during early encoding versus late encoding. Overall, our results provide new insights into how visuo-perceptual processes contribute to different components of complex memory.

Do eye movements enhance visual memory retrieval?

When remembering an object at a given location, participants tend to return their gaze to that location even after the object has disappeared, known as Looking-at-Nothing (LAN). However, it is unclear whether LAN is associated with better memory performance. Previous studies reporting beneficial effects of LAN have often not systematically manipulated or assessed eye movements. We asked 20 participants to remember the location and identity of eight objects arranged in a circle, shown for 5 s. Participants were prompted to judge whether a location statement (e.g., "Star Right") was correct or incorrect, or referred to a previously unseen object. During memory retrieval, participants either fixated in the screen center or were free to move their eyes. Results reveal no difference in memory accuracy and response time between free-viewing and fixation while a LAN effect was found for saccades during free viewing, but not for microsaccades during fixation. Memory performance was better in those free-viewing trials in which participants made a saccade to the critical location, and scaled with saccade accuracy. These results indicate that saccade kinematics might be related to both memory performance and memory retrieval processes, but the strength of their link would differ between individuals and task demands.

Computational discrimination between natural images based on gaze during mental imagery

When retrieving image from memory, humans usually move their eyes spontaneously as if the image were in front of them. Such eye movements correlate strongly with the spatial layout of the recalled image content and function as memory cues facilitating the retrieval procedure. However, how close the correlation is between imagery eye movements and the eye movements while looking at the original image is unclear so far. In this work we first quantify the similarity of eye movements between recalling an image and encoding the same image, followed by the investigation on whether comparing such pairs of eye movements can be used for computational image retrieval. Our results show that computational image retrieval based on eye movements during spontaneous imagery is feasible. Furthermore, we show that such a retrieval approach can be generalized to unseen images.

Eye Movement Reinstatement and Neural Reactivation During Mental Imagery

Half a century ago, Donald Hebb posited that mental imagery is a constructive process that emulates perception. Specifically, Hebb claimed that visual imagery results from the reactivation of neural activity associated with viewing images. He also argued that neural reactivation and imagery benefit from the re-enactment of eye movement patterns that first occurred at viewing (fixation reinstatement). To investigate these claims, we applied multivariate pattern analyses to functional MRI (fMRI) and eye tracking data collected while healthy human participants repeatedly viewed and visualized complex images. We observed that the specificity of neural reactivation correlated positively with vivid imagery and with memory for stimulus image details. Moreover, neural reactivation correlated positively with fixation reinstatement, meaning that image-specific eye movements accompanied image-specific patterns of brain activity during visualization. These findings support the conception of mental imagery as a simulation of perception, and provide evidence consistent with the supportive role of eye movement in neural reactivation.

Titta: A toolbox for creating PsychToolbox and Psychopy experiments with Tobii eye trackers

We present Titta, an open-source toolbox for controlling eye trackers manufactured by Tobii AB from MATLAB and Python. The toolbox provides a wrapper around the Tobii Pro SDK, providing a convenient graphical participant setup, calibration and validation interface implemented using the PsychToolbox and PsychoPy toolboxes. The toolbox furthermore enables MATLAB and Python experiments to communicate with Tobii Pro Lab through the TalkToProLab tool. This enables experiments to be created and run using the freedom of MATLAB and Python, while the recording can be visualized and analyzed in Tobii Pro Lab. All screen-mounted Tobii eye trackers that are supported by the Tobii Pro SDK are also supported by Titta. At the time of writing, these are the Spectrum, Nano, TX300, T60XL, X3-120, X2-60, X2-30, X60, X120, T60 and T120 from Tobii Pro, and the 4C from Tobii Tech.

Spatial language difficulties reflect the structure of intact spatial representation: Evidence from high-functioning autism

Previous studies have shown that the basic properties of the visual representation of space are reflected in spatial language. This close relationship between linguistic and non-linguistic spatial systems has been observed both in typical development and in some developmental disorders. Here we provide novel evidence for structural parallels along with a degree of autonomy between these two systems among individuals with Autism Spectrum Disorder, a developmental disorder with uneven cognitive and linguistic profiles. In four experiments, we investigated language and memory for locations organized around an axis-based reference system. Crucially, we also recorded participants' eye movements during the tasks in order to provide new insights into the online processes underlying spatial thinking. Twenty-three intellectually high-functioning individuals with autism (HFA) and 23 typically developing controls (TD), all native speakers of Norwegian matched on chronological age and cognitive abilities, participated in the studies. The results revealed a well-preserved axial reference system in HFA and weakness in the representation of direction within the axis, which was especially evident in spatial language. Performance on the non-linguistic tasks did not differ between HFA and control participants, and we observed clear structural parallels between spatial language and spatial representation in both groups. However, there were some subtle differences in the use of spatial language in HFA compared to TD, suggesting that despite the structural parallels, some aspects of spatial language in HFA deviated from the typical pattern. These findings provide novel insights into the prominence of the axial reference systems in non-linguistic spatial representations and spatial language, as well as the possibility that the two systems are, to some degree, autonomous.

The function of "looking-at-nothing" for sequential sensorimotor tasks: Eye movements to remembered action-target locations

When performing manual actions, eye movements precede hand movements to target locations: Before we grasp an object, we look at it. Eye-hand guidance is even preserved when visual targets are unavailable, e.g., grasping behind an occlusion. This "looking-atnothing" behavior might be functional, e.g., as "deictic pointer" for manual control or as memory-retrieval cue, or a by-product of automatization. Here, it is studied if looking at empty locations before acting on them is beneficial for sensorimotor performance. In five experiments, participants completed a click sequence on eight visual targets for 0-100 trials while they had either to fixate on the screen center or could move their eyes freely. During 50-100 consecutive trials, participants clicked the same sequence on a blank screen with free or fixed gaze. During both phases, participants looked at target locations when gaze shifts were allowed. With visual targets, target fixations led to faster, more precise clicking, fewer errors, and sparser cursor-paths than central fixation. Without visual information, a tiny free-gaze benefit could sometimes be observed and was rather a memory than a motor-calculation benefit. Interestingly, central fixation during learning forced early explicit encoding causing a strong benefit for acting on remembered targets later, independent of whether eyes moved then.

Eye Movements Actively Reinstate Spatiotemporal Mnemonic Content

Eye movements support memory encoding by binding distinct elements of the visual world into coherent representations. However, the role of eye movements in memory retrieval is less clear. We propose that eye movements play a functional role in retrieval by reinstating the encoding context. By overtly shifting attention in a manner that broadly recapitulates the spatial locations and temporal order of encoded content, eye movements facilitate access to, and reactivation of, associated details. Such mnemonic gaze reinstatement may be obligatorily recruited when task demands exceed cognitive resources, as is often observed in older adults. We review research linking gaze reinstatement to retrieval, describe the neural integration between the oculomotor and memory systems, and discuss implications for models of oculomotor control, memory, and aging.

Spontaneous eye movements during focused-attention mindfulness meditation

Oculometric measures have been proven to be useful markers of mind-wandering during visual tasks such as reading. However, little is known about ocular activity during mindfulness meditation, a mental practice naturally involving mind-wandering episodes. In order to explore this issue, we extracted closed-eyes ocular movement measurements via a covert technique (EEG recordings) from expert meditators during two repetitions of a 7-minute mindfulness meditation session, focusing on the breath, and two repetitions of a 7-minute instructed mind-wandering task. Power spectral density was estimated on both the vertical and horizontal components of eye movements. The results show a significantly smaller average amplitude of eye movements in the delta band (1–4 Hz) during mindfulness meditation than instructed mind-wandering. Moreover, participants’ meditation expertise correlated significantly with this average amplitude during both tasks, with more experienced meditators generally moving their eyes less than less experienced meditators. These findings suggest the potential use of this measure to detect mind-wandering episodes during mindfulness meditation and to assess meditation performance.

Recurrence quantification analysis of eye movements during mental imagery

Several studies demonstrated similarities of eye fixations during mental imagery and visual perception but-to our knowledge-the temporal characteristics of eye movements during imagery have not yet been considered in detail. To fill this gap, the same data is analyzed with conventional spatial techniques such as analysis of areas of interest (AOI), ScanMatch, and MultiMatch and with recurrence quantification analysis (RQA), a new way of analyzing gaze data by tracking re-fixations and their temporal dynamics. Participants viewed and afterwards imagined three different kinds of pictures (art, faces, and landscapes) while their eye movements were recorded. While fixation locations during imagery were related to those during perception, participants returned more often to areas they had previously looked at during imagery and their scan paths were more clustered and more repetitive when compared to visual perception. Furthermore, refixations of the same area occurred sooner after initial fixation during mental imagery. The results highlight not only content-driven spatial similarities between imagery and perception but also shed light on the processes of mental imagery maintenance and interindividual differences in these processes.

Less imageable words lead to more looks to blank locations during memory retrieval

People revisit spatial locations of visually encoded information when they are asked to retrieve that information, even when the visual image is no longer present. Such “looking at nothing” during retrieval is likely modulated by memory load (i.e., mental effort to maintain and reconstruct information) and the strength of mental representations. We investigated whether words that are more difficult to remember also lead to more looks to relevant, blank locations. Participants were presented four nouns on a two by two grid. A number of lexico-semantic variables were controlled to form high-difficulty and low-difficulty noun sets. Results reveal more frequent looks to blank locations during retrieval of high-difficulty nouns compared to low-difficulty ones. Mixed-effects modelling demonstrates that imagery-related semantic factors (imageability and concreteness) predict looking at nothing during retrieval. Results provide the first direct evidence that looking at nothing is modulated by word difficulty and in particular, word imageability. Overall, the research provides substantial support to the integrated memory account for linguistic stimuli and looking at nothing as a form of mental imagery.

Mental Size Scaling of Three-Dimensional Objects Perceived Visually or Tactilely

The issue of pictorial or propositional nature of imagery remains unresolved. To take a step forward into the debate, we conducted a systematic evaluation of time and accuracy of mental scaling in sighted people. Participants viewed or touched three-dimensional objects and then had to imagine them in a resized version, depending on a given scale. Both the mental scaling time and the estimated object size were measured. To promote verbal or perceptual strategies, the size was estimated verbally or bimanually, respectively. It was found that time taken for mental scaling is a linear function of decreasing and increasing scale and that the modality of perception did not influence the time taken to perform the operation. The results contribute to the knowledge of object size estimation by revealing the interaction between the modality of the object perception and the accuracy of size estimation by sighted adults. The accuracy of estimation was greater when the imagery representation was based on visual rather than tactile perception, but only in the case of verbal size assessment. Verbal height estimation in centimeters showed a tendency towards underestimation, while bimanually estimated sizes tended to be overestimated. The results indicate that people can use pictorial as well as prepositional strategies, depending on the task.

Watching diagnoses develop: Eye movements reveal symptom processing during diagnostic reasoning

Finding a probable explanation for observed symptoms is a highly complex task that draws on information retrieval from memory. Recent research suggests that observed symptoms are interpreted in a way that maximizes coherence for a single likely explanation. This becomes particularly clear if symptom sequences support more than one explanation. However, there are no existing process data available that allow coherence maximization to be traced in ambiguous diagnostic situations, where critical information has to be retrieved from memory. In this experiment, we applied memory indexing, an eye-tracking method that affords rich time-course information concerning memory-based cognitive processing during higher order thinking, to reveal symptom processing and the preferred interpretation of symptom sequences. Participants first learned information about causes and symptoms presented in spatial frames. Gaze allocation to emptied spatial frames during symptom processing and during the diagnostic response reflected the subjective status of hypotheses held in memory and the preferred interpretation of ambiguous symptoms. Memory indexing traced how the diagnostic decision developed and revealed instances of hypothesis change and biases in symptom processing. Memory indexing thus provided direct online evidence for coherence maximization in processing ambiguous information.

Gaze patterns reveal how situation models and text representations contribute to episodic text memory

When recalling something you have previously read, to what degree will such episodic remembering activate a situation model of described events versus a memory representation of the text itself? The present study was designed to address this question by recording eye movements of participants who recalled previously read texts while looking at a blank screen. An accumulating body of research has demonstrated that spontaneous eye movements occur during episodic memory retrieval and that fixation locations from such gaze patterns to a large degree overlap with the visuospatial layout of the recalled information. Here we used this phenomenon to investigate to what degree participants' gaze patterns corresponded with the visuospatial configuration of the text itself versus a visuospatial configuration described in it. The texts to be recalled were scene descriptions, where the spatial configuration of the scene content was manipulated to be either congruent or incongruent with the spatial configuration of the text itself. Results show that participants' gaze patterns were more likely to correspond with a visuospatial representation of the described scene than with a visuospatial representation of the text itself, but also that the contribution of those representations of space is sensitive to the text content. This is the first demonstration that eye movements can be used to discriminate on which representational level texts are remembered and the findings provide novel insight into the underlying dynamics in play.

"Looking-at-nothing" during sequential sensorimotor actions: Long-term memory-based eye scanning of remembered target locations

Before acting humans saccade to a target object to extract relevant visual information. Even when acting on remembered objects, locations previously occupied by relevant objects are fixated during imagery and memory tasks - a phenomenon called "looking-at-nothing". While looking-at-nothing was robustly found in tasks encouraging declarative memory built-up, results are mixed in the case of procedural sensorimotor tasks. Eye-guidance to manual targets in complete darkness was observed in a task practiced for days beforehand, while investigations using only a single session did not find fixations to remembered action targets. Here, it is asked whether looking-at-nothing can be found in a single sensorimotor session and thus independent from sleep consolidation, and how it progresses when visual information is repeatedly unavailable. Eye movements were investigated in a computerized version of the trail making test. Participants clicked on numbered circles in ascending sequence. Fifty trials were performed with the same spatial arrangement of 9 visual targets to enable long-term memory consolidation. During 50 consecutive trials, participants had to click the remembered target sequence on an empty screen. Participants scanned the visual targets and also the empty target locations sequentially with their eyes, however, the latter less precise than the former. Over the course of the memory trials, manual and oculomotor sequential target scanning became more similar to the visual trials. Results argue for robust looking-at-nothing during procedural sensorimotor tasks provided that long-term memory information is sufficient.

Eye-Tracking Causality

How do people make causal judgments? What role, if any, does counterfactual simulation play? Counterfactual theories of causal judgments predict that people compare what actually happened with what would have happened if the candidate cause had been absent. Process theories predict that people focus only on what actually happened, to assess the mechanism linking candidate cause and outcome. We tracked participants' eye movements while they judged whether one billiard ball caused another one to go through a gate or prevented it from going through. Both participants' looking patterns and their judgments demonstrated that counterfactual simulation played a critical role. Participants simulated where the target ball would have gone if the candidate cause had been removed from the scene. The more certain participants were that the outcome would have been different, the stronger the causal judgments. These results provide the first direct evidence for spontaneous counterfactual simulation in an important domain of high-level cognition.

Beyond eye gaze: What else can eyetracking reveal about cognition and cognitive development?

This review provides an introduction to two eyetracking measures that can be used to study cognitive development and plasticity: pupil dilation and spontaneous blink rate. We begin by outlining the rich history of gaze analysis, which can reveal the current focus of attention as well as cognitive strategies. We then turn to the two lesser-utilized ocular measures. Pupil dilation is modulated by the brain's locus coeruleus-norepinephrine system, which controls physiological arousal and attention, and has been used as a measure of subjective task difficulty, mental effort, and neural gain. Spontaneous eyeblink rate correlates with levels of dopamine in the central nervous system, and can reveal processes underlying learning and goal-directed behavior. Taken together, gaze, pupil dilation, and blink rate are three non-invasive and complementary measures of cognition with high temporal resolution and well-understood neural foundations. Here we review the neural foundations of pupil dilation and blink rate, provide examples of their usage, describe analytic methods and methodological considerations, and discuss their potential for research on learning, cognitive development, and plasticity.

Object Recognition in Mental Representations: Directions for Exploring Diagnostic Features through Visual Mental Imagery

One of the fundamental goals of object recognition research is to understand how a cognitive representation produced from the output of filtered and transformed sensory information facilitates efficient viewer behavior. Given that mental imagery strongly resembles perceptual processes in both cortical regions and subjective visual qualities, it is reasonable to question whether mental imagery facilitates cognition in a manner similar to that of perceptual viewing: via the detection and recognition of distinguishing features. Categorizing the feature content of mental imagery holds potential as a reverse pathway by which to identify the components of a visual stimulus which are most critical for the creation and retrieval of a visual representation. This review will examine the likelihood that the information represented in visual mental imagery reflects distinctive object features thought to facilitate efficient object categorization and recognition during perceptual viewing. If it is the case that these representational features resemble their sensory counterparts in both spatial and semantic qualities, they may well be accessible through mental imagery as evaluated through current investigative techniques. In this review, methods applied to mental imagery research and their findings are reviewed and evaluated for their efficiency in accessing internal representations, and implications for identifying diagnostic features are discussed. An argument is made for the benefits of combining mental imagery assessment methods with diagnostic feature research to advance the understanding of visual perceptive processes, with suggestions for avenues of future investigation.

Recalling what was where when seeing nothing there

So-called "looks-at-nothing" have previously been used to show that recalling what also elicits the recall of where this was. Here, we present evidence from an eye-tracking study which shows that disrupting looks to "there" does not disrupt recalling what was there, nor do (anticipatory) looks to "there" facilitate recalling what was there. Therefore, our results suggest that recalling where does not recall what.

Using space to represent categories: insights from gaze position

We investigated the boundaries among imagery, memory, and perception by measuring gaze during retrieved versus imagined visual information. Eye fixations during recall were bound to the location at which a specific stimulus was encoded. However, eye position information generalized to novel objects of the same category that had not been seen before. For example, encoding an image of a dog in a specific location enhanced the likelihood of looking at the same location during subsequent mental imagery of other mammals. The results suggest that eye movements can also be launched by abstract representations of categories and not exclusively by a single episode or a specific visual exemplar.