Keeping an eye on serial order: Ocular movements bind space and time

Inter-individual variability (but intra-individual stability) of overt versus covert rehearsal strategies in a digital Corsi task

The Corsi (block-tapping) paradigm is a classic and well-established visuospatial working memory task in humans involving internal computations (memorizing of item sequences, organizing and updating the memorandum, and recall processes), as well as both overt and covert shifts of attention to facilitate rehearsal, serving to maintain the Corsi sequences during the retention phase. Here, we introduce a novel digital version of a Corsi task in which i) the difficulty of the memorandum (using sequence lengths ranging from 3 to 8) was controlled, ii) the execution of overt and/or covert attention as well as the visuospatial working memory load during the retention phase was manipulated, and iii) shifts of attention were quantified in all experimental phases. With this, we present behavioral data that demonstrate, characterize, and classify the individual effects of overt and covert strategies used as a means of encoding and rehearsal. In a full within-subject design, we tested 28 participants who had to solve three different Corsi conditions. While in condition A neither of the two strategies were restricted, in condition B the overt and in condition C the overt as well as the covert strategies were suppressed. Analyzing Corsi span, (eye) exploration index, and pupil size (change), data clearly show a continuum between overt and covert strategies over all participants (indicating inter-individual variability). Further, all participants showed stable strategy choice (indicating intra-individual stability), meaning that the preferred strategy was maintained in all three conditions, phases, and sequence lengths of the experiment.

Spatial-positional associations in short-term memory can vanish in long-term memory

Studies on the SPoARC effect have shown that serial information is spatially processed in working memory. However, it remains unknown whether these spatial-positional associations are durable or only temporary. This study aimed at investigating whether spatialization would persist when a sequence presented repeatedly is expected to be chunked. If chunked, the items could be unified spatially and their spatialization could vanish. Thirty-seven participants performed a spatialization task which was remotely inspired by the Hebb repetition paradigm. A sequence of four stimuli presented individually in the middle of a computer screen was repeated throughout the task. After each sequence, participants had to decide whether a probe belonged to the series using two lateralized response keys. The results showed no spatialization for these repetitive sequences, on average. Moreover, further analysis revealed that the effect was detectable at the beginning of the task, suggesting that the more the sequence was repeated, the less participants spatialized information from left to right. These findings show that associations created in working memory between items and space can vanish in repeated sequences: we discuss the idea that working memory progressively saves on spatialization once a sequence is chunked in long-term memory.

Optimized experimental designs to best detect spatial positional association of response codes in working memory

The SPoARC (Spatial Positional Association of Response Codes) effect refers to spatialization of information in working memory. Among the potential factors that could influence how order is mapped onto a mental space during the recognition process, we selected the following two factors: i) the type of stimuli, in particular their verbal vs. visual aspects and ii) the number of probes. In this study, 137 participants memorized sequences of either words or pictures and subsequently performed a recognition test for which they responded using lateralized keys. For half of the participants, only one probe was presented after each sequence, whereas the other half was administered several probes. A significantly greater number of participants presented a SPoARC using a single probe. We discuss that spatialization is best detected when the sequence is scanned only once. Results also showed no difference between the two types of stimuli (i.e., verbal vs. visual). This finding raises the question of the respective roles of verbalization and visualization in the SPoARC.

Spatial metaphors and the design of everyday things

People use space (e.g., left-right, up-down) to think about a variety of non-spatial concepts like time, number, similarity, and emotional valence. These spatial metaphors can be used to inform the design of user interfaces, which visualize many of these concepts in space. Traditionally, researchers have relied on patterns in language to discover habits of metaphorical thinking. However, advances in cognitive science have revealed that many spatial metaphors remain unspoken, shaping people's preferences, memories, and actions independent of language - and even in contradiction to language. Here we argue that cognitive science can impact our everyday lives by informing the design of physical and digital objects via the spatial metaphors in people's minds. We propose a simple principle for predicting which spatial metaphors organize people's non-spatial concepts based on the structure of their linguistic, cultural, and bodily experiences. By leveraging the latent metaphorical structure of people's minds, we can design objects and interfaces that help people think.

A SPoARC of Music: Musicians Spatialize Melodies but not All-Comers

Recent studies on the spatial positional associated response codes (SPoARC) effect have shown that when Western adults are asked to keep in mind sequences of verbal items, they mentally spatialize them along the horizontal axis, with the initial items being associated with the left and the last items being associated with the right. The origin of this mental line is still debated, but it has been theorized that it necessitates specific spatial cognitive structures to emerge, which are built through expertise. This hypothesis is examined by testing for the first time whether Western individuals spatialize melodies from left to right and whether expertise in the musical domain is necessary for this effect to emerge. Two groups (musicians and non-musicians) of participants were asked to memorize sequences of four musical notes and to indicate if a subsequent probe was part of the sequence by pressing a "yes" key or a "no" key with the left or right index finger. Left/right-hand key assignment was reversed at mid-experiment. The results showed a SPoARC effect only for the group of musicians. Moreover, no association between pitch and hand responses was observed in either of the two groups. These findings suggest a crucial role of expertise in the SPoARC effect.

Spatialization in working memory and its relation to math anxiety

Working memory (WM) is one of the most important cognitive functions that may play a role in the relation between math anxiety (MA) and math performance. The processing efficiency theory proposes that rumination and worrisome thoughts (induced by MA) result in less available WM resources (which are needed to solve math problems). At the same time, high MA individuals have lower verbal and spatial WM capacity in general. Extending these findings, we found that MA is also linked to the spatial coding of serial order in verbal WM: subjects who organize sequences from left-to-right in verbal WM show lower levels of MA compared with those who do not spatialize. Furthermore, these spatial coders have higher verbal WM capacity, better numerical order judgment abilities, and higher math scores. These findings suggest that spatially structuring the verbal mind is a promising cognitive correlate of MA and opens new avenues for exploring causal links between elementary cognitive processes and MA.

Eye-movements reveal the serial position of the attended item in verbal working memory

The problem of how the mind can retain sequentially organized information has a long research tradition that remains unresolved. While various computational models propose a mechanism of binding serial order information to position markers, the representational nature and processes that operate on these position markers are not clear. Recent behavioral work suggests that space is used to mark positions in serial order and that this process is governed by spatial attention. Based on the assumption that brain areas controlling spatial attention are also involved in saccadic planning, we continuously tracked the eye-movements as a direct measure of the spatial attention during retrieval from a verbal WM sequence. Participants memorized a sequence of auditory numbers. During retention, they heard a number-cue that did or did not belong to the memorized set. After this number-cue, a target-beep could be presented to which they had to respond if the number-cue belonged to the memorized sequence. In Experiment 1, the target-beep was either presented to the left or right ear, and in Experiment 2 bilaterally (removing any spatial aspect). We tested the hypothesis that systematic eye-movements are made when people retrieve items of sequences of auditory words and found that the retrieval of begin items resulted in leftward eye-movements and the retrieval of end items in rightward eye-movements. These observations indicate that the oculomotor system is also involved in the serial order processes in verbal WM thereby providing a promising novel approach to get insight into abstract cognitive processes.

Early is left and up: Saccadic responses reveal horizontal and vertical spatial associations of serial order in working memory

Maintaining serial order in working memory is crucial for cognition. Recent theories propose that serial information is achieved by positional coding of items on a spatial frame of reference. In line with this, an early-left and late-right spatial-positional association of response code (SPoARC) effect has been established. Various theoretical accounts have been put forward to explain the SPoARC effect (the mental whiteboard hypothesis, conceptual metaphor theory, polarity correspondence, or the indirect spatial-numerical association effect). Crucially, while all these accounts predict a left-to-right orientation of the SPoARC effect, they make different predictions regarding the direction of a possible vertical SPoARC effect. In this study, we therefore investigated SPoARC effects along the horizontal and vertical spatial dimension by means of saccadic responses. We replicated the left-to-right horizontal SPoARC effect and established for the first time an up-to-down vertical SPoARC effect. The direction of the vertical SPoARC effect was in contrast to that predicted by metaphor theory, polarity correspondence, or by the indirect spatial-numerical association effect. Rather, our results support the mental whiteboard-hypothesis, according to which positions can be flexibly coded on an internal space depending on the task demands. We also found that the strengths of the horizontal and vertical SPoARC effects were correlated, showing that some people are more prone than others to use spatial references for position coding. Our results therefore suggest that context templates used for position marking are not necessarily spatial in nature but depend on individual strategy preferences.

Bootstrapping the visuospatial bootstrapping effect and testing its spatialisation

When sequences of digits are visually presented within a numerical keypad on a screen, memory span increases, this effect was named visuospatial bootstrapping. The aim of the first experiment was to know if this effect could emerge without presenting a keypad on the screen. For this purpose, a three-phase experiment was designed. During phase 1, the immediate serial recall of two groups of participants was compared (pre-training): the first group saw sequences of one-digit numbers displayed on a screen within a keypad (the keypad group) whereas the second group heard the (same) sequences (the auditive group). During phase 2, all participants underwent a training session to help them visualise in their mind a keypad. Finally, in the third phase, participants were tested again with an immediate serial recall task (post-training). Results showed that both groups had comparable performance in post-training indicating that the visuospatial bootstrapping could be obtained without displaying a numerical keypad. The second experiment also involved a keypad group and an auditive group and was designed to investigate their spatial representation. Results showed that both groups spatialised the digits following the keypad spatial configuration: digits 1-4-7 were associated to left, 2-5-8 to middle and 3-6-9 to right.

Eye movements during visual imagery and perception show spatial correspondence but have unique temporal signatures

Eye fixation patterns during mental imagery are similar to those during perception of the same picture, suggesting that oculomotor mechanisms play a role in mental imagery (i.e., the "looking at nothing" effect). Previous research has focused on the spatial similarities of eye movements during perception and mental imagery. The primary aim of this study was to assess whether the spatial similarity translates to the temporal domain. We used recurrence quantification analysis (RQA) to assess the temporal structure of eye fixations in visual perception and mental imagery and we compared the temporal as well as the spatial characteristics in mental imagery with perception by means of Bayesian hierarchical regression models. We further investigated how person and picture-specific characteristics contribute to eye movement behavior in mental imagery. Working memory capacity and mental imagery abilities were assessed to either predict gaze dynamics in visual imagery or to moderate a possible correspondence between spatial or temporal gaze dynamics in perception and mental imagery. We were able to show the spatial similarity of fixations between visual perception and imagery and we provide first evidence for its moderation by working memory capacity. Interestingly, the temporal gaze dynamics in mental imagery were unrelated to those in perception and their variance between participants was not explained by variance in visuo-spatial working memory capacity or vividness of mental images. The semantic content of the imagined pictures was the only meaningful predictor of temporal gaze dynamics. The spatial correspondence reflects shared spatial structure of mental images and perceived pictures, while the unique temporal gaze behavior could be driven by generation, maintenance and protection processes specific to visual imagery. The unique temporal gaze dynamics offer a window to new insights into the genuine process of mental imagery independent of its similarity to perception.

Spatialization in working memory: can individuals reverse the cultural direction of their thoughts?

A recent study based on the SPoARC effect (spatial position association response codes) showed that culture heavily shapes cognition and more specifically the way thought is organized; when Western adults are asked to keep in mind a sequence of colors, they mentally organize them from left to right, whereas right-to-left reading/writing adults spatialize them in the opposite direction. Here, we investigate if the spontaneous direction of spatialization in Westerners can be reversed. Lists of five consonants were presented auditorily at a rate of 3 s per item, participants were asked to mentally organize the memoranda from right to left. Each list was followed by a probe. Participants had to indicate whether the probe was part of the sequence by pressing a "yes" key or a "no" key with the left or right index finger. Left/right-hand key assignment was switched after half of the trials were completed. The results showed a reverse SPoARC effect that was comparable in magnitude to the spontaneous left-to-right SPoARC effect found in a previous study. Overall, our results suggest that individuals can reverse the cultural direction of their thoughts.

Do Writing Directions Influence How People Map Space onto Time?

Abstract. The causal role of a unidirectional orthography in shaping speakers’ mental representations of time seems to be well established by many psychological experiments. However, the question of whether bidirectional writing systems in some languages can also produce such an impact on temporal cognition remains unresolved. To address this issue, the present study focused on Japanese and Taiwanese, both of which have a similar mix of texts written horizontally from left to right (HLR) and vertically from top to bottom (VTB). Two experiments were performed which recruited Japanese and Taiwanese speakers as participants. Experiment 1 used an explicit temporal arrangement design, and Experiment 2 measured implicit space-time associations in participants along the horizontal (left/right) and the vertical (up/down) axis. Converging evidence gathered from the two experiments demonstrate that neither Japanese speakers nor Taiwanese speakers aligned their vertical representations of time with the VTB writing orientation. Along the horizontal axis, only Japanese speakers encoded elapsing time into a left-to-right linear layout, which was commensurate with the HLR writing direction. Therefore, two distinct writing orientations of a language could not bring about two coexisting mental time lines. Possible theoretical implications underlying the findings are discussed.

Sequential Effects in SNARC

Small and large numbers are typically associated with the left and right side of space, respectively. We conducted an online version of the classical Spatial-Numerical Association of Response Codes (SNARC) paradigm in 604 subjects in order to analyse how previous trials and responses affect SNARC. Our results point to a strong inversion of number-space associations (left/large and right/small) when the last trial was incoherent - i.e. when a response with the left hand was made to a large number or vice-versa. In addition, we demonstrate that the congruency of trials beyond just the last two can influence SNARC, providing empirical support for an important assumption of a working memory account of spatial-numerical associations. Finally, we show that sequential effects in SNARC can be captured by a simple exponential filter, known to underpin sequential effects across a range of stimuli detection and perceptual two-alternative forced choice decision tasks. Our findings point to universal mechanisms responsible for the processing of sequences from perception to cognition.

Systematic spatial patterns of the sense of familiarity: Hierarchical modelling based on eye-tracking experiments

Using different types of stimuli, such as pictures, horizontally written Japanese words, and vertically written Japanese words, this study investigated the spatial patterns of the sense of familiarity within the visual field. The perceptual asymmetry theory predicted that stimuli in the lower visual field would be processed more fluently and would therefore be perceived as more familiar. The working memory theory, originally proposed in space-number research, envisaged type-specific spatial patterns for different stimuli. Participants made old/new recognition memory judgements for stimuli, presented at random positions, while their eye movements were recorded. The observed spatial patterns changed according to the stimulus type (e.g., "more left = older" for horizontally written words and "upper = older" for vertically written words), and this flexibility is encapsulated by the working memory theory as follows: (a) stimulus-type-specific spatial configurations are encoded in long-term memory on the basis of one's experience (e.g., vertically written words are empirically associated with the "upper = older" spatial configuration), (b) the presentation of a stimulus automatically cues the temporal activation of the associated spatial configuration in working memory, and (c) the referential process between the stimulus and configuration unconsciously affects the viewer's sense of familiarity.

The spatial representation of number, time, and serial order following sensory deprivation: A systematic review

The spatial representation of numerical and temporal information is thought to be rooted in our multisensory experiences. Accordingly, we may expect visual or auditory deprivation to affect the way we represent numerical magnitude and time spatially. Here, we systematically review recent findings on how blind and deaf individuals represent abstract concepts such as magnitude and time (e.g., past/future, serial order of events) in a spatial format. Interestingly, available evidence suggests that sensory deprivation does not prevent the spatial "re-mapping" of abstract information, but differences compared to normally sighted and hearing individuals may emerge depending on the specific dimension considered (i.e., numerical magnitude, time as past/future, serial order). Herein we discuss how the study of sensory deprived populations may shed light on the specific, and possibly distinct, mechanisms subserving the spatial representation of these concepts. Furthermore, we pinpoint unresolved issues that need to be addressed by future studies to grasp a full understanding of the spatial representation of abstract information associated with visual and auditory deprivation.

Hemispatial neglect and serial order in verbal working memory

Working memory refers to our ability to actively maintain and process a limited amount of information during a brief period of time. Often, not only the information itself but also its serial order is crucial for good task performance. It was recently proposed that serial order is grounded in spatial cognition. Here, we compared performance of a group of right hemisphere-damaged patients with hemispatial neglect to healthy controls in verbal working memory tasks. Participants memorized sequences of consonants at span level and had to judge whether a target consonant belonged to the memorized sequence (item task) or whether a pair of consonants were presented in the same order as in the memorized sequence (order task). In line with this idea that serial order is grounded in spatial cognition, we found that neglect patients made significantly more errors in the order task than in the item task compared to healthy controls. Furthermore, this deficit seemed functionally related to neglect severity and was more frequently observed following right posterior brain damage. Interestingly, this specific impairment for serial order in verbal working memory was not lateralized. We advance the hypotheses of a potential contribution to the deficit of serial order in neglect patients of either or both (1) reduced spatial working memory capacity that enables to keep track of the spatial codes that provide memorized items with a positional context, (2) a spatial compression of these codes in the intact representational space.

Grounding Verbal Working Memory: The Case of Serial Order

The maintenance of serial order in verbal working memory (WM) is a major unsolved puzzle in cognitive science. Here we review a series of studies showing that serial order in verbal WM closely interacts with spatial processing. Accordingly, we outline the “mental whiteboard hypothesis,” which postulates that serial order in verbal WM is grounded in the spatial attention system. Specifically, serial context in verbal WM is provided by binding the memoranda to coordinates within an internal, spatially defined system within which (internal) spatial attention is at play to the purpose of searching for and retrieving information. Challenges and opportunities to be considered in future studies are discussed.

Order Information in Verbal Working Memory Shifts the Subjective Midpoint in Both the Line Bisection and the Landmark Tasks

A largely substantiated view in the domain of working memory is that the maintenance of serial order is achieved by generating associations of each item with an independent representation of its position, so-called position markers. Recent studies reported that the ordinal position of an item in verbal working memory interacts with spatial processing. This suggests that position markers might be spatial in nature. However, these interactions were so far observed in tasks implying a clear binary categorization of space (i.e., with left and right responses or targets). Such binary categorizations leave room for alternative interpretations, such as congruency between non-spatial categorical codes for ordinal position (e.g., begin and end) and spatial categorical codes for response (e.g., left and right). Here we discard this interpretation by providing evidence that this interaction can also be observed in a task that draws upon a continuous processing of space, the line bisection task. Specifically, bisections are modulated by ordinal position in verbal working memory, with lines bisected more towards the right after retrieving items from the end compared to the beginning of the memorized sequence. This supports the idea that position markers are intrinsically spatial in nature.

The Automatic Mapping of Magnitude to Temporal Order Is Space-Dependent

Past research has shown that performance in ordinal magnitude tasks is enhanced when stimuli are presented in ascending order, suggesting that magnitude is mapped to temporal order, with small magnitude associated with early and large with late presentation. The present study addresses the automaticity of this effect and its limitations. We used the "same/different" task for numbers (Experiment 1) and physical sizes of shapes (Experiment 2) as well as identity of shapes (Experiment 3). The advantage for stimuli in ascending order was found for both numbers and physical sizes of shapes. However, it was limited to specific conditions - when magnitude processing was required for the task and when a "different" response was mapped to the right hand side. Thus, it seems that the automatic mapping of magnitude to temporal order is dependent on the mapping of magnitude to space.

Manual actions cover symbolic distances at different speed

A privileged way of representing numbers in the human mind is along an oriented mental number line. Activation of this representation has been proposed to account for the impact of numbers on motor tasks, such as on grasping, pointing, and eye movements. Here we evaluated the impact of numbers on motor control, by exploiting the evidence that the speed reached by the manual connection of two points is correlated with their physical distance. We reasoned that, if irrelevant numbers induce a mis-perception of the distance between two points, this should be reflected in the movement speed. Results showed a speed difference in the manual connection of two numerically close numbers (i.e., connected slower) and two numerically distant numbers (i.e., connected faster), placed at equal physical distance. This representational length effect indicates not only that symbolic distance modulates speed movement as physical distance does, but suggests that the impact of numbers on action planning does not only involve action initiation but it extends to the definition of kinematic parameters. More generally, the reported findings show that the representation of numbers along a mental space affects our behaviour in the physical space.

How Does Working Memory Enable Number-Induced Spatial Biases?

Number-space associations are a robust observation, but their underlying mechanisms remain debated. Two major accounts have been identified. First, spatial codes may constitute an intrinsic part of number representations stored in the brain – a perspective most commonly referred to as the Mental Number Line account. Second, spatial codes may be generated at the level of working memory when number (or other) representations are coordinated in function of a specific task. The aim of the current paper is twofold. First, whereas a pure Mental Number Line account cannot capture the complexity of observations reported in the literature, we here explore if and how a pure working memory account can suffice. Second, we make explicit (more than in our earlier work) the potential building blocks of such a working memory account, thereby providing clear and concrete foci for empirical efforts to test the feasibility of the account.

A helping hand putting in order: Visuomotor routines organize numerical and non-numerical sequences in space

Theories of embodied cognition emphasize the importance of sensorimotor schemas linked to external world experience for representing conceptual knowledge. Accordingly, some researchers have proposed that the spatial representation of numerical and non-numerical sequences relies on visuomotor routines, like reading habit and finger counting. There is a growing interest in how these two routines contribute to the spatial representation of ordinal sequences, although no investigation has so far directly compared them. The present study aims to investigate how these routines contribute to represent ordinal information in space. To address this issue, bilingual participants reading either from left-to-right or right-to-left were required to map ordinal information to all fingers of their right dominant hand. Critically, we manipulated both the direction of the mapping and the language of the verbal information. More specifically, a finger-mapping compatibility task was adopted in three experiments to explore the spatial representation of numerical (digit numbers and number words) and non-numerical (days of the week, presented in Hebrew and in English) sequences. Results showed that numerical information was preferentially mapped according to participants' finger counting habits, regardless of hand posture (prone and supine), number notation and reading habit. However, for non-numerical ordinal sequences, reading and finger counting directions both contributed to determine a preferential spatial mapping. These findings indicate that abstract knowledge representation relies on multiple over-trained visuomotor routines. More generally, these results highlight the capacity of our cognitive system to flexibly represent abstract ordered information, by relying on different directional experiences (finger counting, reading direction) depending on the stimuli and on the task at hand.