From numerosity to ordinal rank: a gain-field model of serial order representation in cortical working memory

A geometrical solution underlies general neural principle for serial ordering

A general mathematical description of how the brain sequentially encodes knowledge remains elusive. We propose a linear solution for serial learning tasks, based on the concept of mixed selectivity in high-dimensional neural state spaces. In our framework, neural representations of items in a sequence are projected along a "geometric" mental line learned through classical conditioning. The model successfully solves serial position tasks and explains behaviors observed in humans and animals during transitive inference tasks amidst noisy sensory input and stochastic neural activity. This approach extends to recurrent neural networks performing motor decision tasks, where the same geometric mental line correlates with motor plans and modulates network activity according to the symbolic distance between items. Serial ordering is thus predicted to emerge as a monotonic mapping between sensory input and behavioral output, highlighting a possible pivotal role for motor-related associative cortices in transitive inference tasks.

Mixed selectivity: Cellular computations for complexity

The property of mixed selectivity has been discussed at a computational level and offers a strategy to maximize computational power by adding versatility to the functional role of each neuron. Here, we offer a biologically grounded implementational-level mechanistic explanation for mixed selectivity in neural circuits. We define pure, linear, and nonlinear mixed selectivity and discuss how these response properties can be obtained in simple neural circuits. Neurons that respond to multiple, statistically independent variables display mixed selectivity. If their activity can be expressed as a weighted sum, then they exhibit linear mixed selectivity; otherwise, they exhibit nonlinear mixed selectivity. Neural representations based on diverse nonlinear mixed selectivity are high dimensional; hence, they confer enormous flexibility to a simple downstream readout neural circuit. However, a simple neural circuit cannot possibly encode all possible mixtures of variables simultaneously, as this would require a combinatorially large number of mixed selectivity neurons. Gating mechanisms like oscillations and neuromodulation can solve this problem by dynamically selecting which variables are mixed and transmitted to the readout.

Brain-imaging evidence for compression of binary sound sequences in human memory

According to the language-of-thought hypothesis, regular sequences are compressed in human memory using recursive loops akin to a mental program that predicts future items. We tested this theory by probing memory for 16-item sequences made of two sounds. We recorded brain activity with functional MRI and magneto-encephalography (MEG) while participants listened to a hierarchy of sequences of variable complexity, whose minimal description required transition probabilities, chunking, or nested structures. Occasional deviant sounds probed the participants' knowledge of the sequence. We predicted that task difficulty and brain activity would be proportional to the complexity derived from the minimal description length in our formal language. Furthermore, activity should increase with complexity for learned sequences, and decrease with complexity for deviants. These predictions were upheld in both fMRI and MEG, indicating that sequence predictions are highly dependent on sequence structure and become weaker and delayed as complexity increases. The proposed language recruited bilateral superior temporal, precentral, anterior intraparietal, and cerebellar cortices. These regions overlapped extensively with a localizer for mathematical calculation, and much less with spoken or written language processing. We propose that these areas collectively encode regular sequences as repetitions with variations and their recursive composition into nested structures.

Semi-orthogonal subspaces for value mediate a tradeoff between binding and generalization

When choosing between options, we must associate their values with the action needed to select them. We hypothesize that the brain solves this binding problem through neural population subspaces. To test this hypothesis, we examined neuronal responses in five reward-sensitive regions in macaques performing a risky choice task with sequential offers. Surprisingly, in all areas, the neural population encoded the values of offers presented on the left and right in distinct subspaces. We show that the encoding we observe is sufficient to bind the values of the offers to their respective positions in space while preserving abstract value information, which may be important for rapid learning and generalization to novel contexts. Moreover, after both offers have been presented, all areas encode the value of the first and second offers in orthogonal subspaces. In this case as well, the orthogonalization provides binding. Our binding-by-subspace hypothesis makes two novel predictions borne out by the data. First, behavioral errors should correlate with putative spatial (but not temporal) misbinding in the neural representation. Second, the specific representational geometry that we observe across animals also indicates that behavioral errors should increase when offers have low or high values, compared to when they have medium values, even when controlling for value difference. Together, these results support the idea that the brain makes use of semi-orthogonal subspaces to bind features together.

ROSE: A Neurocomputational Architecture for Syntax

A comprehensive model of natural language processing in the brain must accommodate four components: representations, operations, structures and encoding. It further requires a principled account of how these different components mechanistically, and causally, relate to each another. While previous models have isolated regions of interest for structure-building and lexical access, and have utilized specific neural recording measures to expose possible signatures of syntax, many gaps remain with respect to bridging distinct scales of analysis that map onto these four components. By expanding existing accounts of how neural oscillations can index various linguistic processes, this article proposes a neurocomputational architecture for syntax, termed the ROSE model (Representation, Operation, Structure, Encoding). Under ROSE, the basic data structures of syntax are atomic features, types of mental representations (R), and are coded at the single-unit and ensemble level. Elementary computations (O) that transform these units into manipulable objects accessible to subsequent structure-building levels are coded via high frequency broadband γ activity. Low frequency synchronization and cross-frequency coupling code for recursive categorial inferences (S). Distinct forms of low frequency coupling and phase-amplitude coupling (δ-θ coupling via pSTS-IFG; θ-γ coupling via IFG to conceptual hubs in lateral and ventral temporal cortex) then encode these structures onto distinct workspaces (E). Causally connecting R to O is spike-phase/LFP coupling; connecting O to S is phase-amplitude coupling; connecting S to E is a system of frontotemporal traveling oscillations; connecting E back to lower levels is low-frequency phase resetting of spike-LFP coupling. This compositional neural code has important implications for algorithmic accounts, since it makes concrete predictions for the appropriate level of study for psycholinguistic parsing models. ROSE is reliant on neurophysiologically plausible mechanisms, is supported at all four levels by a range of recent empirical research, and provides an anatomically precise and falsifiable grounding for the basic property of natural language syntax: hierarchical, recursive structure-building.

Replay and compositional computation

Replay in the brain has been viewed as rehearsal or, more recently, as sampling from a transition model. Here, we propose a new hypothesis: that replay is able to implement a form of compositional computation where entities are assembled into relationally bound structures to derive qualitatively new knowledge. This idea builds on recent advances in neuroscience, which indicate that the hippocampus flexibly binds objects to generalizable roles and that replay strings these role-bound objects into compound statements. We suggest experiments to test our hypothesis, and we end by noting the implications for AI systems which lack the human ability to radically generalize past experience to solve new problems.

Subspace orthogonalization as a mechanism for binding values to space

When choosing between options, we must solve an important binding problem. The values of the options must be associated with information about the action needed to select them. We hypothesize that the brain solves this binding problem through use of distinct population subspaces. To test this hypothesis, we examined the responses of single neurons in five reward-sensitive regions in rhesus macaques performing a risky choice task. In all areas, neurons encoded the value of the offers presented on both the left and the right side of the display in semi-orthogonal subspaces, which served to bind the values of the two offers to their positions in space. Supporting the idea that this orthogonalization is functionally meaningful, we observed a session-to-session covariation between choice behavior and the orthogonalization of the two value subspaces: trials with less orthogonalized subspaces were associated with greater likelihood of choosing the less valued option. Further inspection revealed that these semi-orthogonal subspaces arose from a combination of linear and nonlinear mixed selectivity in the neural population. We show this combination of selectivity balances reliable binding with an ability to generalize value across different spatial locations. These results support the hypothesis that semi-orthogonal subspaces support reliable binding, which is essential to flexible behavior in the face of multiple options.

The contribution of episodic long-term memory to working memory for bindings

The present experiments support two conclusions about the capacity limit of working memory (WM). First, they provide evidence for the Binding Hypothesis, WM capacity is limited by interference between bindings but not items. Second, they show that episodic LTM contributes substantially to binding memory when the capacity of WM is stretched to the limit by larger set sizes. We tested immediate memory for sets of word-picture pairs. With increasing set size, memory for bindings declined more precipitously than memory for items, as predicted from the binding hypothesis. Yet, at higher set sizes performance was more stable than expected from a capacity limited memory, suggesting a contribution of episodic long-term memory (LTM) to circumvent the WM capacity limit. In support of that hypothesis, we show a double dissociation of contributions of WM and episodic LTM to binding memory: Performance at set sizes larger than 3 was specifically affected by proactive interference - but were immune to influences from a distractor-filled delay. In contrast, performance at set size 2 was unaffected by proactive interference but harmed by a distractor-filled delay.

Association between different sensory modalities based on concurrent time series data obtained by a collaborative reservoir computing model

Humans perceive the external world by integrating information from different modalities, obtained through the sensory organs. However, the aforementioned mechanism is still unclear and has been a subject of widespread interest in the fields of psychology and brain science. A model using two reservoir computing systems, i.e., a type of recurrent neural network trained to mimic each other's output, can detect stimulus patterns that repeatedly appear in a time series signal. We applied this model for identifying specific patterns that co-occur between information from different modalities. The model was self-organized by specific fluctuation patterns that co-occurred between different modalities, and could detect each fluctuation pattern. Additionally, similarly to the case where perception is influenced by synchronous/asynchronous presentation of multimodal stimuli, the model failed to work correctly for signals that did not co-occur with corresponding fluctuation patterns. Recent experimental studies have suggested that direct interaction between different sensory systems is important for multisensory integration, in addition to top-down control from higher brain regions such as the association cortex. Because several patterns of interaction between sensory modules can be incorporated into the employed model, we were able to compare the performance between them; the original version of the employed model incorporated such an interaction as the teaching signals for learning. The performance of the original and alternative models was evaluated, and the original model was found to perform the best. Thus, we demonstrated that feedback of the outputs of appropriately learned sensory modules performed the best when compared to the other examined patterns of interaction. The proposed model incorporated information encoded by the dynamic state of the neural population and the interactions between different sensory modules, both of which were based on recent experimental observations; this allowed us to study the influence of the temporal relationship and frequency of occurrence of multisensory signals on sensory integration, as well as the nature of interaction between different sensory signals.

Thinking about order: a review of common processing of magnitude and learned orders in animals

Rich behavioral and neurobiological evidence suggests cognitive and neural overlap in how quantitatively comparable dimensions such as quantity, time, and space are processed in humans and animals. While magnitude domains such as physical magnitude, time, and space represent information that can be quantitatively compared (4 "is half of" 8), they also represent information that can be organized ordinally (1→2→3→4). Recent evidence suggests that the common representations seen across physical magnitude, time, and space domains in humans may be due to their common ordinal features rather than their common quantitative features, as these common representations appear to extend beyond magnitude domains to include learned orders. In this review, we bring together separate lines of research on multiple ordinal domains including magnitude-based and learned orders in animals to explore the extent to which there is support for a common cognitive process underlying ordinal processing. Animals show similarities in performance patterns across natural quantitatively comparable ordered domains (physical magnitude, time, space, dominance) and learned orders (acquired through transitive inference or simultaneous chaining). Additionally, they show transfer and interference across tasks within and between ordinal domains that support the theory of a common ordinal representation across domains. This review provides some support for the development of a unified theory of ordinality and suggests areas for future research to better characterize the extent to which there are commonalities in cognitive processing of ordinal information generally.

Geometrical representation of serial order in working memory

Encoding a sequence relies on one's memory for ordinal succession of events and is critical for episodic memory, spatial navigation, language, and other cognitive functions. Investigating the neural mechanisms underlying sequence working memory in the macaque prefrontal cortex, Xie et al. (Science, 375, 632-639, 2022) uncovered a novel integrated representation of temporal and spatial information in different subspaces of a high-dimensional neural state space, offering broad implications across comparative cognition and neuroscience.

Digital computing through randomness and order in neural networks

We propose that coding and decoding in the brain are achieved through digital computation using three principles: relative ordinal coding of inputs, random connections between neurons, and belief voting. Due to randomization and despite the coarseness of the relative codes, we show that these principles are sufficient for coding and decoding sequences with error-free reconstruction. In particular, the number of neurons needed grows linearly with the size of the input repertoire growing exponentially. We illustrate our model by reconstructing sequences with repertoires on the order of a billion items. From this, we derive the Shannon equations for the capacity limit to learn and transfer information in the neural population, which is then generalized to any type of neural network. Following the maximum entropy principle of efficient coding, we show that random connections serve to decorrelate redundant information in incoming signals, creating more compact codes for neurons and therefore, conveying a larger amount of information. Henceforth, despite the unreliability of the relative codes, few neurons become necessary to discriminate the original signal without error. Finally, we discuss the significance of this digital computation model regarding neurobiological findings in the brain and more generally with artificial intelligence algorithms, with a view toward a neural information theory and the design of digital neural networks.

The Functional Interactions between Cortical Regions through Theta-Gamma Coupling during Resting-State and a Visual Working Memory Task

Theta phase-gamma amplitude coupling (TGC) plays an important role in several different cognitive processes. Although spontaneous brain activity at the resting state is crucial in preparing for cognitive performance, the functional role of resting-state TGC remains unclear. To investigate the role of resting-state TGC, electroencephalogram recordings were obtained for 56 healthy volunteers while they were in the resting state, with their eyes closed, and then when they were engaged in a retention interval period in the visual memory task. The TGCs of the two different conditions were calculated and compared. The results indicated that the modulation index of TGC during the retention interval of the visual working memory (VWM) task was not higher than that during the resting state; however, the topographical distribution of TGC during the resting state was negatively correlated with TGC during VWM task at the local level. The topographical distribution of TGC during the resting state was negatively correlated with TGC coordinates’ engagement of brain areas in local and large-scale networks and during task performance at the local level. These findings support the view that TGC reflects information-processing and signal interaction across distant brain areas. These results demonstrate that TGC could explain the efficiency of competing brain networks.

Geometry of sequence working memory in macaque prefrontal cortex

How the brain stores a sequence in memory remains largely unknown. We investigated the neural code underlying sequence working memory using two-photon calcium imaging to record thousands of neurons in the prefrontal cortex of macaque monkeys memorizing and then reproducing a sequence of locations after a delay. We discovered a regular geometrical organization: The high-dimensional neural state space during the delay could be decomposed into a sum of low-dimensional subspaces, each storing the spatial location at a given ordinal rank, which could be generalized to novel sequences and explain monkey behavior. The rank subspaces were distributed across large overlapping neural groups, and the integration of ordinal and spatial information occurred at the collective level rather than within single neurons. Thus, a simple representational geometry underlies sequence working memory.

Working Memory for Spatial Sequences: Developmental and Evolutionary Factors in Encoding Ordinal and Relational Structures

Sequence learning is a ubiquitous facet of human and animal cognition. Here, using a common sequence reproduction task, we investigated whether and how the ordinal and relational structures linking consecutive elements are acquired by human adults, children, and macaque monkeys. While children and monkeys exhibited significantly lower precision than adults for spatial location and temporal order information, only monkeys appeared to exceedingly focus on the first item. Most importantly, only humans, regardless of age, spontaneously extracted the spatial relations between consecutive items and used a chunking strategy to compress sequences in working memory. Monkeys did not detect such relational structures, even after extensive training. Monkey behavior was captured by a conjunctive coding model, whereas a chunk-based conjunctive model explained more variance in humans. These age- and species-related differences are indicative of developmental and evolutionary mechanisms of sequence encoding and may provide novel insights into the uniquely human cognitive capacities.SIGNIFICANCE STATEMENT Sequence learning, the ability to encode the order of discrete elements and their relationships presented within a sequence, is a ubiquitous facet of cognition among humans and animals. By exploring sequence-processing abilities at different human developmental stages and in nonhuman primates, we found that only humans, regardless of age, spontaneously extracted the spatial relations between consecutive items and used an internal language to compress sequences in working memory. The findings provided insights into understanding the origins of sequence capabilities in humans and how they evolve through development to identify the unique aspects of human cognitive capacity, which includes the comprehension, learning, and production of sequences, and perhaps, above all, language processing.

When A is greater than B: Interactions between magnitude and serial order

Processing ordinal information is an important aspect of cognitive ability, yet the nature of such ordinal representations remains largely unclear. Previously, it has been suggested that ordinal position is coded as magnitude, but this claim has not yet received direct empirical support. This study examined the nature of ordinal representations using a Stroop-like letter order judgment task. If ordinal position is coded as magnitude, then letter ordering and font size should interact. Experiments 1 and 2 identified a significant interaction between letter size and ordering. Specifically, a facilitation effect was observed for alphabetically ordered sequences with decreasing font size (e.g., B C D). This suggests an overlap in the mechanisms for order and magnitude processing. The finding also suggests that earlier ranks may be represented as "more" in such a magnitude-based code, and vice versa for later ranks.

The domain-specificity of serial order working memory

Making a turn while driving is simple: turn on the indicator, check for cars, then turn. Two types of information are required to perform this sequence of events: information about the items (e.g., the correct indicator), and the serial order of those items (e.g., checking before turning rather than vice-versa). Previous research has found distinct working memory capacities (WMCs) for item and serial order information in both verbal and nonverbal domains. The current study investigates whether the serial order WMC is shared for sequences from different content domains. One hundred and fifty-three participants performed sequence matching tasks with verbal (letters and words) and nonverbal (locations and arrows) stimuli. The accuracy of detecting mismatched item-identity and serial order information in sequences was used to operationalize item and order WMC. Using structural equation modeling analyses, we directly compared models that included either domain-specific or domain-general serial order WMC latent variables, finding that models with domain-specific serial order WMC latent variables for verbal and nonverbal materials fit the data better than models with domain-general latent variables. The findings support the hypothesis that there are separate capacities for serial order working memory depending on the type of material being ordered.

Working Memory and Cross-Frequency Coupling of Neuronal Oscillations

Working memory (WM) is the active retention and processing of information over a few seconds and is considered an essential component of cognitive function. The reduced WM capacity is a common feature in many diseases, such as schizophrenia, attention deficit hyperactivity disorder (ADHD), mild cognitive impairment (MCI), and Alzheimer's disease (AD). The theta-gamma neural code is an essential component of memory representations in the multi-item WM. A large body of studies have examined the association between cross-frequency coupling (CFC) across the cerebral cortices and WM performance; electrophysiological data together with the behavioral results showed the associations between CFC and WM performance. The oscillatory entrainment (sensory, non-invasive electrical/magnetic, and invasive electrical) remains the key method to investigate the causal relationship between CFC and WM. The frequency-tuned non-invasive brain stimulation is a promising way to improve WM performance in healthy and non-healthy patients with cognitive impairment. The WM performance is sensitive to the phase and rhythm of externally applied stimulations. CFC-transcranial-alternating current stimulation (CFC-tACS) is a recent approach in neuroscience that could alter cognitive outcomes. The studies that investigated (1) the association between CFC and WM and (2) the brain stimulation protocols that enhanced WM through modulating CFC by the means of the non-invasive brain stimulation techniques have been included in this review. In principle, this review can guide the researchers to identify the most prominent form of CFC associated with WM processing (e.g., theta/gamma phase-amplitude coupling), and to define the previously published studies that manipulate endogenous CFC externally to improve WM. This in turn will pave the path for future studies aimed at investigating the CFC-tACS effect on WM. The CFC-tACS protocols need to be thoroughly studied before they can be considered as therapeutic tools in patients with WM deficits.

Cerebral representation of sequence patterns across multiple presentation formats

The ability to detect the abstract pattern underlying a temporal sequence of events is crucial to many human activities, including language and mathematics, but its cortical correlates remain poorly understood. It is also unclear whether repeated exposure to the same sequence of sensory stimuli is sufficient to induce the encoding of an abstract amodal representation of the pattern. Using functional MRI, we probed the existence of such abstract codes for sequential patterns, their localization in the human brain, and their relation to existing language and math-responsive networks. We used a passive sequence violation paradigm, in which a given sequence is repeatedly presented before rare deviant sequences are introduced. We presented two binary patterns, AABB and ABAB, in four presentation formats, either visual or auditory, and either cued by the identity of the stimuli or by their spatial location. Regardless of the presentation format, a habituation to the repeated pattern and a response to pattern violations were seen in a set of inferior frontal, intraparietal and temporal areas. Within language areas, such pattern-violation responses were only found in the inferior frontal gyrus (IFG), whereas all math-responsive regions responded to pattern changes. Most of these regions also responded whenever the modality or the cue changed, suggesting a general sensitivity to violation detection. Thus, the representation of sequence patterns appears to be distributed, yet to include a core set of abstract amodal regions, particularly the IFG.

Eye-movements reveal the serial position of the attended item in verbal working memory

The problem of how the mind can retain sequentially organized information has a long research tradition that remains unresolved. While various computational models propose a mechanism of binding serial order information to position markers, the representational nature and processes that operate on these position markers are not clear. Recent behavioral work suggests that space is used to mark positions in serial order and that this process is governed by spatial attention. Based on the assumption that brain areas controlling spatial attention are also involved in saccadic planning, we continuously tracked the eye-movements as a direct measure of the spatial attention during retrieval from a verbal WM sequence. Participants memorized a sequence of auditory numbers. During retention, they heard a number-cue that did or did not belong to the memorized set. After this number-cue, a target-beep could be presented to which they had to respond if the number-cue belonged to the memorized sequence. In Experiment 1, the target-beep was either presented to the left or right ear, and in Experiment 2 bilaterally (removing any spatial aspect). We tested the hypothesis that systematic eye-movements are made when people retrieve items of sequences of auditory words and found that the retrieval of begin items resulted in leftward eye-movements and the retrieval of end items in rightward eye-movements. These observations indicate that the oculomotor system is also involved in the serial order processes in verbal WM thereby providing a promising novel approach to get insight into abstract cognitive processes.

Early is left and up: Saccadic responses reveal horizontal and vertical spatial associations of serial order in working memory

Maintaining serial order in working memory is crucial for cognition. Recent theories propose that serial information is achieved by positional coding of items on a spatial frame of reference. In line with this, an early-left and late-right spatial-positional association of response code (SPoARC) effect has been established. Various theoretical accounts have been put forward to explain the SPoARC effect (the mental whiteboard hypothesis, conceptual metaphor theory, polarity correspondence, or the indirect spatial-numerical association effect). Crucially, while all these accounts predict a left-to-right orientation of the SPoARC effect, they make different predictions regarding the direction of a possible vertical SPoARC effect. In this study, we therefore investigated SPoARC effects along the horizontal and vertical spatial dimension by means of saccadic responses. We replicated the left-to-right horizontal SPoARC effect and established for the first time an up-to-down vertical SPoARC effect. The direction of the vertical SPoARC effect was in contrast to that predicted by metaphor theory, polarity correspondence, or by the indirect spatial-numerical association effect. Rather, our results support the mental whiteboard-hypothesis, according to which positions can be flexibly coded on an internal space depending on the task demands. We also found that the strengths of the horizontal and vertical SPoARC effects were correlated, showing that some people are more prone than others to use spatial references for position coding. Our results therefore suggest that context templates used for position marking are not necessarily spatial in nature but depend on individual strategy preferences.

Numeral order and the operationalization of the numerical system

Recent years have witnessed an increase in research on how numeral ordering skills relate to children's and adults' mathematics achievement both cross-sectionally and longitudinally. Nonetheless, it remains unknown which core competency numeral ordering tasks measure, which cognitive mechanisms underlie performance on these tasks, and why numeral ordering skills relate to arithmetic and math achievement. In the current study, we focused on the processes underlying decision-making in the numeral order judgement task with triplets to investigate these questions. A drift-diffusion model for two-choice decisions was fit to data from 97 undergraduates. Findings aligned with the hypothesis that numeral ordering skills reflected the operationalization of the numerical system, where small numbers provide more evidence of an ordered response than large numbers. Furthermore, the pattern of findings suggested that arithmetic achievement was associated with the accuracy of the ordinal representations of numbers.

Mental compression of spatial sequences in human working memory using numerical and geometrical primitives

How does the human brain store sequences of spatial locations? We propose that each sequence is internally compressed using an abstract, language-like code that captures its numerical and geometrical regularities. We exposed participants to spatial sequences of fixed length but variable regularity while their brain activity was recorded using magneto-encephalography. Using multivariate decoders, each successive location could be decoded from brain signals, and upcoming locations were anticipated prior to their actual onset. Crucially, sequences with lower complexity, defined as the minimal description length provided by the formal language, led to lower error rates and to increased anticipations. Furthermore, neural codes specific to the numerical and geometrical primitives of the postulated language could be detected, both in isolation and within the sequences. These results suggest that the human brain detects sequence regularities at multiple nested levels and uses them to compress long sequences in working memory.

Spatial Attention in Serial Order Working Memory: An EEG Study

Theoretical models explaining serial order processing link order information to specified position markers. However, the precise characteristics of position marking have remained largely elusive. Recent studies have shown that space is involved in marking serial position of items in verbal working memory (WM). Furthermore, it has been suggested, but not proven, that accessing these items involves horizontal shifts of spatial attention. We used continuous electroencephalography recordings to show that memory search in serial order verbal WM involves spatial attention processes that share the same electrophysiological signatures as those operating on the visuospatial WM and external space. Accessing an item from a sequence in verbal WM induced posterior “early directing attention negativity” and “anterior directing attention negativity” contralateral to the position of the item in mental space (i.e., begin items on the left; end items on the right). In the frequency domain, we observed posterior alpha suppression contralateral to the position of the item. Our results provide clear evidence for the involvement of spatial attention in retrieving serial information from verbal WM. Implications for WM models are discussed.

Spatialization in working memory: can individuals reverse the cultural direction of their thoughts?

A recent study based on the SPoARC effect (spatial position association response codes) showed that culture heavily shapes cognition and more specifically the way thought is organized; when Western adults are asked to keep in mind a sequence of colors, they mentally organize them from left to right, whereas right-to-left reading/writing adults spatialize them in the opposite direction. Here, we investigate if the spontaneous direction of spatialization in Westerners can be reversed. Lists of five consonants were presented auditorily at a rate of 3 s per item, participants were asked to mentally organize the memoranda from right to left. Each list was followed by a probe. Participants had to indicate whether the probe was part of the sequence by pressing a "yes" key or a "no" key with the left or right index finger. Left/right-hand key assignment was switched after half of the trials were completed. The results showed a reverse SPoARC effect that was comparable in magnitude to the spontaneous left-to-right SPoARC effect found in a previous study. Overall, our results suggest that individuals can reverse the cultural direction of their thoughts.

The spatial coding mechanism of ordinal symbols: a study based on the ordinal position effect

The ordinal position effect refers to a phenomenon in which items positioned early in an ordinal sequence receive a faster response with the left key than with the right key, and the opposite response pattern occurs when items are positioned later in an ordinal sequence. Previous studies have suggested that ordinal symbols are spatially represented from left to right, thus leading to the ordinal position effect; however, the spatial coding mechanism of ordinal symbols remains unclear. Therefore, the present study explored the ordinal position effect as an index to judge the spatial coding of ordinal symbols, and three experiments were performed to investigate the spatial coding mechanism of ordinal symbols. In particular, a novel transitory ordinal sequence was induced by presenting successive dots of different colors centrally (Experiment 1), from left to right or from right to left (Experiments 2 and 3), and participants were asked to memorize the successive dots in the correct order. Then, the participants were asked to press a key to provide a response corresponding to a probe dot's ordinal position (Experiments 1 and 2) or its spatial location (Experiment 3). The following results were identified: (1) The ordinal position effect occurred when responses were based on the ordinal position regardless of the presentation direction, and (2) the ordinal position effect was overridden when responses were based on the spatial locations of the ordinal symbols. From these results, we concluded that the spatial coding of ordinal symbols is flexible and that ordinal symbols are encoded depending on the specific experimental context.

Gated spiking neural network using Iterative Free-Energy Optimization and rank-order coding for structure learning in memory sequences (INFERNO GATE)

We present a framework based on iterative free-energy optimization with spiking neural networks for modeling the fronto-striatal system (PFC-BG) for the generation and recall of audio memory sequences. In line with neuroimaging studies carried out in the PFC, we propose a genuine coding strategy using the gain-modulation mechanism to represent abstract sequences based solely on the rank and location of items within them. Based on this mechanism, we show that we can construct a repertoire of neurons sensitive to the temporal structure in sequences from which we can represent any novel sequences. Free-energy optimization is then used to explore and to retrieve the missing indices of the items in the correct order for executive control and compositionality. We show that the gain-modulation mechanism permits the network to be robust to variabilities and to have long-term dependencies as it implements a gated recurrent neural network. This model, called Inferno Gate, is an extension of the neural architecture Inferno standing for Iterative Free-Energy Optimization of Recurrent Neural Networks with Gating or Gain-modulation. In experiments performed with an audio database of ten thousand MFCC vectors, Inferno Gate is capable of encoding efficiently and retrieving chunks of fifty items length. We then discuss the potential of our network to model the features of working memory in the PFC-BG loop for structural learning, goal-direction and hierarchical reinforcement learning.

Temporal-order information can be maintained in non-conscious working memory

Classical theories hold conscious perception and working memory to be tightly interwoven. Recent work has challenged this assumption, demonstrating that information may be stored for several seconds without any subjective awareness. Does such non-conscious working memory possess the same functional properties as conscious working memory? Here, we probe whether non-conscious working memory can maintain multiple items and their temporal order. In a visual masking task with a delayed response, 38 participants were asked to retain the location and order of presentation of two sequentially flashed spatial positions, and retrieve both after a 2.5 second delay. Even when subjective visibility was nil, subjects' objective forced-choice performance exceeded chance level and, crucially, distinct retrieval of the first and second location was observed on both conscious and non-conscious trials. Non-conscious working memory may therefore store two items in proper temporal order. These findings can be explained by recent models of activity-silent working memory.

The Mechanism of the Ordinal Position Effect: Stability Across Sense Modalities and the Hands Crossed Context

The ordinal position effect posits that items positioned earlier in an ordinal sequence are responded to faster with the left key than the right key, and items positioned later in an ordinal sequence are responded to faster with the right key than the left key. Although the mechanism of the ordinal position effect has been investigated in many studies, it is unclear whether the ordinal position effect can extend to the auditory modality and the hands crossed context. Therefore, the present study employed days as the order information to investigate this question. Days were visually or acoustically displayed on a screen in random order, and participants were instructed to judge whether the probe day they perceived was before or after the current day (days-relevant task) or to identify the color or voice of the probe day they perceived (days-irrelevant task). The results indicate the following: (a) The days before the current day were responded to faster with the left key than the right key, and the days after the current day were responded to faster with the right key than the left key, both when the days-relevant task and the days-irrelevant task were performed, regardless of the sense modality. (b) The ordinal position effect for judgments of days was also obtained in the auditory modality even when the hands were crossed. These results indicate that the ordinal position effect can extend to the auditory modality and the hands crossed context, similar to the spatial-numerical association of response codes effect of numbers.

Fast and flexible sequence induction in spiking neural networks via rapid excitability changes

Cognitive flexibility likely depends on modulation of the dynamics underlying how biological neural networks process information. While dynamics can be reshaped by gradually modifying connectivity, less is known about mechanisms operating on faster timescales. A compelling entrypoint to this problem is the observation that exploratory behaviors can rapidly cause selective hippocampal sequences to 'replay' during rest. Using a spiking network model, we asked whether simplified replay could arise from three biological components: fixed recurrent connectivity; stochastic 'gating' inputs; and rapid gating input scaling via long-term potentiation of intrinsic excitability (LTP-IE). Indeed, these enabled both forward and reverse replay of recent sensorimotor-evoked sequences, despite unchanged recurrent weights. LTP-IE 'tags' specific neurons with increased spiking probability under gating input, and ordering is reconstructed from recurrent connectivity. We further show how LTP-IE can implement temporary stimulus-response mappings. This elucidates a novel combination of mechanisms that might play a role in rapid cognitive flexibility.

Functionally distinct contributions of parietal cortex to a numerical landmark task: An fMRI study

This study aimed at establishing the neural basis of magnitude processing of multiple numbers from working memory. We designed a numerical landmark task and embedded it in a fragmented trial event-related fMRI design, allowing to separate encoding from decision processing. An attentional localiser task not involving numbers allowed further functional specification. The results show that in a numerical landmark task the right anterior intraparietal sulcus is involved in number encoding while more posterior parietal regions, bilateral superior parietal lobule and right inferior parietal lobule, provide domain-general support in the form of constructing a working memory representation or orienting spatial attention within that mental representation during number comparison. The results are in line with earlier studies reporting a functional distinction between anterior and posterior parietal contributions to number processing and further specify their role at a functional level.

Mechanism of the SNARC Effect in Numerical Magnitude, Time Sequence, and Spatial Sequence Tasks: Involvement of LTM and WM

The spatial-numerical association of response codes (SNARC) effect refers to the phenomenon that responses involving small numbers are faster with the left hand and responses involving large numbers are faster with the right hand. Previous studies have investigated the mechanism of the SNARC effect only when the time sequence is induced by centrally presented successive numbers. No study has investigated the mechanism of the SNARC effect when the spatial sequence is induced. Given that the induction of a spatial sequence together with a time sequence provides a new temporary reference for the serial order to be coded in working memory (WM), it would be interesting to examine the SNARC effect when both the time sequence and spatial sequence are induced. Therefore, a novel priming paradigm that simultaneously induced the time sequence and spatial sequence was employed in the present study to investigate the mechanism of the SNARC effect. Specifically, the time sequence and spatial sequence were induced by presenting a series of self-paced successive numbers, centrally or in a left-to-right or right-to-left direction, on the screen. Following the presentation of successive numbers, the probe number was centrally presented on the screen and university students were asked to distinguish to which time sequence or spatial sequence the probe number belonged by pressing a specified key of a qwerty keyboard. The results indicated that (1) the SNARC effect simultaneously appeared in the processing of the number magnitude and time sequence when only the time sequence was induced. (2) The SNARC effect disappeared in the processing of the time sequence; however, the SNARC effect coexisted in the processing of the numerical magnitude and spatial sequence when the spatial sequence was induced and participants performed a time sequence relevant task. (3) The SNARC effect coexisted in the processing of the numerical magnitude, time sequence, and spatial sequence when the spatial sequence was induced and participants performed a spatial sequence relevant task. Based on these results, we conclude that whether the SNARC effect coexists in the processing of the numerical magnitude, the time sequence and spatial sequence were influenced by the spatial sequence and relevant task. The results further support the mental whiteboard hypothesis and extended the WM account. Implications for theories on the SNARC effect were discussed.

Order Information in Verbal Working Memory Shifts the Subjective Midpoint in Both the Line Bisection and the Landmark Tasks

A largely substantiated view in the domain of working memory is that the maintenance of serial order is achieved by generating associations of each item with an independent representation of its position, so-called position markers. Recent studies reported that the ordinal position of an item in verbal working memory interacts with spatial processing. This suggests that position markers might be spatial in nature. However, these interactions were so far observed in tasks implying a clear binary categorization of space (i.e., with left and right responses or targets). Such binary categorizations leave room for alternative interpretations, such as congruency between non-spatial categorical codes for ordinal position (e.g., begin and end) and spatial categorical codes for response (e.g., left and right). Here we discard this interpretation by providing evidence that this interaction can also be observed in a task that draws upon a continuous processing of space, the line bisection task. Specifically, bisections are modulated by ordinal position in verbal working memory, with lines bisected more towards the right after retrieving items from the end compared to the beginning of the memorized sequence. This supports the idea that position markers are intrinsically spatial in nature.

Numerical Proportion Representation: A Neurocomputational Account

Proportion representation is an emerging subdomain in numerical cognition. However, its nature and its correlation with simple number representation remain elusive, especially at the theoretical level. To fill this gap, we propose a gain-field model of proportion representation to shed light on the neural and computational basis of proportion representation. The model is based on two well-supported neuroscientific findings. The first, gain modulation, is a general mechanism for information integration in the brain; the second relevant finding is how simple quantity is neurally represented. Based on these principles, the model accounts for recent relevant proportion representation data at both behavioral and neural levels. The model further addresses two key computational problems for the cognitive processing of proportions: invariance and generalization. Finally, the model provides pointers for future empirical testing.

A shared representation of order between encoding and recognition in visual short-term memory

Many complex tasks require people to bind individual events into a sequence that can be held in short term memory (STM). For this purpose information about the order of the individual events in the sequence needs to be maintained in an active and accessible form in STM over a period of few seconds. Here we investigated how the temporal order information is shared between the presentation and response phases of an STM task. We trained a classification algorithm on the fMRI activity patterns from the presentation phase of the STM task to predict the order of the items during the subsequent recognition phase. While voxels in a number of brain regions represented positional information during either presentation and recognition phases, only voxels in the lateral prefrontal cortex (PFC) and the anterior temporal lobe (ATL) represented position consistently across task phases. A shared positional code in the ATL might reflect verbal recoding of visual sequences to facilitate the maintenance of order information over several seconds.

Iterative free-energy optimization for recurrent neural networks (INFERNO)

The intra-parietal lobe coupled with the Basal Ganglia forms a working memory that demonstrates strong planning capabilities for generating robust yet flexible neuronal sequences. Neurocomputational models however, often fails to control long range neural synchrony in recurrent spiking networks due to spontaneous activity. As a novel framework based on the free-energy principle, we propose to see the problem of spikes' synchrony as an optimization problem of the neurons sub-threshold activity for the generation of long neuronal chains. Using a stochastic gradient descent, a reinforcement signal (presumably dopaminergic) evaluates the quality of one input vector to move the recurrent neural network to a desired activity; depending on the error made, this input vector is strengthened to hill-climb the gradient or elicited to search for another solution. This vector can be learned then by one associative memory as a model of the basal-ganglia to control the recurrent neural network. Experiments on habit learning and on sequence retrieving demonstrate the capabilities of the dual system to generate very long and precise spatio-temporal sequences, above two hundred iterations. Its features are applied then to the sequential planning of arm movements. In line with neurobiological theories, we discuss its relevance for modeling the cortico-basal working memory to initiate flexible goal-directed neuronal chains of causation and its relation to novel architectures such as Deep Networks, Neural Turing Machines and the Free-Energy Principle.

Domain-Generality of Timing-Based Serial Order Processes in Short-Term Memory: New Insights from Musical and Verbal Domains

Several models in the verbal domain of short-term memory (STM) consider a dissociation between item and order processing. This view is supported by data demonstrating that different types of time-based interference have a greater effect on memory for the order of to-be-remembered items than on memory for the items themselves. The present study investigated the domain-generality of the item versus serial order dissociation by comparing the differential effects of time-based interfering tasks, such as rhythmic interference and articulatory suppression, on item and order processing in verbal and musical STM domains. In Experiment 1, participants had to maintain sequences of verbal or musical information in STM, followed by a probe sequence, this under different conditions of interference (no-interference, rhythmic interference, articulatory suppression). They were required to decide whether all items of the probe list matched those of the memory list (item condition) or whether the order of the items in the probe sequence matched the order in the memory list (order condition). In Experiment 2, participants performed a serial order probe recognition task for verbal and musical sequences ensuring sequential maintenance processes, under no-interference or rhythmic interference conditions. For Experiment 1, serial order recognition was not significantly more impacted by interfering tasks than was item recognition, this for both verbal and musical domains. For Experiment 2, we observed selective interference of the rhythmic interference condition on both musical and verbal order STM tasks. Overall, the results suggest a similar and selective sensitivity to time-based interference for serial order STM in verbal and musical domains, but only when the STM tasks ensure sequential maintenance processes.

The neuroscience of working memory capacity and training

Working memory - the ability to maintain and manipulate information over a period of seconds - is a core component of higher cognitive functions. The storage capacity of working memory is limited but can be expanded by training, and evidence of the neural mechanisms underlying this effect is accumulating. Human imaging studies and neurophysiological recordings in non-human primates, together with computational modelling studies, reveal that training increases the activity of prefrontal neurons and the strength of connectivity in the prefrontal cortex and between the prefrontal and parietal cortex. Dopaminergic transmission could have a facilitatory role. These changes more generally inform us of the plasticity of higher cognitive functions.

Distinctiveness as a function of spatial expansion in verbal working memory: comment on Kreitz, Furley, Memmert, and Simons (2015)

In a recent study, Kreitz et al. (Psychological Research 79:1034-1041, 2015) reported on a relationship between verbal working memory capacity and visuo-spatial attentional breadth. The authors hinted at attentional control to be the major link underlying this relationship. We put forward an alternative explanation by framing it within the context of a recent theory on serial order in memory: verbal item sequences entering in working memory are coded by adding a spatial context that can be derived from reading/writing habits. The observation by Kreitz et al. (Psychological Research 79:1034-1041, 2015) enriches this framework by suggesting that a larger visuo-spatial attentional breadth allows for internal coding of the verbal items in a more (spatially) distinct manner-thereby increasing working memory performance. As such, Kreitz et al. (Psychological Research 79:1034-1041, 2015) is the first study revealing a functional link between visuo-spatial attentional breadth and verbal working memory size, which strengthens spatial accounts of serial order coding in working memory.

Modality independence of order coding in working memory: Evidence from cross-modal order interference at recall

Working memory researchers do not agree on whether order in serial recall is encoded by dedicated modality-specific systems or by a more general modality-independent system. Although previous research supports the existence of autonomous modality-specific systems, it has been shown that serial recognition memory is prone to cross-modal order interference by concurrent tasks. The present study used a serial recall task, which was performed in a single-task condition and in a dual-task condition with an embedded memory task in the retention interval. The modality of the serial task was either verbal or visuospatial, and the embedded tasks were in the other modality and required either serial or item recall. Care was taken to avoid modality overlaps during presentation and recall. In Experiment 1, visuospatial but not verbal serial recall was more impaired when the embedded task was an order than when it was an item task. Using a more difficult verbal serial recall task, verbal serial recall was also more impaired by another order recall task in Experiment 2. These findings are consistent with the hypothesis of modality-independent order coding. The implications for views on short-term recall and the multicomponent view of working memory are discussed.

Serial position markers in space: visuospatial priming of serial order working memory retrieval

Most general theories on serial order working memory (WM) assume the existence of position markers that are bound to the to-be-remembered items to keep track of the serial order. So far, the exact cognitive/neural characteristics of these markers have remained largely underspecified, while direct empirical evidence for their existence is mostly lacking. In the current study we demonstrate that retrieval from verbal serial order WM can be facilitated or hindered by spatial cuing: begin elements of a verbal WM sequence are retrieved faster after cuing the left side of space, while end elements are retrieved faster after cuing the right side of space. In direct complement to our previous work--where we showed the reversed impact of WM retrieval on spatial processing--we argue that the current findings provide us with a crucial piece of evidence suggesting a direct and functional involvement of space in verbal serial order WM. We outline the idea that serial order in verbal WM is coded within a spatial coordinate system with spatial attention being involved when searching through WM, and we discuss how this account can explain several hallmark observations related to serial order WM.

Finding the answer in space: the mental whiteboard hypothesis on serial order in working memory

Various prominent models on serial order coding in working memory (WM) build on the notion that serial order is achieved by binding the various items to-be-maintained to fixed position markers. Despite being relatively successful in accounting for empirical observations and some recent neuro-imaging support, these models were largely formulated on theoretical grounds and few specifications have been provided with respect to the cognitive and/or neural nature of these position markers. Here we outline a hypothesis on a novel candidate mechanism to substantiate the notion of serial position markers. Specifically, we propose that serial order WM is grounded in the spatial attention system: (I) The position markers that provide multi-item WM with a serial context should be understood as coordinates within an internal, spatially defined system; (II) internal spatial attention is involved in searching through the resulting serial order representation; and (III) retrieval corresponds to selection by spatial attention. We sketch the available empirical support and discuss how the hypothesis may provide a parsimonious framework from which to understand a broad range of observations across behavioral, neural and neuropsychological domains. Finally, we pinpoint what we believe are major questions for future research inspired by the hypothesis.