Mental compression of spatial sequences in human working memory using numerical and geometrical primitives

Symbolic metaprogram search improves learning efficiency and explains rule learning in humans

Throughout their lives, humans seem to learn a variety of rules for things like applying category labels, following procedures, and explaining causal relationships. These rules are often algorithmically rich but are nonetheless acquired with minimal data and computation. Symbolic models based on program learning successfully explain rule-learning in many domains, but performance degrades quickly as program complexity increases. It remains unclear how to scale symbolic rule-learning methods to model human performance in challenging domains. Here we show that symbolic search over the space of metaprograms-programs that revise programs-dramatically improves learning efficiency. On a behavioral benchmark of 100 algorithmically rich rules, this approach fits human learning more accurately than alternative models while also using orders of magnitude less search. The computation required to match median human performance is consistent with conservative estimates of human thinking time. Our results suggest that metaprogram-like representations may help human learners to efficiently acquire rules.

Shared structure facilitates working memory of multiple sequences

Daily experiences often involve the processing of multiple sequences, yet storing them challenges the limited capacity of working memory (WM). To achieve efficient memory storage, relational structures shared by sequences would be leveraged to reorganize and compress information. Here, participants memorized a sequence of items with different colors and spatial locations and later reproduced the full color and location sequences one after another. Crucially, we manipulated the consistency between location and color sequence trajectories. First, sequences with consistent trajectories demonstrate improved memory performance and a trajectory correlation between reproduced color and location sequences. Second, sequences with consistent trajectories show neural reactivation of common trajectories, and display spontaneous replay of color sequences when recalling locations. Finally, neural reactivation correlates with WM behavior. Our findings suggest that a shared common structure is leveraged for the storage of multiple sequences through compressed encoding and neural replay, together facilitating efficient information organization in WM.

Working memory capacity for continuous events: The root of temporal compression in episodic memory?

Remembering the unfolding of past episodes usually takes less time than their actual duration. In this study, we evaluated whether such temporal compression emerges when continuous events are too long to be fully held in working memory. To do so, we asked 90 young adults to watch and mentally replay video clips showing people performing a continuous action (e.g., turning a car jack) that lasted 3, 6, 9, 12, or 15 s. For each clip, participants had to carefully watch the event and then to mentally replay it as accurately and precisely as possible. Results showed that mental replay durations increased with event duration but in a non-linear manner: they were close to the actual event duration for short videos (3-9 s), but significantly smaller for longer videos (12 and 15 s). These results suggest that working memory is temporally limited in its capacity to represent continuous events, which could in part explain why the unfolding of events is temporally compressed in episodic memory.

Space as a scaffold for rotational generalisation of abstract concepts

Learning invariances allows us to generalise. In the visual modality, invariant representations allow us to recognise objects despite translations or rotations in physical space. However, how we learn the invariances that allow us to generalise abstract patterns of sensory data ('concepts') is a longstanding puzzle. Here, we study how humans generalise relational patterns in stimulation sequences that are defined by either transitions on a nonspatial two-dimensional feature manifold, or by transitions in physical space. We measure rotational generalisation, i.e., the ability to recognise concepts even when their corresponding transition vectors are rotated. We find that humans naturally generalise to rotated exemplars when stimuli are defined in physical space, but not when they are defined as positions on a nonspatial feature manifold. However, if participants are first pre-trained to map auditory or visual features to spatial locations, then rotational generalisation becomes possible even in nonspatial domains. These results imply that space acts as a scaffold for learning more abstract conceptual invariances.

Tracking the neural codes for words and phrases during semantic composition, working-memory storage, and retrieval

The ability to compose successive words into a meaningful phrase is a characteristic feature of human cognition, yet its neural mechanisms remain incompletely understood. Here, we analyze the cortical mechanisms of semantic composition using magnetoencephalography (MEG) while participants read one-word, two-word, and five-word noun phrases and compared them with a subsequent image. Decoding of MEG signals revealed three processing stages. During phrase comprehension, the representation of individual words was sustained for a variable duration depending on phrasal context. During the delay period, the word code was replaced by a working-memory code whose activation increased with semantic complexity. Finally, the speed and accuracy of retrieval depended on semantic complexity and was faster for surface than for deep semantic properties. In conclusion, we propose that the brain initially encodes phrases using factorized dimensions for successive words but later compresses them in working memory and requires a period of decompression to access them.

Neural tracking of continuous acoustics: properties, speech-specificity and open questions

Human speech is a particularly relevant acoustic stimulus for our species, due to its role of information transmission during communication. Speech is inherently a dynamic signal, and a recent line of research focused on neural activity following the temporal structure of speech. We review findings that characterise neural dynamics in the processing of continuous acoustics and that allow us to compare these dynamics with temporal aspects in human speech. We highlight properties and constraints that both neural and speech dynamics have, suggesting that auditory neural systems are optimised to process human speech. We then discuss the speech-specificity of neural dynamics and their potential mechanistic origins and summarise open questions in the field.

Neurocognitive Assessment of Mathematics-Related Capacities in Neurosurgical Patients

A precise neuropsychological assessment is of the utmost importance for neurosurgical patients undergoing the surgical excision of cerebral lesions. The assessment of mathematical abilities is usually limited to arithmetical operations while other fundamental visuo-spatial aspects closely linked to mathematics proficiency, such as the perception of numerical quantities and geometrical reasoning, are completely neglected. We evaluated these abilities with two objective and reproducible psychophysical tests, measuring numerosity perception and non-symbolic geometry, respectively. We tested sixteen neuro-oncological patients before the operation and six after the operation with classical neuropsychological tests and with two psychophysical tests. The scores of the classical neuropsychological tests were very heterogeneous, possibly due to the distinct location and histology of the tumors that might have spared (or not) brain areas subserving these abilities or allowed for plastic reorganization. Performance in the two non-symbolic tests reflected, on average, the presumed functional role of the lesioned areas, with participants with parietal and frontal lesions performing worse on these tests than patients with occipital and temporal lesions. Single-case analyses not only revealed some interesting exceptions to the group-level results (e.g., patients with parietal lesions performing well in the numerosity test), but also indicated that performance in the two tests was independent of non-verbal reasoning and visuo-spatial working memory. Our results highlight the importance of assessing non-symbolic numerical and geometrical abilities to complement typical neuropsychological batteries. However, they also suggest an avoidance of reliance on an excessively rigid localizationist approach when evaluating the neuropsychological profile of oncological patients.

Brain-imaging evidence for compression of binary sound sequences in human memory

According to the language-of-thought hypothesis, regular sequences are compressed in human memory using recursive loops akin to a mental program that predicts future items. We tested this theory by probing memory for 16-item sequences made of two sounds. We recorded brain activity with functional MRI and magneto-encephalography (MEG) while participants listened to a hierarchy of sequences of variable complexity, whose minimal description required transition probabilities, chunking, or nested structures. Occasional deviant sounds probed the participants' knowledge of the sequence. We predicted that task difficulty and brain activity would be proportional to the complexity derived from the minimal description length in our formal language. Furthermore, activity should increase with complexity for learned sequences, and decrease with complexity for deviants. These predictions were upheld in both fMRI and MEG, indicating that sequence predictions are highly dependent on sequence structure and become weaker and delayed as complexity increases. The proposed language recruited bilateral superior temporal, precentral, anterior intraparietal, and cerebellar cortices. These regions overlapped extensively with a localizer for mathematical calculation, and much less with spoken or written language processing. We propose that these areas collectively encode regular sequences as repetitions with variations and their recursive composition into nested structures.

Can compression take place in working memory without a central contribution of long-term memory?

Information is easier to remember when it is recognized as structured. One explanation for this benefit is that people represent structured information in a compressed form, thus reducing memory load. However, the contribution of long-term memory and working memory to compression are not yet disentangled. Previous work has mostly produced evidence that long-term memory is the main source of compression. In the present work, we reveal two signatures of compression in working memory using a large-scale naturalistic data set from a science museum. Analyzing data from more than 32,000 memory trials, in which people attempted to recall briefly displayed sequences of colors, we examined how the estimated compressibility of each sequence predicted memory performance. Besides finding that compressibility predicted memory performance, we found that greater compressibility of early subsections of sequences predicted better memory for later subsections, and that mis-recalled sequences were simpler than the originals. These findings suggest that (1) more compressibility reduces memory load, leaving space for additional information; (2) memory errors are not random and instead reflect compression gone awry. Together, these findings suggest that compression can take place in working memory. This may enable efficient storage on the spot without direct contributions from long-term memory. However, we also discuss ways long-term memory could explain our findings.

Natural logic and baby LoTH

Language-of-thought hypothesis (LoTH) is having a profound impact on cognition studies. However, much remains unknown about its basic primitives and generative operations. Infant studies are fundamental, but methodologically very challenging. By distilling potential primitives from work in natural-language semantics, an approach beyond the corset of standard formal logic may be undertaken. Still, the road ahead is challenging and long.

Dimensionality and Ramping: Signatures of Sentence Integration in the Dynamics of Brains and Deep Language Models

A sentence is more than the sum of its words: its meaning depends on how they combine with one another. The brain mechanisms underlying such semantic composition remain poorly understood. To shed light on the neural vector code underlying semantic composition, we introduce two hypotheses: (1) the intrinsic dimensionality of the space of neural representations should increase as a sentence unfolds, paralleling the growing complexity of its semantic representation; and (2) this progressive integration should be reflected in ramping and sentence-final signals. To test these predictions, we designed a dataset of closely matched normal and jabberwocky sentences (composed of meaningless pseudo words) and displayed them to deep language models and to 11 human participants (5 men and 6 women) monitored with simultaneous MEG and intracranial EEG. In both deep language models and electrophysiological data, we found that representational dimensionality was higher for meaningful sentences than jabberwocky. Furthermore, multivariate decoding of normal versus jabberwocky confirmed three dynamic patterns: (1) a phasic pattern following each word, peaking in temporal and parietal areas; (2) a ramping pattern, characteristic of bilateral inferior and middle frontal gyri; and (3) a sentence-final pattern in left superior frontal gyrus and right orbitofrontal cortex. These results provide a first glimpse into the neural geometry of semantic integration and constrain the search for a neural code of linguistic composition.SIGNIFICANCE STATEMENT Starting from general linguistic concepts, we make two sets of predictions in neural signals evoked by reading multiword sentences. First, the intrinsic dimensionality of the representation should grow with additional meaningful words. Second, the neural dynamics should exhibit signatures of encoding, maintaining, and resolving semantic composition. We successfully validated these hypotheses in deep neural language models, artificial neural networks trained on text and performing very well on many natural language processing tasks. Then, using a unique combination of MEG and intracranial electrodes, we recorded high-resolution brain data from human participants while they read a controlled set of sentences. Time-resolved dimensionality analysis showed increasing dimensionality with meaning, and multivariate decoding allowed us to isolate the three dynamical patterns we had hypothesized.

Spontaneous supra-modal encoding of number in the infant brain

The core knowledge hypothesis postulates that infants automatically analyze their environment along abstract dimensions, including numbers. According to this view, approximate numbers should be encoded quickly, pre-attentively, and in a supra-modal manner by the infant brain. Here, we directly tested this idea by submitting the neural responses of sleeping 3-month-old infants, measured with high-density electroencephalography (EEG), to decoders designed to disentangle numerical and non-numerical information. The results show the emergence, in approximately 400 ms, of a decodable number representation, independent of physical parameters, that separates auditory sequences of 4 vs. 12 tones and generalizes to visual arrays of 4 vs. 12 objects. Thus, the infant brain contains a number code that transcends sensory modality, sequential or simultaneous presentation, and arousal state.

Continual task learning in natural and artificial agents

How do humans and other animals learn new tasks? A wave of brain recording studies has investigated how neural representations change during task learning, with a focus on how tasks can be acquired and coded in ways that minimise mutual interference. We review recent work that has explored the geometry and dimensionality of neural task representations in neocortex, and computational models that have exploited these findings to understand how the brain may partition knowledge between tasks. We discuss how ideas from machine learning, including those that combine supervised and unsupervised learning, are helping neuroscientists understand how natural tasks are learned and coded in biological brains.

The best game in town: The reemergence of the language-of-thought hypothesis across the cognitive sciences

Mental representations remain the central posits of psychology after many decades of scrutiny. However, there is no consensus about the representational format(s) of biological cognition. This paper provides a survey of evidence from computational cognitive psychology, perceptual psychology, developmental psychology, comparative psychology, and social psychology, and concludes that one type of format that routinely crops up is the language-of-thought (LoT). We outline six core properties of LoTs: (i) discrete constituents; (ii) role-filler independence; (iii) predicate-argument structure; (iv) logical operators; (v) inferential promiscuity; and (vi) abstract content. These properties cluster together throughout cognitive science. Bayesian computational modeling, compositional features of object perception, complex infant and animal reasoning, and automatic, intuitive cognition in adults all implicate LoT-like structures. Instead of regarding LoT as a relic of the previous century, researchers in cognitive science and philosophy-of-mind must take seriously the explanatory breadth of LoT-based architectures. We grant that the mind may harbor many formats and architectures, including iconic and associative structures as well as deep-neural-network-like architectures. However, as computational/representational approaches to the mind continue to advance, classical compositional symbolic structures - that is, LoTs - only prove more flexible and well-supported over time.

Reactivating ordinal position information from auditory sequence memory in human brains

Retaining a sequence of events in their order is a core ability of many cognitive functions, such as speech recognition, movement control, and episodic memory. Although content representations have been widely studied in working memory (WM), little is known about how ordinal position information of an auditory sequence is retained in the human brain as well as its coding characteristics. In fact, there is still a lack of an efficient approach to directly accessing the stored ordinal position code during WM retention. Here, 31 participants performed an auditory sequence WM task with their brain activities recorded using electroencephalography (EEG). We developed new triggering events that could successfully reactivate neural representations of ordinal position during the delay period. Importantly, the ordinal position reactivation is further related to recognition behavior, confirming its indexing of WM storage. Furthermore, the ordinal position code displays an intriguing “stable-dynamic” format, i.e. undergoing the same dynamic neutral trajectory in the multivariate neural space during both encoding and retention (whenever reactivated). Overall, our results provide an effective approach to accessing the behaviorally-relevant ordinal position information in auditory sequence WM and reveal its new temporal characteristics.

A language of thought for the mental representation of geometric shapes

In various cultures and at all spatial scales, humans produce a rich complexity of geometric shapes such as lines, circles or spirals. Here, we propose that humans possess a language of thought for geometric shapes that can produce line drawings as recursive combinations of a minimal set of geometric primitives. We present a programming language, similar to Logo, that combines discrete numbers and continuous integration to form higher-level structures based on repetition, concatenation and embedding, and we show that the simplest programs in this language generate the fundamental geometric shapes observed in human cultures. On the perceptual side, we propose that shape perception in humans involves searching for the shortest program that correctly draws the image (program induction). A consequence of this framework is that the mental difficulty of remembering a shape should depend on its minimum description length (MDL) in the proposed language. In two experiments, we show that encoding and processing of geometric shapes is well predicted by MDL. Furthermore, our hypotheses predict additive laws for the psychological complexity of repeated, concatenated or embedded shapes, which we confirm experimentally.

Problems and Mysteries of the Many Languages of Thought

"What is the structure of thought?" is as central a question as any in cognitive science. A classic answer to this question has appealed to a Language of Thought (LoT). We point to emerging research from disparate branches of the field that supports the LoT hypothesis, but also uncovers diversity in LoTs across cognitive systems, stages of development, and species. Our letter formulates open research questions for cognitive science concerning the varieties of rules and representations that underwrite various LoT-based systems and how these variations can help researchers taxonomize cognitive systems.

Functional neuroimaging in psychiatry and the case for failing better

Psychiatric disorders encompass complex aberrations of cognition and affect and are among the most debilitating and poorly understood of any medical condition. Current treatments rely primarily on interventions that target brain function (drugs) or learning processes (psychotherapy). A mechanistic understanding of how these interventions mediate their therapeutic effects remains elusive. From the early 1990s, non-invasive functional neuroimaging, coupled with parallel developments in the cognitive neurosciences, seemed to signal a new era of neurobiologically grounded diagnosis and treatment in psychiatry. Yet, despite three decades of intense neuroimaging research, we still lack a neurobiological account for any psychiatric condition. Likewise, functional neuroimaging plays no role in clinical decision making. Here, we offer a critical commentary on this impasse and suggest how the field might fare better and deliver impactful neurobiological insights.

Symbols and mental programs: a hypothesis about human singularity

Natural language is often seen as the single factor that explains the cognitive singularity of the human species. Instead, we propose that humans possess multiple internal languages of thought, akin to computer languages, which encode and compress structures in various domains (mathematics, music, shape…). These languages rely on cortical circuits distinct from classical language areas. Each is characterized by: (i) the discretization of a domain using a small set of symbols, and (ii) their recursive composition into mental programs that encode nested repetitions with variations. In various tasks of elementary shape or sequence perception, minimum description length in the proposed languages captures human behavior and brain activity, whereas non-human primate data are captured by simpler nonsymbolic models. Our research argues in favor of discrete symbolic models of human thought.

Decoding cognition from spontaneous neural activity

In human neuroscience, studies of cognition are rarely grounded in non-task-evoked, 'spontaneous' neural activity. Indeed, studies of spontaneous activity tend to focus predominantly on intrinsic neural patterns (for example, resting-state networks). Taking a 'representation-rich' approach bridges the gap between cognition and resting-state communities: this approach relies on decoding task-related representations from spontaneous neural activity, allowing quantification of the representational content and rich dynamics of such activity. For example, if we know the neural representation of an episodic memory, we can decode its subsequent replay during rest. We argue that such an approach advances cognitive research beyond a focus on immediate task demand and provides insight into the functional relevance of the intrinsic neural pattern (for example, the default mode network). This in turn enables a greater integration between human and animal neuroscience, facilitating experimental testing of theoretical accounts of intrinsic activity, and opening new avenues of research in psychiatry.

Working Memory for Spatial Sequences: Developmental and Evolutionary Factors in Encoding Ordinal and Relational Structures

Sequence learning is a ubiquitous facet of human and animal cognition. Here, using a common sequence reproduction task, we investigated whether and how the ordinal and relational structures linking consecutive elements are acquired by human adults, children, and macaque monkeys. While children and monkeys exhibited significantly lower precision than adults for spatial location and temporal order information, only monkeys appeared to exceedingly focus on the first item. Most importantly, only humans, regardless of age, spontaneously extracted the spatial relations between consecutive items and used a chunking strategy to compress sequences in working memory. Monkeys did not detect such relational structures, even after extensive training. Monkey behavior was captured by a conjunctive coding model, whereas a chunk-based conjunctive model explained more variance in humans. These age- and species-related differences are indicative of developmental and evolutionary mechanisms of sequence encoding and may provide novel insights into the uniquely human cognitive capacities.SIGNIFICANCE STATEMENT Sequence learning, the ability to encode the order of discrete elements and their relationships presented within a sequence, is a ubiquitous facet of cognition among humans and animals. By exploring sequence-processing abilities at different human developmental stages and in nonhuman primates, we found that only humans, regardless of age, spontaneously extracted the spatial relations between consecutive items and used an internal language to compress sequences in working memory. The findings provided insights into understanding the origins of sequence capabilities in humans and how they evolve through development to identify the unique aspects of human cognitive capacity, which includes the comprehension, learning, and production of sequences, and perhaps, above all, language processing.