A shared representation of order between encoding and recognition in visual short-term memory

Reactivating ordinal position information from auditory sequence memory in human brains

Retaining a sequence of events in their order is a core ability of many cognitive functions, such as speech recognition, movement control, and episodic memory. Although content representations have been widely studied in working memory (WM), little is known about how ordinal position information of an auditory sequence is retained in the human brain as well as its coding characteristics. In fact, there is still a lack of an efficient approach to directly accessing the stored ordinal position code during WM retention. Here, 31 participants performed an auditory sequence WM task with their brain activities recorded using electroencephalography (EEG). We developed new triggering events that could successfully reactivate neural representations of ordinal position during the delay period. Importantly, the ordinal position reactivation is further related to recognition behavior, confirming its indexing of WM storage. Furthermore, the ordinal position code displays an intriguing “stable-dynamic” format, i.e. undergoing the same dynamic neutral trajectory in the multivariate neural space during both encoding and retention (whenever reactivated). Overall, our results provide an effective approach to accessing the behaviorally-relevant ordinal position information in auditory sequence WM and reveal its new temporal characteristics.

The extended present: an informational context for perception

Several previous authors have proposed a kind of specious or subjective present moment that covers a few seconds of recent information. This article proposes a new hypothesis about the subjective present, renamed the extended present, defined not in terms of time covered but as a thematically connected information structure held in working memory and in transiently accessible form in long-term memory. The three key features of the extended present are that information in it is thematically connected, both internally and to current attended perceptual input, it is organised in a hierarchical structure, and all information in it is marked with temporal information, specifically ordinal and duration information. Temporal boundaries to the information structure are determined by hierarchical structure processing and by limits on processing and storage capacity. Supporting evidence for the importance of hierarchical structure analysis is found in the domains of music perception, speech and language processing, perception and production of goal-directed action, and exact arithmetical calculation. Temporal information marking is also discussed and a possible mechanism for representing ordinal and duration information on the time scale of the extended present is proposed. It is hypothesised that the extended present functions primarily as an informational context for making sense of current perceptual input, and as an enabler for perception and generation of complex structures and operations in language, action, music, exact calculation, and other domains.

Distinct Neural Representations of Content and Ordinal Structure in Auditory Sequence Memory

Two forms of information, frequency (content) and ordinal position (structure), have to be stored when retaining a sequence of auditory tones in working memory (WM). However, the neural representations and coding characteristics of content and structure, particularly during WM maintenance, remain elusive. Here, in two EEG studies in human participants (both sexes), by transiently perturbing the "activity-silent" WM retention state and decoding the reactivated WM information, we demonstrate that content and structure are stored in a dissociative manner with distinct characteristics throughout WM process. First, each tone in the sequence is associated with two codes in parallel, characterizing its frequency and ordinal position, respectively. Second, during retention, a structural retrocue successfully reactivates structure but not content, whereas a following white noise triggers content but not structure. Third, structure representation remains stable, whereas content code undergoes a dynamic transformation through memory progress. Finally, the noise-triggered content reactivations during retention correlate with subsequent WM behavior. Overall, our results support distinct content and structure representations in auditory WM and provide an efficient approach to access the silently stored WM information in the human brain. The dissociation of content and structure could facilitate efficient memory formation via generalizing stable structure to new auditory contents.SIGNIFICANCE STATEMENT In memory experiences, contents do not exist independently but are linked with each other via ordinal structure. For instance, recalling a piece of favorite music relies on correct ordering (sequence structure) of musical tones (content). How are the structure and content for an auditory temporally structured experience maintained in working memory? Here, by using impulse-response approach and time-resolved representational dissimilarity analysis on human EEG recordings in an auditory working memory task, we reveal that content and structure are stored in a dissociated way, which would facilitate efficient and rapid memory formation through generalizing stable structure knowledge to new auditory inputs.

Neural Representations of Task Context and Temporal Order During Action Sequence Execution

Routine action sequences can share a great deal of similarity in terms of their stimulus response mappings. As a consequence, their correct execution relies crucially on the ability to preserve contextual and temporal information. However, there are few empirical studies on the neural mechanism and the brain areas maintaining such information. To address this gap in the literature, we recently recorded the blood-oxygen level dependent (BOLD) response in a newly developed coffee-tea making task. The task involves the execution of four action sequences that each comprise six consecutive decision states, which allows for examining the maintenance of contextual and temporal information. Here, we report a reanalysis of this dataset using a data-driven approach, namely multivariate pattern analysis, that examines context-dependent neural activity across several predefined regions of interest. Results highlight involvement of the inferior-temporal gyrus and lateral prefrontal cortex in maintaining temporal and contextual information for the execution of hierarchically organized action sequences. Furthermore, temporal information seems to be more strongly encoded in areas over the left hemisphere.

Age differences in the fronto-striato-parietal network underlying serial ordering

Maintaining the ability to arrange thoughts and actions in an appropriate serial order is crucial for complex behavior. We aimed to investigate age differences in the fronto-striato-parietal network underlying serial ordering using functional magnetic resonance imaging. We exposed 25 young and 27 older healthy adults to a digit ordering task, where they had to reorder and recall sequential digits or simply to recall them. We detected a network comprising of the lateral and medial prefrontal, posterior parietal, and striatal regions. In young adults, the prefrontal and parietal regions were more activated and more strongly connected with the supplementary motor area for "reorder & recall" than "pure recall" trials (psychophysiological interaction, PPI). In older adults, the prefrontal and parietal activations were elevated, but the PPI was attenuated. Individual adults who had a stronger PPI performed more accurately in "reorder & recall" trials. The decreased PPI appeared to be compensated by increased physiological correlations between the prefrontal/parietal cortex and the striatum, and by that between the striatum and the supplementary motor area.

Even an activated long-term memory system still needs a separate short-term store: A reply to Cowan (2019)

In Norris (2017), I explained why the notion of activated LTM (long-term memory) combined with a focus of attention was unable to perform the computations required to support short-term memory (STM) and argued that those extra computations must require a separate STM system. Cowan (2019) made the alternative proposal that this full set of computations is better conceptualized as a unitary system of activated LTM. To this he added a pointer system, the ability to perform variable binding, and an unspecified model of STM that acts as a front end to LTM. This appears to be simply an exercise in relabeling. Furthermore, without a computational specification of how the components work, the model lacks the ability to simulate even the most basic STM phenomena. If the model were specified in more detail it seems almost inevitable that it would contain something instantly recognizable as an STM system. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Short-term memory based on activated long-term memory: A review in response to Norris (2017)

Short-term memory (STM), the limited information temporarily in a state of heightened accessibility, includes just-presented events and recently retrieved information. Norris (2017) argued for a prominent class of theories in which STM depends on the brain keeping a separate copy of new information, and against alternatives in which the information is held only in a portion of long-term memory (LTM) that is currently activated (aLTM). Here I question premises of Norris' case for separate-copy theories in the following ways. (a) He did not allow for implications of the common assumption (e.g., Cowan, 1999; Cowan & Chen, 2009) that aLTM can include new, rapidly formed LTM records of a trial within an STM task. (b) His conclusions from pathological cases of impaired STM along with intact LTM are tenuous; these rare cases can be explained by impairments in encoding, processing, or retrieval related to LTM rather than passive maintenance. (c) Although Norris reasonably allowed structured pointers to aLTM instead of separate copies of the actual item representations in STM, the same structured pointers may well be involved in long-term learning. (d) Last, models of STM storage can serve as the front end of an LTM learning system rather than being separate. I summarize evidence for these premises and an updated version of an alternative theory in which storage depends on aLTM (newly clarified), and, embedded within it, information enhanced by the current focus of attention (Cowan, 1988, 1999), with no need for a separate STM copy. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Decoding hierarchical control of sequential behavior in oscillatory EEG activity

Despite strong theoretical reasons for assuming that abstract representations organize complex action sequences in terms of subplans (chunks) and sequential positions, we lack methods to directly track such content-independent, hierarchical representations in humans. We applied time-resolved, multivariate decoding analysis to the pattern of rhythmic EEG activity that was registered while participants planned and executed individual elements from pre-learned, structured sequences. Across three experiments, the theta and alpha-band activity coded basic elements and abstract control representations, in particular, the ordinal position of basic elements, but also the identity and position of chunks. Further, a robust representation of higher level, chunk identity information was only found in individuals with above-median working memory capacity, potentially providing a neural-level explanation for working-memory differences in sequential performance. Our results suggest that by decoding oscillatory activity we can track how the cognitive system traverses through the states of a hierarchical control structure.

The Ebb and Flow of Experience Determines the Temporal Structure of Memory

Everyday life consists of a continuous stream of information, yet somehow we remember the past as distinct episodic events. Prominent models posit that event segmentation is driven by erroneous predictions about how current experiences are unfolding. Yet this perspective fails to explain how memories become integrated or separated in the absence of prior knowledge. Here, we propose that contextual stability dictates the temporal organization of events in episodic memory. To support this view, we summarize new findings showing that neural measures of event organization index how ongoing changes in external contextual cues and internal representations of time influence different forms of episodic memory.

Reading positional codes with fMRI: Problems and solutions

Neural mechanisms which bind items into sequences have been investigated in a large body of research in animal neurophysiology and human neuroimaging. However, a major problem in interpreting this data arises from a fact that several unrelated processes, such as memory load, sensory adaptation, and reward expectation, also change in a consistent manner as the sequence unfolds. In this paper we use computational simulations and data from two fMRI experiments to show that a host of unrelated neural processes can masquerade as sequence representations. We show that dissociating such unrelated processes from a dedicated sequence representation is an especially difficult problem for fMRI data, which is almost exclusively the modality used in human experiments. We suggest that such fMRI results must be treated with caution and in many cases the assumed neural representation might actually reflect unrelated processes.