Resource allocation models of auditory working memory

Heterogeneous Forgetting Rates and Greedy Allocation in Slot-Based Memory Networks Promotes Signal Retention

A key question in the neuroscience of memory encoding pertains to the mechanisms by which afferent stimuli are allocated within memory networks. This issue is especially pronounced in the domain of working memory, where capacity is finite. Presumably the brain must embed some "policy" by which to allocate these mnemonic resources in an online manner in order to maximally represent and store afferent information for as long as possible and without interference from subsequent stimuli. Here, we engage this question through a top-down theoretical modeling framework. We formally optimize a gating mechanism that projects afferent stimuli onto a finite number of memory slots within a recurrent network architecture. In the absence of external input, the activity in each slot attenuates over time (i.e., a process of gradual forgetting). It turns out that the optimal gating policy consists of a direct projection from sensory activity to memory slots, alongside an activity-dependent lateral inhibition. Interestingly, allocating resources myopically (greedily with respect to the current stimulus) leads to efficient utilization of slots over time. In other words, later-arriving stimuli are distributed across slots in such a way that the network state is minimally shifted and so prior signals are minimally "overwritten." Further, networks with heterogeneity in the timescales of their forgetting rates retain stimuli better than those that are more homogeneous. Our results suggest how online, recurrent networks working on temporally localized objectives without high-level supervision can nonetheless implement efficient allocation of memory resources over time.

Not fully remembered, but not forgotten: interfering sounds worsen but do not eliminate the representation of pitch in working memory

Recent research has begun measuring auditory working memory with a continuous adjustment task in which listeners adjust attributes of a sound to match a stimulus presented earlier. This approach captures auditory memory's continuous nature better than standard change detection paradigms that collect binary ("same or different") memory measurements. In two experiments, we assessed the impact of different interference stimuli (multitone complexes vs. white noise vs. silence) on the precision and accuracy of participants' reproductions of pitch from memory. Participants were presented with a target multitone complex stimulus followed by eight successive interference signals. Across trials, these signals alternated between additional multitone complexes, randomly generated white noise samples, or (in Experiment 2) silence. This was followed by a response period where participants adjusted the pitch of a response stimulus using a MIDI touchpad to match the target. Experiment 1 found a significant effect of interference type on performance, with tone interference signals producing the greatest impairments to participants' accuracy and precision compared to white noise. Interestingly, it also found a compression in the participants' responses, with overestimations of low-frequency targets and underestimations for high-frequency targets. Experiment 2 replicated results from Experiment 1, with an additional silence condition showing the best performance, suggesting that non-tonal signals also generate interference. In general, results support a shared resource model of working memory with a limited capacity that can be flexibly allocated to hold items in memory with varying levels of fidelity. Interference does not appear to knock items out of a fixed precision slot, but rather robs a portion of capacity from stored items.

Neural dynamics underlying successful auditory short-term memory performance

Listeners often operate in complex acoustic environments, consisting of many concurrent sounds. Accurately encoding and maintaining such auditory objects in short-term memory is crucial for communication and scene analysis. Yet, the neural underpinnings of successful auditory short-term memory (ASTM) performance are currently not well understood. To elucidate this issue, we presented a novel, challenging auditory delayed match-to-sample task while recording MEG. Human participants listened to 'scenes' comprising three concurrent tone pip streams. The task was to indicate, after a delay, whether a probe stream was present in the just-heard scene. We present three key findings: First, behavioural performance revealed faster responses in correct versus incorrect trials as well as in 'probe present' versus 'probe absent' trials, consistent with ASTM search. Second, successful compared with unsuccessful ASTM performance was associated with a significant enhancement of event-related fields and oscillatory activity in the theta, alpha and beta frequency ranges. This extends previous findings of an overall increase of persistent activity during short-term memory performance. Third, using distributed source modelling, we found these effects to be confined mostly to sensory areas during encoding, presumably related to ASTM contents per se. Parietal and frontal sources then became relevant during the maintenance stage, indicating that effective STM operation also relies on ongoing inhibitory processes suppressing task-irrelevant information. In summary, our results deliver a detailed account of the neural patterns that differentiate successful from unsuccessful ASTM performance in the context of a complex, multi-object auditory scene.

Auditory short-term memory for pitch loses precision over time

The impact of retention interval duration on the fidelity of pitch memory was investigated. Listeners heard "target" pure tones, followed by a retention interval (2-8 s), then a response period in which the frequency of a novel sound was adjusted to match their memory of the target. The variability of pitch matches increased with retention interval duration. Supplemental analyses of the most accurate trials and temporal dynamics of matching suggest that decreasing precision was not due to differences in complete forgetting among intervals. Results suggest that the precision of short-term memory for pitch may continuously degrade over time.

Shared cognitive resources between memory and attention during sound-sequence encoding

You are on the phone, walking down a street. This daily situation calls for selective attention, allowing you to ignore surrounding irrelevant sounds, while trying to encode in memory the relevant information from the phone. Attention and memory are indeed two cognitive functions that are interacting constantly. However, their interaction is not yet well characterized during sound-sequence encoding. We independently manipulated both selective attention and working memory in a delayed-matching-to-sample of two tone-series, played successively in one ear. During the first melody presentation (memory encoding), weakly or highly distracting melodies were played in the other ear. Detection of the difference between the two comparison melodies could be easy or difficult, requiring low- or high-precision encoding, i.e., low or high memory load. Sixteen non-musician and 16 musician participants performed this new task. As expected, both groups of participants were less accurate in the difficult memory task and in difficult-to-ignore distractor conditions. Importantly, an interaction between memory-task difficulty and distractor difficulty was found in both groups. Non-musicians presented less difference between easy and difficult-to-ignore distractors in the difficult than in the easy memory task. On the contrary, musicians, with better performance than non-musicians, showed a greater difference between easy and difficult-to-ignore distractors in the difficult than in the easy memory task. In a second experiment including trials without a distractor, we could show that these effects are in line with the cognitive load theory. Taken together, these results speak for shared cognitive resources between working memory and attention during sound-sequence encoding.

Distractor probabilities modulate flanker task performance

Expectations about upcoming events help humans to effectively filter out potential distractors and respond more efficiently to task-relevant inputs. While previous work has emphasized the role of expectations about task-relevant inputs, less is known about the role that expectations play in suppressing specific distractors. To address this question, we manipulated the probabilities of different flanker configurations in the Eriksen flanker task. Across four studies, we found robust evidence for sensitivity to the probability of flankers, with an approximately logarithmic relationship between the likelihood of a particular flanker configuration and the accuracy of subjects' responses. Subjects were also sensitive to length of runs of repeated targets, but minimally sensitive to length of runs of repeated flankers. Two studies used chevron stimuli, and two used letters (confirming that results generalize with greater dissimilarity between stimuli). Expanding the set of stimuli (thus reducing the dominance of any one exemplar) eliminated the effect. Our findings suggest that expectations about distractors form in response to statistical regularities at multiple timescales, and that their effects are strongest when stimuli are geometrically similar and subjects are able to respond to trials quickly. Unexpected distractors could disrupt performance, most likely via a form of attentional capture. This work demonstrates how expectations can influence attention in complex cognitive settings, and illuminates the multiple, nested factors that contribute.

The perceived present: What is it, and what is it there for?

It is proposed that the perceived present is not a moment in time, but an information structure comprising an integrated set of products of perceptual processing. All information in the perceived present carries an informational time marker identifying it as "present". This marker is exclusive to information in the perceived present. There are other kinds of time markers, such as ordinality ("this stimulus occurred before that one") and duration ("this stimulus lasted for 50 ms"). These are different from the "present" time marker and may be attached to information regardless of whether it is in the perceived present or not. It is proposed that the perceived present is a very short-term and very high-capacity holding area for perceptual information. The maximum holding time for any given piece of information is ~100 ms: This is affected by the need to balance the value of informational persistence for further processing against the problem of obsolescence of the information. The main function of the perceived present is to facilitate access by other specialized, automatic processes.

Effects of white noise on word recall performance and brain activity in healthy adolescents with normal and low auditory working memory

The present study examined the impact of white noise on word recall performance and brain activity in 40 healthy adolescents, split in two groups (normal and low) depending on their auditory working memory capacity (AWMC). Using functional magnetic resonance imaging, participants performed a backward recall task under four different signal-to-noise ratio (SNR) conditions: 15, 10, 5, and 0-dB SNR. Behaviorally, normal AWMC individuals scored significantly higher than low AWMC individuals across noise levels. Whole-brain analyses showed brain activation not to be statistically different between groups across noise levels. In the normal group, a significant positive relationship was found between performance and number of activated voxels in the right superior frontal gyrus. In the low group, significant positive correlations were found between performance and number of activated voxels in left superior frontal gyrus, left inferior frontal gyrus, and left anterior cingulate cortex. These findings suggest that the strategic structure involved in the enhancement of AWM performance may differ in normal and low AWMC individuals.

Distinct role of flexible and stable encodings in sequential working memory

The serial-position effect in working memory is considered important for studying how a sequence of sensory information can be retained and manipulated simultaneously in neural memory circuits. Here, via a precise analysis of the primacy and recency effects in human psychophysical experiments, we propose that stable and flexible codings take distinct roles of retaining and updating information in working memory, and that their combination induces serial-position effects spontaneously. We found that stable encoding retains memory to induce the primacy effect, while flexible encoding used for learning new inputs induces the recency effect. A model simulation based on human data, confirmed that a neural network with both flexible and stable synapses could reproduce the major characteristics of serial-position effects. Our new prediction, that the control of resource allocation by flexible-stable coding balance can modulate memory performance in sequence-specific manner, was supported by pre-cued memory performance data in humans.

Working Memory in Alzheimer's Disease and Parkinson's Disease

Working memory impairments are frequently observed in patients with Alzheimer's disease (AD) and Parkinson's disease (PD). Recent research suggests that the mechanisms underlying these deficits might be dissociable using sensitive tasks, specifically those that rely on the reproduction of the exact quality of features held in memory.In patients with AD, working memory impairments are mainly due to an increase in misbinding errors. They arise when patients misremember which features (e.g., color, orientation, shape, and location) belong to different objects held in memory. Hence, they erroneously report features that belong to items in memory other than the one they are probed on. This misbinding of features that belong to different objects in memory can be considered a form of interference between stored items. Such binding errors are evident even in presymptomatic individuals with familial AD (due to gene mutations) who do not have AD yet. Overall, these findings are in line with the role of the medial temporal lobes, and specifically the hippocampus, in retention of feature bindings, regardless of retention duration, i.e., in both short- or long-term memory.Patients with PD, on the other hand, do not show increased misbinding. Their working memory deficits are associated with making more random errors or guesses. These random responses are not modulated by manipulations of their dopaminergic medication and hence may reflect involvement of non-dopaminergic neurotransmitters in this deficit. In addition, patients with PD demonstrate impairments in gating of information into relevant vs. irrelevant items in memory, a cognitive operation that is modulated by dopaminergic manipulation in line with a frontal executive effect of this neurotransmitter. Thus, although AD and PD are both associated with working memory impairments, these surface manifestations appear to be underpinned by very different mechanisms.

Temporal Expectation Modulates the Cortical Dynamics of Short-Term Memory

Increased memory load is often signified by enhanced neural oscillatory power in the alpha range (8-13 Hz), which is taken to reflect inhibition of task-irrelevant brain regions. The corresponding neural correlates of memory decay, however, are not yet well understood. In the current study, we investigated auditory short-term memory decay in humans using a delayed matching-to-sample task with pure-tone sequences. First, in a behavioral experiment, we modeled memory performance over six different delay-phase durations. Second, in a MEG experiment, we assessed alpha-power modulations over three different delay-phase durations. In both experiments, the temporal expectation for the to-be-remembered sound was manipulated so that it was either temporally expected or not. In both studies, memory performance declined over time, but this decline was weaker when the onset time of the to-be-remembered sound was expected. Similarly, patterns of alpha power in and alpha-tuned connectivity between sensory cortices changed parametrically with delay duration (i.e., decrease in occipitoparietal regions, increase in temporal regions). Temporal expectation not only counteracted alpha-power decline in heteromodal brain areas (i.e., supramarginal gyrus), but also had a beneficial effect on memory decay, counteracting memory performance decline. Correspondingly, temporal expectation also boosted alpha connectivity within attention networks known to play an active role during memory maintenance. The present data show how patterns of alpha power orchestrate short-term memory decay and encourage a more nuanced perspective on alpha power across brain space and time beyond its inhibitory role.SIGNIFICANCE STATEMENT Our sensory memories of the physical world fade quickly. We show here that this decay of short-term memory can be counteracted by so-called temporal expectation; that is, knowledge of when to expect a sensory event that an individual must remember. We also show that neural oscillations in the "alpha" (8-13 Hz) range index both the degree of memory decay (for brief sound patterns) and the respective memory benefit from temporal expectation. Spatially distributed cortical patterns of alpha power show opposing effects in auditory versus visual sensory cortices. Moreover, alpha-tuned connectivity changes within supramodal attention networks reflect the allocation of neural resources as short-term memory representations fade.

Musical and verbal short-term memory: insights from neurodevelopmental and neurological disorders

Auditory short-term memory (STM) is a fundamental ability to make sense of auditory information as it unfolds over time. Whether separate STM systems exist for different types of auditory information (music and speech, in particular) is a matter of debate. The present paper reviews studies that have investigated both musical and verbal STM in healthy individuals and in participants with neurodevelopmental and neurological disorders. Overall, the results are in favor of only partly shared networks for musical and verbal STM. Evidence for a distinction in STM for the two materials stems from (1) behavioral studies in healthy participants, in particular from the comparison between nonmusicians and musicians; (2) behavioral studies in congenital amusia, where a selective pitch STM deficit is observed; and (3) studies in brain-damaged patients with cases of double dissociation. In this review we highlight the need for future studies comparing STM for the same perceptual dimension (e.g., pitch) in different materials (e.g., music and speech), as well as for studies aiming at a more insightful characterization of shared and distinct mechanisms for speech and music in the different components of STM, namely encoding, retention, and retrieval.

Temporal precision and the capacity of auditory–verbal short-term memory

The capacity of serially ordered auditory–verbal short-term memory (AVSTM) is sensitive to the timing of the material to be stored, and both temporal processing and AVSTM capacity are implicated in the development of language. We developed a novel “rehearsal-probe” task to investigate the relationship between temporal precision and the capacity to remember serial order. Participants listened to a sub-span sequence of spoken digits and silently rehearsed the items and their timing during an unfilled retention interval. After an unpredictable delay, a tone prompted report of the item being rehearsed at that moment. An initial experiment showed cyclic distributions of item responses over time, with peaks preserving serial order and broad, overlapping tails. The spread of the response distributions increased with additional memory load and correlated negatively with participants’ auditory digit spans. A second study replicated the negative correlation and demonstrated its specificity to AVSTM by controlling for differences in visuo-spatial STM and nonverbal IQ. The results are consistent with the idea that a common resource underpins both the temporal precision and capacity of AVSTM. The rehearsal-probe task may provide a valuable tool for investigating links between temporal processing and AVSTM capacity in the context of speech and language abilities.

Temporal Processing in Audition: Insights from Music

•

What music psychology reveals about the natural bounds of human temporal processing.

•

Psychoacoustics of beat perception.

•

Neurophysiology of beat perception.

•

Predictable timing in auditory perception.

•

Neural mechanisms of timing.

Music is a curious example of a temporally patterned acoustic stimulus, and a compelling pan-cultural phenomenon. This review strives to bring some insights from decades of music psychology and sensorimotor synchronization (SMS) literature into the mainstream auditory domain, arguing that musical rhythm perception is shaped in important ways by temporal processing mechanisms in the brain. The feature that unites these disparate disciplines is an appreciation of the central importance of timing, sequencing, and anticipation. Perception of musical rhythms relies on an ability to form temporal predictions, a general feature of temporal processing that is equally relevant to auditory scene analysis, pattern detection, and speech perception. By bringing together findings from the music and auditory literature, we hope to inspire researchers to look beyond the conventions of their respective fields and consider the cross-disciplinary implications of studying auditory temporal sequence processing.

We begin by highlighting music as an interesting sound stimulus that may provide clues to how temporal patterning in sound drives perception. Next, we review the SMS literature and discuss possible neural substrates for the perception of, and synchronization to, musical beat. We then move away from music to explore the perceptual effects of rhythmic timing in pattern detection, auditory scene analysis, and speech perception. Finally, we review the neurophysiology of general timing processes that may underlie aspects of the perception of rhythmic patterns. We conclude with a brief summary and outlook for future research.

The Persistence of Memory: How the Brain Encodes Time in Memory

Time and memory are inextricably linked, but it is far from clear how event durations and temporal sequences are encoded in memory. In this review, we focus on resource allocation models of working memory which suggest that memory resources can be flexibly distributed amongst several items such that the precision of working memory decreases with the number of items to be encoded. This type of model is consistent with human performance in working memory tasks based on visual, auditory as well as temporal stimulus patterns. At the neural-network level, we focus on excitatory-inhibitory oscillatary processes that are able to encode both interval timing and working memory in a coupled excitatory-inhibitory network. This modification of the striatal beat-frequency model of interval timing shows how memories for multiple time intervals are represented by neural oscillations and can also be used to explain the mechanisms of resource allocation in working memory.

Short-term memory for spatial, sequential and duration information

•

Analog report methods provide novel insights on STM for space and time.

•

Space and time may be used to bind features in STM.

•

The hippocampus is involved in object-location binding in STM.

Space and time appear to play key roles in the way that information is organized in short-term memory (STM). Some argue that they are crucial contexts within which other stored features are embedded, allowing binding of information that belongs together within STM. Here we review recent behavioral, neurophysiological and imaging studies that have sought to investigate the nature of spatial, sequential and duration representations in STM, and how these might break down in disease. Findings from these studies point to an important role of the hippocampus and other medial temporal lobe structures in aspects of STM, challenging conventional accounts of involvement of these regions in only long-term memory.

Toward a taxonomic model of attention in effortful listening

In recent years, there has been increasing interest in studying listening effort. Research on listening effort intersects with the development of active theories of speech perception and contributes to the broader endeavor of understanding speech perception within the context of neuroscientific theories of perception, attention, and effort. Due to the multidisciplinary nature of the problem, researchers vary widely in their precise conceptualization of the catch-all term listening effort. Very recent consensus work stresses the relationship between listening effort and the allocation of cognitive resources, providing a conceptual link to current cognitive neuropsychological theories associating effort with the allocation of selective attention. By linking listening effort to attentional effort, we enable the application of a taxonomy of external and internal attention to the characterization of effortful listening. More specifically, we use a vectorial model to decompose the demand causing listening effort into its mutually orthogonal external and internal components and map the relationship between demanded and exerted effort by means of a resource-limiting term that can represent the influence of motivation as well as vigilance and arousal. Due to its quantitative nature and easy graphical interpretation, this model can be applied to a broad range of problems dealing with listening effort. As such, we conclude that the model provides a good starting point for further research on effortful listening within a more differentiated neuropsychological framework.

Working Memory for Sequences of Temporal Durations Reveals a Volatile Single-Item Store

When a sequence is held in working memory, different items are retained with differing fidelity. Here we ask whether a sequence of brief time intervals that must be remembered show recency effects, similar to those observed in verbal and visuospatial working memory. It has been suggested that prioritizing some items over others can be accounted for by a "focus of attention," maintaining some items in a privileged state. We therefore also investigated whether such benefits are vulnerable to disruption by attention or expectation. Participants listened to sequences of one to five tones, of varying durations (200 ms to 2 s). Subsequently, the length of one of the tones in the sequence had to be reproduced by holding a key. The discrepancy between the reproduced and actual durations quantified the fidelity of memory for auditory durations. Recall precision decreased with the number of items that had to be remembered, and was better for the first and last items of sequences, in line with set-size and serial position effects seen in other modalities. To test whether attentional filtering demands might impair performance, an irrelevant variation in pitch was introduced in some blocks of trials. In those blocks, memory precision was worse for sequences that consisted of only one item, i.e., the smallest memory set-size. Thus, when irrelevant information was present, the benefit of having only one item in memory is attenuated. Finally we examined whether expectation could interfere with memory. On half the trials, the number of items in the upcoming sequence was cued. When the number of items was known in advance, performance was paradoxically worse when the sequence consisted of only one item. Thus the benefit of having only one item to remember is stronger when it is unexpectedly the only item. Our results suggest that similar mechanisms are used to hold auditory time durations in working memory, as for visual or verbal stimuli. Further, solitary items were remembered better when more items were expected, but worse when irrelevant features were present. This suggests that the "privileged" state of one item in memory is particularly volatile and susceptible to interference.

Brain Bases of Working Memory for Time Intervals in Rhythmic Sequences

Perception of auditory time intervals is critical for accurate comprehension of natural sounds like speech and music. However, the neural substrates and mechanisms underlying the representation of time intervals in working memory are poorly understood. In this study, we investigate the brain bases of working memory for time intervals in rhythmic sequences using functional magnetic resonance imaging. We used a novel behavioral paradigm to investigate time-interval representation in working memory as a function of the temporal jitter and memory load of the sequences containing those time intervals. Human participants were presented with a sequence of intervals and required to reproduce the duration of a particular probed interval. We found that perceptual timing areas including the cerebellum and the striatum were more or less active as a function of increasing and decreasing jitter of the intervals held in working memory respectively whilst the activity of the inferior parietal cortex is modulated as a function of memory load. Additionally, we also analyzed structural correlations between gray and white matter density and behavior and found significant correlations in the cerebellum and the striatum, mirroring the functional results. Our data demonstrate neural substrates of working memory for time intervals and suggest that the cerebellum and the striatum represent core areas for representing temporal information in working memory.