Working memory for time intervals in auditory rhythmic sequences

Working memory for time intervals: Another manifestation of the central tendency effect

The relationship between working memory and time perception has been typically investigated using dual-task paradigms (e.g., testing timing performance during a concurrent task). To our knowledge, none of these studies used time intervals as the target stimulus to be remembered. The current study investigated the working memory for time intervals by asking participants to reproduce durations they experienced at different orders in a series of experienced intervals (n-back task). One of the experiments was conducted online and the other one in the lab setting. Results showed a central tendency bias and additive elongation of time reproductions with increasing working memory load. Our results also showed that participants assigned different weights to experienced intervals based on their order of presentation (higher weight to the target interval). We conclude that the recall of intervals from working memory under high cognitive load leads to a central tendency effect, which is known to be induced by the temporal context and present particularly in aging and in those with Parkinson's disease.

Sensorimotor and working memory systems jointly support development of perceptual rhythm processing

We studied the role of sensorimotor and working memory systems in supporting development of perceptual rhythm processing with 119 participants aged 7-12 years. Children were assessed for their abilities in sensorimotor synchronization (SMS; beat tapping), auditory working memory (AWM; digit span), and rhythm discrimination (RD; same/different judgment on a pair of musical rhythm sequences). Multiple regression analysis revealed that children's RD performance was independently predicted by higher beat tapping consistency and greater digit span score, with all other demographic variables (age, sex, socioeconomic status, music training) controlled. The association between RD and SMS was more robust in the slower tempos (60 and 100 beats-per-minute (BPM)) than faster ones (120 and 180 BPM). Critically, the relation of SMS to RD was moderated by age in that RD performance was predicted by beat tapping consistency in younger children (age: 7-9 years), but not in older children (age: 10-12 years). AWM was the only predictor of RD in older children. Together, the current findings demonstrate that the sensorimotor and working memory systems jointly support RD processing during middle-to-late childhood and that the degree of association between the two systems and perceptual rhythm processing is shifted before entering into early adolescence.

Exposure to multisensory and visual static or moving stimuli enhances processing of nonoptimal visual rhythms

Research has shown that visual moving and multisensory stimuli can efficiently mediate rhythmic information. It is possible, therefore, that the previously reported auditory dominance in rhythm perception is due to the use of nonoptimal visual stimuli. Yet it remains unknown whether exposure to multisensory or visual-moving rhythms would benefit the processing of rhythms consisting of nonoptimal static visual stimuli. Using a perceptual learning paradigm, we tested whether the visual component of the multisensory training pair can affect processing of metric simple two integer-ratio nonoptimal visual rhythms. Participants were trained with static (AVstat), moving-inanimate (AVinan), or moving-animate (AVan) visual stimuli along with auditory tones and a regular beat. In the pre- and posttraining tasks, participants responded whether two static-visual rhythms differed or not. Results showed improved posttraining performance for all training groups irrespective of the type of visual stimulation. To assess whether this benefit was auditory driven, we introduced visual-only training with a moving or static stimulus and a regular beat (Vinan). Comparisons between Vinan and Vstat showed that, even in the absence of auditory information, training with visual-only moving or static stimuli resulted in an enhanced posttraining performance. Overall, our findings suggest that audiovisual and visual static or moving training can benefit processing of nonoptimal visual rhythms.

Tapping to hip-hop: Effects of cognitive load, arousal, and musical meter on time experiences

Experiences of time vary intra- and interindividually, depending on factors such as attentional resource allocation and arousal. Music as a temporal art that is structured by multiple temporal layers is ideal for investigating human time experiences. The current study used examples of hip-hop music that varied in arousal but were constant in tempo. Participants judged the passage of time to be quicker when cognitive load was high in a dual-task condition, and perceived duration to be shorter when performing a concurrent motor task (tapping along with the music). Perceived musical arousal did not affect subjective time. Attending to a higher metrical level by tapping with half notes resulted in shorter duration estimates and a quicker passage of time, compared to tapping with eighth notes of the same music. Results were not influenced by spontaneous motor tempo, musical expertise, preference or familiarity with the music. Taken together, these findings indicate consistent effects of cognitive load and attention to meter on time experiences.

Disentangling the effects of modality, interval length and task difficulty on the accuracy and precision of older adults in a rhythmic reproduction task

Studies on the functional quality of the internal clock that governs the temporal processing of older adults have demonstrated mixed results as to whether they perceive and produce time slower, faster, or equally well as younger adults. These mixed results are due to a multitude of methodologies applied to study temporal processing: many tasks demand different levels of cognitive ability. To investigate the temporal accuracy and precision of older adults, in Experiment 1, we explored the age-related differences in rhythmic continuation task taking into consideration the effects of attentional resources required by the stimulus (auditory vs. visual; length of intervals). In Experiment 2, we added a dual task to explore the effect of attentional resources required by the task. Our findings indicate that (1) even in an inherently automatic rhythmic task, where older and younger adult’s general accuracy is comparable, accuracy but not precision is altered by the stimulus properties and (2) an increase in task load can magnify age-related differences in both accuracy and precision.

Gradual decay and sudden death of short-term memory for pitch

The ability to discriminate frequency differences between pure tones declines as the duration of the interstimulus interval (ISI) increases. The conventional explanation for this finding is that pitch representations gradually decay from auditory short-term memory. Gradual decay means that internal noise increases with increasing ISI duration. Another possibility is that pitch representations experience "sudden death," disappearing without a trace from memory. Sudden death means that listeners guess (respond at random) more often when the ISIs are longer. Since internal noise and guessing probabilities influence the shape of psychometric functions in different ways, they can be estimated simultaneously. Eleven amateur musicians performed a two-interval, two-alternative forced-choice frequency-discrimination task. The frequencies of the first tones were roved, and frequency differences and ISI durations were manipulated across trials. Data were analyzed using Bayesian models that simultaneously estimated internal noise and guessing probabilities. On average across listeners, internal noise increased monotonically as a function of increasing ISI duration, suggesting that gradual decay occurred. The guessing rate decreased with an increasing ISI duration between 0.5 and 2 s but then increased with further increases in ISI duration, suggesting that sudden death occurred but perhaps only at longer ISIs. Results are problematic for decay-only models of discrimination and contrast with those from a study on visual short-term memory, which found that over similar durations, visual representations experienced little gradual decay yet substantial sudden death.

Older adults preserve accuracy but not precision in explicit and implicit rhythmic timing

Aging brings with it several forms of neurophysiological and cognitive deterioration, but whether a decline in temporal processing is part of the aging process is unclear. The current study investigated whether this timing deficit has a cause independent of those of memory and attention using rhythmic stimuli that reduce the demand for these higher cognitive functions. In Study 1, participants took part in two rhythmic timing tasks: explicit and implicit. Participants had to distinguish regular from irregular sequences while processing temporal information explicitly or implicitly. Results showed that while the accuracy in the implicit timing task was preserved, older adults had more noise in their performance in the explicit and implicit tasks. In Study 2, participants took part in a dual-implicit task to explore whether the performance of temporal tasks differed with increasing task difficulty. We found that increasing task difficulty magnifies age-related differences.

Temporal attention is not affected by working memory load

Temporal attention refers to the ability to orient attention in time, which serves to enhance performance such as target detection and discrimination and is a fundamental component of cognitive function. Although some research indicates that temporal attention ability is affected by working memory updating, it is unclear whether temporal attention is also affected by the availability of working memory stores. To address this, participants were presented a dual-task paradigm requiring zero, three, or six digits to be held in working memory while engaged in a temporally cued visual discrimination task. Results show that working memory load did not differentially affect the ability to benefit from predictive temporal cues during the visual discrimination task. This indicates that temporal attention is not affected by available working memory stores. Interestingly, posterior beta band (12-30 Hz) activity was differentially modulated by temporal attention and working memory load, such that it decreased prior to expected targets and increased with load. Analysis across participants indicated that those individuals who exhibited greater temporal attention-based modulation of beta activity (i.e., predictive < neutrally cued) displayed improved discrimination performance, but also yielded lowered working memory accuracy. Thus, the ability to benefit from temporal attention processes while multitasking comes at the cost of lowered secondary task performance. Together, these results indicate that available working memory stores do not affect temporal attention ability. Rather, limitations in divided attention ability result in a performance cost that prioritizes one task over another, which may be indexed by beta band activity.

Understanding the computation of time using neural network models

Perceiving, maintaining, and using time intervals in working memory are crucial for animals to anticipate or act correctly at the right time in the ever-changing world. Here, we systematically study the underlying neural mechanisms by training recurrent neural networks to perform temporal tasks or complex tasks in combination with spatial information processing and decision making. We found that neural networks perceive time through state evolution along stereotypical trajectories and produce time intervals by scaling evolution speed. Temporal and nontemporal information is jointly coded in a way that facilitates decoding generalizability. We also provided potential sources for the temporal signals observed in nontiming tasks. Our study revealed the computational principles of a number of experimental phenomena and provided several predictions.

To maximize future rewards in this ever-changing world, animals must be able to discover the temporal structure of stimuli and then anticipate or act correctly at the right time. How do animals perceive, maintain, and use time intervals ranging from hundreds of milliseconds to multiseconds in working memory? How is temporal information processed concurrently with spatial information and decision making? Why are there strong neuronal temporal signals in tasks in which temporal information is not required? A systematic understanding of the underlying neural mechanisms is still lacking. Here, we addressed these problems using supervised training of recurrent neural network models. We revealed that neural networks perceive elapsed time through state evolution along stereotypical trajectory, maintain time intervals in working memory in the monotonic increase or decrease of the firing rates of interval-tuned neurons, and compare or produce time intervals by scaling state evolution speed. Temporal and nontemporal information is coded in subspaces orthogonal with each other, and the state trajectories with time at different nontemporal information are quasiparallel and isomorphic. Such coding geometry facilitates the decoding generalizability of temporal and nontemporal information across each other. The network structure exhibits multiple feedforward sequences that mutually excite or inhibit depending on whether their preferences of nontemporal information are similar or not. We identified four factors that facilitate strong temporal signals in nontiming tasks, including the anticipation of coming events. Our work discloses fundamental computational principles of temporal processing, and it is supported by and gives predictions to a number of experimental phenomena.

Temporal Processing in Audition: Insights from Music

Music is a curious example of a temporally patterned acoustic stimulus, and a compelling pan-cultural phenomenon. This review strives to bring some insights from decades of music psychology and sensorimotor synchronization (SMS) literature into the mainstream auditory domain, arguing that musical rhythm perception is shaped in important ways by temporal processing mechanisms in the brain. The feature that unites these disparate disciplines is an appreciation of the central importance of timing, sequencing, and anticipation. Perception of musical rhythms relies on an ability to form temporal predictions, a general feature of temporal processing that is equally relevant to auditory scene analysis, pattern detection, and speech perception. By bringing together findings from the music and auditory literature, we hope to inspire researchers to look beyond the conventions of their respective fields and consider the cross-disciplinary implications of studying auditory temporal sequence processing. We begin by highlighting music as an interesting sound stimulus that may provide clues to how temporal patterning in sound drives perception. Next, we review the SMS literature and discuss possible neural substrates for the perception of, and synchronization to, musical beat. We then move away from music to explore the perceptual effects of rhythmic timing in pattern detection, auditory scene analysis, and speech perception. Finally, we review the neurophysiology of general timing processes that may underlie aspects of the perception of rhythmic patterns. We conclude with a brief summary and outlook for future research.

Multiple Time Intervals of Visual Events Are Represented as Discrete Items in Working Memory

Previous studies on time perception and temporal memory have focused primarily on single time intervals; it is still unclear how multiple time intervals are perceived and maintained in working memory. In the present study, using Sternberg's item recognition task, we compared the working memory of multiple items with different time intervals and visual textures, for sub- and supra-second ranges, and investigated the characteristics of working memory representation in the framework of the signal detection theory. In Experiments 1-3, gratings with different spatial frequencies and time intervals were sequentially presented as study items, followed by another grating as a probe. Participants determined whether the probe matched one of the study gratings, in either the temporal dimension or in the visual dimension. The results exhibited typical working memory characteristics such as the effects of memory load, serial position, and similarity between probe and study gratings, similarly, to the time intervals and visual textures. However, there were some differences between the two conditions. Specifically, the recency effect for time intervals was smaller, or even absent, as compared to that for visual textures. Further, as compared with visual textures, sub-second intervals were more likely to be judged as remembered in working memory. In addition, we found interactions between visual texture memory and time interval memory, and such visual-interval binding differed between sub- and supra-second ranges. Our results indicate that multiple time intervals are stored as discrete items in working memory, similarly, to visual texture memory, but the former might be more susceptible to decay than the latter. The differences in the binding between sub- and supra-second ranges imply that working memory for sub- and supra-second ranges may differ in the relatively higher decision stage.

Beta oscillatory power modulation reflects the predictability of pitch change

Humans process highly dynamic auditory information in real time, and regularities in stimuli such as speech and music can aid such processing by allowing sensory predictions for upcoming events. Auditory sequences contain information about both the identity of sounds (what) and their timing (when they occur). Temporal prediction in isochronous sequences is reflected in neural oscillatory power modulation in the beta band (∼20 Hz). Specifically, power decreases (desynchronization) after tone onset and then increases (resynchronization) to reach a maximum around the expected time of the next tone. The current study investigates whether the predictability of the pitch of a tone (what) is also reflected in beta power modulation. We presented two isochronous auditory oddball sequences, each with 20% of tones at a deviant pitch. In one sequence the deviant tones occurred regularly every fifth tone (predictably), but in the other sequence they occurred pseudorandomly (unpredictably). We recorded the electroencephalogram (EEG) while participants listened passively to these sequences. The results showed that auditory beta power desynchronization was larger prior to a predictable than an unpredictable pitch change. A single-trial correlation analysis using linear mixed-effect (LME) models further showed that the deeper the pre-deviant beta desynchronization depth, the smaller the event-related P3a amplitude following the deviant, and this effect only occurred when the pitch change was predictable. Given that P3a is associated with attentional response to prediction error, larger beta desynchronization depth indicates better prediction of an upcoming deviant pitch. Thus, these findings suggest that beta oscillations reflect predictions for what in additional to when during dynamic auditory information processing.

The effect of attention and working memory on the estimation of elapsed time

Psychological models of time perception involve attention and memory: while attention typically regulates the flow of events, memory maintains timed events or intervals. The precise, and possibly distinct, roles of attention and memory in time perception remain debated. In this behavioral study, we tested 48 participants in a prospective duration estimation task while they fully attended to time or performed a working memory (WM) task. We report that paying attention to time lengthened perceived duration in the range of seconds to minutes, whereas diverting attention away from time shortened perceived duration. The overestimation due to attending to time did not scale with durations. To the contrary, increasing WM load systematically decreased subjective duration and this effect scaled with durations. Herein, we discuss the dissociation between attention and WM in timing and scalar variability from the perspective of Bayesian models of time estimations.

Temporal precision and the capacity of auditory–verbal short-term memory

The capacity of serially ordered auditory–verbal short-term memory (AVSTM) is sensitive to the timing of the material to be stored, and both temporal processing and AVSTM capacity are implicated in the development of language. We developed a novel “rehearsal-probe” task to investigate the relationship between temporal precision and the capacity to remember serial order. Participants listened to a sub-span sequence of spoken digits and silently rehearsed the items and their timing during an unfilled retention interval. After an unpredictable delay, a tone prompted report of the item being rehearsed at that moment. An initial experiment showed cyclic distributions of item responses over time, with peaks preserving serial order and broad, overlapping tails. The spread of the response distributions increased with additional memory load and correlated negatively with participants’ auditory digit spans. A second study replicated the negative correlation and demonstrated its specificity to AVSTM by controlling for differences in visuo-spatial STM and nonverbal IQ. The results are consistent with the idea that a common resource underpins both the temporal precision and capacity of AVSTM. The rehearsal-probe task may provide a valuable tool for investigating links between temporal processing and AVSTM capacity in the context of speech and language abilities.

The dopaminergic system dynamic in the time perception: a review of the evidence

Dopaminergic system plays a key role in perception, which is an important executive function of the brain. Modulation in dopaminergic system forms an important biochemical underpinning of neural mechanisms of time perception in a very wide range, from milliseconds to seconds to longer daily rhythms. Distinct types of temporal experience are poorly understood, and the relationship between processing of different intervals by the brain has received little attention. A comprehensive understanding of interval timing functions should be sought within a wider context of temporal processing, involving genetic aspects, pharmacological models, cognitive aspects, motor control and the neurological diseases with impaired dopaminergic system. Particularly, an unexplored question is whether the role of dopamine in interval timing can be integrated with the role of dopamine in non-interval timing temporal components. In this review, we explore a wider perspective of dopaminergic system, involving genetic polymorphisms, pharmacological models, executive functions and neurological diseases on the time perception. We conclude that the dopaminergic system has great participation in impact on time perception and neurobiological basis of the executive functions and neurological diseases.

The Persistence of Memory: How the Brain Encodes Time in Memory

Time and memory are inextricably linked, but it is far from clear how event durations and temporal sequences are encoded in memory. In this review, we focus on resource allocation models of working memory which suggest that memory resources can be flexibly distributed amongst several items such that the precision of working memory decreases with the number of items to be encoded. This type of model is consistent with human performance in working memory tasks based on visual, auditory as well as temporal stimulus patterns. At the neural-network level, we focus on excitatory-inhibitory oscillatary processes that are able to encode both interval timing and working memory in a coupled excitatory-inhibitory network. This modification of the striatal beat-frequency model of interval timing shows how memories for multiple time intervals are represented by neural oscillations and can also be used to explain the mechanisms of resource allocation in working memory.

Auditory memory for random time patterns

The acquisition of auditory memory for temporal patterns was investigated. The temporal patterns were random sequences of irregularly spaced clicks. Participants performed a task previously used to study auditory memory for noise [Agus, Thorpe, and Pressnitzer (2010). Neuron 66, 610-618]. The memory for temporal patterns displayed strong similarities with the memory for noise: temporal patterns were learnt rapidly, in an unsupervised manner, and could be distinguished from statistically matched patterns after learning. There was, however, a qualitative difference from the memory for noise. For temporal patterns, no memory transfer was observed after time reversals, showing that both the time intervals and their order were represented in memory. Remarkably, learning was observed over a broad range of time scales, which encompassed rhythm-like and buzz-like temporal patterns. Temporal patterns present specific challenges to the neural mechanisms of plasticity, because the information to be learnt is distributed over time. Nevertheless, the present data show that the acquisition of novel auditory memories can be as efficient for temporal patterns as for sounds containing additional spectral and spectro-temporal cues, such as noise. This suggests that the rapid formation of memory traces may be a general by-product of repeated auditory exposure.

Working Memory for Sequences of Temporal Durations Reveals a Volatile Single-Item Store

When a sequence is held in working memory, different items are retained with differing fidelity. Here we ask whether a sequence of brief time intervals that must be remembered show recency effects, similar to those observed in verbal and visuospatial working memory. It has been suggested that prioritizing some items over others can be accounted for by a "focus of attention," maintaining some items in a privileged state. We therefore also investigated whether such benefits are vulnerable to disruption by attention or expectation. Participants listened to sequences of one to five tones, of varying durations (200 ms to 2 s). Subsequently, the length of one of the tones in the sequence had to be reproduced by holding a key. The discrepancy between the reproduced and actual durations quantified the fidelity of memory for auditory durations. Recall precision decreased with the number of items that had to be remembered, and was better for the first and last items of sequences, in line with set-size and serial position effects seen in other modalities. To test whether attentional filtering demands might impair performance, an irrelevant variation in pitch was introduced in some blocks of trials. In those blocks, memory precision was worse for sequences that consisted of only one item, i.e., the smallest memory set-size. Thus, when irrelevant information was present, the benefit of having only one item in memory is attenuated. Finally we examined whether expectation could interfere with memory. On half the trials, the number of items in the upcoming sequence was cued. When the number of items was known in advance, performance was paradoxically worse when the sequence consisted of only one item. Thus the benefit of having only one item to remember is stronger when it is unexpectedly the only item. Our results suggest that similar mechanisms are used to hold auditory time durations in working memory, as for visual or verbal stimuli. Further, solitary items were remembered better when more items were expected, but worse when irrelevant features were present. This suggests that the "privileged" state of one item in memory is particularly volatile and susceptible to interference.

A Citation-Based Analysis and Review of Significant Papers on Timing and Time Perception

Time is an important dimension of brain function, but little is yet known about the underlying cognitive principles and neurobiological mechanisms. The field of timing and time perception has witnessed tremendous growth and multidisciplinary interest in the recent years with the advent of modern neuroimaging and neurophysiological approaches. In this article, I used a data mining approach to analyze the timing literature published by a select group of researchers (n = 202) during the period 2000–2015 and highlight important reviews as well as empirical articles that meet the criterion of a minimum of 100 citations. The qualifying articles (n = 150) are listed in a table along with key details such as number of citations, names of authors, year and journal of publication as well as a short summary of the findings of each study. The results of such a data-driven approach to literature review not only serve as a useful resource to any researcher interested in timing, but also provides a means to evaluate key papers that have significantly influenced the field and summarize recent progress and popular research trends in the field. Additionally, such analyses provides food for thought about future scientific directions and raises important questions about improving organizational structures to boost open science and progress in the field. I discuss exciting avenues for future research that have the potential to significantly advance our understanding of the neurobiology of timing, and propose the establishment of a new society, the Timing Research Forum, to promote open science and collaborative work within the highly diverse and multidisciplinary community of researchers in the field of timing and time perception.

Brain Bases of Working Memory for Time Intervals in Rhythmic Sequences

Perception of auditory time intervals is critical for accurate comprehension of natural sounds like speech and music. However, the neural substrates and mechanisms underlying the representation of time intervals in working memory are poorly understood. In this study, we investigate the brain bases of working memory for time intervals in rhythmic sequences using functional magnetic resonance imaging. We used a novel behavioral paradigm to investigate time-interval representation in working memory as a function of the temporal jitter and memory load of the sequences containing those time intervals. Human participants were presented with a sequence of intervals and required to reproduce the duration of a particular probed interval. We found that perceptual timing areas including the cerebellum and the striatum were more or less active as a function of increasing and decreasing jitter of the intervals held in working memory respectively whilst the activity of the inferior parietal cortex is modulated as a function of memory load. Additionally, we also analyzed structural correlations between gray and white matter density and behavior and found significant correlations in the cerebellum and the striatum, mirroring the functional results. Our data demonstrate neural substrates of working memory for time intervals and suggest that the cerebellum and the striatum represent core areas for representing temporal information in working memory.

Resource allocation models of auditory working memory

Auditory working memory (WM) is the cognitive faculty that allows us to actively hold and manipulate sounds in mind over short periods of time. We develop here a particular perspective on WM for non-verbal, auditory objects as well as for time based on the consideration of possible parallels to visual WM. In vision, there has been a vigorous debate on whether WM capacity is limited to a fixed number of items or whether it represents a limited resource that can be allocated flexibly across items. Resource allocation models predict that the precision with which an item is represented decreases as a function of total number of items maintained in WM because a limited resource is shared among stored objects. We consider here auditory work on sequentially presented objects of different pitch as well as time intervals from the perspective of dynamic resource allocation. We consider whether the working memory resource might be determined by perceptual features such as pitch or timbre, or bound objects comprising multiple features, and we speculate on brain substrates for these behavioural models. This article is part of a Special Issue entitled SI: Auditory working memory.