Interference effects in short-term memory for timbre

Beyond Recognition: Visual Contributions to Verbal Working Memory

PURPOSE

It is well recognized that adding the visual to the acoustic speech signal improves recognition when the acoustic signal is degraded, but how that visual signal affects postrecognition processes is not so well understood. This study was designed to further elucidate the relationships among auditory and visual codes in working memory, a postrecognition process.

DESIGN

In a main experiment, 80 young adults with normal hearing were tested using an immediate serial recall paradigm. Three types of signals were presented (unprocessed speech, vocoded speech, and environmental sounds) in three conditions (audio-only, audio-video with dynamic visual signals, and audio-picture with static visual signals). Three dependent measures were analyzed: (a) magnitude of the recency effect, (b) overall recall accuracy, and (c) response times, to assess cognitive effort. In a follow-up experiment, 30 young adults with normal hearing were tested largely using the same procedures, but with a slight change in order of stimulus presentation.

RESULTS

The main experiment produced three major findings: (a) unprocessed speech evoked a recency effect of consistent magnitude across conditions; vocoded speech evoked a recency effect of similar magnitude to unprocessed speech only with dynamic visual (lipread) signals; environmental sounds never showed a recency effect. (b) Dynamic and static visual signals enhanced overall recall accuracy to a similar extent, and this enhancement was greater for vocoded speech and environmental sounds than for unprocessed speech. (c) All visual signals reduced cognitive load, except for dynamic visual signals with environmental sounds. The follow-up experiment revealed that dynamic visual (lipread) signals exerted their effect on the vocoded stimuli by enhancing phonological quality.

CONCLUSIONS

Acoustic and visual signals can combine to enhance working memory operations, but the source of these effects differs for phonological and nonphonological signals. Nonetheless, visual information can support better postrecognition processes for patients with hearing loss.

Gradual decay and sudden death of short-term memory for pitch

The ability to discriminate frequency differences between pure tones declines as the duration of the interstimulus interval (ISI) increases. The conventional explanation for this finding is that pitch representations gradually decay from auditory short-term memory. Gradual decay means that internal noise increases with increasing ISI duration. Another possibility is that pitch representations experience "sudden death," disappearing without a trace from memory. Sudden death means that listeners guess (respond at random) more often when the ISIs are longer. Since internal noise and guessing probabilities influence the shape of psychometric functions in different ways, they can be estimated simultaneously. Eleven amateur musicians performed a two-interval, two-alternative forced-choice frequency-discrimination task. The frequencies of the first tones were roved, and frequency differences and ISI durations were manipulated across trials. Data were analyzed using Bayesian models that simultaneously estimated internal noise and guessing probabilities. On average across listeners, internal noise increased monotonically as a function of increasing ISI duration, suggesting that gradual decay occurred. The guessing rate decreased with an increasing ISI duration between 0.5 and 2 s but then increased with further increases in ISI duration, suggesting that sudden death occurred but perhaps only at longer ISIs. Results are problematic for decay-only models of discrimination and contrast with those from a study on visual short-term memory, which found that over similar durations, visual representations experienced little gradual decay yet substantial sudden death.

Musical and verbal short-term memory: insights from neurodevelopmental and neurological disorders

Auditory short-term memory (STM) is a fundamental ability to make sense of auditory information as it unfolds over time. Whether separate STM systems exist for different types of auditory information (music and speech, in particular) is a matter of debate. The present paper reviews studies that have investigated both musical and verbal STM in healthy individuals and in participants with neurodevelopmental and neurological disorders. Overall, the results are in favor of only partly shared networks for musical and verbal STM. Evidence for a distinction in STM for the two materials stems from (1) behavioral studies in healthy participants, in particular from the comparison between nonmusicians and musicians; (2) behavioral studies in congenital amusia, where a selective pitch STM deficit is observed; and (3) studies in brain-damaged patients with cases of double dissociation. In this review we highlight the need for future studies comparing STM for the same perceptual dimension (e.g., pitch) in different materials (e.g., music and speech), as well as for studies aiming at a more insightful characterization of shared and distinct mechanisms for speech and music in the different components of STM, namely encoding, retention, and retrieval.

Short-term memory stores organized by information domain

Vision and audition have complementary affinities, with vision excelling in spatial resolution and audition excelling in temporal resolution. Here, we investigated the relationships among the visual and auditory modalities and spatial and temporal short-term memory (STM) using change detection tasks. We created short sequences of visual or auditory items, such that each item within a sequence arose at a unique spatial location at a unique time. On each trial, two successive sequences were presented; subjects attended to either space (the sequence of locations) or time (the sequence of inter item intervals) and reported whether the patterns of locations or intervals were identical. Each subject completed blocks of unimodal trials (both sequences presented in the same modality) and crossmodal trials (Sequence 1 visual, Sequence 2 auditory, or vice versa) for both spatial and temporal tasks. We found a strong interaction between modality and task: Spatial performance was best on unimodal visual trials, whereas temporal performance was best on unimodal auditory trials. The order of modalities on crossmodal trials also mattered, suggesting that perceptual fidelity at encoding is critical to STM. Critically, no cost was attributable to crossmodal comparison: In both tasks, performance on crossmodal trials was as good as or better than on the weaker unimodal trials. STM representations of space and time can guide change detection in either the visual or the auditory modality, suggesting that the temporal or spatial organization of STM may supersede sensory-specific organization.

Nonverbal auditory working memory: Can music indicate the capacity?

Different working memory (WM) mechanisms that underlie words, tones, and timbres have been proposed in previous studies. In this regard, the present study developed a WM test with nonverbal sounds and compared it to the conventional verbal WM test. A total of twenty-five, non-music major, right-handed college students were presented with four different types of sounds (words, syllables, pitches, timbres) that varied from two to eight digits in length. Both accuracy and oxygenated hemoglobin (oxyHb) were measured. The results showed significant effects of number of targets on accuracy and sound type on oxyHb. A further analysis showed prefrontal asymmetry with pitch being processed by the right hemisphere (RH) and timbre by the left hemisphere (LH). These findings suggest a potential for employing musical sounds (i.e., pitch and timbre) as a complementary stimuli for conventional nonverbal WM tests, which can additionally examine its asymmetrical roles in the prefrontal regions.

The Item versus the Object in Memory: On the Implausibility of Overwriting As a Mechanism for Forgetting in Short-Term Memory

The nature of forgetting in short-term memory remains a disputed topic, with much debate focussed upon whether decay plays a fundamental role (Berman et al., 2009; Altmann and Schunn, 2012; Barrouillet et al., 2012; Neath and Brown, 2012; Oberauer and Lewandowsky, 2013; Ricker et al., 2014) but much less focus on other plausible mechanisms. One such mechanism of long-standing in auditory memory is overwriting (e.g., Crowder and Morton, 1969) in which some aspects of a representation are "overwritten" and rendered inaccessible by the subsequent presentation of a further item. Here, we review the evidence for different forms of overwriting (at the feature and item levels) and examine the plausibility of this mechanism both as a form of auditory memory and when viewed in the context of a larger hearing, speech and language understanding system.

Resource allocation models of auditory working memory

Auditory working memory (WM) is the cognitive faculty that allows us to actively hold and manipulate sounds in mind over short periods of time. We develop here a particular perspective on WM for non-verbal, auditory objects as well as for time based on the consideration of possible parallels to visual WM. In vision, there has been a vigorous debate on whether WM capacity is limited to a fixed number of items or whether it represents a limited resource that can be allocated flexibly across items. Resource allocation models predict that the precision with which an item is represented decreases as a function of total number of items maintained in WM because a limited resource is shared among stored objects. We consider here auditory work on sequentially presented objects of different pitch as well as time intervals from the perspective of dynamic resource allocation. We consider whether the working memory resource might be determined by perceptual features such as pitch or timbre, or bound objects comprising multiple features, and we speculate on brain substrates for these behavioural models. This article is part of a Special Issue entitled SI: Auditory working memory.

What is a melody? On the relationship between pitch and brightness of timbre

Previous studies showed that the perceptual processing of sound sequences is more efficient when the sounds vary in pitch than when they vary in loudness. We show here that sequences of sounds varying in brightness of timbre are processed with the same efficiency as pitch sequences. The sounds used consisted of two simultaneous pure tones one octave apart, and the listeners’ task was to make same/different judgments on pairs of sequences varying in length (one, two, or four sounds). In one condition, brightness of timbre was varied within the sequences by changing the relative level of the two pure tones. In other conditions, pitch was varied by changing fundamental frequency, or loudness was varied by changing the overall level. In all conditions, only two possible sounds could be used in a given sequence, and these two sounds were equally discriminable. When sequence length increased from one to four, discrimination performance decreased substantially for loudness sequences, but to a smaller extent for brightness sequences and pitch sequences. In the latter two conditions, sequence length had a similar effect on performance. These results suggest that the processes dedicated to pitch and brightness analysis, when probed with a sequence-discrimination task, share unexpected similarities.

Pitch and timbre interfere when both are parametrically varied

Pitch and timbre perception are both based on the frequency content of sound, but previous perceptual experiments have disagreed about whether these two dimensions are processed independently from each other. We tested the interaction of pitch and timbre variations using sequential comparisons of sound pairs. Listeners judged whether two sequential sounds were identical along the dimension of either pitch or timbre, while the perceptual distances along both dimensions were parametrically manipulated. Pitch and timbre variations perceptually interfered with each other and the degree of interference was modulated by the magnitude of changes along the un-attended dimension. These results show that pitch and timbre are not orthogonal to each other when both are assessed with parametrically controlled variations.

Retention of Gap Length in Normal-Hearing Listeners

Listener retention of silent, gap-length duration was studied. Just noticeable differences (JNDs) for gap length within standard and comparison stimuli were obtained for intervals with and without intervening noise bursts, including a condition with gapped intervening bursts. Outcomes indicate that gap duration itself can be determinant. Also, JNDs were similar whether intervening stimuli were present or absent, differing from results reported for pitch, loudness, and timbre retention. The latter suggests additional/alternative cortical resources might be employed for retention of auditory–temporal information.

Resource allocation and prioritization in auditory working memory

A prevalent view of working memory (WM) considers it to be capacity-limited, fixed to a set number of items. However, recent shared resource models of WM have challenged this “quantized” account using measures of recall precision. Although this conceptual framework can account for several features of visual WM, it remains to be established whether it also applies to auditory WM.

We used a novel pitch-matching paradigm to probe participants’ memory of pure tones in sequences of varying length, and measured their precision of recall. Crucially, this provides an index of the variability of memory representation around its true value, rather than a binary “yes/no” recall measure typically used in change detection paradigms. We show that precision of auditory WM varies with both memory load and serial order. Moreover, auditory WM resources can be prioritized to cued items, improving precision of recall, but with a concomitant cost to other items, consistent with a resource model account.

Working memory for pitch, timbre, and words

Aiming to further our understanding of fundamental mechanisms of auditory working memory (WM), the present study compared performance for three auditory materials (words, tones, timbres). In a forward recognition task (Experiment 1) participants indicated whether the order of the items in the second sequence was the same as in the first sequence. In a backward recognition task (Experiment 2) participants indicated whether the items of the second sequence were played in the correct backward order. In Experiment 3 participants performed an articulatory suppression task during the retention delay of the backward task. To investigate potential length effects the number of items per sequence was manipulated. Overall findings underline the benefit of a cross-material experimental approach and suggest that human auditory WM is not a unitary system. Whereas WM processes for timbres differed from those for tones and words, similarities and differences were observed for words and tones: Both types of stimuli appear to rely on rehearsal mechanisms, but might differ in the involved sensorimotor codes.

Summary statistics in auditory perception

Sensory signals are transduced at high resolution, but their structure must be stored in a more compact format. Here we provide evidence that the auditory system summarizes the temporal details of sounds using time-averaged statistics. We measured discrimination of 'sound textures' that were characterized by particular statistical properties, as normally result from the superposition of many acoustic features in auditory scenes. When listeners discriminated examples of different textures, performance improved with excerpt duration. In contrast, when listeners discriminated different examples of the same texture, performance declined with duration, a paradoxical result given that the information available for discrimination grows with duration. These results indicate that once these sounds are of moderate length, the brain's representation is limited to time-averaged statistics, which, for different examples of the same texture, converge to the same values with increasing duration. Such statistical representations produce good categorical discrimination, but limit the ability to discern temporal detail.

Effect of acoustic similarity on short-term auditory memory in the monkey

Recent evidence suggests that the monkey's short-term memory in audition depends on a passively retained sensory trace as opposed to a trace reactivated from long-term memory for use in working memory. Reliance on a passive sensory trace could render memory particularly susceptible to confusion between sounds that are similar in some acoustic dimension. If so, then in delayed matching-to-sample, the monkey's performance should be predicted by the similarity in the salient acoustic dimension between the sample and subsequent test stimulus, even at very short delays. To test this prediction and isolate the acoustic features relevant to short-term memory, we examined the pattern of errors made by two rhesus monkeys performing a serial, auditory delayed match-to-sample task with interstimulus intervals of 1 s. The analysis revealed that false-alarm errors did indeed result from similarity-based confusion between the sample and the subsequent nonmatch stimuli. Manipulation of the stimuli showed that removal of spectral cues was more disruptive to matching behavior than removal of temporal cues. In addition, the effect of acoustic similarity on false-alarm response was stronger at the first nonmatch stimulus than at the second one. This pattern of errors would be expected if the first nonmatch stimulus overwrote the sample's trace, and suggests that the passively retained trace is not only vulnerable to similarity-based confusion but is also highly susceptible to overwriting.

Auditory Displays for In-Vehicle Technologies

Modern vehicle cockpits have begun to incorporate a number of information-rich techno-logies, including systems to enhance and improve driving and navigation performance and also driving-irrelevant information systems. The visually intensive nature of the driving task requires these systems to adopt primarily nonvisual means of information display, and the auditory modality represents an obvious alternative to vision for interacting with in-vehicle technologies (IVTs). Although the literature on auditory displays has grown tremendously in recent decades, to date, few guidelines or recommendations exist to aid in the design of effective auditory displays for IVTs. This chapter provides an overview of the current state of research and practice with auditory displays for IVTs. The role of basic auditory capabilities and limitations as they relate to in-vehicle auditory display design are discussed. Extant systems and prototypes are reviewed, and when possible, design recommendations are made. Finally, research needs and an iterative design process to meet those needs are discussed.

Dissociable systems of working memory for rhythm and melody

Specialized neural systems are engaged by the rhythmic and melodic components of music. Here, we used PET to measure regional cerebral blood flow (rCBF) in a working memory task for sequences of rhythms and melodies, which were presented in separate blocks. Healthy subjects, without musical training, judged whether a target rhythm or melody was identical to a series of subsequently presented rhythms or melodies. When contrasted with passive listening to rhythms, working memory for rhythm activated the cerebellar hemispheres and vermis, right anterior insular cortex, and left anterior cingulate gyrus. These areas were not activated in a contrast between passive listening to rhythms and a non-auditory control, indicating their role in the temporal processing that was specific to working memory for rhythm. The contrast between working memory for melody and passive listening to melodies activated mainly a right-hemisphere network of frontal, parietal, and temporal cortices: areas involved in pitch processing and auditory working memory. Overall, these results demonstrate that rhythm and melody have unique neural signatures not only in the early stages of auditory processing, but also at the higher cognitive level of working memory.

Comparisons of Complex Sounds across Extended Retention Intervals Survives Reading Aloud

A simple experimental arrangement is designed to foil verbal rehearsal during an extended (from 5 to 30 s) retention interval across which participants attempt to discriminate two periodic complex sounds. Sounds have an abstract timbre that does not lend itself to verbal labeling, they differ across trials so that no ‘standard’ comparison stimulus is built up by the participants, and the spectral change to be discriminated is very slight and therefore does not shift the stimulus into a new verbal category. And, crucially, in one experimental condition, participants read aloud during most of the retention interval. Despite these precautions, performance is robust across the extended retention interval. The inference is that one form of auditory memory does not require verbal rehearsal. Nevertheless, modest forgetting occurred. Whatever form memory takes in this situation, it is not totally secure from disruption.

Rapid Communication: Interference in Short-Term Auditory Memory

Although interference is a well-established forgetting function in short-term auditory memory, an adequate understanding of its underlying mechanisms and time course has yet to be attained. The present study therefore aimed to explore these issues in memory for timbre. Listeners compared standard and comparison complex tones, having distinct timbres (four components varying in frequency), over a 4.7-s retention interval and made a same–different response. This interval either was silent or included one of 15 distractor tones occurring 0 ms, 100 ms, or 1,200 ms after the standard. These distractors varied in the extent to which the frequencies of their component tones were shared with the standard. Performance in comparing the two tones was significantly impaired by distractors composed of novel frequencies, regardless of the temporal position at which the distractor occurred. These results were fully compatible with the recent timbre memory model (McKeown & Wellsted, 2009) and suggested that interference in auditory memory operates via a feature-overwriting mechanism.

Congenital amusia: a short-term memory deficit for non-verbal, but not verbal sounds

Congenital amusia refers to a lifelong disorder of music processing and is linked to pitch-processing deficits. The present study investigated congenital amusics' short-term memory for tones, musical timbres and words. Sequences of five events (tones, timbres or words) were presented in pairs and participants had to indicate whether the sequences were the same or different. The performance of congenital amusics confirmed a memory deficit for tone sequences, but showed normal performance for word sequences. For timbre sequences, amusics' memory performance was impaired in comparison to matched controls. Overall timbre performance was found to be correlated with melodic contour processing (as assessed by the Montreal Battery of Evaluation of Amusia). The present findings show that amusics' deficits extend to non-verbal sound material other than pitch, in this case timbre, while not affecting memory for verbal material. This is in line with previous suggestions about the domain-specificity of congenital amusia.

Auditory memory: a comparison between humans and starlings

In this study, we compare the processing of acoustic signals in European starlings (Sturnus vulgaris) and in human listeners by observing the decay of short-term auditory memory in delayed non-matching-to-sample experiments. A series of identical "sample" stimuli and a final "test" stimulus were separated by variable delays (1 to 180.1 s). Subjects had to classify sample and test stimuli as being either the same or different. Test stimuli were pure tones that differed in a single signal feature, i.e., frequency, and song motifs that differed in multiple signal characteristics. We have tested several predictions concerning the memory performance of starlings and humans and we obtained the following outcome: (1) In contrast to our expectation, signal complexity had no effect. The overall analysis of the starling data did not show differences in memory performance for signals differing in single or multiple signal features. (2) Starling and human data supported the hypothesis that auditory memory impairs with increasing delay. This was also seen when interfering noise was added to the delay periods in an additional series with human subjects. (3) The starling data showed that the repetition of sample stimuli improved memory performance, compared to only a single presentation. Human memory performance, however, was similar for a single and for the repeated presentation of signals. (4) Differences in salience between sample and test stimuli were positively related to memory performance only for tonal stimuli but not for song motifs. Results are discussed with respect to a model based on signal detection theory and to requirements for the analysis of natural communication signals.

Temporal interval production and short-term memory

Interference with time estimation from concurrent nontemporal processing has been shown to depend on the short-term memory requirements of the concurrent task (Fortin & Breton, 1995; Fortin, Rousseau, Bourque, & Kirouac, 1993). In particular, it has been claimed that active processing of information in short-term memory produces interference, whereas simply maintaining information does not. Here, four experiments are reported in which subjects were trained to produce a 2,500-msec interval and then perform concurrent memory tasks. Interference with timing was demonstrated for concurrent memory tasks involving only maintenance. In one experiment, increasing set size in a pitch memory task systematically lengthened temporal production. Two further experiments suggested that this was due to a specific interaction between the short-term memory requirements of the pitch task and those of temporal production. In the final experiment, subjects performed temporal production while concurrently remembering the durations of a set of tones. Interference with interval production was comparable to that produced by the pitch memory task. Results are discussed in terms of a pacemaker-counter model of temporal processing, in which the counter component is supported by short-term memory.

Influence of tonal context and timbral variation on perception of pitch

In this study, spectral timbre's effect on pitch perception is examined in varying contexts. In two experiments, subjects detected pitch deviations of tones differing in brightness in an isolated context in which they compared two tones, in a tone-series context in which they judged whether the last tone of a simple sequence was in or out of tune, and in a melodic context in which they determined whether the last note of familiar melodies was in or out of tune. Timbre influenced pitch judgments in all the conditions, but increasing tonal context allowed the subjects to extract pitch information more accurately. This appears to be due to two factors: (1) The presence of extra tones creates a stronger reference point from which to judge pitch, and (2) the melodies' tonal structure gives more cues that facilitate pitch extraction, even in the face of conflicting spectral information.

Similarity, invariance, and musical variation

Perceptual similarity underlies a number of important psychological properties of musical materials, including perceptual invariance under transformation, categorization, recognition, and the sense of familiarity. Mental processes involved in the perception of musical similarity may be an integral part of the functional logic of music composition and thus underly important aspects of musical experience. How much and in what ways can musical materials be varied and still be considered as perceptually related or as belonging to the same category? The notions of musical material, musical variation, perceptual similarity and invariance, and form-bearing dimensions are considered in this light. Recent work on similarity perception has demonstrated that the transformation space for a given musical material is limited by several factors ranging from degree of match of the values of auditory attributes of the events composing the sequences to their relations of various levels of abstraction and to the degree that the transformation respects the grammar of the musical system within which the material was composed. These notions and results are considered in the light of future directions of research, particularly concerning the role of similarity and invariance in the understanding of musical form during listening.

Memory for pitch versus memory for loudness

The decays of pitch traces and loudness traces in short-term auditory memory were compared in forced-choice discrimination experiments. The two stimuli presented on each trial were separated by a variable delay (D); they consisted of pure tones, series of resolved harmonics, or series of unresolved harmonics mixed with lowpass noise. A roving procedure was employed in order to minimize the influence of context coding. During an initial phase of each experiment, frequency and intensity discrimination thresholds [P(C) = 0.80] were measured with an adaptive staircase method while D was fixed at 0.5 s. The corresponding physical differences (in cents or dB) were then constantly presented at four values of D: 0.5, 2, 5, and 10 s. In the case of intensity discrimination, performance (d') markedly decreased when D increased from 0.5 to 2 s, but was not further reduced when D was longer. In the case of frequency discrimination, the decline of performance as a function of D was significantly less abrupt. This divergence suggests that pitch and loudness are processed in separate modules of auditory memory.