Incremental planning in sequence production

Repetition costs in sequence chunking

We examined how flexibly we plan sequences of actions when we switch between multiple action sequences. Mastering a sequential skill is assumed to involve integrating successive actions into groups known as chunks that can be efficiently planned and smoothly executed. Chunking is suggested by gains in planning efficiency for long compared to short action sequences following practice and learning associations between actions and perceptual outcomes. Less is understood about how efficiently we plan sequential chunks when we switch between multiple action sequences. Do we plan learned chunks less efficiently when we switch to a different action sequence? We examined this question by comparing the initiation and execution latencies of long versus short action sequences, performed from memory, when sequences switched or repeated across trials. Additionally, each action within the sequences generated predictable perceptual outcomes that were either spatially compatible or spatially incompatible with the action sequences. Results suggested repetition costs (instead of benefits) when performing long sequences. Repetition, as opposed to switching, prolonged initiation and increased the error rate of long compared to short sequences. We attribute these results to the flexible coordination of chunk planning and execution. Repetition may prolong advanced planning of long sequences in order to resolve conflict between multiple chunks, and switching may allow the planning of later chunks to be postponed until execution. We propose that the chunking of action sequences can both facilitate and interfere with action-switching performance.

Task-irrelevant auditory metre shapes visuomotor sequential learning

The ability to learn and reproduce sequences is fundamental to every-day life, and deficits in sequential learning are associated with developmental disorders such as specific language impairment. Individual differences in sequential learning are usually investigated using the serial reaction time task (SRTT), wherein a participant responds to a series of regularly timed, seemingly random visual cues that in fact follow a repeating deterministic structure. Although manipulating inter-cue interval timing has been shown to adversely affect sequential learning, the role of metre (the patterning of salience across time) remains unexplored within the regularly timed, visual SRTT. The current experiment consists of an SRTT adapted to include task-irrelevant auditory rhythms conferring a sense of metre. We predicted that (1) participants' (n = 41) reaction times would reflect the auditory metric structure; (2) that disrupting the correspondence between the learned visual sequence and auditory metre would impede performance; and (3) that individual differences in sensitivity to rhythm would predict the magnitude of these effects. Altering the relationship via a phase shift between the trained visual sequence and auditory metre slowed reaction times. Sensitivity to rhythm was predictive of reaction times over all. In an exploratory analysis, we, moreover, found that approximately half of participants made systematically different responses to visual cues on the basis of the cues' position within the auditory metre. We demonstrate the influence of auditory temporal structures on visuomotor sequential learning in a widely used task where metre and timing are rarely considered. The current results indicate sensitivity to metre as a possible latent factor underpinning individual differences in SRTT performance.

Hysteresis in motor and language production

Hysteresis in motor planning and syntactic priming in language planning refer to the influence of prior production history on current production behaviour. Computational efficiency accounts of action hysteresis and theoretical accounts of syntactic priming both argue that reusing an existing plan is less costly than generating a novel plan. Despite these similarities across motor and language frameworks, research on planning in these domains has largely been conducted independently. The current study adapted an existing language paradigm to mirror the incremental nature of a manual motor task to investigate the presence of parallel hysteresis effects across domains. We observed asymmetries in production choice for both the motor and language tasks that resulted from the influence of prior history. Furthermore, these hysteresis effects were more exaggerated for subordinate production forms implicating an inverse preference effect that spanned domain. Consistent with computational efficiency accounts, across both task participants exhibited reaction time savings on trials in which they reused a recent production choice. Together, these findings lend support to the broader notion that there are common production biases that span both motor and language domains.

Music rhythm perception and production relate to treatment response in schizophrenia

Accumulating evidence indicates that pathophysiology of schizophrenia involves abnormalities in the dopamine and glutamatergic neuronal systems. Antipsychotic medications are currently used to normalize dopaminergic function for schizophrenia. However, approximately 30 % of the patients have no response to antipsychotic medications, which is classified as treatment-resistant schizophrenia (TRS). Furthermore, dopamine and glutamate levels in the neural basis have been reported to differ between TRS and non-TRS. In this study, we assumed that these differences may affect music rhythm perception and production abilities between the two groups. We examined fifty-seven schizophrenia (26 TRS, 31 non-TRS) and thirty-one healthy controls (HCs) by using the Harvard Beat Assessment Test (H-BAT). As a result, we found that rhythm production was worse in patients with TRS compared to patients with non-TRS and HCs, while no difference was observed between patients with non-TRS and HCs. In addition, rhythm perception and production abilities were impaired in the whole patient group compared with HCs. Furthermore, in the patient group, the deficits were correlated with cognitive impairments. Collectively, these results suggest that patients with schizophrenia may have rhythm processing deficits, with particular a rhythm production problem in the TRS group.

Lateral Prefrontal Cortex Is a Hub for Music Production from Structural Rules to Movements

Complex sequential behaviors, such as speaking or playing music, entail flexible rule-based chaining of single acts. However, it remains unclear how the brain translates abstract structural rules into movements. We combined music production with multimodal neuroimaging to dissociate high-level structural and low-level motor planning. Pianists played novel musical chord sequences on a muted MR-compatible piano by imitating a model hand on screen. Chord sequences were manipulated in terms of musical harmony and context length to assess structural planning, and in terms of fingers used for playing to assess motor planning. A model of probabilistic sequence processing confirmed temporally extended dependencies between chords, as opposed to local dependencies between movements. Violations of structural plans activated the left inferior frontal and middle temporal gyrus, and the fractional anisotropy of the ventral pathway connecting these two regions positively predicted behavioral measures of structural planning. A bilateral frontoparietal network was instead activated by violations of motor plans. Both structural and motor networks converged in lateral prefrontal cortex, with anterior regions contributing to musical structure building, and posterior areas to movement planning. These results establish a promising approach to study sequence production at different levels of action representation.

Temporal grouping effects in verbal and musical short-term memory: Is serial order representation domain-general?

The question of the domain-general versus domain-specific nature of the serial order mechanisms involved in short-term memory is currently under debate. The present study aimed at addressing this question through the study of temporal grouping effects in short-term memory tasks with musical material, a domain which has received little interest so far. The goal was to determine whether positional coding-currently the best account of grouping effect in verbal short-term memory-represents a viable mechanism to explain grouping effects in the musical domain. In a first experiment, non-musicians performed serial reconstruction of 6-tone sequences, where half of the sequences was grouped by groups of three items and the other half presented at a regular pace. The overall data pattern suggests that temporal grouping exerts on tone sequences reconstruction the same effects as in the verbal domain, except for ordering errors which were not characterised by the typical increase of interpositions. This pattern has been replicated in two additional experiments with verbal material, using the same grouping structure as in the musical experiment. The findings support that verbal and musical short-term memory domains are characterised by similar temporal grouping effects for the recall of 6-item lists grouped by three, but it also suggests the existence of boundary condition to observe an increase in interposition errors predicted by positional theories.

Rats' use of hierarchical organization in serial pattern learning

Many organisms show aptitude for learning and performing patterned sequences. However, we do not yet have a complete account of how they accomplish this. One of the most successful is Restle's hierarchical model, which supposes organisms represent sequences using the simplest form available through using hierarchies to organize the sequences' elements. In two experiments, we evaluated if Restle's model accurately accounted for rats' patterned sequence learning. First, we compared rats' learning of a runs pattern with three levels of hierarchical organization with no violations of pattern structure to learning of the same pattern with one violation of pattern structure in the pattern's second half. Restle's model predicts rats should have more difficulty learning the violation pattern and that rats learning the pattern with the violation in the second half of the pattern should attempt to reflect that violation to the first half. The results indicated that rats had more difficulty learning the violation pattern and that inserting a violation in the second half of the pattern led rats to make significantly more errors on the second half of the pattern relative to the first half of the pattern. Rats did not insert the violation from the second half of the pattern into the first half of the pattern. In a second experiment, we continued to explore if Restle's model accurately accounted for rats' pattern learning using more complex patterns. The present results of both experiments and prior work with humans were inconsistent with Restle's model, suggesting more work is needed to develop a model that accurately accounts for rats' learning of complex patterned sequences.

Not all errors are alike: modulation of error-related neural responses in musical joint action

During joint action, the sense of agency enables interaction partners to implement corrective and adaptive behaviour in response to performance errors. When agency becomes ambiguous (e.g. when action similarity encourages perceptual self-other overlap), confusion as to who produced what may disrupt this process. The current experiment investigated how ambiguity of agency affects behavioural and neural responses to errors in a joint action domain where self-other overlap is common: musical duos. Pairs of pianists performed piano pieces in synchrony, playing either the same pitches (ambiguous agency) or different pitches (unambiguous agency) while electroencephalography (EEG) was recorded for each individual. Behavioural and event-related potential results showed no effects of the agency manipulation but revealed differences in how distinct error types are processed. Self-produced 'wrong note' errors (substitutions) were left uncorrected, showed post-error slowing and elicited an error-related negativity (ERN) peaking before erroneous keystrokes (pre-ERN). In contrast, self-produced 'extra note' errors (additions) exhibited pre-error slowing, error and post-error speeding, were rapidly corrected and elicited the ERN. Other-produced errors evoked a feedback-related negativity but no behavioural effects. Overall findings shed light upon how the nervous system supports fluent interpersonal coordination in real-time joint action by employing distinct mechanisms to manage different types of errors.

Accessibility and Historical Change: An Emergent Cluster Led Uncles and Aunts to Become Aunts and Uncles

There are times when a curiously odd relic of language presents us with a thread, which when pulled, reveals deep and general facts about human language. This paper unspools such a case. Prior to 1930, English speakers uniformly preferred male-before-female word order in conjoined nouns such as uncles and aunts; nephews and nieces; men and women. Since then, at least a half dozen items have systematically reversed their preferred order (e.g., aunts and uncles, nieces and nephews) while others have not (men and women). We review evidence that the unusual reversals began with mother and dad(dy) and spread to semantically and morphologically related binomials over a period of decades. The present work proposes that three aspects of cognitive accessibility combine to quantify the probability of A&B order: (1) the relative accessibility of the A&B terms individually, (2) competition from B&A order, and critically, (3) cluster strength (i.e., similarity to related A'&B' cases). The emergent cluster of female-first binomials highlights the influence of semantic neighborhoods in memory retrieval. We suggest that cognitive accessibility can be used to predict the word order of both familiar and novel binomials generally, as well as the diachronic change focused on here.

Advances in the Neurocognition of Music and Language

Neurocomparative music and language research has seen major advances over the past two decades. The goal of this Special Issue “Advances in the Neurocognition of Music and Language” was to showcase the multiple neural analogies between musical and linguistic information processing, their entwined organization in human perception and cognition and to infer the applicability of the combined knowledge in pedagogy and therapy. Here, we summarize the main insights provided by the contributions and integrate them into current frameworks of rhythm processing, neuronal entrainment, predictive coding and cognitive control.

Generalization of novel sensorimotor associations among pianists and non-pianists: more evidence that musical training effects are constrained

In the process of acquiring musical skills, such as playing the piano, we develop sensorimotor associations between motor movements and perception of pitch. Previous research suggests that these acquired associations are relatively inflexible and show limited generalizability to performance under novel conditions. The current study investigated whether piano training constrains the ability to generalize learning based on an unfamiliar (inverted) pitch mapping, using a transfer-of-training paradigm (Palmer and Meyer in Psychol Sci 11:63-68, 2000). Pianists and non-pianists learned a training melody by ear with normal (higher pitches to the right) or inverted (higher pitches to the left) pitch mapping. After training, participants completed a generalization test in which they listened to and then immediately reproduced four types of melodies that varied in their similarity to the melody used during training and were based on the same, a similar, an inverted, or a different pitch pattern. The feedback mapping during the generalization test matched training. Overall, pianists produced fewer errors and required fewer training trials than non-pianists. However, benefits of training were absent for pianists who trained with inverted feedback when they attempted to reproduce a melody with a different structure than the melody used for training. This suggests that piano experience may constrain one's ability to generalize learning that is based on novel sensorimotor associations.

Music predictability and liking enhance pupil dilation and promote motor learning in non-musicians

Humans can anticipate music and derive pleasure from it. Expectations facilitate the learning of movements associated with anticipated events, and they are also linked with reward, which may further facilitate learning of the anticipated rewarding events. The present study investigates the synergistic effects of predictability and hedonic responses to music on arousal and motor-learning in a naïve population. Novel melodies were manipulated in their overall predictability (predictable/unpredictable) as objectively defined by a model of music expectation, and ranked as high/medium/low liked based on participants' self-reports collected during an initial listening session. During this session, we also recorded ocular pupil size as an implicit measure of listeners' arousal. During the following motor task, participants learned to play target notes of the melodies on a keyboard (notes were of similar motor and musical complexity across melodies). Pupil dilation was greater for liked melodies, particularly when predictable. Motor performance was facilitated in predictable rather than unpredictable melodies, but liked melodies were learned even in the unpredictable condition. Low-liked melodies also showed learning but mostly in participants with higher scores of task perceived competence. Taken together, these results highlight  the effects of stimuli predictability on learning, which can be however overshadowed by the effects of stimulus liking or task-related intrinsic motivation.

Pitch-specific contributions of auditory imagery and auditory memory in vocal pitch imitation

Vocal imitation guides both music and language development. Despite the developmental significance of this behavior, a sizable minority of individuals are inaccurate at vocal pitch imitation. Although previous research suggested that inaccurate pitch imitation results from deficient sensorimotor associations between pitch perception and vocal motor planning, the cognitive processes involved in sensorimotor translation are not clearly defined. In the present research, we investigated the roles of basic cognitive processes in the vocal imitation of pitch, as well as the degree to which these processes rely on pitch-specific resources. In the present study, participants completed a battery of pitch and verbal tasks to measure pitch perception, pitch and verbal auditory imagery, pitch and verbal auditory short-term memory, and pitch imitation ability. Information on participants' music background was collected, as well. Pitch imagery, pitch short-term memory, pitch discrimination ability, and musical experience were unique predictors of pitch imitation ability. Furthermore, pitch imagery was a partial mediator of the relationship between pitch short-term memory and pitch imitation ability. These results indicate that vocal imitation recruits cognitive processes that rely on at least partially separate neural resources for pitch and verbal representations.

Individual differences in musical training and executive functions: A latent variable approach

Learning and performing music draw on a host of cognitive abilities, and previous research has postulated that musicians might have advantages in related cognitive processes. One such aspect of cognition that may be related to musical training is executive functions (EFs), a set of top-down processes that regulate behavior and cognition according to task demands. Previous studies investigating the link between musical training and EFs have yielded mixed results and are difficult to compare. In part, this is because most studies have looked at only one specific cognitive process, and even studies looking at the same process have used different experimental tasks. Furthermore, most correlational studies have used different "musician" and "non-musician" categorizations for their comparisons, so generalizing the findings is difficult. The present study provides a more comprehensive assessment of how individual differences in musical training relate to latent measures of three separable aspects of EFs. We administered a well-validated EF battery containing multiple tasks tapping the EF components of inhibition, shifting, and working memory updating (Friedman et al. in Journal of Experimental Psychology: General, 137, 201-225, 2008), as well as a comprehensive, continuous measure of musical training and sophistication (Müllensiefen et al., in PLoS ONE, 9, e89642, 2014). Musical training correlated with some individual EF tasks involving inhibition and working memory updating, but not with individual tasks involving shifting. However, musical training only predicted the latent variable of working memory updating, but not the latent variables of inhibition or shifting after controlling for IQ, socioeconomic status, and handedness. Although these data are correlational, they nonetheless suggest that musical experience places particularly strong demands specifically on working memory updating processes.

Tonal and textural influences on musical sight-reading

Two experiments investigated the impact of two structural factors-musical tonality and musical texture-on pianists' ability to play by sight without prior preparation, known as musical sight-reading. Tonality refers to the cognitive organization of tones around a central reference pitch, whereas texture refers to the organization of music in terms of the simultaneous versus successive onsets of tones as well as the number of hands (unimanual versus bimanual) involved in performance. Both experiments demonstrated that tonality and texture influenced sight-reading. For tonality, both studies found that errors in performance increased for passages with lesser perceived psychological stability (i.e., minor and atonal passages) relative to greater perceived stability (i.e., major passages). For texture, both studies found that errors in performance increased for passages that were more texturally complex, requiring two-handed versus one-handed performance, with some additional evidence that the relative simultaneity of note onsets (primarily simultaneous versus primarily successive) also influenced errors. These experiments are interpreted within a perception-action framework of music performance, highlighting influences of both top-down cognitive factors and bottom-up motoric processes on sight-reading behavior.

Ears, heads, and eyes: When singers synchronise

We examined the relationship between endogenous rhythms, auditory and visual cues, and body movement in the temporal coordination of duet singers. Sixteen pairs of experienced vocalists sang a familiar melody in Solo and two Duet conditions. Vocalists sang together in Unison (simultaneously producing identical pitches) and Round Duet conditions (one vocalist, the Follower, producing pitches at an eight-tone delay from their partner, the Leader) while facing Inward (full visual cues) and Outward (reduced visual cues). Larger tempo differences in partners' spontaneous (temporally unconstrained) Solo performances were associated with larger asynchrony in Duet performances, consistent with coupling predictions for oscillators with similar natural frequencies. Vocalists were slightly but consistently more synchronous in Duets when facing their partner (Inward) than when facing Outward; Unison and Round performances were equally synchronous. The greater difficulty of Rounds production was evidenced in vocalists' slower performance rates and more variable head movements; Followers directed their head gaze away from their partner and used bobbing head movements to mark the musical beat. The strength of Followers' head movements corresponded to the amount of tone onset asynchrony with their partners, indicating a strong association between timing and movement under increased attentional and working memory demands in music performance.

Electrical Brain Responses Reveal Sequential Constraints on Planning during Music Performance

Elements in speech and music unfold sequentially over time. To produce sentences and melodies quickly and accurately, individuals must plan upcoming sequence events, as well as monitor outcomes via auditory feedback. We investigated the neural correlates of sequential planning and monitoring processes by manipulating auditory feedback during music performance. Pianists performed isochronous melodies from memory at an initially cued rate while their electroencephalogram was recorded. Pitch feedback was occasionally altered to match either an immediately upcoming Near-Future pitch (next sequence event) or a more distant Far-Future pitch (two events ahead of the current event). Near-Future, but not Far-Future altered feedback perturbed the timing of pianists' performances, suggesting greater interference of Near-Future sequential events with current planning processes. Near-Future feedback triggered a greater reduction in auditory sensory suppression (enhanced response) than Far-Future feedback, reflected in the P2 component elicited by the pitch event following the unexpected pitch change. Greater timing perturbations were associated with enhanced cortical sensory processing of the pitch event following the Near-Future altered feedback. Both types of feedback alterations elicited feedback-related negativity (FRN) and P3a potentials and amplified spectral power in the theta frequency range. These findings suggest similar constraints on producers' sequential planning to those reported in speech production.

Shared periodic performer movements coordinate interactions in duo improvisations

Human interaction involves the exchange of temporally coordinated, multimodal cues. Our work focused on interaction in the visual domain, using music performance as a case for analysis due to its temporally diverse and hierarchical structures. We made use of two improvising duo datasets-(i) performances of a jazz standard with a regular pulse and (ii) non-pulsed, free improvizations-to investigate whether human judgements of moments of interaction between co-performers are influenced by body movement coordination at multiple timescales. Bouts of interaction in the performances were manually annotated by experts and the performers' movements were quantified using computer vision techniques. The annotated interaction bouts were then predicted using several quantitative movement and audio features. Over 80% of the interaction bouts were successfully predicted by a broadband measure of the energy of the cross-wavelet transform of the co-performers' movements in non-pulsed duos. A more complex model, with multiple predictors that captured more specific, interacting features of the movements, was needed to explain a significant amount of variance in the pulsed duos. The methods developed here have key implications for future work on measuring visual coordination in musical ensemble performances, and can be easily adapted to other musical contexts, ensemble types and traditions.

Domain-Generality of Timing-Based Serial Order Processes in Short-Term Memory: New Insights from Musical and Verbal Domains

Several models in the verbal domain of short-term memory (STM) consider a dissociation between item and order processing. This view is supported by data demonstrating that different types of time-based interference have a greater effect on memory for the order of to-be-remembered items than on memory for the items themselves. The present study investigated the domain-generality of the item versus serial order dissociation by comparing the differential effects of time-based interfering tasks, such as rhythmic interference and articulatory suppression, on item and order processing in verbal and musical STM domains. In Experiment 1, participants had to maintain sequences of verbal or musical information in STM, followed by a probe sequence, this under different conditions of interference (no-interference, rhythmic interference, articulatory suppression). They were required to decide whether all items of the probe list matched those of the memory list (item condition) or whether the order of the items in the probe sequence matched the order in the memory list (order condition). In Experiment 2, participants performed a serial order probe recognition task for verbal and musical sequences ensuring sequential maintenance processes, under no-interference or rhythmic interference conditions. For Experiment 1, serial order recognition was not significantly more impacted by interfering tasks than was item recognition, this for both verbal and musical domains. For Experiment 2, we observed selective interference of the rhythmic interference condition on both musical and verbal order STM tasks. Overall, the results suggest a similar and selective sensitivity to time-based interference for serial order STM in verbal and musical domains, but only when the STM tasks ensure sequential maintenance processes.

Auditory N1 reveals planning and monitoring processes during music performance

The current study investigated the relationship between planning processes and feedback monitoring during music performance, a complex task in which performers prepare upcoming events while monitoring their sensory outcomes. Theories of action planning in auditory-motor production tasks propose that the planning of future events co-occurs with the perception of auditory feedback. This study investigated the neural correlates of planning and feedback monitoring by manipulating the contents of auditory feedback during music performance. Pianists memorized and performed melodies at a cued tempo in a synchronization-continuation task while the EEG was recorded. During performance, auditory feedback associated with single melody tones was occasionally substituted with tones corresponding to future (next), present (current), or past (previous) melody tones. Only future-oriented altered feedback disrupted behavior: Future-oriented feedback caused pianists to slow down on the subsequent tone more than past-oriented feedback, and amplitudes of the auditory N1 potential elicited by the tone immediately following the altered feedback were larger for future-oriented than for past-oriented or noncontextual (unrelated) altered feedback; larger N1 amplitudes were associated with greater slowing following altered feedback in the future condition only. Feedback-related negativities were elicited in all altered feedback conditions. In sum, behavioral and neural evidence suggests that future-oriented feedback disrupts performance more than past-oriented feedback, consistent with planning theories that posit similarity-based interference between feedback and planning contents. Neural sensory processing of auditory feedback, reflected in the N1 ERP, may serve as a marker for temporal disruption caused by altered auditory feedback in auditory-motor production tasks.

Relating Language and Music Skills in Young Children: A First Approach to Systemize and Compare Distinct Competencies on Different Levels

Children in transition from kindergarten to school develop fundamental skills important for the acquisition of reading and writing. Previous research pointed toward substantial correlations between specific language- and music-related competencies as well as positive transfer effects from music on pre-literacy skills. However, until now the relationship between diverse music and language competencies remains unclear. In the present study, we used a comprehensive approach to clarify the relationships between a broad variety of language and music skills on different levels, not only between but also within domains. In order to do so, we selected representative language- and music-related competencies and systematically compared the performance of N = 44 5- to 7-year-old children with a control group of N = 20 young adults aged from 20 to 30. Competencies were organized in distinct levels according to varying units of vowels/sounds, words or syllables/short melodic or rhythmic phrases, syntax/harmony and context of a whole story/song to test for their interrelatedness within each domain. Following this, we conducted systematic correlation analyses between the competencies of both domains. Overall, selected competencies appeared to be appropriate for the measurement of language and music skills in young children with reference to comprehension, difficulty and a developmental perspective. In line with a hierarchical model of skill acquisition, performance on lower levels was predictive for the performance on higher levels within domains. Moreover, correlations between domains were stronger for competencies reflecting a similar level of cognitive processing, as expected. In conclusion, a systematic comparison of various competencies on distinct levels according to varying units turned out to be appropriate regarding comparability and interrelatedness. Results are discussed with regard to similarities and differences in the development of language and music skills as well as in terms of implications for further research on transfer effects from music on language.

Flexibility of Expressive Timing in Repeated Musical Performances

Performances by soloists in the Western classical tradition are normally highly prepared, yet must sound fresh and spontaneous. How do musicians manage this? We tested the hypothesis that they achieve the necessary spontaneity by varying the musical gestures that express their interpretation of a piece. We examined the tempo arches produced by final slowing at the ends of phrases in performances of J. S. Bach's No. 6 (Prelude) for solo cello (12 performances) and the Italian Concerto (Presto) for solo piano (eight performances). The performances were given by two experienced concert soloists during a short time period (3½ months for the Prelude, 2 weeks for the Presto) after completing their preparations for public performance. We measured the tempo of each bar or half-bar, and the stability of tempo across performances (difference of the tempo of each bar/half bar from each of the other performances). There were phrase arches for both tempo and stability with slower, less stable tempi at beginnings and ends of phrases and faster, more stable tempi mid-phrase. The effects of practice were complex. Tempo decreased overall with practice, while stability increased in some bars and decreased in others. One effect of practice may be to imbue well-learned, automatic motor sequences with freshness and spontaneity through cognitive control at phrase boundaries where slower tempi and decreased stability provide opportunities for slower cognitive processes to modulate rapid automatic motor sequences.

Sequence Memory in Music Performance

How do people remember and produce complex sequences like music or speech? Music provides an example of excellent sequence memory under fast performance conditions; novices as well as skilled musicians can perform memorized music rapidly, without making mistakes. In addition, musical pitches repeat often within a melodic sequence in different orders, yet people do not confuse the sequential ordering; temporal properties of musical pitches aid sequence memory. I describe a contextual model of sequence memory that is sensitive to the rate at which musical sequences are produced and to individual differences among performers. Age and musical experience differentiate adults' and children's memory for musical sequences during performance. Performers' memory for the sequential structure of one melody transfers or generalizes to other melodies in terms of the sequence of pitch events, their temporal properties, and their movements. Motion-analysis techniques provide further views of the time course of the cognitive processes that make sequence memory for music so accurate.

Tuning the mind: Exploring the connections between musical ability and executive functions

A growing body of research suggests that musical experience and ability are related to a variety of cognitive abilities, including executive functioning (EF). However, it is not yet clear if these relationships are limited to specific components of EF, limited to auditory tasks, or reflect very general cognitive advantages. This study investigated the existence and generality of the relationship between musical ability and EFs by evaluating the musical experience and ability of a large group of participants and investigating whether this predicts individual differences on three different components of EF - inhibition, updating, and switching - in both auditory and visual modalities. Musical ability predicted better performance on both auditory and visual updating tasks, even when controlling for a variety of potential confounds (age, handedness, bilingualism, and socio-economic status). However, musical ability was not clearly related to inhibitory control and was unrelated to switching performance. These data thus show that cognitive advantages associated with musical ability are not limited to auditory processes, but are limited to specific aspects of EF. This supports a process-specific (but modality-general) relationship between musical ability and non-musical aspects of cognition.

Word Detection in Sung and Spoken Sentences in Children With Typical Language Development or With Specific Language Impairment

Background: Previous studies have reported that children score better in language tasks using sung rather than spoken stimuli. We examined word detection ease in sung and spoken sentences that were equated for phoneme duration and pitch variations in children aged 7 to 12 years with typical language development (TLD) as well as in children with specific language impairment (SLI ), and hypothesized that the facilitation effect would vary with language abilities. Method: In Experiment 1, 69 children with TLD (7–10 years old) detected words in sentences that were spoken, sung on pitches extracted from speech, and sung on original scores. In Experiment 2, we added a natural speech rate condition and tested 68 children with TLD (7–12 years old). In Experiment 3, 16 children with SLI and 16 age-matched children with TLD were tested in all four conditions. Results: In both TLD groups, older children scored better than the younger ones. The matched TLD group scored higher than the SLI group who scored at the level of the younger children with TLD . None of the experiments showed a facilitation effect of sung over spoken stimuli. Conclusions: Word detection abilities improved with age in both TLD and SLI groups. Our findings are compatible with the hypothesis of delayed language abilities in children with SLI , and are discussed in light of the role of durational prosodic cues in words detection.

Syntax in Action Has Priority over Movement Selection in Piano Playing: An ERP Study

Complex human behavior is hierarchically organized. Whether or not syntax plays a role in this organization is currently under debate. The present ERP study uses piano performance to isolate syntactic operations in action planning and to demonstrate their priority over nonsyntactic levels of movement selection. Expert pianists were asked to execute chord progressions on a mute keyboard by copying the posture of a performing model hand shown in sequences of photos. We manipulated the final chord of each sequence in terms of Syntax (congruent/incongruent keys) and Manner (conventional/unconventional fingering), as well as the strength of its predictability by varying the length of the Context (five-chord/two-chord progressions). The production of syntactically incongruent compared to congruent chords showed a response delay that was larger in the long compared to the short context. This behavioral effect was accompanied by a centroparietal negativity in the long but not in the short context, suggesting that a syntax-based motor plan was prepared ahead. Conversely, the execution of the unconventional manner was not delayed as a function of Context and elicited an opposite electrophysiological pattern (a posterior positivity). The current data support the hypothesis that motor plans operate at the level of musical syntax and are incrementally translated to lower levels of movement selection.

Context and meter enhance long-range planning in music performance

Neural responses demonstrate evidence of resonance, or oscillation, during the production of periodic auditory events. Music contains periodic auditory events that give rise to a sense of beat, which in turn generates a sense of meter on the basis of multiple periodicities. Metrical hierarchies may aid memory for music by facilitating similarity-based associations among sequence events at different periodic distances that unfold in longer contexts. A fundamental question is how metrical associations arising from a musical context influence memory during music performance. Longer contexts may facilitate metrical associations at higher hierarchical levels more than shorter contexts, a prediction of the range model, a formal model of planning processes in music performance (Palmer and Pfordresher, 2003; Pfordresher et al., 2007). Serial ordering errors, in which intended sequence events are produced in incorrect sequence positions, were measured as skilled pianists performed musical pieces that contained excerpts embedded in long or short musical contexts. Pitch errors arose from metrically similar positions and further sequential distances more often when the excerpt was embedded in long contexts compared to short contexts. Musicians' keystroke intensities and error rates also revealed influences of metrical hierarchies, which differed for performances in long and short contexts. The range model accounted for contextual effects and provided better fits to empirical findings when metrical associations between sequence events were included. Longer sequence contexts may facilitate planning during sequence production by increasing conceptual similarity between hierarchically associated events. These findings are consistent with the notion that neural oscillations at multiple periodicities may strengthen metrical associations across sequence events during planning.

Temporal coordination in joint music performance: effects of endogenous rhythms and auditory feedback

Many behaviors require that individuals coordinate the timing of their actions with others. The current study investigated the role of two factors in temporal coordination of joint music performance: differences in partners’ spontaneous (uncued) rate and auditory feedback generated by oneself and one’s partner. Pianists performed melodies independently (in a Solo condition), and with a partner (in a duet condition), either at the same time as a partner (Unison), or at a temporal offset (Round), such that pianists heard their partner produce a serially shifted copy of their own sequence. Access to self-produced auditory information during duet performance was manipulated as well: Performers heard either full auditory feedback (Full), or only feedback from their partner (Other). Larger differences in partners’ spontaneous rates of Solo performances were associated with larger asynchronies (less effective synchronization) during duet performance. Auditory feedback also influenced temporal coordination of duet performance: Pianists were more coordinated (smaller tone onset asynchronies and more mutual adaptation) during duet performances when self-generated auditory feedback aligned with partner-generated feedback (Unison) than when it did not (Round). Removal of self-feedback disrupted coordination (larger tone onset asynchronies) during Round performances only. Together, findings suggest that differences in partners’ spontaneous rates of Solo performances, as well as differences in self- and partner-generated auditory feedback, influence temporal coordination of joint sensorimotor behaviors.

Long-range correlation properties in timing of skilled piano performance: the influence of auditory feedback and deep brain stimulation

Unintentional timing deviations during musical performance can be conceived of as timing errors. However, recent research on humanizing computer-generated music has demonstrated that timing fluctuations that exhibit long-range temporal correlations (LRTC) are preferred by human listeners. This preference can be accounted for by the ubiquitous presence of LRTC in human tapping and rhythmic performances. Interestingly, the manifestation of LRTC in tapping behavior seems to be driven in a subject-specific manner by the LRTC properties of resting-state background cortical oscillatory activity. In this framework, the current study aimed to investigate whether propagation of timing deviations during the skilled, memorized piano performance (without metronome) of 17 professional pianists exhibits LRTC and whether the structure of the correlations is influenced by the presence or absence of auditory feedback. As an additional goal, we set out to investigate the influence of altering the dynamics along the cortico-basal-ganglia-thalamo-cortical network via deep brain stimulation (DBS) on the LRTC properties of musical performance. Specifically, we investigated temporal deviations during the skilled piano performance of a non-professional pianist who was treated with subthalamic-deep brain stimulation (STN-DBS) due to severe Parkinson's disease, with predominant tremor affecting his right upper extremity. In the tremor-affected right hand, the timing fluctuations of the performance exhibited random correlations with DBS OFF. By contrast, DBS restored long-range dependency in the temporal fluctuations, corresponding with the general motor improvement on DBS. Overall, the present investigations demonstrate the presence of LRTC in skilled piano performances, indicating that unintentional temporal deviations are correlated over a wide range of time scales. This phenomenon is stable after removal of the auditory feedback, but is altered by STN-DBS, which suggests that cortico-basal ganglia-thalamocortical circuits play a role in the modulation of the serial correlations of timing fluctuations exhibited in skilled musical performance.

New learning of music after bilateral medial temporal lobe damage: evidence from an amnesic patient

Damage to the hippocampus impairs the ability to acquire new declarative memories, but not the ability to learn simple motor tasks. An unresolved question is whether hippocampal damage affects learning for music performance, which requires motor processes, but in a cognitively complex context. We studied learning of novel musical pieces by sight-reading in a newly identified amnesic, LSJ, who was a skilled amateur violist prior to contracting herpes simplex encephalitis. LSJ has suffered virtually complete destruction of the hippocampus bilaterally, as well as extensive damage to other medial temporal lobe structures and the left anterior temporal lobe. Because of LSJ's rare combination of musical training and near-complete hippocampal destruction, her case provides a unique opportunity to investigate the role of the hippocampus for complex motor learning processes specifically related to music performance. Three novel pieces of viola music were composed and closely matched for factors contributing to a piece's musical complexity. LSJ practiced playing two of the pieces, one in each of the two sessions during the same day. Relative to a third unpracticed control piece, LSJ showed significant pre- to post-training improvement for the two practiced pieces. Learning effects were observed both with detailed analyses of correctly played notes, and with subjective whole-piece performance evaluations by string instrument players. The learning effects were evident immediately after practice and 14 days later. The observed learning stands in sharp contrast to LSJ's complete lack of awareness that the same pieces were being presented repeatedly, and to the profound impairments she exhibits in other learning tasks. Although learning in simple motor tasks has been previously observed in amnesic patients, our results demonstrate that non-hippocampal structures can support complex learning of novel musical sequences for music performance.

A conceptual review on action-perception coupling in the musicians' brain: what is it good for?

Experience with a sensorimotor task, such as practicing a piano piece, leads to strong coupling of sensory (visual or auditory) and motor cortices. Here we review behavioral and neurophysiological (M/EEG, TMS and fMRI) research exploring this topic using the brain of musicians as a model system. Our review focuses on a recent body of evidence suggesting that this form of coupling might have (at least) two cognitive functions. First, it leads to the generation of equivalent predictions (concerning both when and what event is more likely to occur) during both perception and production of music. Second, it underpins the common coding of perception and action that supports the integration of the motor output of multiple musicians’ in the context of joint musical tasks. Essentially, training-based coupling of perception and action might scaffold the human ability to represent complex (structured) actions and to entrain multiple agents—via reciprocal prediction and adaptation—in the pursuit of shared goals.

Making and monitoring errors based on altered auditory feedback

Previous research has demonstrated that altered auditory feedback (AAF) disrupts music performance and causes disruptions in both action planning and the perception of feedback events. It has been proposed that this disruption occurs because of interference within a shared representation for perception and action (Pfordresher, 2006). Studies reported here address this claim from the standpoint of error monitoring. In Experiment 1 participants performed short melodies on a keyboard while hearing no auditory feedback, normal auditory feedback, or alterations to feedback pitch on some subset of events. Participants overestimated error frequency when AAF was present but not for normal feedback. Experiment 2 introduced a concurrent load task to determine whether error monitoring requires executive resources. Although the concurrent task enhanced the effect of AAF, it did not alter participants' tendency to overestimate errors when AAF was present. A third correlational study addressed whether effects of AAF are reduced for a subset of the population who may lack the kind of perception/action associations that lead to AAF disruption: poor-pitch singers. Effects of manipulations similar to those presented in Experiments 1 and 2 were reduced for these individuals. We propose that these results are consistent with the notion that AAF interference is based on associations between perception and action within a forward internal model of auditory-motor relationships.

Translating working memory into action: behavioral and neural evidence for using motor representations in encoding visuo-spatial sequences

The neurobiological organization of action-oriented working memory is not well understood. To elucidate the neural correlates of translating visuo-spatial stimulus sequences into delayed (memory-guided) sequential actions, we measured brain activity using functional magnetic resonance imaging while participants encoded sequences of four to seven dots appearing on fingers of a left or right schematic hand. After variable delays, sequences were to be reproduced with the corresponding fingers. Recall became less accurate with longer sequences and was initiated faster after long delays. Across both hands, encoding and recall activated bilateral prefrontal, premotor, superior and inferior parietal regions as well as the basal ganglia, whereas hand-specific activity was found (albeit to a lesser degree during encoding) in contralateral premotor, sensorimotor, and superior parietal cortex. Activation differences after long versus short delays were restricted to motor-related regions, indicating that rehearsal during long delays might have facilitated the conversion of the memorandum into concrete motor programs at recall. Furthermore, basal ganglia activity during encoding selectively predicted correct recall. Taken together, the results suggest that to-be-reproduced visuo-spatial sequences are encoded as prospective action representations (motor intentions), possibly in addition to retrospective sensory codes. Overall, our study supports and extends multi-component models of working memory, highlighting the notion that sensory input can be coded in multiple ways depending on what the memorandum is to be used for.

Everyday robotic action: lessons from human action control

Robots are increasingly capable of performing everyday human activities such as cooking, cleaning, and doing the laundry. This requires the real-time planning and execution of complex, temporally extended sequential actions under high degrees of uncertainty, which provides many challenges to traditional approaches to robot action control. We argue that important lessons in this respect can be learned from research on human action control. We provide a brief overview of available psychological insights into this issue and focus on four principles that we think could be particularly beneficial for robot control: the integration of symbolic and subsymbolic planning of action sequences, the integration of feedforward and feedback control, the clustering of complex actions into subcomponents, and the contextualization of action-control structures through goal representations.

Estimating working memory capacity for lists of nonverbal sounds

Working memory (WM) capacity limit has been extensively studied in the domains of visual and verbal stimuli. Previous studies have suggested a fixed WM capacity of typically about three or four items, on the basis of the number of items in working memory reaching a plateau after several items as the set size increases. However, the fixed WM capacity estimate appears to rely on categorical information in the stimulus set (Olsson & Poom Proceedings of the National Academy of Sciences 102:8776-8780, 2005). We designed a series of experiments to investigate nonverbal auditory WM capacity and its dependence on categorical information. Experiments 1 and 2 used simple tones and revealed capacity limit of up to two tones following a 6-s retention interval. Importantly, performance was significantly higher at set sizes 2, 3, and 4 when the frequency difference between target and test tones was relatively large. In Experiment 3, we added categorical information to the simple tones, and the effect of tone change magnitude decreased. Maximal capacity for each individual was just over three sounds, in the range of typical visual procedures. We propose that two types of information, categorical and detailed acoustic information, are kept in WM and that categorical information is critical for high WM performance.

Musical expertise and the ability to imagine loudness

Most perceived parameters of sound (e.g. pitch, duration, timbre) can also be imagined in the absence of sound. These parameters are imagined more veridically by expert musicians than non-experts. Evidence for whether loudness is imagined, however, is conflicting. In music, the question of whether loudness is imagined is particularly relevant due to its role as a principal parameter of performance expression. This study addressed the hypothesis that the veridicality of imagined loudness improves with increasing musical expertise. Experts, novices and non-musicians imagined short passages of well-known classical music under two counterbalanced conditions: 1) while adjusting a slider to indicate imagined loudness of the music and 2) while tapping out the rhythm to indicate imagined timing. Subtests assessed music listening abilities and working memory span to determine whether these factors, also hypothesised to improve with increasing musical expertise, could account for imagery task performance. Similarity between each participant's imagined and listening loudness profiles and reference recording intensity profiles was assessed using time series analysis and dynamic time warping. The results suggest a widespread ability to imagine the loudness of familiar music. The veridicality of imagined loudness tended to be greatest for the expert musicians, supporting the predicted relationship between musical expertise and musical imagery ability.

Speed, accuracy, and serial order in sequence production

The production of complex sequences like music or speech requires the rapid and temporally precise production of events (e.g., notes and chords), often at fast rates. Memory retrieval in these circumstances may rely on the simultaneous activation of both the current event and the surrounding context (Lashley, 1951). We describe an extension to a model of incremental retrieval in sequence production (Palmer & Pfordresher, 2003) that incorporates this logic to predict overall error rates and speed-accuracy trade-offs, as well as types of serial ordering errors. The model-assumes that retrieval of the current event is influenced by activations of surrounding events. Activations of surrounding events increase over time, such that both the accessibility of distant events and overall accuracy increases at slower production rates. The model's predictions were tested in an experiment in which pianists performed unfamiliar music at 8 different tempi. Model fits to speed-accuracy data and to serial ordering errors support model predictions. Parameter fits to individual data further suggest that working memory contributes to the retrieval of serial order and overall accuracy is influenced in addition by motor dexterity and domain-specific skill.

Activation of learned action sequences by auditory feedback

The present research addressed whether auditory feedback associated with a learned action sequence can activate the action representation of that sequence. Nonpianist participants learned to perform two melodies at a piano keyboard repeatedly during a trial. The participants heard feedback either from the melody they were performing (normal feedback) or from the other learned melody (termed alternate feedback). An additional tone functioned as an instruction cue to either switch melodies or continue the current melody. Following the instruction cues, participants typically paused just before switching, and paused similarly during trials with a continue cue. Participants paused longer after a continue cue when they experienced alternate rather than normal feedback. This effect was specific to instruction cues positioned at weak metrical accents--positions at which participants were less likely to switch overall. Feedback did not influence timing on switch trials. These findings indicate that influences of auditory feedback can activate learned action sequences, leading to longer latencies associated with cue evaluation.

Rate effects on timing, key velocity, and finger kinematics in piano performance

We examined the effect of rate on finger kinematics in goal-directed actions of pianists. In addition, we evaluated whether movement kinematics can be treated as an indicator of personal identity. Pianists' finger movements were recorded with a motion capture system while they performed melodies from memory at different rates. Pianists' peak finger heights above the keys preceding keystrokes increased as tempo increased, and were attained about one tone before keypress. These rate effects were not simply due to a strategy to increase key velocity (associated with tone intensity) of the corresponding keystroke. Greater finger heights may compensate via greater tactile feedback for a speed-accuracy tradeoff that underlies the tendency toward larger temporal variability at faster tempi. This would allow pianists to maintain high temporal accuracy when playing at fast rates. In addition, finger velocity and accelerations as pianists' fingers approached keys were sufficiently unique to allow pianists' identification with a neural-network classifier. Classification success was higher in pianists with more extensive musical training. Pianists' movement “signatures” may reflect unique goal-directed movement kinematic patterns, leading to individualistic sound.

Sequential performance in young and older adults: evidence of chunking and inhibition

Two experiments were conducted to examine possible sources of age-related decline in sequential performance: age differences in sequence representation, retrieval of sequence elements, and efficiency of inhibitory processes. Healthy young and older participants learned a sequence of eight animal drawings in fixed order, then monitored for these targets within trials of mis-ordered stimuli, responding only when targets were shown in the correct order. Responses were slower for odd numbered targets, suggesting that participants spontaneously organized the sequence in two-element chunks. Perseverations (responses to previously relevant targets) served as an index of inhibitory inefficiency. Efficiency of chunk retrieval and self-inhibition were lower for older than for younger adults. Increasing environmental support in Experiment 2 through overt articulation of current chunk elements showed a pattern of results similar to Experiment 1, with particular benefit for older adults. The findings suggest an underlying susceptibility to interference in old age.

Nobody is perfect: ERP effects prior to performance errors in musicians indicate fast monitoring processes

BACKGROUND

One central question in the context of motor control and action monitoring is at what point in time errors can be detected. Previous electrophysiological studies investigating this issue focused on brain potentials elicited after erroneous responses, mainly in simple speeded response tasks. In the present study, we investigated brain potentials before the commission of errors in a natural and complex situation.

METHODOLOGY/PRINCIPAL FINDINGS

Expert pianists bimanually played scales and patterns while the electroencephalogram (EEG) was recorded. Event-related potentials (ERPs) were computed for correct and incorrect performances. Results revealed differences already 100 ms prior to the onset of a note (i.e., prior to auditory feedback). We further observed that erroneous keystrokes were delayed in time and pressed more slowly.

CONCLUSIONS

Our data reveal neural mechanisms in musicians that are able to detect errors prior to the execution of erroneous movements. The underlying mechanism probably relies on predictive control processes that compare the predicted outcome of an action with the action goal.