Sensorimotor Grounding of Musical Embodiment and the Role of Prediction: A Review

Sensorimotor Synchronization and Neural Entrainment to Imagined Rhythms in Individuals With Proficient Imagery Ability

Sensorimotor synchronization (SMS) is the temporal coordination of motor movements with external or imagined stimuli. Finger-tapping studies indicate better SMS performance with auditory or tactile stimuli compared to visual. However, SMS with a visual rhythm can be improved by enriching stimulus properties (e.g., spatiotemporal content) or individual differences (e.g., one's vividness of auditory imagery). We previously showed that higher self-reported vividness of auditory imagery led to more consistent synchronization-continuation performance when participants continued without a guiding visual rhythm. Here, we examined the contribution of imagery to the SMS performance of proficient imagers, including an auditory or visual distractor task during the continuation phase. While the visual distractor task had minimal effect, SMS consistency was significantly worse when the auditory distractor task was present. Our electroencephalography analysis revealed beat-related neural entrainment, only when the visual or auditory distractor tasks were present. During continuation with the auditory distractor task, the neural entrainment showed an occipital electrode distribution, suggesting the involvement of visual imagery. Unique to SMS continuation with the auditory distractor task, we found neural and sub-vocal (measured with electromyography) entrainment at the three-beat pattern frequency. In this most difficult condition, proficient imagers employed both beat- and pattern-related imagery strategies. However, this combination was insufficient to restore SMS consistency to that observed with visual or no distractor task. Our results suggest that proficient imagers effectively utilized beat-related imagery in one modality when imagery in another modality was limited.

Sensorimotor synchronization with visual, auditory, and tactile modalities

While it is well known that humans are highly responsive to rhythm, the factors that influence our ability to synchronize remain unclear. In the current study, we examined how stimulus modality and rhythmic deviation, along with the synchronizer's level of musicality, impacted sensorimotor synchronization (SMS). Utilizing a finger-tapping task and three sensory modalities (visual, auditory, and tactile), we manipulated rhythmic deviation by varying the temporal position, intensity, and availability of cues across four deviation levels. Additionally, to determine our participants' musical familiarity and aptitude, we administered the Goldsmiths Musical Sophistication Index (Gold-MSI) questionnaire. We found that SMS to external rhythmic stimuli was significantly more precise for auditory and tactile than for visual sequences. Further, we found SMS consistency significantly decreased in all modalities with increased rhythmic deviation, suggesting rhythmic deviation directly relates to SMS difficulty. Moreover, a significant correlation was found between Gold-MSI scores and SMS consistency in the most rhythmically deviant level, such that the higher one's musical general sophistication score, the greater one's SMS ability. This held for all three modalities. Combined, these findings suggest that rhythmic synchronization performance is affected not only by the modality and rhythmic deviation of the stimuli but also by the musical general sophistication of the synchronizer.

Cross-Modal Musical Expectancy in Complex Sound Music: A Grounded Theory

Expectancy is a core mechanism for constructing affective and cognitive experiences of music. However, research on musical expectations has been largely founded upon the perception of tonal music. Therefore, it is still to be determined how this mechanism explains the cognition of sound-based acoustic and electroacoustic music, such as complex sound music (CSM). Additionally, the dominant methodologies have consisted of well-controlled experimental designs with low ecological validity that have overlooked the listening experience as described by the listeners. This paper presents results concerning musical expectancy from a qualitative research project that investigated the listening experiences of 15 participants accustomed to CSM listening. Corbin and Strauss' (2015) grounded theory was used to triangulate data from interviews along with musical analyses of the pieces chosen by the participants to describe their listening experiences. Cross-modal musical expectancy (CMME) emerged from the data as a subcategory that explained prediction through the interaction of multimodal elements beyond just the acoustic properties of music. The results led to hypothesise that multimodal information coming from sounds, performance gestures, and indexical, iconic, and conceptual associations re-enact cross-modal schemata and episodic memories where real and imagined sounds, objects, actions, and narratives interrelate to give rise to CMME processes. This construct emphasises the effect of CSM's subversive acoustic features and performance practices on the listening experience. Further, it reveals the multiplicity of factors involved in musical expectancy, such as cultural values, subjective musical and non-musical experiences, music structure, listening situation, and psychological mechanisms. Following these ideas, CMME is conceived as a grounded cognition process.

Caressed by music: Related preferences for velocity of touch and tempo of music?

Given that both hearing and touch are 'mechanical senses' that respond to physical pressure or mechanical energy and that individuals appear to have a characteristic internal or spontaneous tempo, individual preferences in musical and touch rhythms might be related. We explored this in two experiments probing individual preferences for tempo in the tactile and auditory modalities. Study 1 collected ratings of received stroking on the forearm and measured the velocity the participants used for stroking a fur. Music tempo preferences were assessed as mean beats per minute of individually selected music pieces and via the adjustment of experimenter-selected music to a preferred tempo. Heart rate was recorded to measure levels of physiological arousal. We found that the preferred tempo of favorite (self-selected) music correlated positively with the velocity with which each individual liked to be touched. In Study 2, participants rated videos of repeated touch on someone else's arm and videos of a drummer playing with brushes on a snare drum, both at a variety of tempos. We found that participants with similar rating patterns for the different stroking speeds did not show similar rating patterns for the different music beats. The results suggest that there may be a correspondence between preferences for favorite music and felt touch, but this is either weak or it cannot be evoked effectively with vicarious touch and/or mere drum beats. Thus, if preferences for touch and music are related, this is likely to be dependent on the specific type of stimulation.

The dynamics of musical participation

In this paper we argue that our comprehension of musical participation-the complex network of interactive dynamics involved in collaborative musical experience-can benefit from an analysis inspired by the existing frameworks of dynamical systems theory and coordination dynamics. These approaches can offer novel theoretical tools to help music researchers describe a number of central aspects of joint musical experience in greater detail, such as prediction, adaptivity, social cohesion, reciprocity, and reward. While most musicians involved in collective forms of musicking already have some familiarity with these terms and their associated experiences, we currently lack an analytical vocabulary to approach them in a more targeted way. To fill this gap, we adopt insights from these frameworks to suggest that musical participation may be advantageously characterized as an open, non-equilibrium, dynamical system. In particular, we suggest that research informed by dynamical systems theory might stimulate new interdisciplinary scholarship at the crossroads of musicology, psychology, philosophy, and cognitive (neuro)science, pointing toward new understandings of the core features of musical participation.

Musical Interaction Reveals Music as Embodied Language

Life and social sciences often focus on the social nature of music (and language alike). In biology, for example, the three main evolutionary hypotheses about music (i.e., sexual selection, parent-infant bond, and group cohesion) stress its intrinsically social character (Honing et al., 2015). Neurobiology thereby has investigated the neuronal and hormonal underpinnings of musicality for more than two decades (Chanda and Levitin, 2013; Salimpoor et al., 2015; Mehr et al., 2019). In line with these approaches, the present paper aims to suggest that the proper way to capture the social interactive nature of music (and, before it, musicality), is to conceive of it as an embodied language, rooted in culturally adapted brain structures (Clarke et al., 2015; D'Ausilio et al., 2015). This proposal heeds Ian Cross' call for an investigation of music as an "interactive communicative process" rather than "a manifestation of patterns in sound" (Cross, 2014), with an emphasis on its embodied and predictive (coding) aspects (Clark, 2016; Leman, 2016; Koelsch et al., 2019). In the present paper our goal is: (i) to propose a framework of music as embodied language based on a review of the major concepts that define joint musical action, with a particular emphasis on embodied music cognition and predictive processing, along with some relevant neural underpinnings; (ii) to summarize three experiments conducted in our laboratories (and recently published), which provide evidence for, and can be interpreted according to, the new conceptual framework. In doing so, we draw on both cognitive musicology and neuroscience to outline a comprehensive framework of musical interaction, exploring several aspects of making music in dyads, from a very basic proto-musical action, like tapping, to more sophisticated contexts, like playing a jazz standard and singing a hocket melody. Our framework combines embodied and predictive features, revolving around the concept of joint agency (Pacherie, 2012; Keller et al., 2016; Bolt and Loehr, 2017). If social interaction is the "default mode" by which human brains communicate with their environment (Hari et al., 2015), music and musicality conceived of as an embodied language may arguably provide a route toward its navigation.

Musical novices perform with equal accuracy when learning to drum alone or with a peer

The capacity of expert musicians to coordinate with each other when playing in ensembles or rehearsing has been widely investigated. However, little is known about the ability of novices to achieve satisfactory coordinated behaviour when making music together. We tested whether performance accuracy differs when novices play a newly learned drumming pattern with another musically untrained individual (duo group) or alone (solo group). A comparison between musical outcomes of the two groups revealed no significant differences concerning performative accuracy. An additional, exploratory examination of the degree of mutual influence between members of the duos suggested that they reciprocally affected each other when playing together. These findings indicate that a responsive auditory feedback involving surprises introduced by human errors could be part of pedagogical settings that employ repetition or imitation, thereby facilitating coordination among novices in a less prescribed fashion.

Eye gaze as a means of giving and seeking information during musical interaction

During skilled music ensemble performance, a multi-layered network of interaction processes allows musicians to negotiate common interpretations of ambiguously-notated music in real-time. This study investigated the conditions that encourage visual interaction during duo performance. Duos recorded performances of a new piece before and after a period of rehearsal. Mobile eye tracking and motion capture were used in combination to map uni- and bidirectional eye gaze patterns. Musicians watched each other more during temporally-unstable passages than during regularly-timed passages. They also watched each other more after rehearsal than before. Duo musicians may seek visual interaction with each other primarily, but not exclusively, when coordination is threatened by temporal instability. Visual interaction increases as musicians become familiar with the piece, suggesting that they visually monitor each other once a shared interpretation of the piece is established. Visual monitoring of co-performers' movements and attention may facilitate feelings of engagement and high-level creative collaboration.

Collaborative Musical Creativity: How Ensembles Coordinate Spontaneity

Music performance is inherently social. Most music is performed in groups, and even soloists are subject to influence from a (real or imagined) audience. It is also inherently creative. Performers are called upon to interpret notated music, improvise new musical material, adapt to unexpected playing conditions, and accommodate technical errors. The focus of this paper is how creativity is distributed across members of a music ensemble as they perform these tasks. Some aspects of ensemble performance have been investigated extensively in recent years as part of the broader literature on joint action (e.g., the processes underlying sensorimotor synchronization). Much of this research has been done under highly controlled conditions, using tasks that generate reliable results, but capture only a small part of ensemble performance as it occurs naturalistically. Still missing from this literature is an explanation of how ensemble musicians perform in conditions that require creative interpretation, improvisation, and/or adaptation: how do they coordinate the production of something new? Current theories of creativity endorse the idea that dynamic interaction between individuals, their actions, and their social and material environments underlies creative performance. This framework is much in line with the embodied music cognition paradigm and the dynamical systems perspective on ensemble coordination. This review begins by situating the concept of collaborative musical creativity in the context of embodiment. Progress that has been made toward identifying the mechanisms that underlie collaborative creativity in music performance is then assessed. The focus is on the possible role of musical imagination in facilitating performer flexibility, and on the forms of communication that are likely to support the coordination of creative musical output. Next, emergence and group flow–constructs that seem to characterize ensemble performance at its peak–are considered, and some of the conditions that may encourage periods of emergence or flow are identified. Finally, it is argued that further research is needed to (1) demystify the constructs of emergence and group flow, clarifying their effects on performer experience and listener response, (2) determine how constrained musical imagination is by perceptual experience and understand people's capacity to depart from familiar frameworks and imagine new sounds and sound structures, and (3) assess the technological developments that are supposed to facilitate or enhance musical creativity, and determine what effect they have on the processes underlying creative collaboration.

Creating Time: Social Collaboration in Music Improvisation

Musical collaboration emerges from the complex interaction of environmental and informational constraints, including those of the instruments and the performance context. Music improvisation in particular is more like everyday interaction in that dynamics emerge spontaneously without a rehearsed score or script. We examined how the structure of the musical context affords and shapes interactions between improvising musicians. Six pairs of professional piano players improvised with two different backing tracks while we recorded both the music produced and the movements of their heads, left arms, and right arms. The backing tracks varied in rhythmic and harmonic information, from a chord progression to a continuous drone. Differences in movement coordination and playing behavior were evaluated using the mathematical tools of complex dynamical systems, with the aim of uncovering the multiscale dynamics that characterize musical collaboration. Collectively, the findings indicated that each backing track afforded the emergence of different patterns of coordination with respect to how the musicians played together, how they moved together, as well as their experience collaborating with each other. Additionally, listeners' experiences of the music when rating audio recordings of the improvised performances were related to the way the musicians coordinated both their playing behavior and their bodily movements. Accordingly, the study revealed how complex dynamical systems methods (namely recurrence analysis) can capture the turn-taking dynamics that characterized both the social exchange of the music improvisation and the sounds of collaboration more generally. The study also demonstrated how musical improvisation provides a way of understanding how social interaction emerges from the structure of the behavioral task context.

Enhancement of Pleasure during Spontaneous Dance

Dancing emphasizes the motor expression of emotional experiences. The bodily expression of emotions can modulate the subjective experience of emotions, as when adopting emotion-specific postures and faces. Thus, dancing potentially offers a ground for emotional coping through emotional enhancement and regulation. Here we investigated the emotional responses to music in individuals without any prior dance training while they either freely danced or refrained from movement. Participants were also tested while imitating their own dance movements but in the absence of music as a control condition. Emotional ratings and cardio-respiratory measures were collected following each condition. Dance movements were recorded using motion capture. We found that emotional valence was increased specifically during spontaneous dance of groovy excerpts, compared to both still listening and motor imitation. Furthermore, parasympathetic-related heart rate variability (HRV) increased during dance compared to motor imitation. Nevertheless, subjective and physiological arousal increased during movement production, regardless of whether participants were dancing or imitating. Significant correlations were found between inter-individual differences in the emotions experienced during dance and whole-body acceleration profiles. The combination of movement and music during dance results in a distinct state characterized by acutely heightened pleasure, which is of potential interest for the use of dance in therapeutic settings.

Unimodal and cross-modal prediction is enhanced in musicians

Musical training involves exposure to complex auditory and visual stimuli, memorization of elaborate sequences, and extensive motor rehearsal. It has been hypothesized that such multifaceted training may be associated with differences in basic cognitive functions, such as prediction, potentially translating to a facilitation in expert musicians. Moreover, such differences might generalize to non-auditory stimuli. This study was designed to test both hypotheses. We implemented a cross-modal attentional cueing task with auditory and visual stimuli, where a target was preceded by compatible or incompatible cues in mainly compatible (80% compatible, predictable) or random blocks (50% compatible, unpredictable). This allowed for the testing of prediction skills in musicians and controls. Musicians showed increased sensitivity to the statistical structure of the block, expressed as advantage for compatible trials (disadvantage for incompatible trials), but only in the mainly compatible (predictable) blocks. Controls did not show this pattern. The effect held within modalities (auditory, visual), across modalities, and when controlling for short-term memory capacity. These results reveal a striking enhancement in cross-modal prediction in musicians in a very basic cognitive task.

Assessing the Impact of Movement Consequences on the Development of Early Reaching in Infancy

Prior research on infant reaching has shown that providing infants with repeated opportunities to reach for objects aids the emergence and progression of reaching behavior. This study investigated the effect of movement consequences on the process of learning to reach in pre-reaching infants. Thirty-five infants aged 2.9 months at the onset of the study were randomly assigned to 1 of 3 groups. Two groups received a 14-day intervention to distinct reaching tasks: (1) in a contingent group, a toy target moved and sounded upon contact only, and (2) in a continuous group, the toy moved and sounded continuously, independent of hand-toy contact. A third control group did not receive any intervention; this group's performance was assessed only on 2 days at a 15-day interval. Results revealed that infants in the contingent group made the most progress over time compared to the two other groups. Infants in this group made significantly more overall contacts with the sounding/moving toy, and they increased their rate of visually attended target contacts relative to non-visually attended target contacts compared to the continuous and control groups. Infants in the continuous group did not differ from the control group on the number of hand-toy contacts nor did they show a change in visually attended target versus non-visually attended target contacts ratio over time. However, they did show an increase in movement speed, presumably in an attempt to attain the moving toy. These findings highlight the importance of contingent movement consequences as a critical reinforcer for the selection of action and motor learning in early development. Through repeated opportunities to explore movement consequences, infants discover and select movements that are most successful to the task-at-hand. This study further demonstrates that distinct sensory-motor experiences can have a significant impact on developmental trajectories and can influence the skills young infants will discover through their interactions with their surroundings.