The importance of integration and top-down salience when listening to complex multi-part musical stimuli

Motivated cognitive control during cued anticipation and receipt of unfamiliar musical themes: An fMRI study

Principal themes, particularly choruses in pop songs, hold a central place in human music. Singing along with a familiar chorus tends to elicit pleasure and a sense of belonging, especially in group settings. These principal themes, which frequently serve as musical rewards, are commonly preceded by distinctive musical cues. Such cues guide listeners' attention and amplify their motivation to receive the impending themes. Despite the significance of cue-theme sequences in music, the neural mechanisms underlying the processing of these sequences in unfamiliar songs remain underexplored. To fill this research gap, we employed fMRI to examine neural activity during the cued anticipation of unfamiliar musical themes and the subsequent receipt of their opening phrase. Twenty-three Taiwanese participants underwent fMRI scans while listening to excerpts of Korean slow pop songs unfamiliar to them, with lyrics they could not understand. Our findings revealed distinct temporal dynamics in lateral frontal activity, with posterior regions being more active during theme anticipation and anterior regions during theme receipt. During anticipation, participants reported substantial increases in arousal levels, aligning with the observed enhanced activity in the midbrain, ventral striatum, inferior frontal junction, and premotor regions. We posit that when motivational musical cues are detected, the ventral striatum and inferior frontal junction played a role in attention allocation, while premotor regions may be engaged in monitoring the theme's entry. Notably, both the anticipation and receipt of themes were associated with pronounced activity in the frontal eye field, dorsolateral prefrontal cortex, posterior parietal cortex, dorsal caudate, and salience network. Overall, our results highlight that within a naturalistic music-listening context, the dynamic interplay between the frontoparietal, dopaminergic midbrain-striatal, and salience networks could allow for precise adjustments of control demands based on the cue-theme structure in unfamiliar songs.

Virtual reality-based music attention training for acquired brain injury: A protocol for randomized cross-over trial

Attention training is the primary step in the rehabilitation for patients with acquired brain injury (ABI). While active music performance has been reported to aid neural and functional recovery, its efficacy for patients with ABI remains uncertain due to methodological concerns. The purpose of the study is to develop a virtual reality-based music attention training (VR-MAT), which utilizes a visually guided, bilateral drumming in an immersive environment to train attention and executive functions. We also aims to examine the feasibility and effectiveness of the VR-MAT with a small sample size of participants (3-60 months after ABI, N = 20 approximately). Participants will be randomly assigned to either a waitlist control or music group, in which VR-MAT will take place five times weekly over 4 weeks (randomized crossover design). The evaluation of VR-MAT performance will include accuracy and response time in music responses. Neurocognitive outcome measures will be administered to quantify pre-post changes in attention, working memory, and executive functions. Additionally, functional near-infrared spectroscopy will be employed to explore the relationships between musical behavior, neurocognitive function, and neurophysiological responses.

Prediction errors arising from switches between major and minor modes in music: An fMRI study

The major and minor modes in Western music have positive and negative connotations, respectively. The present fMRI study examined listeners' neural responses to switches between major and minor modes. We manipulated the final chords of J. S. Bach's keyboard pieces so that each major-mode passage ended with either the major (Major-Major) or minor (Major-Minor) tonic chord, and each minor-mode passage ended with either the minor (Minor-Minor) or major (Minor-Major) tonic chord. If the final major and minor chords have positive and negative reward values respectively, the Major-Minor and Minor-Major stimuli would cause negative and positive reward prediction errors (RPEs) respectively in a listener's brain. We found that activity in a frontoparietal network was significantly higher for Major-Minor than for Major-Major. Based on previous research, these results support the idea that a major-to-minor switch causes negative RPE. The contrast of Minor-Major minus Minor-Minor yielded activation in the ventral insula and visual cortex, speaking against the idea that a minor-to-major switch causes positive RPE. We discuss our results in relation to executive functions and the emotional connotations of major versus minor modes.

Alpha oscillations related to self-other integration and distinction during live orchestral performance: A naturalistic case study

Ensemble music performance requires musicians to achieve precise interpersonal coordination while maintaining autonomous control over their own actions. To do so, musicians dynamically shift between integrating other performers' actions into their own action plans and maintaining a distinction between their own and others' actions. Research in laboratory settings has shown that this dynamic process of self-other integration and distinction is indexed by sensorimotor alpha oscillations. The purpose of the current descriptive case study was to examine oscillations related to self-other integration and distinction in a naturalistic performance context. We measured alpha activity from four violinists during a concert hall performance of a 60-musician orchestra. We selected a musical piece from the orchestra's repertoire and, before analyzing alpha activity, performed a score analysis to divide the piece into sections that were expected to strongly promote self-other integration and distinction. In line with previous laboratory findings, performers showed suppressed and enhanced alpha activity during musical sections that promoted self-other integration and distinction, respectively. The current study thus provides preliminary evidence that findings from carefully controlled laboratory experiments generalize to complex real-world performance. Its findings also suggest directions for future research and potential applications of interest to musicians, music educators, and music therapists.

Criterion-Related Validation of a Music-Based Attention Assessment for Individuals with Traumatic Brain Injury

The music-based attention assessment (MAA) is a melody contour identification task that evaluates different types of attention. Previous studies have examined the psychometric and physiological validity of the MAA across various age groups in clinical and typical populations. The purpose of this study was to confirm the MAA's criterion validity in individuals with traumatic brain injury (TBI) and to correlate this with standardized neuropsychological measurements. The MAA and various neurocognitive tests (i.e., the Wechsler adult intelligence scale DST, Delis-Kaplan executive functioning scale color-word interference test, and Conner's continuous performance test) were administered to 38 patients within two weeks prior to or post to the MAA administration. Significant correlations between MAA and neurocognitive batteries were found, indicating the potential of MAA as a valid measure of different types of attention deficits. An additional multiple regression analysis revealed that MAA was a significant factor in predicting attention ability.

Vocal melody and musical background are simultaneously processed by the brain for musical predictions

Musical pleasure is related to the capacity to predict and anticipate the music. By recording early cerebral responses of 16 participants with electroencephalography during periods of silence inserted in known and unknown songs, we aimed to measure the contribution of different musical attributes to musical predictions. We investigated the mismatch between past encoded musical features and the current sensory inputs when listening to lyrics associated with vocal melody, only background instrumental material, or both attributes grouped together. When participants were listening to chords and lyrics for known songs, the brain responses related to musical violation produced event-related potential responses around 150-200 ms that were of a larger amplitude than for chords or lyrics only. Microstate analysis also revealed that for chords and lyrics, the global field power had an increased stability and a longer duration. The source localization identified that the right superior temporal and frontal gyri and the inferior and medial frontal gyri were activated for a longer time for chords and lyrics, likely caused by the increased complexity of the stimuli. We conclude that grouped together, a broader integration and retrieval of several musical attributes at the same time recruit larger neuronal networks that lead to more accurate predictions.

Modulating Cortical Instrument Representations During Auditory Stream Segregation and Integration With Polyphonic Music

Numerous neuroimaging studies demonstrated that the auditory cortex tracks ongoing speech and that, in multi-speaker environments, tracking of the attended speaker is enhanced compared to the other irrelevant speakers. In contrast to speech, multi-instrument music can be appreciated by attending not only on its individual entities (i.e., segregation) but also on multiple instruments simultaneously (i.e., integration). We investigated the neural correlates of these two modes of music listening using electroencephalography (EEG) and sound envelope tracking. To this end, we presented uniquely composed music pieces played by two instruments, a bassoon and a cello, in combination with a previously validated music auditory scene analysis behavioral paradigm (Disbergen et al., 2018). Similar to results obtained through selective listening tasks for speech, relevant instruments could be reconstructed better than irrelevant ones during the segregation task. A delay-specific analysis showed higher reconstruction for the relevant instrument during a middle-latency window for both the bassoon and cello and during a late window for the bassoon. During the integration task, we did not observe significant attentional modulation when reconstructing the overall music envelope. Subsequent analyses indicated that this null result might be due to the heterogeneous strategies listeners employ during the integration task. Overall, our results suggest that subsequent to a common processing stage, top-down modulations consistently enhance the relevant instrument's representation during an instrument segregation task, whereas such an enhancement is not observed during an instrument integration task. These findings extend previous results from speech tracking to the tracking of multi-instrument music and, furthermore, inform current theories on polyphonic music perception.

Multi-Voiced Music Bypasses Attentional Limitations in the Brain

Attentional limits make it difficult to comprehend concurrent speech streams. However, multiple musical streams are processed comparatively easily. Coherence may be a key difference between music and stimuli like speech, which does not rely on the integration of multiple streams for comprehension. The musical organization between melodies in a composition may provide a cognitive scaffold to overcome attentional limitations when perceiving multiple lines of music concurrently. We investigated how listeners attend to multi–voiced music, examining biological indices associated with processing structured versus unstructured music. We predicted that musical structure provides coherence across distinct musical lines, allowing listeners to attend to simultaneous melodies, and that a lack of organization causes simultaneous melodies to be heard as separate streams. Musician participants attended to melodies in a Coherent music condition featuring flute duets and a Jumbled condition where those duets were manipulated to eliminate coherence between the parts. Auditory–evoked cortical potentials were collected to a tone probe. Analysis focused on the N100 response which is primarily generated within the auditory cortex and is larger for attended versus ignored stimuli. Results suggest that participants did not attend to one line over the other when listening to Coherent music, instead perceptually integrating the streams. Yet, for the Jumbled music, effects indicate that participants attended to one line while ignoring the other, abandoning their integration. Our findings lend support for the theory that musical organization aids attention when perceiving multi–voiced music.

EEG Hyperconnectivity Study on Saxophone Quartet Playing in Ensemble

A professional quartet of saxophonists playing in ensemble provides a perfect scenario to study the eventual occurrence of synchronous oscillatory brain activity across subjects. Here, we applied hyperscanning methodologies for simultaneously recordings of electroencephalographic (EEG) signals from four professional saxophonists while they observ an audiovideo recording of their own previous musical performance. An ad-hoc musical composition was written for the study. At debriefing, the subjects were asked to answer two questionnaires to assess their empathy trait and the musical leadership. In order to estimate the hyperconnectivity of each musician we proposed a measure which combines phase synchronization index of brain oscillations and graph theory framework. The inter-connectivity level of each musician was statistically compared. Statistical results revealed a significant lower hyperconnectivity in the left Brodmann area 44 for the Soprano with respect to the other three members. Recent theories attributed this brain region (Broca's area) to music generation, empathy processes and communication. We hypothesize a relationship between brain-to-brain connectivity level and the musical role within the quartet.

Assessing Top-Down and Bottom-Up Contributions to Auditory Stream Segregation and Integration With Polyphonic Music

Polyphonic music listening well exemplifies processes typically involved in daily auditory scene analysis situations, relying on an interactive interplay between bottom-up and top-down processes. Most studies investigating scene analysis have used elementary auditory scenes, however real-world scene analysis is far more complex. In particular, music, contrary to most other natural auditory scenes, can be perceived by either integrating or, under attentive control, segregating sound streams, often carried by different instruments. One of the prominent bottom-up cues contributing to multi-instrument music perception is their timbre difference. In this work, we introduce and validate a novel paradigm designed to investigate, within naturalistic musical auditory scenes, attentive modulation as well as its interaction with bottom-up processes. Two psychophysical experiments are described, employing custom-composed two-voice polyphonic music pieces within a framework implementing a behavioral performance metric to validate listener instructions requiring either integration or segregation of scene elements. In Experiment 1, the listeners' locus of attention was switched between individual instruments or the aggregate (i.e., both instruments together), via a task requiring the detection of temporal modulations (i.e., triplets) incorporated within or across instruments. Subjects responded post-stimulus whether triplets were present in the to-be-attended instrument(s). Experiment 2 introduced the bottom-up manipulation by adding a three-level morphing of instrument timbre distance to the attentional framework. The task was designed to be used within neuroimaging paradigms; Experiment 2 was additionally validated behaviorally in the functional Magnetic Resonance Imaging (fMRI) environment. Experiment 1 subjects (N = 29, non-musicians) completed the task at high levels of accuracy, showing no group differences between any experimental conditions. Nineteen listeners also participated in Experiment 2, showing a main effect of instrument timbre distance, even though within attention-condition timbre-distance contrasts did not demonstrate any timbre effect. Correlation of overall scores with morph-distance effects, computed by subtracting the largest from the smallest timbre distance scores, showed an influence of general task difficulty on the timbre distance effect. Comparison of laboratory and fMRI data showed scanner noise had no adverse effect on task performance. These Experimental paradigms enable to study both bottom-up and top-down contributions to auditory stream segregation and integration within psychophysical and neuroimaging experiments.

Audience gaze while appreciating a multipart musical performance

Visual information has been observed to be crucial for audience members during musical performances. The present study used an eye tracker to investigate audience members' gazes while appreciating an audiovisual musical ensemble performance, based on evidence of the dominance of musical part in auditory attention when listening to multipart music that contains different melody lines and the joint-attention theory of gaze. We presented singing performances, by a female duo. The main findings were as follows: (1) the melody part (soprano) attracted more visual attention than the accompaniment part (alto) throughout the piece, (2) joint attention emerged when the singers shifted their gazes toward their co-performer, suggesting that inter-performer gazing interactions that play a spotlight role mediated performer-audience visual interaction, and (3) musical part (melody or accompaniment) strongly influenced the total duration of gazes among audiences, while the spotlight effect of gaze was limited to just after the singers' gaze shifts.

Rhythm in joint action: psychological and neurophysiological mechanisms for real-time interpersonal coordination

Human interaction often requires simultaneous precision and flexibility in the coordination of rhythmic behaviour between individuals engaged in joint activity, for example, playing a musical duet or dancing with a partner. This review article addresses the psychological processes and brain mechanisms that enable such rhythmic interpersonal coordination. First, an overview is given of research on the cognitive-motor processes that enable individuals to represent joint action goals and to anticipate, attend and adapt to other's actions in real time. Second, the neurophysiological mechanisms that underpin rhythmic interpersonal coordination are sought in studies of sensorimotor and cognitive processes that play a role in the representation and integration of self- and other-related actions within and between individuals' brains. Finally, relationships between social-psychological factors and rhythmic interpersonal coordination are considered from two perspectives, one concerning how social-cognitive tendencies (e.g. empathy) affect coordination, and the other concerning how coordination affects interpersonal affiliation, trust and prosocial behaviour. Our review highlights musical ensemble performance as an ecologically valid yet readily controlled domain for investigating rhythm in joint action.

Brain dynamics that correlate with effects of learning on auditory distance perception

Accuracy in auditory distance perception can improve with practice and varies for sounds differing in familiarity. Here, listeners were trained to judge the distances of English, Bengali, and backwards speech sources pre-recorded at near (2-m) and far (30-m) distances. Listeners' accuracy was tested before and after training. Improvements from pre-test to post-test were greater for forward speech, demonstrating a learning advantage for forward speech sounds. Independent component (IC) processes identified in electroencephalographic (EEG) data collected during pre- and post-testing revealed three clusters of ICs across subjects with stimulus-locked spectral perturbations related to learning and accuracy. One cluster exhibited a transient stimulus-locked increase in 4–8 Hz power (theta event-related synchronization; ERS) that was smaller after training and largest for backwards speech. For a left temporal cluster, 8–12 Hz decreases in power (alpha event-related desynchronization; ERD) were greatest for English speech and less prominent after training. In contrast, a cluster of IC processes centered at or near anterior portions of the medial frontal cortex showed learning-related enhancement of sustained increases in 10–16 Hz power (upper-alpha/low-beta ERS). The degree of this enhancement was positively correlated with the degree of behavioral improvements. Results suggest that neural dynamics in non-auditory cortical areas support distance judgments. Further, frontal cortical networks associated with attentional and/or working memory processes appear to play a role in perceptual learning for source distance.

Segregation and integration of auditory streams when listening to multi-part music

In our daily lives, auditory stream segregation allows us to differentiate concurrent sound sources and to make sense of the scene we are experiencing. However, a combination of segregation and the concurrent integration of auditory streams is necessary in order to analyze the relationship between streams and thus perceive a coherent auditory scene. The present functional magnetic resonance imaging study investigates the relative role and neural underpinnings of these listening strategies in multi-part musical stimuli. We compare a real human performance of a piano duet and a synthetic stimulus of the same duet in a prioritized integrative attention paradigm that required the simultaneous segregation and integration of auditory streams. In so doing, we manipulate the degree to which the attended part of the duet led either structurally (attend melody vs. attend accompaniment) or temporally (asynchronies vs. no asynchronies between parts), and thus the relative contributions of integration and segregation used to make an assessment of the leader-follower relationship. We show that perceptually the relationship between parts is biased towards the conventional structural hierarchy in western music in which the melody generally dominates (leads) the accompaniment. Moreover, the assessment varies as a function of both cognitive load, as shown through difficulty ratings and the interaction of the temporal and the structural relationship factors. Neurally, we see that the temporal relationship between parts, as one important cue for stream segregation, revealed distinct neural activity in the planum temporale. By contrast, integration used when listening to both the temporally separated performance stimulus and the temporally fused synthetic stimulus resulted in activation of the intraparietal sulcus. These results support the hypothesis that the planum temporale and IPS are key structures underlying the mechanisms of segregation and integration of auditory streams, respectively.