Neural correlates of consonance, dissonance, and the hierarchy of musical pitch in the human brainstem

Pleasantness makes a good time: musical consonance shapes interpersonal synchronization in dyadic joint action

Introduction

Music making is a process by which humans across cultures come together to create patterns of sounds that are aesthetically pleasing. What remains unclear is how this aesthetic outcome affects the sensorimotor interaction between participants.

Method

Here we approach this question using an interpersonal sensorimotor synchronization paradigm to test whether the quality of a jointly created chord (consonant vs. dissonant) affects movement coordination. We recruited non-musician participants in dyads to perform a dyadic synchronization-continuation task (dSCT): on each trial, participants first synchronized their movements to a metronome (synchronization phase) and then continued tapping together at the same tempo without the metronome (continuation phase). Each tap yielded a note and participants heard both their own and that of their partner, thus creating a chord that was varied to be either consonant (Perf5 or Maj6) or dissonant (Min2 or Maj2). For each trial, participants also rated the pleasure they felt in creating the sounds together. Additionally, they completed questionnaires about social closeness to the other participant, musical reward sensitivity and musical training.

Results

Results showed that participants' taps were closer in time when they jointly created consonant (high pleasure) vs. dissonant (low pleasure) chords, and that pleasure experienced by the dyad in each trial predicted interpersonal synchronization. However, consonance did not affect individual synchronization with the metronome or individual tapping when the metronome was discontinued. The effect of consonance on synchronization was greater in dyads who reported feeling less close prior to the task.

Discussion

Together, these results highlight the role of consonance in shaping the temporal coordination of our actions with others. More broadly, this work shows that the aesthetic outcome of what we create together affects joint behaviors.

Probing the neural dynamics of musicians' and non-musicians' consonant/dissonant perception: Joint analyses of electrical encephalogram (EEG) and functional magnetic resonance imaging (fMRI)

The perception of two (or more) simultaneous musical notes, depending on their pitch interval(s), could be broadly categorized as consonant or dissonant. Previous literature has suggested that musicians and non-musicians adopt different strategies when discerning music intervals: while musicians rely on the frequency ratios between the two fundamental frequencies, such as "perfect fifth" (3:2) as consonant and "tritone" (45:32) as dissonant intervals; non-musicians may rely on the presence of 'roughness' or 'beats', generated by the difference of fundamental frequencies, as the key elements of 'dissonance'. The separate Event-Related Potential (ERP) differences in N1 and P2 along the midline electrodes provided evidence congruent with such 'separate reliances'. To replicate and to extend, in this study we reran the previous experiment, and separately collected fMRI data of the same protocol (with sparse sampling modifications). The behavioral and EEG results largely corresponded to our previous finding. The fMRI results, with the joint analyses by univariate, psycho-physiological interaction, and representational similarity analysis (RSA) approaches, further reinforce the involvement of central midline-related brain regions, such as ventromedial prefrontal and dorsal anterior cingulate cortex, in consonant/dissonance judgments. The final spatiotemporal searchlight RSA provided convincing evidence that the medial prefrontal cortex, along with the bilateral superior temporal cortex, is the joint locus of midline N1 and dorsal anterior cingulate cortex for the P2 effect (for musicians). Together, these analyses reaffirm that musicians rely more on experience-driven knowledge for consonance/dissonance perception; but also demonstrate the advantages of multiple analyses in constraining the findings from both EEG and fMRI.

Dissonant music engages early visual processing

The neuroscientific examination of music processing in audio-visual contexts offers a valuable framework to assess how auditory information influences the emotional encoding of visual information. Using fMRI during naturalistic film viewing, we investigated the neural mechanisms underlying the effect of music on valence inferences during mental state attribution. Thirty-eight participants watched the same short-film accompanied by systematically controlled consonant or dissonant music. Subjects were instructed to think about the main character's intentions. The results revealed that increasing levels of dissonance led to more negatively valenced inferences, displaying the profound emotional impact of musical dissonance. Crucially, at the neuroscientific level and despite music being the sole manipulation, dissonance evoked the response of the primary visual cortex (V1). Functional/effective connectivity analysis showed a stronger coupling between the auditory ventral stream (AVS) and V1 in response to tonal dissonance and demonstrated the modulation of early visual processing via top-down feedback inputs from the AVS to V1. These V1 signal changes indicate the influence of high-level contextual representations associated with tonal dissonance on early visual cortices, serving to facilitate the emotional interpretation of visual information. Our results highlight the significance of employing systematically controlled music, which can isolate emotional valence from the arousal dimension, to elucidate the brain's sound-to-meaning interface and its distributive crossmodal effects on early visual encoding during naturalistic film viewing.

Neuromagnetic representation of musical roundness in chord progressions

Introduction

Musical roundness perception relies on consonance/dissonance within a rule-based harmonic context, but also on individual characteristics of the listener. The present work tackles these aspects in a combined psychoacoustic and neurophysiological study, taking into account participant's musical aptitude.

Methods

Our paradigm employed cadence-like four-chord progressions, based on Western music theory. Chord progressions comprised naturalistic and artificial sounds; moreover, their single chords varied regarding consonance/dissonance and harmonic function. Thirty participants listened to the chord progressions while their cortical activity was measured with magnetoencephalography; afterwards, they rated the individual chord progressions with respect to their perceived roundness.

Results

Roundness ratings differed according to the degree of dissonance in the dominant chord at the progression's third position; this effect was pronounced in listeners with high musical aptitude. Interestingly, a corresponding pattern occurred in the neuromagnetic N1m response to the fourth chord (i.e., at the progression's resolution), again with somewhat stronger differentiation among musical listeners. The N1m magnitude seemed to increase during chord progressions that were considered particularly round, with the maximum difference after the final chord; here, however, the musical aptitude effect just missed significance.

Discussion

The roundness of chord progressions is reflected in participant's psychoacoustic ratings and in their transient cortical activity, with stronger differentiation among listeners with high musical aptitude. The concept of roundness might help to reframe consonance/dissonance to a more holistic, gestalt-like understanding that covers chord relations in Western music.

Timbral effects on consonance disentangle psychoacoustic mechanisms and suggest perceptual origins for musical scales

The phenomenon of musical consonance is an essential feature in diverse musical styles. The traditional belief, supported by centuries of Western music theory and psychological studies, is that consonance derives from simple (harmonic) frequency ratios between tones and is insensitive to timbre. Here we show through five large-scale behavioral studies, comprising 235,440 human judgments from US and South Korean populations, that harmonic consonance preferences can be reshaped by timbral manipulations, even as far as to induce preferences for inharmonic intervals. We show how such effects may suggest perceptual origins for diverse scale systems ranging from the gamelan's slendro scale to the tuning of Western mean-tone and equal-tempered scales. Through computational modeling we show that these timbral manipulations dissociate competing psychoacoustic mechanisms underlying consonance, and we derive an updated computational model combining liking of harmonicity, disliking of fast beats (roughness), and liking of slow beats. Altogether, this work showcases how large-scale behavioral experiments can inform classical questions in auditory perception.

Subcortical responses to music and speech are alike while cortical responses diverge

Music and speech are encountered daily and are unique to human beings. Both are transformed by the auditory pathway from an initial acoustical encoding to higher level cognition. Studies of cortex have revealed distinct brain responses to music and speech, but differences may emerge in the cortex or may be inherited from different subcortical encoding. In the first part of this study, we derived the human auditory brainstem response (ABR), a measure of subcortical encoding, to recorded music and speech using two analysis methods. The first method, described previously and acoustically based, yielded very different ABRs between the two sound classes. The second method, however, developed here and based on a physiological model of the auditory periphery, gave highly correlated responses to music and speech. We determined the superiority of the second method through several metrics, suggesting there is no appreciable impact of stimulus class (i.e., music vs speech) on the way stimulus acoustics are encoded subcortically. In this study's second part, we considered the cortex. Our new analysis method resulted in cortical music and speech responses becoming more similar but with remaining differences. The subcortical and cortical results taken together suggest that there is evidence for stimulus-class dependent processing of music and speech at the cortical but not subcortical level.

Biological principles for music and mental health

Efforts to integrate music into healthcare systems and wellness practices are accelerating but the biological foundations supporting these initiatives remain underappreciated. As a result, music-based interventions are often sidelined in medicine. Here, I bring together advances in music research from neuroscience, psychology, and psychiatry to bridge music's specific foundations in human biology with its specific therapeutic applications. The framework I propose organizes the neurophysiological effects of music around four core elements of human musicality: tonality, rhythm, reward, and sociality. For each, I review key concepts, biological bases, and evidence of clinical benefits. Within this framework, I outline a strategy to increase music's impact on health based on standardizing treatments and their alignment with individual differences in responsivity to these musical elements. I propose that an integrated biological understanding of human musicality-describing each element's functional origins, development, phylogeny, and neural bases-is critical to advancing rational applications of music in mental health and wellness.

Duplex perception reveals brainstem auditory representations are modulated by listeners' ongoing percept for speech

So-called duplex speech stimuli with perceptually ambiguous spectral cues to one ear and isolated low- versus high-frequency third formant "chirp" to the opposite ear yield a coherent percept supporting their phonetic categorization. Critically, such dichotic sounds are only perceived categorically upon binaural integration. Here, we used frequency-following responses (FFRs), scalp-recorded potentials reflecting phase-locked subcortical activity, to investigate brainstem responses to fused speech percepts and to determine whether FFRs reflect binaurally integrated category-level representations. We recorded FFRs to diotic and dichotic stop-consonants (/da/, /ga/) that either did or did not require binaural fusion to properly label along with perceptually ambiguous sounds without clear phonetic identity. Behaviorally, listeners showed clear categorization of dichotic speech tokens confirming they were heard with a fused, phonetic percept. Neurally, we found FFRs were stronger for categorically perceived speech relative to category-ambiguous tokens but also differentiated phonetic categories for both diotically and dichotically presented speech sounds. Correlations between neural and behavioral data further showed FFR latency predicted the degree to which listeners labeled tokens as "da" versus "ga." The presence of binaurally integrated, category-level information in FFRs suggests human brainstem processing reflects a surprisingly abstract level of the speech code typically circumscribed to much later cortical processing.

The (Co)Evolution of Language and Music Under Human Self-Domestication

Together with language, music is perhaps the most distinctive behavioral trait of the human species. Different hypotheses have been proposed to explain why only humans perform music and how this ability might have evolved in our species. In this paper, we advance a new model of music evolution that builds on the self-domestication view of human evolution, according to which the human phenotype is, at least in part, the outcome of a process similar to domestication in other mammals, triggered by the reduction in reactive aggression responses to environmental changes. We specifically argue that self-domestication can account for some of the cognitive changes, and particularly for the behaviors conducive to the complexification of music through a cultural mechanism. We hypothesize four stages in the evolution of music under self-domestication forces: (1) collective protomusic; (2) private, timbre-oriented music; (3) small-group, pitch-oriented music; and (4) collective, tonally organized music. This line of development encompasses the worldwide diversity of music types and genres and parallels what has been hypothesized for languages. Overall, music diversity might have emerged in a gradual fashion under the effects of the enhanced cultural niche construction as shaped by the progressive decrease in reactive (i.e., impulsive, triggered by fear or anger) aggression and the increase in proactive (i.e., premeditated, goal-directed) aggression.

Duplex perception reveals brainstem auditory representations are modulated by listeners' ongoing percept for speech

So-called duplex speech stimuli with perceptually ambiguous spectral cues to one ear and isolated low- vs. high-frequency third formant "chirp" to the opposite ear yield a coherent percept supporting their phonetic categorization. Critically, such dichotic sounds are only perceived categorically upon binaural integration. Here, we used frequency-following responses (FFRs), scalp-recorded potentials reflecting phase-locked subcortical activity, to investigate brainstem responses to fused speech percepts and to determine whether FFRs reflect binaurally integrated category-level representations. We recorded FFRs to diotic and dichotic stop-consonants (/da/, /ga/) that either did or did not require binaural fusion to properly label along with perceptually ambiguous sounds without clear phonetic identity. Behaviorally, listeners showed clear categorization of dichotic speech tokens confirming they were heard with a fused, phonetic percept. Neurally, we found FFRs were stronger for categorically perceived speech relative to category-ambiguous tokens but also differentiated phonetic categories for both diotically and dichotically presented speech sounds. Correlations between neural and behavioral data further showed FFR latency predicted the degree to which listeners labeled tokens as "da" vs. "ga". The presence of binaurally integrated, category-level information in FFRs suggests human brainstem processing reflects a surprisingly abstract level of the speech code typically circumscribed to much later cortical processing.

Consonance and dissonance perception. A critical review of the historical sources, multidisciplinary findings, and main hypotheses

Revealed more than two millennia ago by Pythagoras, consonance and dissonance (C/D) are foundational concepts in music theory, perception, and aesthetics. The search for the biological, acoustical, and cultural factors that affect C/D perception has resulted in descriptive accounts inspired by arithmetic, musicological, psychoacoustical or neurobiological frameworks without reaching a consensus. Here, we review the key historical sources and modern multidisciplinary findings on C/D and integrate them into three main hypotheses: the vocal similarity hypothesis (VSH), the psychocultural hypothesis (PH), and the sensorimotor hypothesis (SH). By illustrating the hypotheses-related findings, we highlight their major conceptual, methodological, and terminological shortcomings. Trying to provide a unitary framework for C/D understanding, we put together multidisciplinary research on human and animal vocalizations, which converges to suggest that auditory roughness is associated with distress/danger and, therefore, elicits defensive behavioral reactions and neural responses that indicate aversion. We therefore stress the primacy of vocality and roughness as key factors in the explanation of C/D phenomenon, and we explore the (neuro)biological underpinnings of the attraction-aversion mechanisms that are triggered by C/D stimuli. Based on the reviewed evidence, while the aversive nature of dissonance appears as solidly rooted in the multidisciplinary findings, the attractive nature of consonance remains a somewhat speculative claim that needs further investigation. Finally, we outline future directions for empirical research in C/D, especially regarding cross-modal and cross-cultural approaches.

Temporal Pitch Sensitivity in an Animal Model: Psychophysics and Scalp Recordings : Temporal Pitch Sensitivity in Cat

Cochlear implant (CI) users show limited sensitivity to the temporal pitch conveyed by electric stimulation, contributing to impaired perception of music and of speech in noise. Neurophysiological studies in cats suggest that this limitation is due, in part, to poor transmission of the temporal fine structure (TFS) by the brainstem pathways that are activated by electrical cochlear stimulation. It remains unknown, however, how that neural limit might influence perception in the same animal model. For that reason, we developed non-invasive psychophysical and electrophysiological measures of temporal (i.e., non-spectral) pitch processing in the cat. Normal-hearing (NH) cats were presented with acoustic pulse trains consisting of band-limited harmonic complexes that simulated CI stimulation of the basal cochlea while removing cochlear place-of-excitation cues. In the psychophysical procedure, trained cats detected changes from a base pulse rate to a higher pulse rate. In the scalp-recording procedure, the cortical-evoked acoustic change complex (ACC) and brainstem-generated frequency following response (FFR) were recorded simultaneously in sedated cats for pulse trains that alternated between the base and higher rates. The range of perceptual sensitivity to temporal pitch broadly resembled that of humans but was shifted to somewhat higher rates. The ACC largely paralleled these perceptual patterns, validating its use as an objective measure of temporal pitch sensitivity. The phase-locked FFR, in contrast, showed strong brainstem encoding for all tested pulse rates. These measures demonstrate the cat's perceptual sensitivity to pitch in the absence of cochlear-place cues and may be valuable for evaluating neural mechanisms of temporal pitch perception in the feline animal model of stimulation by a CI or novel auditory prostheses.

Rapid Enhancement of Subcortical Neural Responses to Sine-Wave Speech

The efferent auditory nervous system may be a potent force in shaping how the brain responds to behaviorally significant sounds. Previous human experiments using the frequency following response (FFR) have shown efferent-induced modulation of subcortical auditory function online and over short- and long-term time scales; however, a contemporary understanding of FFR generation presents new questions about whether previous effects were constrained solely to the auditory subcortex. The present experiment used sine-wave speech (SWS), an acoustically-sparse stimulus in which dynamic pure tones represent speech formant contours, to evoke FFR_SWS. Due to the higher stimulus frequencies used in SWS, this approach biased neural responses toward brainstem generators and allowed for three stimuli (/bɔ/, /bu/, and /bo/) to be used to evoke FFR_SWSbefore and after listeners in a training group were made aware that they were hearing a degraded speech stimulus. All SWS stimuli were rapidly perceived as speech when presented with a SWS carrier phrase, and average token identification reached ceiling performance during a perceptual training phase. Compared to a control group which remained naïve throughout the experiment, training group FFR_SWS amplitudes were enhanced post-training for each stimulus. Further, linear support vector machine classification of training group FFR_SWS significantly improved post-training compared to the control group, indicating that training-induced neural enhancements were sufficient to bolster machine learning classification accuracy. These results suggest that the efferent auditory system may rapidly modulate auditory brainstem representation of sounds depending on their context and perception as non-speech or speech.

Learning Low-Dimensional Semantics for Music and Language via Multi-Subject fMRI

Embodied Cognition (EC) states that semantics is encoded in the brain as firing patterns of neural circuits, which are learned according to the statistical structure of human multimodal experience. However, each human brain is idiosyncratically biased, according to its subjective experience, making this biological semantic machinery noisy with respect to semantics inherent to media, such as music and language. We propose to represent media semantics using low-dimensional vector embeddings by jointly modeling the functional Magnetic Resonance Imaging (fMRI) activity of several brains via Generalized Canonical Correlation Analysis (GCCA). We evaluate the semantic richness of the resulting latent space in appropriate semantic classification tasks: music genres and language topics. We show that the resulting unsupervised representations outperform the original high-dimensional fMRI voxel spaces in these downstream tasks while being more computationally efficient. Furthermore, we show that joint modeling of several subjects increases the semantic richness of the learned latent vector spaces as the number of subjects increases. Quantitative results and corresponding statistical significance testing demonstrate the instantiation of music and language semantics in the brain, thereby providing further evidence for multimodal embodied cognition as well as a method for extraction of media semantics from multi-subject brain dynamics.

Crossmodal Harmony: Looking for the Meaning of Harmony Beyond Hearing

The notion of harmony was first developed in the context of metaphysics before being applied to the domain of music. However, in recent centuries, the term has often been used to describe especially pleasing combinations of colors by those working in the visual arts too. Similarly, the harmonization of flavors is nowadays often invoked as one of the guiding principles underpinning the deliberate pairing of food and drink. However, beyond the various uses of the term to describe and construct pleasurable unisensory perceptual experiences, it has also been suggested that music and painting may be combined harmoniously (e.g., see the literature on "color music"). Furthermore, those working in the area of "sonic seasoning" sometimes describe certain sonic compositions as harmonizing crossmodally with specific flavor sensations. In this review, we take a critical look at the putative meaning(s) of the term "harmony" when used in a crossmodal, or multisensory, context. Furthermore, we address the question of whether the term's use outside of a strictly unimodal auditory context should be considered literally or merely metaphorically (i.e., as a shorthand to describe those combinations of sensory stimuli that, for whatever reason, appear to go well together, and hence which can be processed especially fluently).

Pleasantness Ratings of Musical Dyads in Cochlear Implant Users

Cochlear implants have been used to restore hearing to more than half a million people around the world. The restored hearing allows most recipients to understand spoken speech without relying on visual cues. While speech comprehension in quiet is generally high for recipients, many complain about the sound of music. The present study examines consonance and dissonance perception in nine cochlear implant users and eight people with no known hearing loss. Participants completed web-based assessments to characterize low-level psychophysical sensitivities to modulation and pitch, as well as higher-level measures of musical pleasantness and speech comprehension in background noise. The underlying hypothesis is that sensitivity to modulation and pitch, in addition to higher levels of musical sophistication, relate to higher-level measures of music and speech perception. This hypothesis tested true with strong correlations observed between measures of modulation and pitch with measures of consonance ratings and speech recognition. Additionally, the cochlear implant users who were the most sensitive to modulations and pitch, and who had higher musical sophistication scores, had similar pleasantness ratings as those with no known hearing loss. The implication is that better coding and focused rehabilitation for modulation and pitch sensitivity will broadly improve perception of music and speech for cochlear implant users.

Neural correlates of acoustic dissonance in music: The role of musicianship, schematic and veridical expectations

In western music, harmonic expectations can be fulfilled or broken by unexpected chords. Musical irregularities in the absence of auditory deviance elicit well-studied neural responses (e.g. ERAN, P3, N5). These responses are sensitive to schematic expectations (induced by syntactic rules of chord succession) and veridical expectations about predictability (induced by experimental regularities). However, the cognitive and sensory contributions to these responses and their plasticity as a result of musical training remains under debate. In the present study, we explored whether the neural processing of pure acoustic violations is affected by schematic and veridical expectations. Moreover, we investigated whether these two factors interact with long-term musical training. In Experiment 1, we registered the ERPs elicited by dissonant clusters placed either at the middle or the ending position of chord cadences. In Experiment 2, we presented to the listeners with a high proportion of cadences ending in a dissonant chord. In both experiments, we compared the ERPs of musicians and non-musicians. Dissonant clusters elicited distinctive neural responses (an early negativity, the P3 and the N5). While the EN was not affected by syntactic rules, the P3a and P3b were larger for dissonant closures than for middle dissonant chords. Interestingly, these components were larger in musicians than in non-musicians, while the N5 was the opposite. Finally, the predictability of dissonant closures in our experiment did not modulate any of the ERPs. Our study suggests that, at early time windows, dissonance is processed based on acoustic deviance independently of syntactic rules. However, at longer latencies, listeners may be able to engage integration mechanisms and further processes of attentional and structural analysis dependent on musical hierarchies, which are enhanced in musicians.

Enhanced brainstem phase-locking in low-level noise reveals stochastic resonance in the frequency-following response (FFR)

In nonlinear systems, the inclusion of low-level noise can paradoxically improve signal detection, a phenomenon known as stochastic resonance (SR). SR has been observed in human hearing whereby sensory thresholds (e.g., signal detection and discrimination) are enhanced in the presence of noise. Here, we asked whether subcortical auditory processing (neural phase locking) shows evidence of SR. We recorded brainstem frequency-following-responses (FFRs) in young, normal-hearing listeners to near-electrophysiological-threshold (40 dB SPL) complex tones composed of 10 iso-amplitude harmonics of 150 Hz fundamental frequency (F0) presented concurrent with low-level noise (+20 to -20 dB SNRs). Though variable and weak across ears, some listeners showed improvement in auditory detection thresholds with subthreshold noise confirming SR psychophysically. At the neural level, low-level FFRs were initially eradicated by noise (expected masking effect) but were surprisingly reinvigorated at select masker levels (local maximum near ∼ 35 dB SPL). These data suggest brainstem phase-locking to near threshold periodic stimuli is enhanced in optimal levels of noise, the hallmark of SR. Our findings provide novel evidence for stochastic resonance in the human auditory brainstem and suggest that under some circumstances, noise can actually benefit both the behavioral and neural encoding of complex sounds.

Cortical hemisphere preference and brainstem ear asymmetry reflect experience-dependent functional modulation of pitch

Temporal attributes of pitch processing at cortical and subcortical levels are differentially weighted and well-coordinated. The question is whether language experience induces functional modulation of hemispheric preference complemented by brainstem ear symmetry for pitch processing. Brainstem frequency-following and cortical pitch responses were recorded concurrently from Mandarin and English participants. A Mandarin syllable with a rising pitch contour was presented to both ears with monaural stimulation. At the cortical level, left ear stimulation in the Chinese group revealed an experience-dependent response for pitch processing in the right hemisphere, consistent with a functionalaccount. The English group revealed a contralateral hemisphere preference consistent with a structuralaccount. At the brainstem level, Chinese participants showed a functional leftward ear asymmetry, whereas English were consistent with a structural account. Overall, language experience modulates both cortical hemispheric preference and brainstem ear asymmetry in a complementary manner to optimize processing of temporal attributes of pitch.

Lemniscal Corticothalamic Feedback in Auditory Scene Analysis

Sound information is transmitted from the ear to central auditory stations of the brain via several nuclei. In addition to these ascending pathways there exist descending projections that can influence the information processing at each of these nuclei. A major descending pathway in the auditory system is the feedback projection from layer VI of the primary auditory cortex (A1) to the ventral division of medial geniculate body (MGBv) in the thalamus. The corticothalamic axons have small glutamatergic terminals that can modulate thalamic processing and thalamocortical information transmission. Corticothalamic neurons also provide input to GABAergic neurons of the thalamic reticular nucleus (TRN) that receives collaterals from the ascending thalamic axons. The balance of corticothalamic and TRN inputs has been shown to refine frequency tuning, firing patterns, and gating of MGBv neurons. Therefore, the thalamus is not merely a relay stage in the chain of auditory nuclei but does participate in complex aspects of sound processing that include top-down modulations. In this review, we aim (i) to examine how lemniscal corticothalamic feedback modulates responses in MGBv neurons, and (ii) to explore how the feedback contributes to auditory scene analysis, particularly on frequency and harmonic perception. Finally, we will discuss potential implications of the role of corticothalamic feedback in music and speech perception, where precise spectral and temporal processing is essential.

Deviant consonance and dissonance capture attention differently only when task demand is high: An ERP study with three-stimulus oddball paradigm

The current study investigated whether consonance (imperfect consonance: major third) and dissonance (minor second) would capture attention differently when they occurred as chords (combinations of two tones) that were deviant from their context. In addition, we also examined how task demand would modulate these chords' attentional capture. For this investigation, we used an auditory three-stimulus oddball paradigm in which these chords were presented as deviant stimuli (5% each) among frequent standard (80%) and infrequent target (10%) pure tones. The task difficulty was manipulated by changing pitch intervals between standard and target tones. The results showed that these chords elicited dual-peak P3a, and that consonance enhanced the late phase of P3a compared to dissonance, only when the task demand was high. These findings revealed that deviant consonance and dissonance captured attention differently; in particular, consonance captured attention more strongly than dissonance, and this effect was induced by high task demand. This attentional capture difference between the chord categories was induced through enhanced focus of attention on the pitch dimension of oddball stimuli. In addition, the deviant chords might have been processed by a mechanism similar to that which processes novel stimuli, and these chords' differences might have affected not the novelty detection process, but a process which orients attentional resources to deviant chords, which were recognized as novel stimuli.

Individual differences in voice adaptability are specifically linked to voice perception skill

There are remarkable individual differences in the ability to recognise individuals by the sound of their voice. Theoretically, this ability is thought to depend on the coding accuracy of voices in a low-dimensional "voice-space". Here we were interested in how adaptive coding of voice identity relates to this variability in skill. In two adaptation experiments we explored first whether the aftereffect size to two familiar vocal identities can predict voice perception ability and second, whether this effect stems from general auditory skill (e.g. discrimination ability for tuning and tempo). Experiment 1 demonstrated that contrastive aftereffect sizes for voice identity predicted voice perception ability. In Experiment 2, we replicated this finding and further established that this effect is unrelated to general auditory abilities or general adaptability of listeners. Our results highlight the important functional role of adaptive coding in voice expertise and suggest that human voice perception is a highly specialised and distinct auditory ability.

Factors influencing classification of frequency following responses to speech and music stimuli

Successful mapping of meaningful labels to sound input requires accurate representation of that sound's acoustic variances in time and spectrum. For some individuals, such as children or those with hearing loss, having an objective measure of the integrity of this representation could be useful. Classification is a promising machine learning approach which can be used to objectively predict a stimulus label from the brain response. This approach has been previously used with auditory evoked potentials (AEP) such as the frequency following response (FFR), but a number of key issues remain unresolved before classification can be translated into clinical practice. Specifically, past efforts at FFR classification have used data from a given subject for both training and testing the classifier. It is also unclear which components of the FFR elicit optimal classification accuracy. To address these issues, we recorded FFRs from 13 adults with normal hearing in response to speech and music stimuli. We compared labeling accuracy of two cross-validation classification approaches using FFR data: (1) a more traditional method combining subject data in both the training and testing set, and (2) a "leave-one-out" approach, in which subject data is classified based on a model built exclusively from the data of other individuals. We also examined classification accuracy on decomposed and time-segmented FFRs. Our results indicate that the accuracy of leave-one-subject-out cross validation approaches that obtained in the more conventional cross-validation classifications while allowing a subject's results to be analysed with respect to normative data pooled from a separate population. In addition, we demonstrate that classification accuracy is highest when the entire FFR is used to train the classifier. Taken together, these efforts contribute key steps toward translation of classification-based machine learning approaches into clinical practice.

Evaluation of the Efficacy of Musical Vibroacupuncture in Pain Relief: A Randomized Controlled Pilot Study

OBJECTIVES

To investigate if skin vibration employing consonant frequencies emitted by skin transducers attached to a combination of acupuncture points and according to musical harmony (musical chord) produces more significant pain relief compared to just a single frequency.

MATERIALS AND METHODS

Skin vibrostimulation produced by five electromagnet transducers was applied at five acupoints traditionally used to pain relief and anxiety in 13 pain-free healthy volunteers using the cold pressor test (CPT). The study consisted of three randomized sessions conducted on alternate days, with participants receiving either simultaneous frequencies of 32, 48, and 64 Hz that equate those used in a musical chord, hereby defined as musical vibroacupuncture (MVA), a single frequency of 32 Hz, set as vibroacupuncture (VA) and sham procedure (SP). CPT scores for pain thresholds and pain tolerance were assessed using repeated-measures ANOVAs. Pain intensity was evaluated using a numerical rating scale (NRS), while sensory and affective aspects of pain were rated using the short-form McGill Pain Questionnaire (SF-MPQ) and State-Trait Anxiety Inventory (STAI) Y-Form.

RESULTS

Pain thresholds did not vary significantly between trials. Pain tolerance scores were markedly higher in MVA compared to baseline (p = 0.0043) or SP (p = 0.006) but not for VA. Pain intensity for MVA also differed significantly from the baseline (p = 0.007) or SP (p = 0.027), but not for VA. No significant differences were found in SF-MPQ and STAI questionnaires.

CONCLUSIONS

These results suggest that MVA effectively increased pain tolerance and reduced pain intensity when compared with all groups, although not significant to the VA group.

Simultaneous consonance in music perception and composition

Simultaneous consonance is a salient perceptual phenomenon corresponding to the perceived pleasantness of simultaneously sounding musical tones. Various competing theories of consonance have been proposed over the centuries, but recently a consensus has developed that simultaneous consonance is primarily driven by harmonicity perception. Here we question this view, substantiating our argument by critically reviewing historic consonance research from a broad variety of disciplines, reanalyzing consonance perception data from 4 previous behavioral studies representing more than 500 participants, and modeling three Western musical corpora representing more than 100,000 compositions. We conclude that simultaneous consonance is a composite phenomenon that derives in large part from three phenomena: interference, periodicity/harmonicity, and cultural familiarity. We formalize this conclusion with a computational model that predicts a musical chord's simultaneous consonance from these three features, and release this model in an open-source R package, incon, alongside 15 other computational models also evaluated in this paper. We hope that this package will facilitate further psychological and musicological research into simultaneous consonance. (PsycINFO Database Record (c) 2020 APA, all rights reserved).

Cortical Patterns for Prediction of Subjective Preference Induced by Chords

To extract an effective feature in prediction of subjective impressions from single-trial neurophysiological recordings, the spatial filter that extracts brain activities related to impressions were constructed using the common spatial pattern (CSP). We focus on subjective preference induced by chords composed of 3 notes with different frequency ratio. Magnetic cortical activities while hearing chords and comparative judgment on pair of them were measured. The predictive model that predicts the scale value of preference was trained using the CSP-based feature for each participant. The result of the evaluation experiment shows that the CSP-based feature improved the mean prediction accuracy in all participants, compared with the other features without spatially filtering. Furthermore, the capability of construction of a spatial filter that extracts cortical activities varying with degree of preference using the comparative judgments was indicated.

Consonance perception beyond the traditional existence region of pitch

Some theories posit that the perception of consonance is based on neural periodicity detection, which is dependent on accurate phase locking of auditory nerve fibers to features of the stimulus waveform. In the current study, 15 listeners were asked to rate the pleasantness of complex tone dyads (2 note chords) forming various harmonic intervals and bandpass filtered in a high-frequency region (all components >5.8 kHz), where phase locking to the rapid stimulus fine structure is thought to be severely degraded or absent. The two notes were presented to opposite ears. Consonant intervals (minor third and perfect fifth) received higher ratings than dissonant intervals (minor second and tritone). The results could not be explained in terms of phase locking to the slower waveform envelope because the preference for consonant intervals was higher when the stimuli were harmonic, compared to a condition in which they were made inharmonic by shifting their component frequencies by a constant offset, so as to preserve their envelope periodicity. Overall the results indicate that, if phase locking is indeed absent at frequencies greater than ∼5 kHz, neural periodicity detection is not necessary for the perception of consonance.

Dissonant endings of chord progressions elicit a larger ERAN than ambiguous endings in musicians

In major-minor tonal music, the hierarchical relationships and patterns of tension/release are essential for its composition and experience. For most listeners, tension leads to an expectation of resolution. Thus, when musical expectations are broken, they are usually perceived as erroneous and elicit specific neural responses such as the early right anterior negativity (ERAN). In the present study, we explored if different degrees of musical violations are processed differently after long-term musical training in comparison to day-to-day exposure. We registered the ERPs elicited by listening to unexpected chords in both musicians and nonmusicians. More specifically, we compared the responses of strong violations by unexpected dissonant endings and mild violations by unexpected but consonant endings (Neapolitan chords). Our results show that, irrespective of training, irregular endings elicited the ERAN. However, the ERAN for dissonant endings was larger in musicians than in nonmusicians. More importantly, we observed a modulation of the neural responses by the degree of violation only in musicians. In this group, the amplitude of the ERAN was larger for strong than for mild violations. These results suggest an early sensitivity of musicians to dissonance, which is processed as less expected than tonal irregularities. We also found that irregular endings elicited a P3 only in musicians. Our study suggests that, even though violations of harmonic expectancies are detected by all listeners, musical training modulates how different violations of the musical context are processed.

Pleasantness Ratings for Harmonic Intervals With Acoustic and Electric Hearing in Unilaterally Deaf Cochlear Implant Patients

Background

Harmony is an important part of tonal music that conveys context, form and emotion. Two notes sounded simultaneously form a harmonic interval. In normal-hearing (NH) listeners, some harmonic intervals (e.g., minor 2nd, tritone, major 7th) typically sound more dissonant than others (e.g., octave, major 3rd, 4th). Because of the limited spectro-temporal resolution afforded by cochlear implants (CIs), music perception is generally poor. However, CI users may still be sensitive to relative dissonance across intervals. In this study, dissonance ratings for harmonic intervals were measured in 11 unilaterally deaf CI patients, in whom ratings from the CI could be compared to those from the normal ear.

Methods

Stimuli consisted of pairs of equal amplitude MIDI piano tones. Intervals spanned a range of two octaves relative to two root notes (F3 or C4). Dissonance was assessed in terms of subjective pleasantness ratings for intervals presented to the NH ear alone, the CI ear alone, and both ears together (NH + CI). Ratings were collected for both root notes for within- and across-octave intervals (1–12 and 13–24 semitones). Participants rated the pleasantness of each interval by clicking on a line anchored with “least pleasant” and “most pleasant.” A follow-up experiment repeated the task with a smaller stimulus set.

Results

With NH-only listening, within-octave intervals minor 2nd, major 2nd, and major 7th were rated least pleasant; major 3rd, 5th, and octave were rated most pleasant. Across-octave counterparts were similarly rated. With CI-only listening, ratings were consistently lower and showed a reduced range. Mean ratings were highly correlated between NH-only and CI-only listening (r = 0.845, p < 0.001). Ratings were similar between NH-only and NH + CI listening, with no significant binaural enhancement/interference. The follow-up tests showed that ratings were reliable for the least and most pleasant intervals.

Discussion

Although pleasantness ratings were less differentiated for the CI ear than the NH ear, there were similarities between the two listening modes. Given the lack of spectro-temporal detail needed for harmonicity-based distinctions, temporal envelope interactions (within and across channels) associated with a perception of roughness may contribute to dissonance perception for harmonic intervals with CI-only listening.

Biometric identification of listener identity from frequency following responses to speech

OBJECTIVE

We investigate the biometric specificity of the frequency following response (FFR), an EEG marker of early auditory processing that reflects phase-locked activity from neural ensembles in the auditory cortex and subcortex (Chandrasekaran and Kraus 2010, Bidelman, 2015a, 2018, Coffey et al 2017b). Our objective is two-fold: demonstrate that the FFR contains information beyond stimulus properties and broad group-level markers, and to assess the practical viability of the FFR as a biometric across different sounds, auditory experiences, and recording days.

APPROACH

We trained the hidden Markov model (HMM) to decode listener identity from FFR spectro-temporal patterns across multiple frequency bands. Our dataset included FFRs from twenty native speakers of English or Mandarin Chinese (10 per group) listening to Mandarin Chinese tones across three EEG sessions separated by days. We decoded subject identity within the same auditory context (same tone and session) and across different stimuli and recording sessions.

MAIN RESULTS

The HMM decoded listeners for averaging sizes as small as one single FFR. However, model performance improved for larger averaging sizes (e.g. 25 FFRs), similarity in auditory context (same tone and day), and lack of familiarity with the sounds (i.e. native English relative to native Chinese listeners). Our results also revealed important biometric contributions from frequency bands in the cortical and subcortical EEG.

SIGNIFICANCE

Our study provides the first deep and systematic biometric characterization of the FFR and provides the basis for biometric identification systems incorporating this neural signal.

Atonal Music: Can Uncertainty Lead to Pleasure?

In recent years, the field of neuroaesthetics has gained considerable attention with music being a favored object of study. The majority of studies concerning music have, however, focused on the experience of Western tonal music (TM), which is characterized by tonal hierarchical organization, a high degree of consonance, and a tendency to provide the listener with a tonic as a reference point during the listening experience. We argue that a narrow focus on Western TM may have led to a one-sided view regarding the qualities of the aesthetic experience of music since Western art music from the 20th and 21st century like atonal music (AM) – while lacking a tonal hierarchical structure, and while being highly dissonant and hard to predict – is nevertheless enjoyed by a group of avid listeners. We propose a research focus that investigates, in particular, the experience of AM as a novel and compelling way with which to enhance our understanding of both the aesthetic appreciation of music and the role of predictive models in the context of musical pleasure. We use music theoretical analysis and music information retrieval methods to demonstrate how AM presents the listener with a highly uncertain auditory environment. Specifically, an analysis of a corpus of 100 musical segments is used to illustrate how tonal classical music and AM differ quantitatively in terms of both key and pulse clarity values. We then examine person related, extrinsic and intrinsic factors, that point to potential mechanisms underlying the appreciation and pleasure derived from AM. We argue that personality traits like “openness to experience,” the framing of AM as art, and the mere exposure effect are key components of such mechanisms. We further argue that neural correlates of uncertainty estimation could represent a central mechanism for engaging with AM and that such contexts engender a comparatively weak predictive model in the listener. Finally we argue that in such uncertain contexts, correct predictions may be more subjectively rewarding than prediction errors since they signal to the individual that their predictive model is improving.

Early neural responses underlie advantages for consonance over dissonance

Consonant musical intervals tend to be more readily processed than dissonant intervals. In the present study, we explore the neural basis for this difference by registering how the brain responds after changes in consonance and dissonance, and how formal musical training modulates these responses. Event-related brain potentials (ERPs) were registered while participants were presented with sequences of consonant intervals interrupted by a dissonant interval, or sequences of dissonant intervals interrupted by a consonant interval. Participants were musicians and non-musicians. Our results show that brain responses triggered by changes in a consonant context differ from those triggered in a dissonant context. Changes in a sequence of consonant intervals are rapidly processed independently of musical expertise, as revealed by a change-related mismatch negativity (MMN, a component of the ERPs triggered by an odd stimulus in a sequence of stimuli) elicited in both musicians and non-musicians. In contrast, changes in a sequence of dissonant intervals elicited a late MMN only in participants with prolonged musical training. These different neural responses might form the basis for the processing advantages observed for consonance over dissonance and provide information about how formal musical training modulates them.

•

We registered ERPs after deviant intervals in consonant and dissonant sequences.

•

Violations of consonant sequences are detected independently of musical expertise.

•

Musical training modulates responses to violations of dissonant sequences.

•

These neural responses might form the basis for a processing advantage of consonance.

Computational Approach to Musical Consonance and Dissonance

In sixth century BC, Pythagoras discovered the mathematical foundation of musical consonance and dissonance. When auditory frequencies in small-integer ratios are combined, the result is a harmonious perception. In contrast, most frequency combinations result in audible, off-centered by-products labeled “beating” or “roughness;” these are reported by most listeners to sound dissonant. In this paper, we consider second-order beats, a kind of beating recognized as a product of neural processing, and demonstrate that the data-driven approach of Recurrence Quantification Analysis (RQA) allows for the reconstruction of the order in which interval ratios are ranked in music theory and harmony. We take advantage of computer-generated sounds containing all intervals over the span of an octave. To visualize second-order beats, we use a glissando from the unison to the octave. This procedure produces a profile of recurrence values that correspond to subsequent epochs along the original signal. We find that the higher recurrence peaks exactly match the epochs corresponding to just intonation frequency ratios. This result indicates a link between consonance and the dynamical features of the signal. Our findings integrate a new element into the existing theoretical models of consonance, thus providing a computational account of consonance in terms of dynamical systems theory. Finally, as it considers general features of acoustic signals, the present approach demonstrates a universal aspect of consonance and dissonance perception and provides a simple mathematical tool that could serve as a common framework for further neuro-psychological and music theory research.

Subcortical sources dominate the neuroelectric auditory frequency-following response to speech

Frequency-following responses (FFRs) are neurophonic potentials that provide a window into the encoding of complex sounds (e.g., speech/music), auditory disorders, and neuroplasticity. While the neural origins of the FFR remain debated, renewed controversy has reemerged after demonstration that FFRs recorded via magnetoencephalography (MEG) are dominated by cortical rather than brainstem structures as previously assumed. Here, we recorded high-density (64 ch) FFRs via EEG and applied state-of-the art source imaging techniques to multichannel data (discrete dipole modeling, distributed imaging, independent component analysis, computational simulations). Our data confirm a mixture of generators localized to bilateral auditory nerve (AN), brainstem inferior colliculus (BS), and bilateral primary auditory cortex (PAC). However, frequency-specific scrutiny of source waveforms showed the relative contribution of these nuclei to the aggregate FFR varied across stimulus frequencies. Whereas AN and BS sources produced robust FFRs up to ∼700 Hz, PAC showed weak phase-locking with little FFR energy above the speech fundamental (100 Hz). Notably, CLARA imaging further showed PAC activation was eradicated for FFRs >150 Hz, above which only subcortical sources remained active. Our results show (i) the site of FFR generation varies critically with stimulus frequency; and (ii) opposite the pattern observed in MEG, subcortical structures make the largest contribution to electrically recorded FFRs (AN ≥ BS > PAC). We infer that cortical dominance observed in previous neuromagnetic data is likely due to the bias of MEG to superficial brain tissue, underestimating subcortical structures that drive most of the speech-FFR. Cleanly separating subcortical from cortical FFRs can be achieved by ensuring stimulus frequencies are >150-200 Hz, above the phase-locking limit of cortical neurons.

Not Cure But Heal: Music and Medicine

Do you know that our soul is composed of harmony? Leonardo Da Vinci Despite evidence for music-specific mechanisms at the level of pitch-pattern representations, the most fascinating aspect of music is its transmodality. Recent psychological and neuroscientific evidence suggest that music is unique in the coupling of perception, cognition, action and emotion. This potentially explains why music has been since time immemorial almost inextricably linked to healing processes and should continue to be.

Vocal similarity predicts the relative attraction of musical chords

The foundations of human music have long puzzled philosophers, mathematicians, psychologists, and neuroscientists. Although virtually all cultures uses combinations of tones as a basis for musical expression, why humans favor some tone combinations over others has been debated for millennia. Here we show that our attraction to specific tone combinations played simultaneously (chords) is predicted by their spectral similarity to voiced speech sounds. This connection between auditory aesthetics and a primary characteristic of vocalization adds to other evidence that tonal preferences arise from the biological advantages of social communication mediated by speech and language.

Musical chords are combinations of two or more tones played together. While many different chords are used in music, some are heard as more attractive (consonant) than others. We have previously suggested that, for reasons of biological advantage, human tonal preferences can be understood in terms of the spectral similarity of tone combinations to harmonic human vocalizations. Using the chromatic scale, we tested this theory further by assessing the perceived consonance of all possible dyads, triads, and tetrads within a single octave. Our results show that the consonance of chords is predicted by their relative similarity to voiced speech sounds. These observations support the hypothesis that the relative attraction of musical tone combinations is due, at least in part, to the biological advantages that accrue from recognizing and responding to conspecific vocal stimuli.

Calculation of the acoustical properties of triadic harmonies

The author reports that the harmonic "tension" and major/minor "valence" of pitch combinations can be calculated directly from acoustical properties without relying on concepts from traditional harmony theory. The capability to compute the well-known types of harmonic triads means that their perception is not simply a consequence of learning an arbitrary cultural "idiom" handed down from the Italian Renaissance. On the contrary, for typical listeners familiar with diatonic music, attention to certain, definable, acoustical features underlies the perception of the valence (modality) and the inherent tension (instability) of three-tone harmonies.

Reduction of the Harmonic Series Influences Musical Enjoyment With Cochlear Implants

OBJECTIVE

Cochlear implantation is associated with poor music perception and enjoyment. Reducing music complexity has been shown to enhance music enjoyment in cochlear implant (CI) recipients. In this study, we assess the impact of harmonic series reduction on music enjoyment.

STUDY DESIGN

Prospective analysis of music enjoyment in normal-hearing (NH) individuals and CI recipients.

SETTING

Single tertiary academic medical center.

PATIENTS

NH adults (N = 20) and CI users (N = 8) rated the Happy Birthday song on three validated enjoyment modalities-musicality, pleasantness, and naturalness.

INTERVENTION

Subjective rating of music excerpts.

MAIN OUTCOME MEASURES

Participants listened to seven different instruments play the melody, each with five levels of harmonic reduction (Full, F3+F2+F1+F0, F2+F1+F0, F1+F0, F0). NH participants listened to the segments both with and without CI simulation. Linear mixed effect models (LME) and likelihood ratio tests were used to assess the impact of harmonic reduction on enjoyment.

RESULTS

NH listeners without simulation rated segments with the first four harmonics (F3+F2+F1+F0) most pleasant and natural (p <0.001, p = 0.004). NH listeners with simulation rated the first harmonic alone (F0) most pleasant and natural (p <0.001, p = 0.003). Their ratings demonstrated a positive linear relationship between harmonic reduction and both pleasantness (slope estimate = 0.030, SE = 0.004, p <0.001, LME) and naturalness (slope estimate = 0.012, SE = 0.003, p = 0.003, LME). CI recipients also found the first harmonic alone (F0) to be most pleasant (p = 0.003), with a positive linear relationship between harmonic reduction and pleasantness (slope estimate = 0.029, SE = 0.008, p <0.001, LME).

CONCLUSION

Harmonic series reduction increases music enjoyment in CI and NH individuals with or without CI simulation. Therefore, minimization of the harmonics may be a useful strategy for enhancing musical enjoyment among both NH and CI listeners.

Dissonance encoding in human inferior colliculus covaries with individual differences in dislike of dissonant music

Harmony is one of the most fundamental elements of music that evokes emotional response. The inferior colliculus (IC) has been known to detect poor agreement of harmonics of sound, that is, dissonance. Electrophysiological evidence has implicated a relationship between a sustained auditory response mainly from the brainstem and unpleasant emotion induced by dissonant harmony. Interestingly, an individual’s dislike of dissonant harmony of an individual correlated with a reduced sustained auditory response. In the current paper, we report novel evidence based on functional magnetic resonance imaging (fMRI) for such a relationship between individual variability in dislike of dissonance and the IC activation. Furthermore, for the first time, we show how dissonant harmony modulates functional connectivity of the IC and its association with behaviourally reported unpleasantness. The current findings support important contributions of low level auditory processing and corticofugal interaction in musical harmony preference.

Successful non-native speech perception is linked to frequency following response phase consistency

Some people who attempt to learn a second language in adulthood meet with greater success than others. The causes driving these individual differences in second language learning skill continue to be debated. In particular, it remains controversial whether robust auditory perception can provide an advantage for non-native speech perception. Here, we tested English speech perception in native Japanese speakers through the use of frequency following responses, the evoked gamma band response, and behavioral measurements. Participants whose neural responses featured less timing jitter from trial to trial performed better on perception of English consonants than participants with more variable neural timing. Moreover, this neural metric predicted consonant perception to a greater extent than did age of arrival and length of residence in the UK, and neural jitter predicted independent variance in consonant perception after these demographic variables were accounted for. Thus, difficulties with auditory perception may be one source of problems learning second languages in adulthood.