Decoding Music-Evoked Emotions in the Auditory and Motor Cortex

Lights on music cognition: A systematic and critical review of fNIRS applications and future perspectives

Research investigating the neural processes related to music perception and production constitutes a well-established field within the cognitive neurosciences. While most neuroimaging tools have limitations in studying the complexity of musical experiences, functional Near-Infrared Spectroscopy (fNIRS) represents a promising, relatively new tool for studying music processes in both laboratory and ecological settings, which is also suitable for both typical and pathological populations across development. Here we systematically review fNIRS studies on music cognition, highlighting prospects and potentialities. We also include an overview of fNIRS basic theory, together with a brief comparison to characteristics of other neuroimaging tools. Fifty-nine studies meeting inclusion criteria (i.e., using fNIRS with music as the primary stimulus) are presented across five thematic sections. Critical discussion of methodology leads us to propose guidelines of good practices aiming for robust signal analyses and reproducibility. A continuously updated world map is proposed, including basic information from studies meeting the inclusion criteria. It provides an organized, accessible, and updatable reference database, which could serve as a catalyst for future collaborations within the community. In conclusion, fNIRS shows potential for investigating cognitive processes in music, particularly in ecological contexts and with special populations, aligning with current research priorities in music cognition.

Choir singing is associated with enhanced structural connectivity across the adult lifespan

The global ageing of populations calls for effective, ecologically valid methods to support brain health across adult life. Previous evidence suggests that music can promote white matter (WM) microstructure and grey matter (GM) volume while supporting auditory and cognitive functioning and emotional well-being as well as counteracting age-related cognitive decline. Adding a social component to music training, choir singing is a popular leisure activity among older adults, but a systematic account of its potential to support healthy brain structure, especially with regard to ageing, is currently missing. The present study used quantitative anisotropy (QA)-based diffusion MRI connectometry and voxel-based morphometry to explore the relationship of lifetime choir singing experience and brain structure at the whole-brain level. Cross-sectional multiple regression analyses were carried out in a large, balanced sample (N = 95; age range 21-88) of healthy adults with varying levels of choir singing experience across the whole age range and within subgroups defined by age (young, middle-aged, and older adults). Independent of age, choir singing experience was associated with extensive increases in WM QA in commissural, association, and projection tracts across the brain. Corroborating previous work, these overlapped with language and limbic networks. Enhanced corpus callosum microstructure was associated with choir singing experience across all subgroups. In addition, choir singing experience was selectively associated with enhanced QA in the fornix in older participants. No associations between GM volume and choir singing were found. The present study offers the first systematic account of amateur-level choir singing on brain structure. While no evidence for counteracting GM atrophy was found, the present evidence of enhanced structural connectivity coheres well with age-typical structural changes. Corroborating previous behavioural studies, the present results suggest that regular choir singing holds great promise for supporting brain health across the adult life span.

Live music stimulates the affective brain and emotionally entrains listeners in real time

Music is powerful in conveying emotions and triggering affective brain mechanisms. Affective brain responses in previous studies were however rather inconsistent, potentially because of the non-adaptive nature of recorded music used so far. Live music instead can be dynamic and adaptive and is often modulated in response to audience feedback to maximize emotional responses in listeners. Here, we introduce a setup for studying emotional responses to live music in a closed-loop neurofeedback setup. This setup linked live performances by musicians to neural processing in listeners, with listeners' amygdala activity was displayed to musicians in real time. Brain activity was measured using functional MRI, and especially amygdala activity was quantified in real time for the neurofeedback signal. Live pleasant and unpleasant piano music performed in response to amygdala neurofeedback from listeners was acoustically very different from comparable recorded music and elicited significantly higher and more consistent amygdala activity. Higher activity was also found in a broader neural network for emotion processing during live compared to recorded music. This finding included observations of the predominance for aversive coding in the ventral striatum while listening to unpleasant music, and involvement of the thalamic pulvinar nucleus, presumably for regulating attentional and cortical flow mechanisms. Live music also stimulated a dense functional neural network with the amygdala as a central node influencing other brain systems. Finally, only live music showed a strong and positive coupling between features of the musical performance and brain activity in listeners pointing to real-time and dynamic entrainment processes.

Decoding the dynamic perception of risk and speed using naturalistic stimuli: A multivariate, whole-brain analysis

Time-resolved decoding of speed and risk perception in car driving is important for understanding the perceptual processes related to driving safety. In this study, we used an fMRI-compatible trackball with naturalistic stimuli to record dynamic ratings of perceived risk and speed and investigated the degree to which different brain regions were able to decode these. We presented participants with first-person perspective videos of cars racing on the same course. These videos varied in terms of subjectively perceived speed and risk profiles, as determined during a behavioral pilot. During the fMRI experiment, participants used the trackball to dynamically rate subjective risk in a first and speed in a second session and assessed overall risk and speed after watching each video. A standard multivariate correlation analysis based on these ratings revealed sparse decodability in visual areas only for the risk ratings. In contrast, the dynamic rating-based correlation analysis uncovered frontal, visual, and temporal region activation for subjective risk and dorsal visual stream and temporal region activation for subjectively perceived speed. Interestingly, further analyses showed that the brain regions for decoding risk changed over time, whereas those for decoding speed remained constant. Overall, our results demonstrate the advantages of time-resolved decoding to help our understanding of the dynamic networks associated with decoding risk and speed perception in realistic driving scenarios.

The transformative power of music: Insights into neuroplasticity, health, and disease

Music is a universal language that can elicit profound emotional and cognitive responses. In this literature review, we explore the intricate relationship between music and the brain, from how it is decoded by the nervous system to its therapeutic potential in various disorders. Music engages a diverse network of brain regions and circuits, including sensory-motor processing, cognitive, memory, and emotional components. Music-induced brain network oscillations occur in specific frequency bands, and listening to one's preferred music can grant easier access to these brain functions. Moreover, music training can bring about structural and functional changes in the brain, and studies have shown its positive effects on social bonding, cognitive abilities, and language processing. We also discuss how music therapy can be used to retrain impaired brain circuits in different disorders. Understanding how music affects the brain can open up new avenues for music-based interventions in healthcare, education, and wellbeing.

Bodily maps of musical sensations across cultures

Emotions, bodily sensations and movement are integral parts of musical experiences. Yet, it remains unknown i) whether emotional connotations and structural features of music elicit discrete bodily sensations and ii) whether these sensations are culturally consistent. We addressed these questions in a cross-cultural study with Western (European and North American, n = 903) and East Asian (Chinese, n = 1035). We precented participants with silhouettes of human bodies and asked them to indicate the bodily regions whose activity they felt changing while listening to Western and Asian musical pieces with varying emotional and acoustic qualities. The resulting bodily sensation maps (BSMs) varied as a function of the emotional qualities of the songs, particularly in the limb, chest, and head regions. Music-induced emotions and corresponding BSMs were replicable across Western and East Asian subjects. The BSMs clustered similarly across cultures, and cluster structures were similar for BSMs and self-reports of emotional experience. The acoustic and structural features of music were consistently associated with the emotion ratings and music-induced bodily sensations across cultures. These results highlight the importance of subjective bodily experience in music-induced emotions and demonstrate consistent associations between musical features, music-induced emotions, and bodily sensations across distant cultures.

Resting-state brain connectivity correlates of musical sophistication

Introduction

A growing body of research has investigated how performing arts training, and more specifically, music training, impacts the brain. Recent meta-analytic work has identified multiple brain areas where activity varies as a function of levels of musical expertise gained through music training. However, research has also shown that musical sophistication may be high even without music training. Thus, we aim to extend previous work by investigating whether the functional connectivity of these areas relates to interindividual differences in musical sophistication, and to characterize differences in connectivity attributed to performing arts training.

Methods

We analyzed resting-state functional magnetic resonance imaging from n = 74 participants, of whom 37 received performing arts training, that is, including a musical instrument, singing, and/or acting, at university level. We used a validated, continuous measure of musical sophistication to further characterize our sample. Following standard pre-processing, fifteen brain areas were identified a priori based on meta-analytic work and used as seeds in separate seed-to-voxel analyses to examine the effect of musical sophistication across the sample, and between-group analyses to examine the effects of performing arts training.

Results

Connectivity of bilateral superior temporal gyrus, bilateral precentral gyrus and cerebellum, and bilateral putamen, left insula, and left thalamus varied with different aspects of musical sophistication. By including these measures of these aspects as covariates in post hoc analyses, we found that connectivity of the right superior temporal gyrus and left precentral gyrus relate to effects of performing arts training beyond effects of individual musical sophistication.

Discussion

Our results highlight the potential role of sensory areas in active engagement with music, the potential role of motor areas in emotion processing, and the potential role of connectivity between putamen and lingual gyrus in general musical sophistication.

Topographic representation of visually evoked emotional experiences in the human cerebral cortex

Affective neuroscience seeks to uncover the neural underpinnings of emotions that humans experience. However, it remains unclear whether an affective space underlies the discrete emotion categories in the human brain, and how it relates to the hypothesized affective dimensions. To address this question, we developed a voxel-wise encoding model to investigate the cortical organization of human emotions. Results revealed that the distributed emotion representations are constructed through a fundamental affective space. We further compared each dimension of this space to 14 hypothesized affective dimensions, and found that many affective dimensions are captured by the fundamental affective space. Our results suggest that emotional experiences are represented by broadly spatial overlapping cortical patterns and form smooth gradients across large areas of the cortex. This finding reveals the specific structure of the affective space and its relationship to hypothesized affective dimensions, while highlighting the distributed nature of emotional representations in the cortex.

Pattern recognition reveals sex-dependent neural substrates of sexual perception

Sex differences in brain activity evoked by sexual stimuli remain elusive despite robust evidence for stronger enjoyment of and interest toward sexual stimuli in men than in women. To test whether visual sexual stimuli evoke different brain activity patterns in men and women, we measured hemodynamic brain activity induced by visual sexual stimuli in two experiments with 91 subjects (46 males). In one experiment, the subjects viewed sexual and nonsexual film clips, and dynamic annotations for nudity in the clips were used to predict hemodynamic activity. In the second experiment, the subjects viewed sexual and nonsexual pictures in an event-related design. Men showed stronger activation than women in the visual and prefrontal cortices and dorsal attention network in both experiments. Furthermore, using multivariate pattern classification we could accurately predict the sex of the subject on the basis of the brain activity elicited by the sexual stimuli. The classification generalized across the experiments indicating that the sex differences were task-independent. Eye tracking data obtained from an independent sample of subjects (N = 110) showed that men looked longer than women at the chest area of the nude female actors in the film clips. These results indicate that visual sexual stimuli evoke discernible brain activity patterns in men and women which may reflect stronger attentional engagement with sexual stimuli in men.

Decoding brain basis of laughter and crying in natural scenes

Laughter and crying are universal signals of prosociality and distress, respectively. Here we investigated the functional brain basis of perceiving laughter and crying using naturalistic functional magnetic resonance imaging (fMRI) approach. We measured haemodynamic brain activity evoked by laughter and crying in three experiments with 100 subjects in each. The subjects i) viewed a 20-minute medley of short video clips, and ii) 30 minutes of a full-length feature film, and iii) listened to 15 minutes of a radio play that all contained bursts of laughter and crying. Intensity of laughing and crying in the videos and radio play was annotated by independent observes, and the resulting time series were used to predict hemodynamic activity to laughter and crying episodes. Multivariate pattern analysis (MVPA) was used to test for regional selectivity in laughter and crying evoked activations. Laughter induced widespread activity in ventral visual cortex and superior and middle temporal and motor cortices. Crying activated thalamus, cingulate cortex along the anterior-posterior axis, insula and orbitofrontal cortex. Both laughter and crying could be decoded accurately (66-77% depending on the experiment) from the BOLD signal, and the voxels contributing most significantly to classification were in superior temporal cortex. These results suggest that perceiving laughter and crying engage distinct neural networks, whose activity suppresses each other to manage appropriate behavioral responses to others' bonding and distress signals.

Functional connectivity profiles of the default mode and visual networks reflect temporal accumulative effects of sustained naturalistic emotional experience

Determining and decoding emotional brain processes under ecologically valid conditions remains a key challenge in affective neuroscience. The current functional Magnetic Resonance Imaging (fMRI) based emotion decoding studies are mainly based on brief and isolated episodes of emotion induction, while sustained emotional experience in naturalistic environments that mirror daily life experiences are scarce. Here we used 12 different 10-minute movie clips as ecologically valid emotion-evoking procedures in n = 52 individuals to explore emotion-specific fMRI functional connectivity (FC) profiles on the whole-brain level at high spatial resolution (432 parcellations including cortical and subcortical structures). Employing machine-learning based decoding and cross validation procedures allowed to investigate FC profiles contributing to classification that can accurately distinguish sustained happiness and sadness and that generalize across subjects, movie clips, and parcellations. Both functional brain network-based and subnetwork-based emotion classification results suggested that emotion manifests as distributed representation of multiple networks, rather than a single functional network or subnetwork. Further, the results showed that the Visual Network (VN) and Default Mode Network (DMN) associated functional networks, especially VN-DMN, exhibited a strong contribution to emotion classification. To further estimate the temporal accumulative effect of naturalistic long-term movie-based video-evoking emotions, we divided the 10-min episode into three stages: early stimulation (1∼200 s), middle stimulation (201∼400 s), and late stimulation (401∼600 s) and examined the emotion classification performance at different stimulation stages. We found that the late stimulation contributes most to the classification (accuracy=85.32%, F1-score=85.62%) compared to early and middle stimulation stages, implying that continuous exposure to emotional stimulation can lead to more intense emotions and further enhance emotion-specific distinguishable representations. The present work demonstrated that sustained happiness and sadness under naturalistic conditions are presented in emotion-specific network profiles and these expressions may play different roles in the generation and modulation of emotions. These findings elucidated the importance of network level adaptations for sustained emotional experiences during naturalistic contexts and open new venues for imaging network level contributions under naturalistic conditions.

Bodily feelings and aesthetic experience of art

Humans all around the world are drawn to creating and consuming art due to its capability to evoke emotions, but the mechanisms underlying art-evoked feelings remain poorly characterised. Here we show how embodiement contributes to emotions evoked by a large database of visual art pieces (n = 336). In four experiments, we mapped the subjective feeling space of art-evoked emotions (n = 244), quantified "bodily fingerprints" of these emotions (n = 615), and recorded the subjects' interest annotations (n = 306) and eye movements (n = 21) while viewing the art. We show that art evokes a wide spectrum of feelings, and that the bodily fingerprints triggered by art are central to these feelings, especially in artworks where human figures are salient. Altogether these results support the model that bodily sensations are central to the aesthetic experience.

Decoding subject's own name in the primary auditory cortex

Current studies have shown that perception of subject's own name (SON) involves multiple multimodal brain regions, while activities in unimodal sensory regions (i.e., primary auditory cortex) and their interaction with multimodal regions during the self-processing remain unclear. To answer this, we combined multivariate pattern analysis and dynamic causal modelling analysis to explore the regional activation pattern and inter-region effective connection during the perception of SON. We found that SON and other names could be decoded from the activation pattern in the primary auditory cortex. In addition, we found an excitatory effect of SON on connections from the anterior insula/inferior frontal gyrus to the primary auditory cortex, and to the temporoparietal junction. Our findings extended the current knowledge of self-processing by showing that primary auditory cortex could discriminate SON from other names. Furthermore, our findings highlighted the importance of influence of the insula on the primary auditory cortex during self-processing.

SF-MVPA: A from raw data to statistical results and surface space-based MVPA toolbox

Compared with traditional volume space-based multivariate pattern analysis (MVPA), surface space-based MVPA has many advantages and has received increasing attention. However, surface space-based MVPA requires considerable programming and is therefore difficult for people without a programming foundation. To address this, we developed a MATLAB toolbox based on a graphical interactive interface (GUI) called surface space-based multivariate pattern analysis (SF-MVPA) in this manuscript. Unlike the traditional MVPA toolboxes, which often only include MVPA calculation processes after data preprocessing, SF-MVPA covers the complete pipeline of surface space-based MVPA, including raw data format conversion, surface reconstruction, functional magnetic resonance (fMRI) data preprocessing, comparative analysis, surface space-based MVPA, leave one-run out cross validation, and family-wise error correction. With SF-MVPA, users can complete the complete pipeline of surface space-based MVPA without programming. In addition, SF-MVPA is designed for parallel computing and hence has high computational efficiency. After introducing SF-MVPA, we analyzed a sample dataset of tonal working memory load. By comparison with another surface space-based MVPA toolbox named CoSMoMVPA, we found that the two toolboxes obtained consistent results. We hope that through this toolbox, users can more easily implement surface space-based MVPA.

Do sparse brain activity patterns underlie human cognition?

Accumulating multivariate pattern analysis (MVPA) results from fMRI studies suggest that information is represented in fingerprint patterns of activations and deactivations during perception, emotions, and cognition. We postulate that these fingerprint patterns might reflect neuronal-population level sparse code documented in two-photon calcium imaging studies in animal models, i.e., information represented in specific and reproducible ensembles of a few percent of active neurons amidst widespread inhibition in neural populations. We suggest that such representations constitute a fundamental organizational principle via interacting across multiple levels of brain hierarchy, thus giving rise to perception, emotions, and cognition.

The rewarding compensatory mechanism of music enhances the sense of fairness

Whether music can influence moral judgment is controversial in the aesthetics and philosophy of music. Aesthetic Autonomy pointed out that music had a morally educational function because of its lyrics or a particular context. The key to resolving the divergence is to select absolute music without lyrics or specific context as the eliciting material. In this study, 84 participants were recruited and randomly divided into three groups to complete the Ultimatum Game (UG) after listening to different stimuli: absolute music, white noise, and no sound. Behavioral results indicated that the participants’ acceptance of unfair offers was significantly lower in the music group. Also, participants in the music group have a shorter reaction time for rejecting an unfair offer than other unfair conditions. However, ERP comparison showed no significant difference in medial frontal negativity (MFN) amplitude, which reflects fairness levels, between the music group and the no sound group for either accepting or rejecting the moderately unfair offer. Brain network analyses revealed that participants in the music group showed stronger activation of rewarding circuits, including the ventral striatum, during the decision-making process of rejecting unfair offers, before the decision especially, compared to the no sound group. These results suggest that absolute music can influence fair decision-making. The reward activated by music compensates participants vicariously for the reward they receive for choosing self-interest in an unfair offer, participants no longer have to choose between self-interest and fairness norms, so the participants reject the unfair offer due to the negative emotions induced by the unfair offer.

A randomised pilot trial of virtual reality-based relaxation for enhancement of perioperative well-being, mood and quality of life

A cancer diagnosis and subsequent treatment can trigger distress, negatively impact coping resources, and affect well-being as well as quality of life. The aim of this pilot study was to investigate feasibility and clinical effects of a VR intervention on quality of life, well-being and mood in cancer patients undergoing surgery compared to a non-VR intervention and a control group. 54 patients with colorectal cancer or liver metastases from colorectal cancer undergoing elective curatively intended surgery were recruited and randomised to one of two intervention groups or a control group receiving standard treatment. Participants assigned to one of the intervention groups either received a VR-based intervention twice daily or listened to music twice daily. Adherence to the intervention was 64.6% in the music group and 81.6% in the VR group. The VR intervention significantly reduced heart rate (- 1.2 bpm; 95% CI - 2.24 to - 0.22; p = 0.02) and respiratory rate (- 0.7 brpm; 95% CI - 1.08 to - 0.25; p = 0.01). Self-reported overall mood improved in both groups (VR: + 0.79 pts; 95% CI 0.37-1.21; p = 0.001; music: + 0.59 pts; 95% CI 0.22-0.97; p = 0.004). There was no difference in quality of life between the three groups. Both interventions groups reported changes in feelings. Adherence rates favoured the VR intervention over the music group. Observed clinical outcomes showed stronger intragroup effects on mood, feelings, and vital signs in the VR group. The study demonstrated feasibility of a VR intervention in cancer patients undergoing surgery and should encourage further research investigating the potential of VR interventions to positively influence well-being and mood in cancer patients.

Classification of emotion categories based on functional connectivity patterns of the human brain

Neurophysiological and psychological models posit that emotions depend on connections across wide-spread corticolimbic circuits. While previous studies using pattern recognition on neuroimaging data have shown differences between various discrete emotions in brain activity patterns, less is known about the differences in functional connectivity. Thus, we employed multivariate pattern analysis on functional magnetic resonance imaging data (i) to develop a pipeline for applying pattern recognition in functional connectivity data, and (ii) to test whether connectivity patterns differ across emotion categories. Six emotions (anger, fear, disgust, happiness, sadness, and surprise) and a neutral state were induced in 16 participants using one-minute-long emotional narratives with natural prosody while brain activity was measured with functional magnetic resonance imaging (fMRI). We computed emotion-wise connectivity matrices both for whole-brain connections and for 10 previously defined functionally connected brain subnetworks and trained an across-participant classifier to categorize the emotional states based on whole-brain data and for each subnetwork separately. The whole-brain classifier performed above chance level with all emotions except sadness, suggesting that different emotions are characterized by differences in large-scale connectivity patterns. When focusing on the connectivity within the 10 subnetworks, classification was successful within the default mode system and for all emotions. We thus show preliminary evidence for consistently different sustained functional connectivity patterns for instances of emotion categories particularly within the default mode system.