The audio-visual integration effect on music emotion: Behavioral and physiological evidence

Infant attention to rhythmic audiovisual synchrony is modulated by stimulus properties

Musical interactions are a common and multimodal part of an infant's daily experiences. Infants hear their parents sing while watching their lips move and see their older siblings dance along to music playing over the radio. Here, we explore whether 8- to 12-month-old infants associate musical rhythms they hear with synchronous visual displays by tracking their dynamic visual attention to matched and mismatched displays. Visual attention was measured using eye-tracking while they attended to a screen displaying two videos of a finger tapping at different speeds. These videos were presented side by side while infants listened to an auditory rhythm (high or low pitch) synchronized with one of the two videos. Infants attended more to the low-pitch trials than to the high-pitch trials but did not display a preference for attending to the synchronous hand over the asynchronous hand within trials. Exploratory evidence, however, suggests that tempo, pitch, and rhythmic complexity interactively engage infants' visual attention to a tapping hand, especially when that hand is aligned with the auditory stimulus. For example, when the rhythm was complex and the auditory stimulus was low in pitch, infants attended to the fast hand more when it aligned with the auditory stream than to misaligned trials. These results suggest that the audiovisual integration in rhythmic non-speech contexts is influenced by stimulus properties.

Increased body movement equals better performance? Not always! Musical style determines motion degree perceived as optimal in music performance

Musicians' body behaviour has a preponderant role in audience perception. We investigated how performers' motion is perceived depending on the musical style and musical expertise. To further explore the effect of visual input, stimuli were presented in audio-only, audio-visual and visual-only conditions. We used motion and audio recordings of expert saxophone players playing two contrasting excerpts (positively and negatively valenced). For each excerpt, stimuli represented five motion degrees with increasing quantity of motion (QoM) and distinct predominant gestures. In the experiment (online and in-person), 384 participants rated performance recordings for expressiveness, professionalism and overall quality. Results revealed that, for the positively valenced excerpt, ratings increased as a function of QoM, whilst for the negatively valenced, the recording with predominant flap motion was favoured. Musicianship did not have a significant effect in motion perception. Concerning multisensory integration, both musicians and non-musicians presented visual dominance in the positively valenced excerpt, whereas in the negatively valenced, musicians shifted to auditory dominance. Our findings demonstrate that musical style not only determines the way observers perceive musicians' movement as adequate, but also that it can promote changes in multisensory integration.

Comparative analysis of noise and music exposure on inflammatory responses on lipopolysaccharide-induced septic rats

OBJECTIVE

Environmental factors such as noise and music can significantly impact physiological responses, including inflammation. This study explored how environmental factors like noise and music affect lipopolysaccharide (LPS)-induced inflammation, with a focus on systemic and organ-specific responses.

MATERIALS AND METHODS

24 Wistar rats were divided into four groups (n = 6 per group): Control group, LPS group, noise-exposed group, and music-exposed group. All rats, except for the Control group, received 10 mg/kg LPS intraperitoneally. The rats in the noise-exposed group were exposed to 95 dB noise, and the music-exposed group listened to Mozart's K. 448 music (65-75 dB) for 1 h daily over 7 days. An enzyme-linked immunosorbent assay was utilized to detect the levels of inflammatory cytokines, including tumor necrosis factor-α (TNF-α) and interleukin-1β (IL-1β), in serum and tissues (lung, liver, and kidney). Western blot examined the phosphorylation levels of nuclear factor-κB (NF-κB) p65 in organ tissues.

RESULTS

Compared with the Control group, LPS-induced sepsis rats displayed a significant increase in the levels of TNF-α and IL-1β in serum, lung, liver, and kidney tissues, as well as a remarkable elevation in the p-NF-κB p65 protein expression in lung, liver, and kidney tissues. Noise exposure further amplified these inflammatory markers, while music exposure reduced them in LPS-induced sepsis rats.

CONCLUSION

Noise exposure exacerbates inflammation by activating the NF-κB pathway, leading to the up-regulation of inflammatory markers during sepsis. On the contrary, music exposure inhibits NF-κB signaling, indicating a potential therapeutic effect in reducing inflammation.

Exploring neurophysiological correlates of visually induced motion sickness using electroencephalography (EEG)

Visually induced motion sickness (VIMS) is a common phenomenon when using visual devices such as smartphones and virtual reality applications, with symptoms including nausea, fatigue, and headache. To date, the neuro-cognitive processes underlying VIMS are not fully understood. Previous studies using electroencephalography (EEG) delivered mixed findings, with some reporting an increase in delta and theta power, and others reporting increases in alpha and beta frequencies. The goal of the study was to gain further insight into EEG correlates for VIMS. Participants viewed a VIMS-inducing visual stimulus, composed of moving black-and-white vertical bars presented on an array of three adjacent monitors. The EEG was recorded during visual stimulation and VIMS ratings were recorded after each trial using the Fast Motion Sickness Scale. Time-frequency analyses were conducted comparing neural activity of participants reporting minimal VIMS (n = 21) and mild-moderate VIMS (n = 12). Results suggested a potential increase in delta power in the centro-parietal regions (CP2) and a decrease in alpha power in the central regions (Cz) for participants experiencing mild-moderate VIMS compared to those with minimal VIMS. Event-related spectral perturbations (ERSPs) suggested that group differences in EEG activity developed with increasing duration of a trial. These results support the hypothesis that the EEG might be sensitive to differences in information processing in VIMS and minimal VIMS contexts, and indicate that it may be possible to identify neurophysiological correlate of VIMS. Differences in EEG activity related to VIMS may reflect differential processing of conflicting visual and vestibular sensory information.

Your ears don't change what your eyes like: People can independently report the pleasure of music and images

Observers can make independent aesthetic judgments of at least two images presented briefly and simultaneously. However, it is unknown whether this is the case for two stimuli of different sensory modalities. Here, we investigated whether individuals can judge auditory and visual stimuli independently, and whether stimulus duration influences such judgments. Participants (N = 120, across two experiments and a replication) saw images of paintings and heard excerpts of music, presented simultaneously for 2 s (Experiment 1) or 5 s (Experiment 2). After the stimuli were presented, participants rated how much pleasure they felt from the stimulus (music, image, or combined pleasure of both, depending on which was cued) on a 9-point scale. Finally, participants completed a baseline rating block where they rated each stimulus in isolation. We used the baseline ratings to predict ratings of audiovisual presentations. Across both experiments, the root mean square errors (RMSEs) obtained from leave-one-out-cross-validation analyses showed that people's ratings of music and images were unbiased by the simultaneously presented other stimulus, and ratings of both were best described as the arithmetic mean of the ratings from the individual presentations at the end of the experiment. This pattern of results replicates previous findings on simultaneously presented images, indicating that participants can ignore the pleasure of an irrelevant stimulus regardless of the sensory modality and duration of stimulus presentation. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Self-construal priming modulates sonic seasoning

Introduction

“Sonic seasoning” is when music influences the real taste experiences of consumers. “Self-construal” is how individuals perceive, understand, and interpret themselves. Numerous studies have shown that independent and interdependent self-construal priming can affect a person's cognition and behavior; however, their moderating effect on the sonic seasoning effect remains unclear.

Methods

This experiment was a 2 (self-construal priming: independent self-construal or interdependent self-construal) × 2 (chocolate: milk chocolate or dark chocolate) × 2 (emotional music: positive emotional music or negative emotional music) mixed design, and explored the moderating role of self-construal priming and the effect of emotional music on taste by comparing participants' evaluations of chocolates while listening to positive or negative music after different levels of self-construal priming.

Results

After initiating independent self-construal, participants increased their ratings of milk chocolate sweetness when listening to music that elicited positive emotions, t(32) = 3.11, p = 0.004, Cohen's d = 0.54, 95% CI = [0.33, 1.61]. In contrast, interdependent self-construal priming led participants to perceive dark chocolate as sweeter when they heard positive music, t(29) = 3.63, p = 0.001, Cohen's d = 0.66, 95%CI = [0.44, 1.56].

Discussion

This study provides evidence for improving people's individual eating experience and enjoyment of food.

Integrative interaction of emotional speech in audio-visual modality

Emotional clues are always expressed in many ways in our daily life, and the emotional information we receive is often represented by multiple modalities. Successful social interactions require a combination of multisensory cues to accurately determine the emotion of others. The integration mechanism of multimodal emotional information has been widely investigated. Different brain activity measurement methods were used to determine the location of brain regions involved in the audio-visual integration of emotional information, mainly in the bilateral superior temporal regions. However, the methods adopted in these studies are relatively simple, and the materials of the study rarely contain speech information. The integration mechanism of emotional speech in the human brain still needs further examinations. In this paper, a functional magnetic resonance imaging (fMRI) study was conducted using event-related design to explore the audio-visual integration mechanism of emotional speech in the human brain by using dynamic facial expressions and emotional speech to express emotions of different valences. Representational similarity analysis (RSA) based on regions of interest (ROIs), whole brain searchlight analysis, modality conjunction analysis and supra-additive analysis were used to analyze and verify the role of relevant brain regions. Meanwhile, a weighted RSA method was used to evaluate the contributions of each candidate model in the best fitted model of ROIs. The results showed that only the left insula was detected by all methods, suggesting that the left insula played an important role in the audio-visual integration of emotional speech. Whole brain searchlight analysis, modality conjunction analysis and supra-additive analysis together revealed that the bilateral middle temporal gyrus (MTG), right inferior parietal lobule and bilateral precuneus might be involved in the audio-visual integration of emotional speech from other aspects.

Developing Serious Video Games to Treat Attention Deficit Hyperactivity Disorder: Tutorial Guide

Video game–based therapeutic interventions have demonstrated some effectiveness in decreasing the symptoms of attention deficit hyperactivity disorder (ADHD). Compared with more traditional strategies within the multimodal treatment of ADHD, video games have certain advantages such as being comfortable, flexible, and cost-efficient. However, establishing the most appropriate type(s) of video games that should be used for this treatment remains a matter of debate, including the commercial existing video games or serious video games that are specifically constructed to target specific disorders. This guide represents a starting point for developing serious video games aimed at treating ADHD. We summarize the key points that need to be addressed to generate an effective and motivating game-based treatment. Following recommendations from the literature to create game-based treatments, we describe the development stages of a serious video game for treating ADHD. Game design should consider the interests of future users; game mechanics should be based on cognitive exercises; and therapeutic mechanisms must include the control of difficulty, engagement, motivation, time constraints, and reinforcement. To elaborate upon this guide, we performed a narrative review focused on the use of video games for the treatment of ADHD, and were inspired by our own experience during the development of the game “The Secret Trail of Moon.”

Controlled audio-visual stimulation for anxiety reduction

BACKGROUND AND OBJECTIVE

Recent clinical data suggest that 75% of patients undergoing surgery are anxious, despite pharmacological measures to relieve anxiety. As an alternative to the administration of drugs, the scientific literature reports the relevant psychophysiological effects of auditory and visual stimulation in reducing preoperative anxiety. The main objective of this study is the development of a portable computer-controlled device for the simultaneous combined administration of audio-visual stimuli and the evaluation of this device through the collection and the statistical analysis of psychophysiological parameters strictly related to the state of anxiety.

METHODS

A new algorithmic approach for the real-time association of sounds and colours is proposed and implemented in a low-cost architectural platform. The combined administration of auditory and visual stimuli is tested on 220 subjects undergoing dental surgery; in particular, psychophysiological parameters are collected and evaluated in four experimental conditions, in order to demonstrate the efficacy of cross-modal stimulation (auditory and visual) compared to non-pharmacological treatments based on monomodal stimuli (auditory or visual).

RESULTS

Non-parametric statistical techniques applied to the recorded experimental data show that the experimental conditions considered significantly differ. Pairwise comparisons between experimental groups show that the combined administration of sounds and colors significantly reduces the level of anxiety, systolic blood pressure and heart rate to a greater extent than monomodal stimulation.

CONCLUSION

The study demonstrates the potential benefits of a device for the combined administration of auditory and visual stimuli. The developed device has proven effective in reducing preoperative anxiety levels, becoming a serious candidate for non-pharmacological therapies. The study also encourages a deeper investigation of models capable of better capturing the potential of cross-modal stimulation, maximizing the desired effects (relaxation, arousal) on patients awaiting specific medical treatments.

Learning Chinese Classical Music with the Aid of Soundscape by Using Intelligent Network

A soundscape is a sound environment of the awareness of auditory perception and social or cultural understanding. To improve the subjective initiative of the Chinese classical music students, this study explores new learning modes and methods using soundscape and investigates its learning effect using intelligent music software. To examine the emotional experience of players in playing before and after learning, 50 students from music majors and 50 from non-music majors were selected. Results show that in the positive and negative emotional classical music experiment, most music majors and non-music majors have weak emotional experience at the beginning, and only a few have a strong emotional experience, which could reach 6 points. In the second scoring, most majors have a score of about 7 points, indicating strong emotional experience and a few have a score of about 4 points and 9 points, representing that there were relatively few majors with weak emotional experience and strong emotional experience. The overall emotional experience score is low in the comparison between non-music majors and music majors, and the second score in the entire experiment is significantly higher than the first score, signifying that the learning effect of players is obvious, and intelligent music software and soundscape play a role in the exploration of Chinese classical music.

Deficits in Sense of Body Ownership, Sensory Processing, and Temporal Perception in Schizophrenia Patients With/Without Auditory Verbal Hallucinations

It has been claimed that individuals with schizophrenia have difficulty in self-recognition and, consequently, are unable to identify the sources of their sensory perceptions or thoughts, resulting in delusions, hallucinations, and unusual experiences of body ownership. The deficits also contribute to the enhanced rubber hand illusion (RHI; a body perception illusion, induced by synchronous visual and tactile stimulation). Evidence based on RHI paradigms is emerging that auditory information can make an impact on the sense of body ownership, which relies on the process of multisensory inputs and integration. Hence, we assumed that auditory verbal hallucinations (AVHs), as an abnormal auditory perception, could be linked with body ownership, and the RHI paradigm could be conducted in patients with AVHs to explore the underlying mechanisms. In this study, we investigated the performance of patients with/without AVHs in the RHI. We administered the RHI paradigm to 80 patients with schizophrenia (47 with AVHs and 33 without AVHs) and 36 healthy controls. We conducted the experiment under two conditions (synchronous and asynchronous) and evaluated the RHI effects by both objective and subjective measures. Both patient groups experienced the RHI more quickly and strongly than HCs. The RHI effects of patients with AVHs were significantly smaller than those of patients without AVHs. Another important finding was that patients with AVHs did not show a reduction in RHI under asynchronous conditions. These results emphasize the disturbances of the sense of body ownership in schizophrenia patients with/without AVHs and the associations with AVHs. Furthermore, it is suggested that patients with AVHs may have multisensory processing dysfunctions and internal timing deficits.

Multisensory integration of musical emotion perception in singing

We investigated how visual and auditory information contributes to emotion communication during singing. Classically trained singers applied two different facial expressions (expressive/suppressed) to pieces from their song and opera repertoire. Recordings of the singers were evaluated by laypersons or experts, presented to them in three different modes: auditory, visual, and audio–visual. A manipulation check confirmed that the singers succeeded in manipulating the face while keeping the sound highly expressive. Analyses focused on whether the visual difference or the auditory concordance between the two versions determined perception of the audio–visual stimuli. When evaluating expressive intensity or emotional content a clear effect of visual dominance showed. Experts made more use of the visual cues than laypersons. Consistency measures between uni-modal and multimodal presentations did not explain the visual dominance. The evaluation of seriousness was applied as a control. The uni-modal stimuli were rated as expected, but multisensory evaluations converged without visual dominance. Our study demonstrates that long-term knowledge and task context affect multisensory integration. Even though singers’ orofacial movements are dominated by sound production, their facial expressions can communicate emotions composed into the music, and observes do not rely on audio information instead. Studies such as ours are important to understand multisensory integration in applied settings.

Human Genomics and the Biocultural Origin of Music

Music is an exclusive feature of humankind. It can be considered as a form of universal communication, only partly comparable to the vocalizations of songbirds. Many trends of research in this field try to address music origins, as well as the genetic bases of musicality. On one hand, several hypotheses have been made on the evolution of music and its role, but there is still debate, and comparative studies suggest a gradual evolution of some abilities underlying musicality in primates. On the other hand, genome-wide studies highlight several genes associated with musical aptitude, confirming a genetic basis for different musical skills which humans show. Moreover, some genes associated with musicality are involved also in singing and song learning in songbirds, suggesting a likely evolutionary convergence between humans and songbirds. This comprehensive review aims at presenting the concept of music as a sociocultural manifestation within the current debate about its biocultural origin and evolutionary function, in the context of the most recent discoveries related to the cross-species genetics of musical production and perception.

Mechanisms of Light and Music Stimulation Controlled by a Person's own Brain and Heart Biopotentials or Those of another Person

The aim of the study was to carry out comparative analysis of effects observed in subjects exposed to light and music stimulation controlled by their own brain and heart biopotentials (closed-loop method) or biopotentials of another person.

MATERIALS AND METHODS

Volunteers under stress participated in two experiments in pairs. In the first experiment, light and music stimulation effects formed in each subject in a pair on the basis of their own brain and heart biopotentials, while in the second experiment, they formed on the basis of biopotentials of the other subject.

RESULTS

Both types of exposure caused reducing the tension of the regulatory systems in the body, reducing stress levels and improving the emotional state due to the mechanisms of multisensory integration and neuroplasticity. A significant increase in the power of the main EEG rhythms, accompanied by significant positive changes in psychological testing results and positive emotional responses to stimulation was observed only during light and music stimulation controlled by the subjects' own brain and heart biopotentials. These data are attributable to the integration of perception and processing of interoceptive signals significant for humans into the resonance mechanisms of the central nervous system, providing normalization of functional state due to stimulation.

CONCLUSION

The data obtained can be used for developing the effective methods of personalized light and music stimulation aimed at timely elimination of functional disorders and returning the human body to homeostasis.