Color, Music, and Emotion: Bach to the Blues

Color and tone color: audiovisual crossmodal correspondences with musical instrument timbre

Crossmodal correspondences, or widely shared tendencies for mapping experiences across sensory domains, are revealed in common descriptors of musical timbre such as bright, dark, and warm. Two experiments are reported in which participants listened to recordings of musical instruments playing major scales, selected colors to match the timbres, and rated the timbres on crossmodal semantic scales. Experiment A used three different keyboard instruments, each played in three pitch registers. Stimuli in Experiment B, representing six different orchestral instruments, were similar to those in Experiment A but were controlled for pitch register. Overall, results were consistent with hypothesized concordances between ratings on crossmodal timbre descriptors and participants' color associations. Semantic ratings predicted the lightness and saturation of colors matched to instrument timbres; effects were larger when both pitch register and instrument type varied (Experiment A) but were still evident when pitch register was held constant (Experiment B). We also observed a weak relationship between participant ratings of musical stimuli on the terms warm and cool and the warmth-coolness of selected colors in Experiment B only. Results were generally consistent with the hypothesis that instrument type and pitch register are related to color choice, though we speculate that these associations may only be relevant for certain instruments. Overall, the results have implications for our understanding the relationship between music and color, suggesting that while timbre/color matching behavior is in many ways diverse, observable trends in strategy can in part be linked to crossmodal timbre semantics.

A multimodal understanding of the role of sound and music in gendered toy marketing

Literature in music theory and psychology shows that, even in isolation, musical sounds can reliably encode gender-loaded messages. Musical material can be imbued with many ideological dimensions and gender is just one of them. Nonetheless, studies of the gendering of music within multimodal communicative events are sparse and lack an encompassing theoretical framework. The present study attempts to address this literature gap by employing a critical quantitative analysis of music in gendered toy marketing, which integrated a content analytical approach with multimodal affective and music-focused perceptual responses. Ratings were collected on a set of 606 commercials spanning a ten-year time frame and strong gender polarization was observed in nearly all of the collected variables. Gendered music styles in toy commercials exhibit synergistic design choices, as music in masculine-targeted adverts was substantially more abrasive-louder, more inharmonious, and more distorted-than in feminine-targeted ones. Thus, toy advertising music appeared deliberately and consistently in line with traditional gender norms. In addition, music perceptual scales and voice-related content analytical variables explain quite well the heavily polarized affective ratings. This study presents a empirical understanding of the gendering of music as constructed within multimodal discourse, reiterating the importance of the sociocultural underpinnings of music cognition. We provided a public repository with all code and data necessary to reproduce the results of this study on github.com/marinelliluca/music-role-gender-marketing.

Sensory translation between audition and vision

Across the millennia, and across a range of disciplines, there has been a widespread desire to connect, or translate between, the senses in a manner that is meaningful, rather than arbitrary. Early examples were often inspired by the vivid, yet mostly idiosyncratic, crossmodal matches expressed by synaesthetes, often exploited for aesthetic purposes by writers, artists, and composers. A separate approach comes from those academic commentators who have attempted to translate between structurally similar dimensions of perceptual experience (such as pitch and colour). However, neither approach has succeeded in delivering consensually agreed crossmodal matches. As such, an alternative approach to sensory translation is needed. In this narrative historical review, focusing on the translation between audition and vision, we attempt to shed light on the topic by addressing the following three questions: (1) How is the topic of sensory translation related to synaesthesia, multisensory integration, and crossmodal associations? (2) Are there common processing mechanisms across the senses that can help to guarantee the success of sensory translation, or, rather, is mapping among the senses mediated by allegedly universal (e.g., amodal) stimulus dimensions? (3) Is the term 'translation' in the context of cross-sensory mappings used metaphorically or literally? Given the general mechanisms and concepts discussed throughout the review, the answers we come to regarding the nature of audio-visual translation are likely to apply to the translation between other perhaps less-frequently studied modality pairings as well.

Inducing Novel Sound-Taste Correspondences via an Associative Learning Task

The interest in crossmodal correspondences, including those involving sounds and involving tastes, has experienced rapid growth in recent years. However, the mechanisms underlying these correspondences are not well understood. In the present study (N = 302), we used an associative learning paradigm, based on previous literature using simple sounds with no consensual taste associations (i.e., square and triangle wave sounds at 200 Hz) and taste words (i.e., sweet and bitter), to test the influence of two potential mechanisms in establishing sound-taste correspondences and investigate whether either learning mechanism could give rise to new and long-lasting associations. Specifically, we examined an emotional mediation account (i.e., using sad and happy emoji facial expressions) and a transitive path (i.e., sound-taste correspondence being mediated by color, using red and black colored squares). The results revealed that the associative learning paradigm mapping the triangle wave tone with a happy emoji facial expression induced a novel crossmodal correspondence between this sound and the word sweet. Importantly, we found that this novel association was still present two months after the experimental learning paradigm. None of the other mappings, emotional or transitive, gave rise to any significant associations between sound and taste. These findings provide evidence that new crossmodal correspondences between sounds and tastes can be created by leveraging the affective connection between both dimensions, helping elucidate the mechanisms underlying these associations. Moreover, these findings reveal that these associations can last for several weeks after the experimental session through which they were induced.

Reducing sweetness expectation in milk tea by crossmodal visuo-auditory interaction

In the realm of healthy dietary choices about reducing sweetness perception, the exploration of crossmodal effects stands as a frequently employed approach. Both music and color can independently influence flavor evaluation and gustatory experience by eliciting emotions. However, less research has been done on the effects of audio-visual crossmodal interactions on sweetness expectations and perceptions. The present study conducted two experiments delving into the crossmodal effect on sweetness expectation and perception of milk tea by manipulating the emotional valence of music and packaging color. The results showed that positive (vs. negative) music led to higher sweetness expectations and perceptions for milk teas with neutral packaging color. Irrespective of music, participants had higher sweetness expectations for milk tea with positive or neutral (vs. negative) packaging colors. The congruence of valence between music and packaging color influenced sweetness perception. Positive (vs. negative) music correlated with a sweeter perception when the packaging color was positive. Exposed to negative music, subjects showed a higher sweetness perception with negative (vs. positive) packaging colors. In conclusion, the results suggest that the valence of music and packaging color crossmodally influence consumers' evaluation of milk tea, and it differs depending on whether it was tasted. Thus, this study has demonstrated the crossmodal influence of music and packaging color, providing valuable implications for healthy eating and marketing applications.

Tasty vibes: Uncovering crossmodal correspondences between tactile vibrations and basic tastes

The interest in crossmodal correspondences individually involving the senses of touch and taste has grown rapidly in the last few decades. Several correspondences involving different tactile dimensions (e.g., hardness/softness, roughness/smoothness) have been uncovered, such as those between sweetness and softness and between roughness and sourness. However, a dimension that has been long overlooked, despite its pervasiveness and importance in everyday experiences, relates to tactile vibrations. The present study aimed to fill this gap and investigate crossmodal correspondences between basic tastes and vibrations. In the present study (N = 72), we uncovered these associations by having participants sample basic taste (i.e., sweet, salty, sour, bitter, umami) aqueous solutions and chose the frequency of vibrations, delivered via a consumer-grade subwoofer wristband on their dominant hand, that they most strongly associated with each taste. We found that sourness was most strongly associated with frequencies around 98 Hz, and that sweetness and umami were associated with frequencies around 77 Hz. These correspondences may, to different extents, be based on affective and semantic mechanisms. The findings have relevant implications for theoretical research on multisensory integration and perception and the potential future applications of these associations, through wearable technologies, to enhance eating experiences and promote healthier eating habits.

Your ears don't change what your eyes like: People can independently report the pleasure of music and images

Observers can make independent aesthetic judgments of at least two images presented briefly and simultaneously. However, it is unknown whether this is the case for two stimuli of different sensory modalities. Here, we investigated whether individuals can judge auditory and visual stimuli independently, and whether stimulus duration influences such judgments. Participants (N = 120, across two experiments and a replication) saw images of paintings and heard excerpts of music, presented simultaneously for 2 s (Experiment 1) or 5 s (Experiment 2). After the stimuli were presented, participants rated how much pleasure they felt from the stimulus (music, image, or combined pleasure of both, depending on which was cued) on a 9-point scale. Finally, participants completed a baseline rating block where they rated each stimulus in isolation. We used the baseline ratings to predict ratings of audiovisual presentations. Across both experiments, the root mean square errors (RMSEs) obtained from leave-one-out-cross-validation analyses showed that people's ratings of music and images were unbiased by the simultaneously presented other stimulus, and ratings of both were best described as the arithmetic mean of the ratings from the individual presentations at the end of the experiment. This pattern of results replicates previous findings on simultaneously presented images, indicating that participants can ignore the pleasure of an irrelevant stimulus regardless of the sensory modality and duration of stimulus presentation. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Development and validation of a highly dynamic and reusable picture-based scale: A new affective measurement tool

Emotion measurement is crucial to conducting emotion research. Numerous studies have extensively employed textual scales for psychological and organizational behavior research. However, emotions are transient states of organisms with relatively short duration, some insurmountable limitations of textual scales have been reported, including low reliability for single measurement or susceptibility to learning effects for multiple repeated use. In the present article, we introduce the Highly Dynamic and Reusable Picture-based Scale (HDRPS), which was randomly generated based on 3,386 realistic, high-quality photographs that are divided into five categories (people, animals, plants, objects, and scenes). Affective ratings of the photographs were gathered from 14 experts and 209 professional judges. The HDRPS was validated using the Self-Assessment Manikin and the PANAS by 751 participants. With an accuracy of 89.73%, this new tool allows researchers to measure individual emotions continuously for their research. The non-commercial use of the HDRPS system can be freely accessible by request at http://syy.imagesoft.cc:8989/Pictures.7z. HDRPS is used for non-commercial academic research only. As some of the images are collected through the open network, it is difficult to trace the source, so please contact the author if there are any copyright issues.

Cross-modal associations between paintings and sounds: Effects of embodiment

The present study investigated cross-modal associations between a series of paintings and sounds. We studied the effects of sound congruency (congruent vs. non-congruent sounds) and embodiment (embodied vs. synthetic sounds) on the evaluation of abstract and figurative paintings. Participants evaluated figurative and abstract paintings paired with congruent and non-congruent embodied and synthetic sounds. They also evaluated the perceived meaningfulness of the paintings, aesthetic value and immersive experience of the paintings. Embodied sounds (sounds associated with bodily sensations, bodily movements and touch) were more strongly associated with figurative paintings, while synthetic sounds (non-embodied sounds) were more strongly associated with abstract paintings. Sound congruency increased the perceived meaningfulness, immersive experience and aesthetic value of paintings. Sound embodiment increased immersive experience of paintings.

Crossmodal correspondences between basic tastes and visual design features: A narrative historical review

People tend to associate abstract visual features with basic taste qualities. This narrative historical review critically evaluates the literature on these associations, often referred to as crossmodal correspondences, between basic tastes and visual design features such as color hue and shape curvilinearity. The patterns, discrepancies, and evolution in the development of the research are highlighted while the mappings that have been reported to date are summarized. The review also reflects on issues of cross-cultural validity and deviations in the matching patterns that are observed when correspondences are assessed with actual tastants versus with verbal stimuli. The various theories that have been proposed to account for different classes of crossmodal correspondence are discussed, among which the statistical and affective (or emotional-mediation) accounts currently appear most promising. Several critical research questions for the future are presented to address the gaps that have been identified in the literature and help validate the popular theories on the origin and operations of visual-taste correspondences.

Virtual Reality Intervention for Managing Apathy in People With Cognitive Impairment: Systematic Review

Background

Apathy is common in people with cognitive impairment. It leads to different consequences, such as more severe cognitive deficits, rapid functional decline, and decreased quality of life. Virtual reality (VR) interventions are increasingly being used to manage apathy in individuals with cognitive impairment. However, reports of VR interventions are scattered across studies, which has hindered the development and use of the interventions.

Objective

This study aimed to systematically review existing evidence on the use of VR interventions for managing apathy in people with cognitive impairment with regard to the effectiveness, contents, and implementation of the interventions.

Methods

The PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines were followed. The PubMed, Embase, CINAHL, and PsycINFO databases were systematically searched for experimental studies published up to March 13, 2022, that reported the effects of VR interventions on apathy in older adults with cognitive impairment. Hand searching and citation chasing were conducted. The results of the included studies were synthesized by using a narrative synthesis. Their quality was appraised by using the Effective Public Health Practice Project quality assessment tool. However, because the VR interventions varied in duration, content, and implementation across studies, a meta-analysis was not conducted.

Results

A total of 22 studies were identified from the databases, of which 6 (27%) met the inclusion criteria. Of these 6 studies, 2 (33%) were randomized controlled trials, 1 (17%) was a controlled clinical trial, and 3 (50%) were quasi-experimental studies. Individual studies showed significant improvement in apathy and yielded within-group medium to large effect sizes. The level of immersion ranged from low to high. Minor adverse effects were reported. The VR content mostly included natural scenes, followed by city views and game-based activities. A background soundtrack was often used with natural scenes. Most (5/6, 83%) of the studies were conducted in a residential care setting and were implemented by health care professionals or researchers. Safety precautions were taken in most (5/6, 83%) of the studies.

Conclusions

Although preliminary evidence shows that VR interventions may be effective and feasible for alleviating apathy in people with cognitive impairment, the methodological limitations in the included studies make it difficult to reach a firm conclusion on these points. The implementation of the interventions was highlighted and discussed. More rigorous studies are encouraged.

Trial Registration

PROSPERO International Prospective Register of Systematic Reviews CRD42021268289; https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42021268289

Illuminating Music: Impact of Color Hue for Background Lighting on Emotional Arousal in Piano Performance Videos

This study sought to determine if hues overlayed on a video recording of a piano performance would systematically influence perception of its emotional arousal level. The hues were artificially added to a series of four short video excerpts of different performances using video editing software. Over two experiments 106 participants were sorted into 4 conditions, with each viewing different combinations of musical excerpts (two excerpts with nominally high arousal and two excerpts with nominally low arousal) and hue (red or blue) combinations. Participants rated the emotional arousal depicted by each excerpt. Results indicated that the overall arousal ratings were consistent with the nominal arousal of the selected excerpts. However, hues added to video produced no significant effect on arousal ratings, contrary to predictions. This could be due to the domination of the combined effects of other channels of information (e.g., the music and player movement) over the emotional effects of the hypothesized influence of hue on perceived performance (red expected to enhance and blue to reduce arousal of the performance). To our knowledge this is the first study to investigate the impact of these hues upon perceived arousal of music performance, and has implications for musical performers and stage lighting. Further research that investigates reactions during live performance and manipulation of a wider range of lighting hues, saturation and brightness levels, and editing techniques, is recommended to further scrutinize the veracity of the findings.

Crossmodal Harmony: Looking for the Meaning of Harmony Beyond Hearing

The notion of harmony was first developed in the context of metaphysics before being applied to the domain of music. However, in recent centuries, the term has often been used to describe especially pleasing combinations of colors by those working in the visual arts too. Similarly, the harmonization of flavors is nowadays often invoked as one of the guiding principles underpinning the deliberate pairing of food and drink. However, beyond the various uses of the term to describe and construct pleasurable unisensory perceptual experiences, it has also been suggested that music and painting may be combined harmoniously (e.g., see the literature on "color music"). Furthermore, those working in the area of "sonic seasoning" sometimes describe certain sonic compositions as harmonizing crossmodally with specific flavor sensations. In this review, we take a critical look at the putative meaning(s) of the term "harmony" when used in a crossmodal, or multisensory, context. Furthermore, we address the question of whether the term's use outside of a strictly unimodal auditory context should be considered literally or merely metaphorically (i.e., as a shorthand to describe those combinations of sensory stimuli that, for whatever reason, appear to go well together, and hence which can be processed especially fluently).

Space oddity: musical syntax is mapped onto visual space

Increasing evidence has uncovered associations between the cognition of abstract schemas and spatial perception. Here we examine such associations for Western musical syntax, tonality. Spatial metaphors are ubiquitous when describing tonality: stable, closural tones are considered to be spatially central and, as gravitational foci, spatially lower. We investigated whether listeners, musicians and nonmusicians, indeed associate tonal relationships with visuospatial dimensions, including spatial height, centrality, laterality, and size, implicitly or explicitly, and whether such mappings are consistent with established metaphors. In the explicit paradigm, participants heard a tonality-establishing prime followed by a probe tone and coupled each probe with a subjectively appropriate location (Exp.1) or size (Exp.4). The implicit paradigm used a version of the Implicit Association Test to examine associations of tonal stability with vertical position (Exp.2), lateral position (Exp3) and size (Exp.5). Tonal stability was indeed associated with perceived physical space: the spatial distances between the locations associated with different scale-degrees significantly correlated with the tonal stability differences between these scale-degrees. However, inconsistently with musical discourse, stable tones were associated with leftward (instead of central) and higher (instead of lower) spatial positions. We speculate that these mappings are influenced by emotion, embodying the "good is up" metaphor, and by the spatial structure of music keyboards. Taken together, the results demonstrate a new type of cross-modal correspondence and a hitherto under-researched connotative function of musical structure. Importantly, the results suggest that the spatial mappings of an abstract domain may be independent of the spatial metaphors used to describe that domain.

Function Follows Form: Using the Aesthetic Association Principle to Enhance Haptic Interface Design

Novel tangible user interface technologies facilitate current trends toward seamless user interfaces. They enable the design of yet unseen interfaces and thus the creation of a new kind of haptic language. In order to use the benefits of a touch-and-feel design for a positive user experience, carefully designed haptic feedback plays an important role by providing aesthetically pleasing and sustainable product features. Haptic feedback may exceed mere acquiring of buttons and input-confirmation but enable orientation and even identification of functionality governed by the haptic impression. We employed the aesthetic association principle as a deeply grounded psychological mechanism that assists effective linkage between haptic form factors and associated functional attributes. In order to illustrate this powerful principle, we analyzed the specific associations between certain main haptic surface qualities and associated functional aspects. In a series of three subsequent studies (Pre-Study 1: perception, Pre-Study 2: similarity, and Main Study: association), we explored paradigmatic associations of that kind to develop guidelines which forms are distinct to be used in interfaces. We show how forms are implicitly categorized into functional qualities (on/off, more-less, selection), using a multidimensional scaling procedure and explore explicit form-functionality associations, using a think-aloud method in the context of an automotive interface. For a series of forms, we revealed clear associative relations to specific functions. We will discuss the general value and opportunities of an association-based approach to user experience in order to create intuitive user interfaces. We will also develop ideas for specific areas of applications.

Explaining Crossmodal Correspondences Between Colours and Tastes

For centuries, if not millennia, people have associated the basic tastes (e.g., sweet, bitter, salty, and sour) with specific colours. While the range of tastes may have changed, and the reasons for wanting to connect the senses in this rather surprising way have undoubtedly differed, there would nevertheless appear to be a surprisingly high degree of consistency regarding this crossmodal mapping among non-synaesthetes that merits further consideration. Traditionally, colour-taste correspondences have often been considered together with odour-colour and flavour-colour correspondences. However, the explanation for these various correspondences with the chemical senses may turn out to be qualitatively different, given the presence of identifiable source objects in the case of food aromas/flavours, but not necessarily in the case of basic tastes. While the internalization of the crossmodal statistics of the environment provides one appealing account for the existence of colour-taste correspondences, emotional mediation may also be relevant. Ultimately, while explaining colour-taste correspondences is of both theoretical and historical interest, the growing awareness of the robustness of colour-taste correspondences would currently seem to be of particular relevance to those working in the fields of design and multisensory experiential marketing.

Verbal and Cross-Modal Ratings of Music: Validation and Application of an Icon-Based Rating Scale

Can music be rated consistently using nonverbal descriptors such as colours and temperatures? 144 participants rated 6 experimenter-selected and 2 self-selected pieces of music along 15 bipolar icon (graphic) scales intended to portray emotions, and sensory experiences consisting of colour, temperature, shape, speed, texture, and weight. Participants also rated the same pieces using bipolar verbal scales which aimed to encompass the concepts represented by the icons (e.g., the word "red" for the colour red). Furthermore, the icons themselves were subjected to open-ended verbal labelling to validate the icon scale. Colour icons spontaneously evoked a cross-modal association on 67% of occasions: blue being cool, and red/orange being warm or hot, and the icon scale had overall good face validity. Music regularly and consistently evoked multisensory associations (using the icon scale) including shapes, colours, weight, and temperatures, in addition to emotions. Cross-modal perception is indicative of music's character rather than the enjoyment of the music. The icon scale provides new insights into music perception and for applications where language skill may limit participant expression.