Crossmodal Harmony: Looking for the Meaning of Harmony Beyond Hearing

Crossmodal associations between naturally occurring tactile and sound textures

This study investigates the crossmodal associations between naturally occurring sound textures and tactile textures. Previous research has demonstrated the association between low-level sensory features of sound and touch, as well as higher-level, cognitively mediated associations involving language, emotions, and metaphors. However, stimuli like textures, which are found in both modalities have received less attention. In this study, we conducted two experiments: a free association task and a two alternate forced choice task using everyday tactile textures and sound textures selected from natural sound categories. The results revealed consistent crossmodal associations reported by participants between the textures of the two modalities. They tended to associate more sound textures (e.g., wood shavings and sandpaper) with tactile surfaces that were rated as harder, rougher, and intermediate on the sticky-slippery scale. While some participants based the auditory-tactile association on sensory features, others made the associations based on semantic relationships, co-occurrence in nature, and emotional mediation. Interestingly, the statistical features of the sound textures (mean, variance, kurtosis, power, autocorrelation, and correlation) did not show significant correlations with the crossmodal associations, indicating a higher-level association. This study provides insights into auditory-tactile associations by highlighting the role of sensory and emotional (or cognitive) factors in prompting these associations.

Sensory translation between audition and vision

Across the millennia, and across a range of disciplines, there has been a widespread desire to connect, or translate between, the senses in a manner that is meaningful, rather than arbitrary. Early examples were often inspired by the vivid, yet mostly idiosyncratic, crossmodal matches expressed by synaesthetes, often exploited for aesthetic purposes by writers, artists, and composers. A separate approach comes from those academic commentators who have attempted to translate between structurally similar dimensions of perceptual experience (such as pitch and colour). However, neither approach has succeeded in delivering consensually agreed crossmodal matches. As such, an alternative approach to sensory translation is needed. In this narrative historical review, focusing on the translation between audition and vision, we attempt to shed light on the topic by addressing the following three questions: (1) How is the topic of sensory translation related to synaesthesia, multisensory integration, and crossmodal associations? (2) Are there common processing mechanisms across the senses that can help to guarantee the success of sensory translation, or, rather, is mapping among the senses mediated by allegedly universal (e.g., amodal) stimulus dimensions? (3) Is the term 'translation' in the context of cross-sensory mappings used metaphorically or literally? Given the general mechanisms and concepts discussed throughout the review, the answers we come to regarding the nature of audio-visual translation are likely to apply to the translation between other perhaps less-frequently studied modality pairings as well.

What, if anything, can be considered an amodal sensory dimension?

The term 'amodal' is a key topic in several different research fields across experimental psychology and cognitive neuroscience, including in the areas of developmental and perception science. However, despite being regularly used in the literature, the term means something different to the researchers working in the different contexts. Many developmental scientists conceive of the term as referring to those perceptual qualities, such as, for example, the size and shape of an object, that can be picked up by multiple senses (e.g., vision and touch potentially providing information relevant to the same physical stimulus/property). However, the amodal label is also widely used in the case of those qualities that are not directly sensory, such as, for example, numerosity, rhythm, synchrony, etc. Cognitive neuroscientists, by contrast, tend to use the term amodal to refer to those central cognitive processes and brain areas that do not appear to be preferentially responsive to a particular sensory modality or to those symbolic or formal representations that essentially lack any modality and that are assumed to play a role in the higher processing of sensory information. Finally, perception scientists sometimes refer to the phenomenon of 'amodal completion', referring to the spontaneous completion of perceptual information that is missing when occluded objects are presented to observers. In this paper, we review the various different ways in which the term 'amodal' has been used in the literature and the evidence supporting the various uses of the term. Morever, we highlight some of the various properties that have been suggested to be 'amodal' over the years. Then, we try to address some of the questions that arise from the reviewed evidence, such as: Do different uses of the 'term' refer to different domains, for example, sensory information, perceptual processes, or perceptual representations? Are there any commonalities among the different uses of the term? To what extent is research on cross-modal associations (or correspondences) related to, or can shed light on, amodality? And how is the notion of amodal related to multisensory integration? Based on the reviewed evidence, it is argued that there is, as yet, no convincing empirical evidence to support the claim that amodal sensory qualities exist. We thus suggest that use of the term amodal would be more meaningful with respect to abstract cognition rather than necessarily sensory perception, the latter being more adequately explained/understood in terms of highly redundant cross-modal correspondences.

Odors modulate color appearance

Our brain constantly combines multisensory information from our surrounding environment. Odors for instance are often perceived with visual cues; these sensations interact to form our own subjective experience. This integration process can have a profound impact on the resulting experience and can alter our subjective reality. Crossmodal correspondences are the consistent associations between stimulus features in different sensory modalities. These correspondences are presumed to be bidirectional in nature and have been shown to influence our perception in a variety of different sensory modalities. Vision is dominant in our multisensory perception and can influence how we perceive information in our other senses, including olfaction. We explored the effect that different odors have on human color perception by presenting olfactory stimuli while asking observers to adjust a color patch to be devoid of hue (neutral gray task). We found a shift in the perceived neutral gray point to be biased toward warmer colors. Four out of five of our odors also trend toward their expected crossmodal correspondences. For instance, when asking observers to perform the neutral gray task while presenting the smell of cherry, the perceptually achromatic stimulus was biased toward a red-brown. Using an achromatic adjustment task, we were able to demonstrate a small but systematic effect of the presence of odors on human color perception.

Crossmodal correspondences between visual features and tastes in preschoolers: an exploratory study

Introduction

Adults possess a natural inclination to associate sensory cues derived from distinct modalities, such as the pairing of sweet with pink. However, studies exploring crossmodal correspondences in children, particularly in the sensory pairing of visual features and tastes, are scant, leaving unanswered questions regarding the developmental trajectory of crossmodal correspondences. The present study investigates whether Japanese preschool children demonstrate specific biases in shape-color, shape-taste, and color-taste associations.

Methods

In a series of in-person experiments, 92 children between 3 to 6 years of age completed matching tasks utilizing paper stimuli.

Results

Children exhibit crossmodal correspondences in shape-color (circle-red and asymmetrical star-yellow), shape-taste (triangle-salty and circle-sweet), and color-taste (yellow-sour, black-bitter, and pink-sweet) associations. Moreover, children's choices are not influenced by their individual preferences.

Discussion

The crossmodal correspondences observed in this study have been observed in previous research on adults from the same (Japanese) culture, although adults showed more crossmodal correspondences than the children in this study (e.g., pink-circle, triangle-sour, and green-bitter). Thus, while some crossmodal correspondences emerge during childhood, others may require additional time to develop, thereby highlighting the importance of understanding the cognitive mechanisms underlying crossmodal correspondences from an ontogenic perspective.

The influence of Bouba- and Kiki-like shape on perceived taste of chocolate pieces

In this paper, we present the findings of a study investigating the impact of shape on the taste perception of chocolate. Previous research has explored the influence of various sensory information on taste perception, but there has been little focus on the effect of food shape being eaten on taste perception. To explore this, we focused on the Bouba-Kiki effect, illustrating an interaction between shape and several modalities, and investigated the effect of Bouba- and Kiki-shaped (rounded and angular) foods eaten on taste perception. We utilized a 3D food printer to produce four different shapes of chocolate pieces based on the Bouba-Kiki. Participants tasted each piece and completed a chocolate flavor questionnaire. With Bayesian analysis, we determined that the Bouba-shaped chocolate pieces were perceived as sweeter than the Kiki-shaped ones, supporting earlier studies on crossmodal correspondences between shape and taste perception. However, there were no significant differences in ratings of other tastes, such as sourness and bitterness. Our research indicates that shape can affect taste perception during consumption and suggests that 3D food printers offer an opportunity to design specific shapes that influence taste experiences.

Self-construal priming modulates sonic seasoning

Introduction

“Sonic seasoning” is when music influences the real taste experiences of consumers. “Self-construal” is how individuals perceive, understand, and interpret themselves. Numerous studies have shown that independent and interdependent self-construal priming can affect a person's cognition and behavior; however, their moderating effect on the sonic seasoning effect remains unclear.

Methods

This experiment was a 2 (self-construal priming: independent self-construal or interdependent self-construal) × 2 (chocolate: milk chocolate or dark chocolate) × 2 (emotional music: positive emotional music or negative emotional music) mixed design, and explored the moderating role of self-construal priming and the effect of emotional music on taste by comparing participants' evaluations of chocolates while listening to positive or negative music after different levels of self-construal priming.

Results

After initiating independent self-construal, participants increased their ratings of milk chocolate sweetness when listening to music that elicited positive emotions, t(32) = 3.11, p = 0.004, Cohen's d = 0.54, 95% CI = [0.33, 1.61]. In contrast, interdependent self-construal priming led participants to perceive dark chocolate as sweeter when they heard positive music, t(29) = 3.63, p = 0.001, Cohen's d = 0.66, 95%CI = [0.44, 1.56].

Discussion

This study provides evidence for improving people's individual eating experience and enjoyment of food.

Caressed by music: Related preferences for velocity of touch and tempo of music?

Given that both hearing and touch are 'mechanical senses' that respond to physical pressure or mechanical energy and that individuals appear to have a characteristic internal or spontaneous tempo, individual preferences in musical and touch rhythms might be related. We explored this in two experiments probing individual preferences for tempo in the tactile and auditory modalities. Study 1 collected ratings of received stroking on the forearm and measured the velocity the participants used for stroking a fur. Music tempo preferences were assessed as mean beats per minute of individually selected music pieces and via the adjustment of experimenter-selected music to a preferred tempo. Heart rate was recorded to measure levels of physiological arousal. We found that the preferred tempo of favorite (self-selected) music correlated positively with the velocity with which each individual liked to be touched. In Study 2, participants rated videos of repeated touch on someone else's arm and videos of a drummer playing with brushes on a snare drum, both at a variety of tempos. We found that participants with similar rating patterns for the different stroking speeds did not show similar rating patterns for the different music beats. The results suggest that there may be a correspondence between preferences for favorite music and felt touch, but this is either weak or it cannot be evoked effectively with vicarious touch and/or mere drum beats. Thus, if preferences for touch and music are related, this is likely to be dependent on the specific type of stimulation.

Tactile perception of auditory roughness

Auditory roughness resulting from fast temporal beatings is often studied by summing two pure tones with close frequencies. Interestingly, the tactile counterpart of auditory roughness can be provided through touch with vibrotactile actuators. However, whether auditory roughness could also be perceived through touch and whether it exhibits similar characteristics are unclear. Here, auditory roughness perception and its tactile counterpart were evaluated using pairs of pure tone stimuli. Results revealed similar roughness curves in both modalities, suggesting similar sensory processing. This study attests to the relevance of such a paradigm for investigating auditory and tactile roughness in a multisensory fashion.

Consonance and dissonance perception. A critical review of the historical sources, multidisciplinary findings, and main hypotheses

Revealed more than two millennia ago by Pythagoras, consonance and dissonance (C/D) are foundational concepts in music theory, perception, and aesthetics. The search for the biological, acoustical, and cultural factors that affect C/D perception has resulted in descriptive accounts inspired by arithmetic, musicological, psychoacoustical or neurobiological frameworks without reaching a consensus. Here, we review the key historical sources and modern multidisciplinary findings on C/D and integrate them into three main hypotheses: the vocal similarity hypothesis (VSH), the psychocultural hypothesis (PH), and the sensorimotor hypothesis (SH). By illustrating the hypotheses-related findings, we highlight their major conceptual, methodological, and terminological shortcomings. Trying to provide a unitary framework for C/D understanding, we put together multidisciplinary research on human and animal vocalizations, which converges to suggest that auditory roughness is associated with distress/danger and, therefore, elicits defensive behavioral reactions and neural responses that indicate aversion. We therefore stress the primacy of vocality and roughness as key factors in the explanation of C/D phenomenon, and we explore the (neuro)biological underpinnings of the attraction-aversion mechanisms that are triggered by C/D stimuli. Based on the reviewed evidence, while the aversive nature of dissonance appears as solidly rooted in the multidisciplinary findings, the attractive nature of consonance remains a somewhat speculative claim that needs further investigation. Finally, we outline future directions for empirical research in C/D, especially regarding cross-modal and cross-cultural approaches.

Visualise the tastes from the label: A study on the taste-colour crossmodal association of crisp and dry

Colour is an important guideline for selection and consumption. It also draws attention to the designers, as some modern design styles require them to illustrate the taste of the product with a limited number of colours. In this case, a precise description of the taste-colour association is required. The present study explored the colour-taste crossmodal association of two tastes, crisp and dry, which are normally found in beers and are the preferred flavours of Chinese consumers. Experiments were carried out to determine the characteristics of the colours associated with the two tastes. And the strength of the tastes perceived from the colours with different hue angles was investigated. The results of this study reveal that the hue and chroma can both affect the perception of these tastes. Both tastes can be perceived from the same colour, but the strength of the taste can be varied from different hues.

Roughness perception: A multisensory/crossmodal perspective

Roughness is a perceptual attribute typically associated with certain stimuli that are presented in one of the spatial senses. In auditory research, the term is typically used to describe the harsh effects that are induced by particular sound qualities (i.e., dissonance) and human/animal vocalizations (e.g., screams, distress cries). In the tactile domain, roughness is a crucial factor determining the perceptual features of a surface. The same feature can also be ascertained visually, by means of the extraction of pattern features that determine the haptic quality of surfaces, such as grain size and density. By contrast, the term roughness has rarely been applied to the description of those stimuli perceived via the chemical senses. In this review, we take a critical look at the putative meaning(s) of the term roughness, when used in both unisensory and multisensory contexts, in an attempt to answer two key questions: (1) Is the use of the term 'roughness' the same in each modality when considered individually? and (2) Do crossmodal correspondences involving roughness match distinct perceptual features or (at least on certain occasions) do they merely pick-up on an amodal property? We start by examining the use of the term in the auditory domain. Next, we summarize the ways in which the term roughness has been used in the literature on tactile and visual perception, and in the domain of olfaction and gustation. Then, we move on to the crossmodal context, reviewing the literature on the perception of roughness in the audiovisual, audiotactile, and auditory-gustatory/olfactory domains. Finally, we highlight some limitations of the reviewed literature and we outline a number of key directions for future empirical research in roughness perception.

Coloured hearing, colour music, colour organs, and the search for perceptually meaningful correspondences between colour and sound

There has long been interest in the nature of the relationship(s) between hue and pitch or, in other words, between colour and musical/pure tones, stretching back at least as far as Newton, Goethe, Helmholtz, and beyond. In this narrative historical review, we take a closer look at the motivations that have lain behind the various assertions that have been made in the literature concerning the analogies, and possible perceptual similarities, between colour and sound. During the last century, a number of experimental psychologists have also investigated the nature of the correspondence between these two primary dimensions of perceptual experience. The multitude of different crossmodal mappings that have been put forward over the centuries are summarized, and a distinction drawn between physical/structural and psychological correspondences. The latter being further sub-divided into perceptual and affective categories. Interest in physical correspondences has typically been motivated by the structural similarities (analogous mappings) between the organization of perceptible dimensions of auditory and visual experience. Emphasis has been placed both on the similarity in terms of the number of basic categories into which pitch and colour can be arranged and also on the fact that both can be conceptualized as circular dimensions. A distinction is drawn between those commentators who have argued for a dimensional alignment of pitch and hue (based on a structural mapping), and those who appear to have been motivated by the existence of specific correspondences between particular pairs of auditory and visual stimuli instead (often, as we will see, based on the idiosyncratic correspondences that have been reported by synaesthetes). Ultimately, though, the emotional-mediation account would currently appear to provide the most parsimonious account for whatever affinity the majority of people experience between musical sounds and colour.