Balloons and bavoons versus spikes and shikes: ERPs reveal shared neural processes for shape-sound-meaning congruence in words, and shape-sound congruence in pseudowords

The Bouba-Kiki effect is predicted by sound properties but not speech properties

Humans robustly associate spiky shapes to words like "Kiki" and round shapes to words like "Bouba." According to a popular explanation, this is because the mouth assumes an angular shape while speaking "Kiki" and a rounded shape for "Bouba." Alternatively, this effect could reflect more general associations between shape and sound that are not specific to mouth shape or articulatory properties of speech. These possibilities can be distinguished using unpronounceable sounds: The mouth-shape hypothesis predicts no Bouba-Kiki effect for these sounds, whereas the generic shape-sound hypothesis predicts a systematic effect. Here, we show that the Bouba-Kiki effect is present for a variety of unpronounceable sounds ranging from reversed words and real object sounds (n = 45 participants) and even pure tones (n = 28). The effect was strongly correlated with the mean frequency of a sound across both spoken and reversed words. The effect was not systematically predicted by subjective ratings of pronounceability or with mouth aspect ratios measured from video. Thus, the Bouba-Kiki effect is explained using simple shape-sound associations rather than using speech properties.

Cognitive neural responses in the semantic comprehension of sound symbolic words and pseudowords

Introduction

Sound symbolism is the phenomenon of sounds having non-arbitrary meaning, and it has been demonstrated that pseudowords with sound symbolic elements have similar meaning to lexical words. It is unclear how the impression given by the sound symbolic elements is semantically processed, in contrast to lexical words with definite meanings. In event-related potential (ERP) studies, phonological mapping negativity (PMN) and N400 are often used as measures of phonological and semantic processing, respectively. Therefore, in this study, we analyze PMN and N400 to clarify the differences between existing sound symbolic words (onomatopoeia or ideophones) and pseudowords in terms of semantic and phonological processing.

Methods

An existing sound symbolic word and pseudowords were presented as an auditory stimulus in combination with a picture of an event, and PMN and N400 were measured while the subjects determined whether the sound stimuli and pictures match or mismatch.

Results

In both the existing word and pseudoword tasks, the amplitude of PMN and N400 increased when the picture of an event and the speech sound did not match. Additionally, compared to the existing words, the pseudowords elicited a greater amplitude for PMN and N400. In addition, PMN latency was delayed in the mismatch condition relative to the match condition for both existing sound symbolic words and pseudowords.

Discussion

We concluded that established sound symbolic words and sound symbolic pseudowords undergo similar semantic processing. This finding suggests that sound symbolism pseudowords are not judged on a simple impression level (e.g., spiky/round) or activated by other words with similar spellings (phonological structures) in the lexicon, but are judged on a similar contextual basis as actual words.

ERPs reveal an iconic relation between sublexical phonology and affective meaning

Classical linguistic theory assumes that formal aspects, like sound, are not internally related to the meaning of words. However, recent research suggests language might code affective meaning such as threat and alert sublexically. Positing affective phonological iconicity as a systematic organization principle of the German lexicon, we calculated sublexical affective values for sub-syllabic phonological word segments from a large-scale affective lexical German database by averaging valence and arousal ratings of all words any phonological segment appears in. We tested word stimuli with either consistent or inconsistent mappings between lexical affective meaning and sublexical affective values (negative-valence/high-arousal vs. neutral-valence/low-arousal) in an EEG visual-lexical-decision task. A mismatch between sublexical and lexical affective values elicited an increased N400 response. These results reveal that systematic affective phonological iconicity - extracted from the lexicon - impacts the extraction of lexical word meaning during reading.

Computer Vision System for Expressing Texture Using Sound-Symbolic Words

The major goals of texture research in computer vision are to understand, model, and process texture and ultimately simulate human visual information processing using computer technologies. The field of computer vision has witnessed remarkable advancements in material recognition using deep convolutional neural networks (DCNNs), which have enabled various computer vision applications, such as self-driving cars, facial and gesture recognition, and automatic number plate recognition. However, for computer vision to "express" texture like human beings is still difficult because texture description has no correct or incorrect answer and is ambiguous. In this paper, we develop a computer vision method using DCNN that expresses texture of materials. To achieve this goal, we focus on Japanese "sound-symbolic" words, which can describe differences in texture sensation at a fine resolution and are known to have strong and systematic sensory-sound associations. Because the phonemes of Japanese sound-symbolic words characterize categories of texture sensations, we develop a computer vision method to generate the phonemes and structure comprising sound-symbolic words that probabilistically correspond to the input images. It was confirmed that the sound-symbolic words output by our system had about 80% accuracy rate in our evaluation.

Neural Basis of the Sound-Symbolic Crossmodal Correspondence Between Auditory Pseudowords and Visual Shapes

Sound symbolism refers to the association between the sounds of words and their meanings, often studied using the crossmodal correspondence between auditory pseudowords, e.g., 'takete' or 'maluma', and pointed or rounded visual shapes, respectively. In a functional magnetic resonance imaging study, participants were presented with pseudoword-shape pairs that were sound-symbolically congruent or incongruent. We found no significant congruency effects in the blood oxygenation level-dependent (BOLD) signal when participants were attending to visual shapes. During attention to auditory pseudowords, however, we observed greater BOLD activity for incongruent compared to congruent audiovisual pairs bilaterally in the intraparietal sulcus and supramarginal gyrus, and in the left middle frontal gyrus. We compared this activity to independent functional contrasts designed to test competing explanations of sound symbolism, but found no evidence for mediation via language, and only limited evidence for accounts based on multisensory integration and a general magnitude system. Instead, we suggest that the observed incongruency effects are likely to reflect phonological processing and/or multisensory attention. These findings advance our understanding of sound-to-meaning mapping in the brain.

Stimulus Parameters Underlying Sound-Symbolic Mapping of Auditory Pseudowords to Visual Shapes

Sound symbolism refers to non-arbitrary mappings between the sounds of words and their meanings and is often studied by pairing auditory pseudowords such as "maluma" and "takete" with rounded and pointed visual shapes, respectively. However, it is unclear what auditory properties of pseudowords contribute to their perception as rounded or pointed. Here, we compared perceptual ratings of the roundedness/pointedness of large sets of pseudowords and shapes to their acoustic and visual properties using a novel application of representational similarity analysis (RSA). Representational dissimilarity matrices (RDMs) of the auditory and visual ratings of roundedness/pointedness were significantly correlated crossmodally. The auditory perceptual RDM correlated significantly with RDMs of spectral tilt, the temporal fast Fourier transform (FFT), and the speech envelope. Conventional correlational analyses showed that ratings of pseudowords transitioned from rounded to pointed as vocal roughness (as measured by the harmonics-to-noise ratio, pulse number, fraction of unvoiced frames, mean autocorrelation, shimmer, and jitter) increased. The visual perceptual RDM correlated significantly with RDMs of global indices of visual shape (the simple matching coefficient, image silhouette, image outlines, and Jaccard distance). Crossmodally, the RDMs of the auditory spectral parameters correlated weakly but significantly with those of the global indices of visual shape. Our work establishes the utility of RSA for analysis of large stimulus sets and offers novel insights into the stimulus parameters underlying sound symbolism, showing that sound-to-shape mapping is driven by acoustic properties of pseudowords and suggesting audiovisual cross-modal correspondence as a basis for language users' sensitivity to this type of sound symbolism.

Electrophysiological Differentiation of the Effects of Stress and Accent on Lexical Integration in Highly Fluent Bilinguals

Individuals who acquire a second language (L2) after infancy often retain features of their native language (L1) accent. Cross-language priming studies have shown negative effects of L1 accent on L2 comprehension, but the role of specific speech features, such as lexical stress, is mostly unknown. Here, we investigate whether lexical stress and accent differently modulate semantic processing and cross-language lexical activation in Welsh–English bilinguals, given that English and Welsh differ substantially in terms of stress realisation. In an L2 cross-modal priming paradigm, we manipulated the stress pattern and accent of spoken primes, whilst participants made semantic relatedness judgments on visual word targets. Event-related brain potentials revealed a main effect of stress on target integration, such that stimuli with stress patterns compatible with either the L1 or L2 required less processing effort than stimuli with stress incompatible with both Welsh and English. An independent cross-language phonological overlap manipulation revealed an interaction between accent and L1 access. Interestingly, although it increased processing effort, incorrect stress did not significantly modulate semantic priming effects or covert access to L1 phonological representations. Our results are consistent with the concept of language-specific stress templates, and suggest that accent and lexical stress affect speech comprehension mechanisms differentially.

Is un stylo sharper than une épée? Investigating the interaction of sound symbolism and grammatical gender in English and French speakers

While the arbitrariness of language has long been considered one of its defining features, there is growing evidence that non-arbitrariness also plays an important role. Here we investigated two sources of non-arbitrariness: systematicity (via grammatical gender) and iconicity (via shape sound symbolism). We manipulated these two elements orthogonally, allowing us to examine the effect of each. In Experiment 1, we found that French speakers associated nonwords containing feminine (masculine) endings with round (sharp) shapes. French speakers also associated nonwords containing round-sounding (sharp-sounding) phonemes with round (sharp) shapes. This was repeated using auditory presentation with both an English-speaking (Experiment 2a) and French-speaking (Experiment 2b) sample. As predicted, the English speakers showed no effects of grammatical gender, while the French speakers did. These results demonstrate that speakers of a language with grammatical gender associate different properties to words belonging to different genders. The results also show that sound symbolism can emerge in stimuli with existing associated information (i.e., endings indicative of grammatical gender, and the association that they evoke). Finally, while previous studies have looked at effects of arbitrary and non-arbitrary mappings contained in a single stimulus, this is the first study to demonstrate that different kinds of non-arbitrary mappings can have an effect when appearing in the same stimulus. Together these results add to our understanding of the importance of non-arbitrariness in language.

Five mechanisms of sound symbolic association

Sound symbolism refers to an association between phonemes and stimuli containing particular perceptual and/or semantic elements (e.g., objects of a certain size or shape). Some of the best-known examples include the mil/mal effect (Sapir, Journal of Experimental Psychology, 12, 225-239, 1929) and the maluma/takete effect (Köhler, 1929). Interest in this topic has been on the rise within psychology, and studies have demonstrated that sound symbolic effects are relevant for many facets of cognition, including language, action, memory, and categorization. Sound symbolism also provides a mechanism by which words' forms can have nonarbitrary, iconic relationships with their meanings. Although various proposals have been put forth for how phonetic features (both acoustic and articulatory) come to be associated with stimuli, there is as yet no generally agreed-upon explanation. We review five proposals: statistical co-occurrence between phonetic features and associated stimuli in the environment, a shared property among phonetic features and stimuli; neural factors; species-general, evolved associations; and patterns extracted from language. We identify a number of outstanding questions that need to be addressed on this topic and suggest next steps for the field.

Sound iconicity of abstract concepts: Place of articulation is implicitly associated with abstract concepts of size and social dominance

The concept of sound iconicity implies that phonemes are intrinsically associated with non-acoustic phenomena, such as emotional expression, object size or shape, or other perceptual features. In this respect, sound iconicity is related to other forms of cross-modal associations in which stimuli from different sensory modalities are associated with each other due to the implicitly perceived correspondence of their primal features. One prominent example is the association between vowels, categorized according to their place of articulation, and size, with back vowels being associated with bigness and front vowels with smallness. However, to date the relative influence of perceptual and conceptual cognitive processing on this association is not clear. To bridge this gap, three experiments were conducted in which associations between nonsense words and pictures of animals or emotional body postures were tested. In these experiments participants had to infer the relation between visual stimuli and the notion of size from the content of the pictures, while directly perceivable features did not support–or even contradicted–the predicted association. Results show that implicit associations between articulatory-acoustic characteristics of phonemes and pictures are mainly influenced by semantic features, i.e., the content of a picture, whereas the influence of perceivable features, i.e., size or shape, is overridden. This suggests that abstract semantic concepts can function as an interface between different sensory modalities, facilitating cross-modal associations.

Glyph guessing for 'oo' and 'ee': spatial frequency information in sound symbolic matching for ancient and unfamiliar scripts

In three experiments, we asked whether diverse scripts contain interpretable information about the speech sounds they represent. When presented with a pair of unfamiliar letters, adult readers correctly guess which is /i/ (the 'ee' sound in 'feet'), and which is /u/ (the 'oo' sound in 'shoe') at rates higher than expected by chance, as shown in a large sample of Singaporean university students (Experiment 1) and replicated in a larger sample of international Internet users (Experiment 2). To uncover what properties of the letters contribute to different scripts' 'guessability,' we analysed the visual spatial frequencies in each letter (Experiment 3). We predicted that the lower spectral frequencies in the formants of the vowel /u/ would pattern with lower spatial frequencies in the corresponding letters. Instead, we found that across all spatial frequencies, the letter with more black/white cycles (i.e. more ink) was more likely to be guessed as /u/, and the larger the difference between the glyphs in a pair, the higher the script's guessability. We propose that diverse groups of humans across historical time and geographical space tend to employ similar iconic strategies for representing speech in visual form, and provide norms for letter pairs from 56 diverse scripts.

To call a cloud ‘cirrus’: sound symbolism in names for categories or items

The aim of the present paper is to experimentally test whether sound symbolism has selective effects on labels with different ranges-of-reference within a simple noun-hierarchy. In two experiments, adult participants learned the make up of two categories of unfamiliar objects (‘alien life forms’), and were passively exposed to either category-labels or item-labels, in a learning-by-guessing categorization task. Following category training, participants were tested on their visual discrimination of object pairs. For different groups of participants, the labels were either congruent or incongruent with the objects. In Experiment 1, when trained on items with individual labels, participants were worse (made more errors) at detecting visual object mismatches when trained labels were incongruent. In Experiment 2, when participants were trained on items in labelled categories, participants were faster at detecting a match if the trained labels were congruent, and faster at detecting a mismatch if the trained labels were incongruent. This pattern of results suggests that sound symbolism in category labels facilitates later similarity judgments when congruent, and discrimination when incongruent, whereas for item labels incongruence generates error in judgements of visual object differences. These findings reveal that sound symbolic congruence has a different outcome at different levels of labelling within a noun hierarchy. These effects emerged in the absence of the label itself, indicating subtle but pervasive effects on visual object processing.

Can a Word Sound Like a Shape Before You Have Seen It? Sound-Shape Mapping Prior to Conscious Awareness

Nonarbitrary mappings between sound and shape (i.e., the bouba-kiki effect) have been shown across different cultures and early in development; however, the level of processing at which this effect arises remains unclear. Here we show that the mapping occurs prior to conscious awareness of the visual stimuli. Under continuous flash suppression, congruent stimuli (e.g., "kiki" inside an angular shape) broke through to conscious awareness faster than incongruent stimuli. This was true even when we trained people to pair unfamiliar letters with auditory word forms, a result showing that the effect was driven by the phonology, not the visual features, of the letters. Furthermore, visibility thresholds of the shapes decreased when they were preceded by a congruent auditory word form in a masking paradigm. Taken together, our results suggest that sound-shape mapping can occur automatically prior to conscious awareness of visual shapes, and that sensory congruence facilitates conscious awareness of a stimulus being present.

From the Bob/Kirk effect to the Benoit/Éric effect: Testing the mechanism of name sound symbolism in two languages

Although it is often assumed that language involves an arbitrary relationship between form and meaning, many studies have demonstrated that nonwords like maluma are associated with round shapes, while nonwords like takete are associated with sharp shapes (i.e., the Maluma/Takete effect, Köhler, 1929/1947). The majority of the research on sound symbolism has used nonwords, but Sidhu and Pexman (2015) recently extended this effect to existing labels: real English first names (i.e., the Bob/Kirk effect). In the present research we tested whether the effects of name sound symbolism generalize to French speakers (Experiment 1) and French names (Experiment 2). In addition, we assessed the underlying mechanism of name sound symbolism, investigating the roles of phonology and orthography in the effect. Results showed that name sound symbolism does generalize to French speakers and French names. Further, this robust effect remained the same when names were presented in a curved vs. angular font (Experiment 3), or when the salience of orthographic information was reduced through auditory presentation (Experiment 4). Together these results suggest that the Bob/Kirk effect is pervasive, and that it is based on fundamental features of name phonemes.