Neural basis of sound-symbolic pseudoword-shape correspondences

Crossmodal correspondences between visual and speech angularity and tactile jaggedness of response key

Several studies reported various crossmodal correspondences related to tactile features. These previous studies have investigated tactile-related correspondences through explicit matching or subjective evaluation tasks, which required participants to recognize relationships between tactile and other sensory features or rate tactile materials on scales with adjective labels related to visual or auditory features. However, these tasks are prone to occur the experimenter-expectancy effects and arbitrary categorization of tactile materials by the labels, making it difficult to assess implicit and non-arbitrary aspects of crossmodal correspondences. To address this, we used a speeded classification task to examine whether the angularity/roundedness of visual and auditory stimuli correspond to tactile jaggedness/fluffiness. Participants distinguished between angularity or roundedness (Experiment 1: visual shapes; Experiment 2: speech sounds) by pressing right- or left-positioned response keys with task-irrelevant jaggedness or fluffiness without prior instruction on which key represented jaggedness/fluffiness. Results showed faster keypresses for jagged/fluffy responses to angular/rounded stimuli, suggesting an implicit correspondence between these sensory features except for the experimenter-expectancy effects and the influence of the labels. Unlike previous studies that examined the correspondence with simple tactile features (e.g., weight, size), our findings suggest that even complex tactile-quality features, such as jaggedness/fluffiness, implicitly correspond to visual and auditory angularity/roundedness.

Multisensory and lexical information in speech perception

Both multisensory and lexical information are known to influence the perception of speech. However, an open question remains: is either source more fundamental to perceiving speech? In this perspective, we review the literature and argue that multisensory information plays a more fundamental role in speech perception than lexical information. Three sets of findings support this conclusion: first, reaction times and electroencephalographic signal latencies indicate that the effects of multisensory information on speech processing seem to occur earlier than the effects of lexical information. Second, non-auditory sensory input influences the perception of features that differentiate phonetic categories; thus, multisensory information determines what lexical information is ultimately processed. Finally, there is evidence that multisensory information helps form some lexical information as part of a phenomenon known as sound symbolism. These findings support a framework of speech perception that, while acknowledging the influential roles of both multisensory and lexical information, holds that multisensory information is more fundamental to the process.