1
|
Chow HM, Ma YK, Tseng CH. Social and communicative not a prerequisite: Preverbal infants learn an abstract rule only from congruent audiovisual dynamic pitch-height patterns. J Exp Child Psychol 2024; 248:106046. [PMID: 39241321 DOI: 10.1016/j.jecp.2024.106046] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/29/2023] [Revised: 07/23/2024] [Accepted: 07/29/2024] [Indexed: 09/09/2024]
Abstract
Learning in the everyday environment often requires the flexible integration of relevant multisensory information. Previous research has demonstrated preverbal infants' capacity to extract an abstract rule from audiovisual temporal sequences matched in temporal synchrony. Interestingly, this capacity was recently reported to be modulated by crossmodal correspondence beyond spatiotemporal matching (e.g., consistent facial emotional expressions or articulatory mouth movements matched with sound). To investigate whether such modulatory influence applies to non-social and non-communicative stimuli, we conducted a critical test using audiovisual stimuli free of social information: visually upward (and downward) moving objects paired with a congruent tone of ascending or incongruent (descending) pitch. East Asian infants (8-10 months old) from a metropolitan area in Asia demonstrated successful abstract rule learning in the congruent audiovisual condition and demonstrated weaker learning in the incongruent condition. This implies that preverbal infants use crossmodal dynamic pitch-height correspondence to integrate multisensory information before rule extraction. This result confirms that preverbal infants are ready to use non-social non-communicative information in serving cognitive functions such as rule extraction in a multisensory context.
Collapse
Affiliation(s)
- Hiu Mei Chow
- Department of Psychology, St. Thomas University, Fredericton, New Brunswick E3B 5G3, Canada
| | - Yuen Ki Ma
- Department of Psychology, The University of Hong Kong, Pokfulam, Hong Kong
| | - Chia-Huei Tseng
- Research Institute of Electrical Communication, Tohoku University, Sendai, Miyagi 980-0812, Japan.
| |
Collapse
|
2
|
Sun Y, Yao L, Fu Q. Crossmodal Correspondence Mediates Crossmodal Transfer from Visual to Auditory Stimuli in Category Learning. J Intell 2024; 12:80. [PMID: 39330459 PMCID: PMC11433196 DOI: 10.3390/jintelligence12090080] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/04/2024] [Revised: 08/12/2024] [Accepted: 08/26/2024] [Indexed: 09/28/2024] Open
Abstract
This article investigated whether crossmodal correspondence, as a sensory translation phenomenon, can mediate crossmodal transfer from visual to auditory stimuli in category learning and whether multimodal category learning can influence the crossmodal correspondence between auditory and visual stimuli. Experiment 1 showed that the category knowledge acquired from elevation stimuli affected the categorization of pitch stimuli when there were robust crossmodal correspondence effects between elevation and size, indicating that crossmodal transfer occurred between elevation and pitch stimuli. Experiments 2 and 3 revealed that the size category knowledge could not be transferred to the categorization of pitches, but interestingly, size and pitch category learning determined the direction of the pitch-size correspondence, suggesting that the pitch-size correspondence was not stable and could be determined using multimodal category learning. Experiment 4 provided further evidence that there was no crossmodal transfer between size and pitch, due to the absence of a robust pitch-size correspondence. These results demonstrated that crossmodal transfer can occur between audio-visual stimuli with crossmodal correspondence, and multisensory category learning can change the corresponding relationship between audio-visual stimuli. These findings suggest that crossmodal transfer and crossmodal correspondence share similar abstract representations, which can be mediated by semantic content such as category labels.
Collapse
Affiliation(s)
- Ying Sun
- State Key Laboratory of Brain and Cognitive Science, Institute of Psychology, Chinese Academy of Sciences, Beijing 100101, China; (Y.S.); (L.Y.)
- University of Chinese Academy of Sciences, Beijing 101408, China
- College of Humanities and Education, Inner Mongolia Medical University, Hohhot 010110, China
| | - Liansheng Yao
- State Key Laboratory of Brain and Cognitive Science, Institute of Psychology, Chinese Academy of Sciences, Beijing 100101, China; (Y.S.); (L.Y.)
- University of Chinese Academy of Sciences, Beijing 101408, China
| | - Qiufang Fu
- State Key Laboratory of Brain and Cognitive Science, Institute of Psychology, Chinese Academy of Sciences, Beijing 100101, China; (Y.S.); (L.Y.)
- University of Chinese Academy of Sciences, Beijing 101408, China
| |
Collapse
|
3
|
Ohtake Y, Tanaka K, Yamamoto K. How many categories are there in crossmodal correspondences? A study based on exploratory factor analysis. PLoS One 2023; 18:e0294141. [PMID: 37963160 PMCID: PMC10645324 DOI: 10.1371/journal.pone.0294141] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/25/2023] [Accepted: 10/25/2023] [Indexed: 11/16/2023] Open
Abstract
Humans naturally associate stimulus features of one sensory modality with those of other modalities, such as associating bright light with high-pitched tones. This phenomenon is called crossmodal correspondence and is found between various stimulus features, and has been suggested to be categorized into several types. However, it is not yet clear whether there are differences in the underlying mechanism between the different kinds of correspondences. This study used exploratory factor analysis to address this question. Through an online experiment platform, we asked Japanese adult participants (Experiment 1: N = 178, Experiment 2: N = 160) to rate the degree of correspondence between two auditory and five visual features. The results of two experiments revealed that two factors underlie the subjective judgments of the audiovisual crossmodal correspondences: One factor was composed of correspondences whose auditory and visual features can be expressed in common Japanese terms, such as the loudness-size and pitch-vertical position correspondences, and another factor was composed of correspondences whose features have no linguistic similarities, such as pitch-brightness and pitch-shape correspondences. These results confirm that there are at least two types of crossmodal correspondences that are likely to differ in terms of language mediation.
Collapse
Affiliation(s)
- Yuka Ohtake
- Graduate School of Human-Environment Studies, Kyushu University, Fukuoka, Japan
- Japan Society for the Promotion of Science, Tokyo, Japan
| | - Kanji Tanaka
- Faculty of Arts and Science, Kyushu University, Fukuoka, Japan
| | - Kentaro Yamamoto
- Faculty of Human-Environment Studies, Kyushu University, Fukuoka, Japan
| |
Collapse
|
4
|
Dolscheid S, Çelik S, Erkan H, Küntay A, Majid A. Children's associations between space and pitch are differentially shaped by language. Dev Sci 2023; 26:e13341. [PMID: 36315982 DOI: 10.1111/desc.13341] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2022] [Revised: 09/08/2022] [Accepted: 10/18/2022] [Indexed: 11/30/2022]
Abstract
Musical properties, such as auditory pitch, are not expressed in the same way across cultures. In some languages, pitch is expressed in terms of spatial height (high vs. low), whereas others rely on thickness vocabulary (thick = low frequency vs. thin = high frequency). We investigated how children represent pitch in the face of this variable linguistic input by examining the developmental trajectory of linguistic and non-linguistic space-pitch associations in children who acquire Dutch (a height-pitch language) or Turkish (a thickness-pitch language). Five-year-olds, 7-year-olds, 9-year-olds, and 11-year-olds were tested for their understanding of pitch terminology and their associations of spatial dimensions with auditory pitch when no language was used. Across tasks, thickness-pitch associations were more robust than height-pitch associations. This was true for Turkish children, and also Dutch children not exposed to thickness-pitch vocabulary. Height-pitch associations, on the other hand, were not reliable-not even in Dutch-speaking children until age 11-the age when they demonstrated full comprehension of height-pitch terminology. Moreover, Turkish-speaking children reversed height-pitch associations. Taken together, these findings suggest thickness-pitch associations are acquired in similar ways by children from different cultures, but the acquisition of height-pitch associations is more susceptible to linguistic input. Overall, then, despite cross-cultural stability in some components, there is variation in how children come to represent musical pitch, one of the building blocks of music. RESEARCH HIGHLIGHTS: Children from diverse cultures differ in their understanding of music vocabulary and in their nonlinguistic associations between spatial dimensions and auditory pitch. Height-pitch mappings are acquired late and require additional scaffolding from language, whereas thickness-pitch mappings are acquired early and are less susceptible to language input. Space-pitch mappings are not static from birth to adulthood, but change over development, suggesting music cognition is shaped by cross-cultural experience.
Collapse
Affiliation(s)
- Sarah Dolscheid
- University of Cologne, Department of Rehabilitation and Special Education, Cologne, Germany
| | | | - Hasan Erkan
- Radboud University, Nijmegen, The Netherlands
| | | | | |
Collapse
|
5
|
Sclafani V, De Pascalis L, Bozicevic L, Sepe A, Ferrari PF, Murray L. Similarities and differences in the functional architecture of mother- infant communication in rhesus macaque and British mother-infant dyads. Sci Rep 2023; 13:13164. [PMID: 37574499 PMCID: PMC10423724 DOI: 10.1038/s41598-023-39623-3] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2023] [Accepted: 07/27/2023] [Indexed: 08/15/2023] Open
Abstract
Similarly to humans, rhesus macaques engage in mother-infant face-to-face interactions. However, no previous studies have described the naturally occurring structure and development of mother-infant interactions in this population and used a comparative-developmental perspective to directly compare them to the ones reported in humans. Here, we investigate the development of infant communication, and maternal responsiveness in the two groups. We video-recorded mother-infant interactions in both groups in naturalistic settings and analysed them with the same micro-analytic coding scheme. Results show that infant social expressiveness and maternal responsiveness are similarly structured in humans and macaques. Both human and macaque mothers use specific mirroring responses to specific infant social behaviours (modified mirroring to communicative signals, enriched mirroring to affiliative gestures). However, important differences were identified in the development of infant social expressiveness, and in forms of maternal responsiveness, with vocal responses and marking behaviours being predominantly human. Results indicate a common functional architecture of mother-infant communication in humans and monkeys, and contribute to theories concerning the evolution of specific traits of human behaviour.
Collapse
Affiliation(s)
- V Sclafani
- Winnicott Research Unit, Department of Psychology, University of Reading, Reading, UK.
- College of Social Sciences, School of Psychology, University of Lincoln, Brayford Pool, Lincoln, LN6 7TS, UK.
| | - L De Pascalis
- Winnicott Research Unit, Department of Psychology, University of Reading, Reading, UK.
- Department of Psychology, Institute of Population Health, University of Liverpool, Liverpool, UK.
- Department of Psychology, University of Bologna, Bologna, Italy.
| | - L Bozicevic
- Winnicott Research Unit, Department of Psychology, University of Reading, Reading, UK
- Department of Primary Care & Mental Health, Institute of Population Health, University of Liverpool, Liverpool, Merseyside, UK
| | - A Sepe
- Department of Medicine and Surgery, University of Parma, Parma, Italy
- Laboratory of Neuro- and Psychophysiology, Department of Neurosciences, KU Leuven Medical School, Leuven, Belgium
| | - P F Ferrari
- Department of Medicine and Surgery, University of Parma, Parma, Italy
- Institut des Sciences Cognitives 'Marc Jeannerod', CNRS, Bron, and Université Claude Bernard Lyon 1, Lyon, France
| | - L Murray
- Winnicott Research Unit, Department of Psychology, University of Reading, Reading, UK
| |
Collapse
|
6
|
Singh M, Mehr SA. Universality, domain-specificity, and development of psychological responses to music. NATURE REVIEWS PSYCHOLOGY 2023; 2:333-346. [PMID: 38143935 PMCID: PMC10745197 DOI: 10.1038/s44159-023-00182-z] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Accepted: 03/30/2023] [Indexed: 12/26/2023]
Abstract
Humans can find music happy, sad, fearful, or spiritual. They can be soothed by it or urged to dance. Whether these psychological responses reflect cognitive adaptations that evolved expressly for responding to music is an ongoing topic of study. In this Review, we examine three features of music-related psychological responses that help to elucidate whether the underlying cognitive systems are specialized adaptations: universality, domain-specificity, and early expression. Focusing on emotional and behavioural responses, we find evidence that the relevant psychological mechanisms are universal and arise early in development. However, the existing evidence cannot establish that these mechanisms are domain-specific. To the contrary, many findings suggest that universal psychological responses to music reflect more general properties of emotion, auditory perception, and other human cognitive capacities that evolved for non-musical purposes. Cultural evolution, driven by the tinkering of musical performers, evidently crafts music to compellingly appeal to shared psychological mechanisms, resulting in both universal patterns (such as form-function associations) and culturally idiosyncratic styles.
Collapse
Affiliation(s)
- Manvir Singh
- Institute for Advanced Study in Toulouse, University of
Toulouse 1 Capitole, Toulouse, France
| | - Samuel A. Mehr
- Yale Child Study Center, Yale University, New Haven, CT,
USA
- School of Psychology, University of Auckland, Auckland,
New Zealand
| |
Collapse
|
7
|
Guedes D, Vaz Garrido M, Lamy E, Pereira Cavalheiro B, Prada M. Crossmodal interactions between audition and taste: A systematic review and narrative synthesis. Food Qual Prefer 2023. [DOI: 10.1016/j.foodqual.2023.104856] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/30/2023]
|
8
|
Shang N, Styles SJ. Implicit Association Test (IAT) Studies Investigating Pitch-Shape Audiovisual Cross-modal Associations Across Language Groups. Cogn Sci 2023; 47:e13221. [PMID: 36607162 PMCID: PMC10078355 DOI: 10.1111/cogs.13221] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/18/2021] [Revised: 11/14/2022] [Accepted: 11/15/2022] [Indexed: 01/07/2023]
Abstract
Previous studies have shown that Chinese speakers and non-Chinese speakers exhibit different patterns of cross-modal congruence for the lexical tones of Mandarin Chinese, depending on which features of the pitch they attend to. But is this pattern of language-specific listening a conscious cultural strategy or an automatic processing effect? If automatic, does it also apply when the same pitch contours no longer sound like speech? Implicit Association Tests (IATs) provide an indirect measure of cross-modal association. In a series of IAT studies, conducted with participants with three kinds of language backgrounds (Chinese-dominant bilinguals, Chinese balanced bilinguals, and English speakers with no Chinese experience) we find language-specific congruence effects for Mandarin lexical tones but not for matched sine-wave stimuli. That is, for linguistic stimuli, non-Chinese speakers show advantages for pitch-height congruence (high-pointy, low-curvy); no congruence effects were found for Chinese speakers. For non-linguistic stimuli, all participant groups showed advantages for pitch-height congruence. The present findings suggest that non-lexical tone congruence (high-pointy, low-curvy) is a basic congruence pattern, and the acquisition of a language with lexical tone can alter this perception.
Collapse
Affiliation(s)
- Nan Shang
- School of Foreign Studies, Northwestern Polytechnical University
| | - Suzy J Styles
- Psychology, School of Social Sciences, Nanyang Technological University.,Centre for Research and Development on Learning, Nanyang Technological University
| |
Collapse
|
9
|
Pitt B, Casasanto D. Spatial metaphors and the design of everyday things. Front Psychol 2022; 13:1019957. [PMID: 36483703 PMCID: PMC9723982 DOI: 10.3389/fpsyg.2022.1019957] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/15/2022] [Accepted: 09/26/2022] [Indexed: 10/24/2023] Open
Abstract
People use space (e.g., left-right, up-down) to think about a variety of non-spatial concepts like time, number, similarity, and emotional valence. These spatial metaphors can be used to inform the design of user interfaces, which visualize many of these concepts in space. Traditionally, researchers have relied on patterns in language to discover habits of metaphorical thinking. However, advances in cognitive science have revealed that many spatial metaphors remain unspoken, shaping people's preferences, memories, and actions independent of language - and even in contradiction to language. Here we argue that cognitive science can impact our everyday lives by informing the design of physical and digital objects via the spatial metaphors in people's minds. We propose a simple principle for predicting which spatial metaphors organize people's non-spatial concepts based on the structure of their linguistic, cultural, and bodily experiences. By leveraging the latent metaphorical structure of people's minds, we can design objects and interfaces that help people think.
Collapse
Affiliation(s)
- Benjamin Pitt
- Department of Psychology, University of California, Berkeley, Berkeley, CA, United States
| | - Daniel Casasanto
- Department of Psychology, Cornell University, Ithaca, NY, United States
| |
Collapse
|
10
|
Boulenger V, Finos L, Koun E, Salemme R, Desoche C, Roy AC. Up right, not right up: Primacy of verticality in both language and movement. Front Hum Neurosci 2022; 16:981330. [PMID: 36248682 PMCID: PMC9558293 DOI: 10.3389/fnhum.2022.981330] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2022] [Accepted: 09/09/2022] [Indexed: 11/28/2022] Open
Abstract
When describing motion along both the horizontal and vertical axes, languages from different families express the elements encoding verticality before those coding for horizontality (e.g., going up right instead of right up). In light of the motor grounding of language, the present study investigated whether the prevalence of verticality in Path expression also governs the trajectory of arm biological movements. Using a 3D virtual-reality setting, we tracked the kinematics of hand pointing movements in five spatial directions, two of which implied the vertical and horizontal vectors equally (i.e., up right +45° and bottom right −45°). Movement onset could be prompted by visual or auditory verbal cues, the latter being canonical in French (“en haut à droite”/up right) or not (“à droite en haut”/right up). In two experiments, analyses of the index finger kinematics revealed a significant effect of gravity, with earlier acceleration, velocity, and deceleration peaks for upward (+45°) than downward (−45°) movements, irrespective of the instructions. Remarkably, confirming the linguistic observations, we found that vertical kinematic parameters occurred earlier than horizontal ones for upward movements, both for visual and congruent verbal cues. Non-canonical verbal instructions significantly affected this temporal dynamic: for upward movements, the horizontal and vertical components temporally aligned, while they reversed for downward movements where the kinematics of the vertical axis was delayed with respect to that of the horizontal one. This temporal dynamic is so deeply anchored that non-canonical verbal instructions allowed for horizontality to precede verticality only for movements that do not fight against gravity. Altogether, our findings provide new insights into the embodiment of language by revealing that linguistic path may reflect the organization of biological movements, giving priority to the vertical axis.
Collapse
Affiliation(s)
- Véronique Boulenger
- Laboratoire Dynamique Du Langage, UMR 5596, CNRS/University Lyon 2, Lyon, France
- *Correspondence: Véronique Boulenger,
| | - Livio Finos
- Department of Statistical Sciences, University of Padua, Padua, Italy
| | - Eric Koun
- Integrative Multisensory Perception Action & Cognition Team (IMPACT), Lyon Neuroscience Research Center, INSERM U1028, CNRS U5292, University Lyon 1, Lyon, France
| | - Roméo Salemme
- Integrative Multisensory Perception Action & Cognition Team (IMPACT), Lyon Neuroscience Research Center, INSERM U1028, CNRS U5292, University Lyon 1, Lyon, France
- Neuro-Immersion, Lyon Neuroscience Research Center, Lyon, France
| | - Clément Desoche
- Integrative Multisensory Perception Action & Cognition Team (IMPACT), Lyon Neuroscience Research Center, INSERM U1028, CNRS U5292, University Lyon 1, Lyon, France
- Neuro-Immersion, Lyon Neuroscience Research Center, Lyon, France
| | - Alice C. Roy
- Laboratoire Dynamique Du Langage, UMR 5596, CNRS/University Lyon 2, Lyon, France
| |
Collapse
|
11
|
Martin L, Marie J, Brun M, de Hevia MD, Streri A, Izard V. Abstract representations of small sets in newborns. Cognition 2022; 226:105184. [PMID: 35671541 PMCID: PMC9289748 DOI: 10.1016/j.cognition.2022.105184] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2021] [Revised: 03/22/2022] [Accepted: 05/26/2022] [Indexed: 11/21/2022]
Abstract
From the very first days of life, newborns are not tied to represent narrow, modality- and object-specific aspects of their environment. Rather, they sometimes react to abstract properties shared by stimuli of very different nature, such as approximate numerosity or magnitude. As of now, however, there is no evidence that newborns possess abstract representations that apply to small sets: in particular, while newborns can match large approximate numerosities across senses, this ability does not extend to small numerosities. In two experiments, we presented newborn infants (N = 64, age 17 to 98 h) with patterned sets AB or ABB simultaneously in the auditory and visual modalities. Auditory patterns were presented as periodic sequences of sounds (AB: triangle-drum-triangle-drum-triangle-drum …; ABB: triangle-drum-drum-triangle-drum-drum-triangle-drum-drum …), and visual patterns as arrays of 2 or 3 shapes (AB: circle-diamond; ABB: circle-diamond-diamond). In both experiments, we found that participants reacted and looked longer when the patterns matched across the auditory and visual modalities – provided that the first stimulus they received was congruent. These findings uncover the existence of yet another type of abstract representations at birth, applying to small sets. As such, they bolster the hypothesis that newborns are endowed with the capacity to represent their environment in broad strokes, in terms of its most abstract properties. This capacity for abstraction could later serve as a scaffold for infants to learn about the particular entities surrounding them. Newborns were presented with auditory and visual patterns (AB vs. ABB). Participants reacted when the patterns presented were congruent across modalities. Newborns possess abstract representations applying to small sets. These representations may encode numerosity and/or repetitions.
Collapse
Affiliation(s)
- Lucie Martin
- Université Paris Cité, CNRS, Integrative Neuroscience and Cognition Center, F-75006 Paris, France
| | - Julien Marie
- Université Paris Cité, CNRS, Integrative Neuroscience and Cognition Center, F-75006 Paris, France
| | - Mélanie Brun
- Université Paris Cité, CNRS, Integrative Neuroscience and Cognition Center, F-75006 Paris, France
| | - Maria Dolores de Hevia
- Université Paris Cité, CNRS, Integrative Neuroscience and Cognition Center, F-75006 Paris, France
| | - Arlette Streri
- Université Paris Cité, CNRS, Integrative Neuroscience and Cognition Center, F-75006 Paris, France
| | - Véronique Izard
- Université Paris Cité, CNRS, Integrative Neuroscience and Cognition Center, F-75006 Paris, France.
| |
Collapse
|
12
|
Spence C. Exploring Group Differences in the Crossmodal Correspondences. Multisens Res 2022; 35:495-536. [PMID: 35985650 DOI: 10.1163/22134808-bja10079] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/09/2022] [Accepted: 07/22/2022] [Indexed: 11/19/2022]
Abstract
There has been a rapid growth of interest amongst researchers in the cross-modal correspondences in recent years. In part, this has resulted from the emerging realization of the important role that the correspondences can sometimes play in multisensory integration. In turn, this has led to an interest in the nature of any differences between individuals, or rather, between groups of individuals, in the strength and/or consensuality of cross-modal correspondences that may be observed in both neurotypically normal groups cross-culturally, developmentally, and across various special populations (including those who have lost a sense, as well as those with autistic tendencies). The hope is that our emerging understanding of such group differences may one day provide grounds for supporting the reality of the various different types of correspondence that have so far been proposed, namely structural, statistical, semantic, and hedonic (or emotionally mediated).
Collapse
Affiliation(s)
- Charles Spence
- Crossmodal Research Laboratory, Department of Experimental Psychology, University of Oxford, New Radcliffe House, Walton Street, Oxford, OX2 6BW, UK
| |
Collapse
|
13
|
Weinberger AB, Gallagher NM, Colaizzi G, Liu N, Parrott N, Fearon E, Shaikh N, Green AE. Analogical mapping across sensory modalities and evidence for a general analogy factor. Cognition 2022; 223:105029. [PMID: 35091260 DOI: 10.1016/j.cognition.2022.105029] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/23/2021] [Revised: 12/20/2021] [Accepted: 01/17/2022] [Indexed: 11/03/2022]
Abstract
Analogy is a central component of human cognition. Analogical "mapping" of similarities between pieces of information present in our experiences supports cognitive and social development, classroom learning, and creative insights and innovation. To date, analogical mapping has primarily been studied within separate modalities of information (e.g., verbal analogies between words, visuo-spatial analogies between objects). However, human experience, in development and adulthood, includes highly variegated information (e.g., words, sounds, objects) received via multiple sensory and information-processing pathways (e.g., visual vs. auditory pathways). Whereas cross-modal correspondences (e.g., between pitch and height) have been observed, the correspondences were between individual items, rather than between relations. Thus, analogical mapping (characterized by second-order relations between relations) has not been directly tested as a basis for cross-modal correspondence. Here, we devised novel cross-modality analogical stimuli (lines-to-sounds, lines-to-words, words-to-sounds) that explicated second-order comparisons between relations. In four samples across three studies-participants demonstrated well-above-chance identification of cross-modal second-order relations, providing robust evidence of analogy across modalities. Further, performance across all analogy types was explained by a single factor, indicating a modality-general analogical ability (i.e., an "analo-g" factor). Analo-g explained performance over-and-above fluid intelligence as well as verbal and spatial abilities, though a stronger relationship to verbal than visuo-spatial ability emerged, consistent with verbal/semantic contributions to analogy. The present data suggests novel questions about our ability to find/learn second-order relations among the diverse information sources that populate human experience, and about cross-modal human and AI analogical mapping in developmental, educational, and creative contexts.
Collapse
Affiliation(s)
- Adam B Weinberger
- Department of Psychology, Georgetown University, USA; Penn Center for Neuroaesthetics, University of Pennsylvania, USA.
| | - Natalie M Gallagher
- Department of Psychology, Georgetown University, USA; Department of Psychology, Princeton University, USA
| | | | - Nathaniel Liu
- Department of Psychology, Georgetown University, USA
| | | | - Edward Fearon
- Department of Psychology, Georgetown University, USA
| | | | - Adam E Green
- Department of Psychology, Georgetown University, USA.
| |
Collapse
|
14
|
Kim H, Lee IK. Studying the Effects of Congruence of Auditory and Visual Stimuli on Virtual Reality Experiences. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 2022; 28:2080-2090. [PMID: 35167477 DOI: 10.1109/tvcg.2022.3150514] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Studies in virtual reality (VR) have introduced numerous multisensory simulation techniques for more immersive VR experiences. However, although they primarily focus on expanding sensory types or increasing individual sensory quality, they lack consensus in designing appropriate interactions between different sensory stimuli. This paper explores how the congruence between auditory and visual (AV) stimuli, which are the sensory stimuli typically provided by VR devices, affects the cognition and experience of VR users as a critical interaction factor in promoting multisensory integration. We defined the types of (in)congruence between AV stimuli, and then designed 12 virtual spaces with different types or degrees of congruence between AV stimuli. We then evaluated the presence, immersion, motion sickness, and cognition changes in each space. We observed the following key findings: 1) there is a limit to the degree of temporal or spatial incongruence that can be tolerated, with few negative effects on user experience until that point is exceeded; 2) users are tolerant of semantic incongruence; 3) a simulation that considers synesthetic congruence contributes to the user's sense of immersion and presence. Based on these insights, we identified the essential considerations for designing sensory simulations in VR and proposed future research directions.
Collapse
|
15
|
Holler J, Drijvers L, Rafiee A, Majid A. Embodied Space-pitch Associations are Shaped by Language. Cogn Sci 2022; 46:e13083. [PMID: 35188682 DOI: 10.1111/cogs.13083] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2021] [Revised: 11/29/2021] [Accepted: 12/02/2021] [Indexed: 11/28/2022]
Abstract
Height-pitch associations are claimed to be universal and independent of language, but this claim remains controversial. The present study sheds new light on this debate with a multimodal analysis of individual sound and melody descriptions obtained in an interactive communication paradigm with speakers of Dutch and Farsi. The findings reveal that, in contrast to Dutch speakers, Farsi speakers do not use a height-pitch metaphor consistently in speech. Both Dutch and Farsi speakers' co-speech gestures did reveal a mapping of higher pitches to higher space and lower pitches to lower space, and this gesture space-pitch mapping tended to co-occur with corresponding spatial words (high-low). However, this mapping was much weaker in Farsi speakers than Dutch speakers. This suggests that cross-linguistic differences shape the conceptualization of pitch and further calls into question the universality of height-pitch associations.
Collapse
Affiliation(s)
- Judith Holler
- Donders Institute for Brain, Cognition & Behaviors, Radboud University.,Language & Cognition and Neurobiology of Language Departments, Max Planck Institute for Psycholinguistics
| | - Linda Drijvers
- Donders Institute for Brain, Cognition & Behaviors, Radboud University.,Language & Cognition and Neurobiology of Language Departments, Max Planck Institute for Psycholinguistics
| | | | - Asifa Majid
- Canter for Language Studies, Radboud University.,Department of Psychology, University of York
| |
Collapse
|
16
|
Speed LJ, Croijmans I, Dolscheid S, Majid A. Crossmodal Associations with Olfactory, Auditory, and Tactile Stimuli in Children and Adults. Iperception 2021; 12:20416695211048513. [PMID: 34900211 PMCID: PMC8652194 DOI: 10.1177/20416695211048513] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2020] [Accepted: 09/06/2021] [Indexed: 11/16/2022] Open
Abstract
People associate information with different senses but the mechanism by which this happens is unclear. Such associations are thought to arise from innate structural associations in the brain, statistical associations in the environment, via shared affective content, or through language. A developmental perspective on crossmodal associations can help determine which explanations are more likely for specific associations. Certain associations with pitch (e.g., pitch-height) have been observed early in infancy, but others may only occur late into childhood (e.g., pitch-size). In contrast, tactile-chroma associations have been observed in children, but not adults. One modality that has received little attention developmentally is olfaction. In the present investigation, we explored crossmodal associations from sound, tactile stimuli, and odor to a range of stimuli by testing a broad range of participants. Across the three modalities, we found little evidence for crossmodal associations in young children. This suggests an account based on innate structures is unlikely. Instead, the number and strength of associations increased over the lifespan. This suggests that experience plays a crucial role in crossmodal associations from sound, touch, and smell to other senses.
Collapse
Affiliation(s)
- Laura J Speed
- Centre for Language Studies, Radboud University, Nijmegen, The Netherlands
| | - Ilja Croijmans
- Faculty of Social and Behavioral Sciences, Utrecht University, Utrecht, The Netherlands
| | - Sarah Dolscheid
- Department of Rehabilitation and Special Education, University of Cologne, Cologne, Germany; Biopsychology & Cognitive Neuroscience, Faculty of Psychology & Sports Science, Bielefeld University, Bielefeld, Germany
| | - Asifa Majid
- Department of Psychology, University of York, York, UK
| |
Collapse
|
17
|
Cuturi LF, Cappagli G, Tonelli A, Cocchi E, Gori M. Perceiving size through sound in sighted and visually impaired children. COGNITIVE DEVELOPMENT 2021. [DOI: 10.1016/j.cogdev.2021.101125] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2022]
|
18
|
Pitt B, Ferrigno S, Cantlon JF, Casasanto D, Gibson E, Piantadosi ST. Spatial concepts of number, size, and time in an indigenous culture. SCIENCE ADVANCES 2021; 7:eabg4141. [PMID: 34380617 PMCID: PMC8357228 DOI: 10.1126/sciadv.abg4141] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/04/2021] [Accepted: 06/28/2021] [Indexed: 05/27/2023]
Abstract
In industrialized groups, adults implicitly map numbers, time, and size onto space according to cultural practices like reading and counting (e.g., from left to right). Here, we tested the mental mappings of the Tsimane', an indigenous population with few such cultural practices. Tsimane' adults spatially arranged number, size, and time stimuli according to their relative magnitudes but showed no directional bias for any domain on any spatial axis; different mappings went in different directions, even in the same participant. These findings challenge claims that people have an innate left-to-right mapping of numbers and that these mappings arise from a domain-general magnitude system. Rather, the direction-specific mappings found in industrialized cultures may originate from direction-agnostic mappings that reflect the correlational structure of the natural world.
Collapse
Affiliation(s)
- Benjamin Pitt
- Department of Psychology, UC Berkeley, Berkeley, CA, USA.
| | - Stephen Ferrigno
- Department of Psychology, Harvard University, Cambridge, MA, USA
| | - Jessica F Cantlon
- Department of Psychology, Carnegie Mellon University, Pittsburgh, PA, USA
| | - Daniel Casasanto
- Department of Human Development, Cornell University, Ithaca, NY, USA
| | - Edward Gibson
- Department of Brain and Cognitive Sciences, MIT, Cambridge, MA, USA
| | | |
Collapse
|
19
|
Rogenmoser L, Arnicane A, Jäncke L, Elmer S. The left dorsal stream causally mediates the tone labeling in absolute pitch. Ann N Y Acad Sci 2021; 1500:122-133. [PMID: 34046902 PMCID: PMC8518498 DOI: 10.1111/nyas.14616] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2021] [Revised: 05/03/2021] [Accepted: 05/07/2021] [Indexed: 11/29/2022]
Abstract
Absolute pitch (AP) refers to the ability to effortlessly identify given pitches without any reference. Correlative evidence suggests that the left posterior dorsolateral prefrontal cortex (DLPFC) is responsible for the process underlying pitch labeling in AP. Here, we measured the sight‐reading performance of right‐handed AP possessors and matched controls under cathodal and sham transcranial direct current stimulation of the left DLPFC. The participants were instructed to report notations as accurately and as fast as possible by playing with their right hand on a piano. The notations were simultaneously presented with distracting auditory stimuli that either matched or mismatched them in different semitone degrees. Unlike the controls, AP possessors revealed an interference effect in that they responded slower in mismatching conditions than in the matching one. Under cathodal stimulation, this interference effect disappeared. These findings confirm that the pitch‐labeling process underlying AP occurs automatically and is largely nonsuppressible when triggered by tone exposure. The improvement of the AP possessors’ sight‐reading performances in response to the suppression of the left DLPFC using cathodal stimulation confirms a causal relationship between this brain structure and pitch labeling.
Collapse
Affiliation(s)
- Lars Rogenmoser
- Department of Medicine, University of Fribourg, Fribourg, Switzerland
| | - Andra Arnicane
- Auditory Research Group Zurich (ARGZ), Division of Neuropsychology, Institute of Psychology, University of Zurich, Zurich, Switzerland
| | - Lutz Jäncke
- Auditory Research Group Zurich (ARGZ), Division of Neuropsychology, Institute of Psychology, University of Zurich, Zurich, Switzerland.,University Research Priority Program (URPP), Dynamics of Healthy Aging, University of Zurich, Zurich, Switzerland
| | - Stefan Elmer
- Auditory Research Group Zurich (ARGZ), Division of Neuropsychology, Institute of Psychology, University of Zurich, Zurich, Switzerland
| |
Collapse
|
20
|
Effects of Musical Training, Timbre, and Response Orientation on the ROMPR Effect. JOURNAL OF COGNITIVE ENHANCEMENT 2021. [DOI: 10.1007/s41465-021-00213-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
|
21
|
Starr A, Cirolia AJ, Tillman KA, Srinivasan M. Spatial Metaphor Facilitates Word Learning. Child Dev 2020; 92:e329-e342. [PMID: 33355926 DOI: 10.1111/cdev.13477] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
Abstract
Why are spatial metaphors, like the use of "high" to describe a musical pitch, so common? This study tested one hundred and fifty-four 3- to 5-year-old English-learning children on their ability to learn a novel adjective in the domain of space or pitch and to extend this adjective to the untrained dimension. Children were more proficient at learning the word when it described a spatial attribute compared to pitch. However, once children learned the word, they extended it to the untrained dimension without feedback. Thus, children leveraged preexisting associations between space and pitch to spontaneously understand new metaphors. These results suggest that spatial metaphors may be common across languages in part because they scaffold children's acquisition of word meanings that are otherwise difficult to learn.
Collapse
|
22
|
Valenzuela J, Alcaraz Carrión D. Temporal Expressions in English and Spanish: Influence of Typology and Metaphorical Construal. Front Psychol 2020; 11:543933. [PMID: 33192788 PMCID: PMC7607253 DOI: 10.3389/fpsyg.2020.543933] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/18/2020] [Accepted: 09/15/2020] [Indexed: 11/24/2022] Open
Abstract
This study investigates how typological and metaphorical construal differences may affect the use and frequency of temporal expressions in English and Spanish. More precisely, we explore whether there are any differences between English, a satellite-framed language, and Spanish, a verb-framed language, in the use of certain temporal linguistic expressions that include a spatial, deictic component (Deictic Time), a purely temporal relation between two events (Sequential Time) or the expression of the duration of an event (Duration). To achieve this, we perform two different types of studies. First, we conduct an informational gain or loss analysis of 1,650 of English-to-Spanish translations extracted from parallel corpora. Secondly, we compare the frequency of 33 English and 27 Spanish temporal expressions in two similar written online corpora (EnTenTen and EsTenTen, respectively) and a television news spoken corpus (NewsScape). Our results suggest that English uses “deictic expressions with directional language” (explicitly stating the spatial location of the temporal event, e.g., back in those days/in the future ahead) much more frequently than Spanish, to the extent that such directional information is often excluded in English-to-Spanish translations. Also, sequential expressions (such as before that/later than) and duration expressions (during the whole day) are much more frequent in Spanish. These usage differences, explained by the variability in motion typology and metaphoric construal, open up the interesting question of how these differences in linguistic usage could affect the conceptualization of time of English and Spanish speakers.
Collapse
|
23
|
Klapman SF, Munn JT, Wilbiks JMP. Response orientation modulates pitch-space relationships: the ROMPR effect. PSYCHOLOGICAL RESEARCH 2020; 85:2197-2212. [PMID: 32729056 DOI: 10.1007/s00426-020-01388-z] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/18/2019] [Accepted: 07/09/2020] [Indexed: 10/23/2022]
Abstract
Congruency between auditory and visuospatial stimuli has been previously shown to affect responses to multisensory stimulus pairs, and congruency between stimuli and response devices may play a role in response speed and accuracy. Across two experiments, we tested whether the accuracy and speed of pitch judgments were affected by a congruent or incongruent paired visual stimulus, and whether the relationship was modulated by response orientation. In Experiment 1, participants using a vertically (transversely) oriented keyboard demonstrated a large crossmodal vertical effect, but a minimal crossmodal horizontal effect. In contrast, Experiment 2 used a horizontally oriented keyboard, while also examining whether musical training impacts pitch judgments. As in the first experiment, we found an effect of response mapping on pitch judgments; these results suggest that vertical visual stimuli are processed automatically, while the effects of horizontal visual stimuli are decisional and require a compatible response orientation. Based on these findings, we propose an effect we call the ROMPR effect: response orientation modulates pitch-space relationships. Unexpectedly, non-musicians demonstrated significant ROMPR effects while trained musicians did not. We suggest that non-musicians are more likely to use visual information when making spatial location judgments of pitch: unlike musicians, they have not been trained to rely exclusively on auditory information during pitch processing. We also discuss alternative explanations of the data: namely, that there is a need to disambiguate audiovisual congruency from visual-response congruency with modulations of experimental design.
Collapse
Affiliation(s)
- Sarah F Klapman
- Department of Neuroscience, Western University, London, ON, Canada.,Department of Psychology, Mount Allison University, Sackville, NB, Canada
| | - Jordan T Munn
- Department of Psychology, Mount Allison University, Sackville, NB, Canada
| | - Jonathan M P Wilbiks
- Department of Psychology, University of New Brunswick, 100 Tucker Park Road, P.O. Box 5050, Saint John, NB, E2L 4L5, Canada. .,Department of Psychology, Mount Allison University, Sackville, NB, Canada.
| |
Collapse
|
24
|
Dolscheid S, Çelik S, Erkan H, Küntay A, Majid A. Space-pitch associations differ in their susceptibility to language. Cognition 2019; 196:104073. [PMID: 31810048 DOI: 10.1016/j.cognition.2019.104073] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/10/2018] [Revised: 09/11/2019] [Accepted: 09/12/2019] [Indexed: 12/01/2022]
Abstract
To what extent are links between musical pitch and space universal, and to what extent are they shaped by language? There is contradictory evidence in support of both universality and linguistic relativity presently, leaving the question open. To address this, speakers of Dutch who talk about pitch in terms of spatial height and speakers of Turkish who use a thickness metaphor were tested in simple nonlinguistic space-pitch association tasks. Both groups showed evidence of a thickness-pitch association, but differed significantly in their height-pitch associations, suggesting the latter may be more susceptible to language. When participants had to match pitches to spatial stimuli where height and thickness were opposed (i.e., a thick line high in space vs. a thin line low in space), Dutch and Turkish differed in their relative preferences. Whereas Turkish participants predominantly opted for a thickness-pitch interpretation-even if this meant a reversal of height-pitch mappings-Dutch participants favored a height-pitch interpretation more often. These findings provide new evidence that speakers of different languages vary in their space-pitch associations, while at the same time showing such associations are not equally susceptible to linguistic influences. Some space-pitch (i.e., height-pitch) associations are more malleable than others (i.e., thickness-pitch).
Collapse
Affiliation(s)
| | | | - Hasan Erkan
- Radboud University, Nijmegen, the Netherlands
| | | | - Asifa Majid
- Radboud University, Nijmegen, the Netherlands; Max Planck Institute for Psycholinguistics, Nijmegen, the Netherlands; University of York, Heslington, UK.
| |
Collapse
|
25
|
Evans KK. The Role of Selective Attention in Cross-modal Interactions between Auditory and Visual Features. Cognition 2019; 196:104119. [PMID: 31751823 DOI: 10.1016/j.cognition.2019.104119] [Citation(s) in RCA: 12] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/02/2016] [Revised: 10/02/2019] [Accepted: 10/25/2019] [Indexed: 11/24/2022]
Abstract
Evans and Treisman (2010) showed systematic interactions between audition and vision when participants made speeded classifications in one modality while supposedly ignoring another. We found perceptual facilitation between high pitch and high visual position, high spatial frequency and small size, and interference between high pitch and low position, low spatial frequency and large size, while the converse was the case between low pitch and the same visual features. The present study examined the role of selective attention in these cross-modal interactions. Participants performed speeded classification or search tasks of low or high load while attempting to ignore irrelevant stimuli in a different modality. In both paradigms, congruency between the visual and the irrelevant auditory stimulus had an equal effect in the low and in the high perceptual load conditions. A third experiment tested divided attention, requiring participants to compare stimuli across modalities and respond to the visual-auditory compound. The congruency effect was as large with attention focused on one modality as when it was divided across both. These findings offer converging evidence that cross-modal interactions between corresponding basic features are independent of selective attention.
Collapse
Affiliation(s)
- Karla K Evans
- University of York, Department of Psychology, Heslington, York, YO10 5DD, United Kingdom.
| |
Collapse
|
26
|
Korzeniowska AT, Root-Gutteridge H, Simner J, Reby D. Audio-visual crossmodal correspondences in domestic dogs ( Canis familiaris). Biol Lett 2019; 15:20190564. [PMID: 31718513 DOI: 10.1098/rsbl.2019.0564] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022] Open
Abstract
Crossmodal correspondences are intuitively held relationships between non-redundant features of a stimulus, such as auditory pitch and visual illumination. While a number of correspondences have been identified in humans to date (e.g. high pitch is intuitively felt to be luminant, angular and elevated in space), their evolutionary and developmental origins remain unclear. Here, we investigated the existence of audio-visual crossmodal correspondences in domestic dogs, and specifically, the known human correspondence in which high auditory pitch is associated with elevated spatial position. In an audio-visual attention task, we found that dogs engaged more with audio-visual stimuli that were congruent with human intuitions (high auditory pitch paired with a spatially elevated visual stimulus) compared to incongruent (low pitch paired with elevated visual stimulus). This result suggests that crossmodal correspondences are not a uniquely human or primate phenomenon and they cannot easily be dismissed as merely lexical conventions (i.e. matching 'high' pitch with 'high' elevation).
Collapse
Affiliation(s)
- A T Korzeniowska
- Mammal Vocal Communication and Cognition Research Group, MULTISENSE Lab, School of Psychology, University of Sussex, Falmer, Brighton BN1 9QH, UK
| | - H Root-Gutteridge
- Mammal Vocal Communication and Cognition Research Group, MULTISENSE Lab, School of Psychology, University of Sussex, Falmer, Brighton BN1 9QH, UK
| | - J Simner
- Mammal Vocal Communication and Cognition Research Group, MULTISENSE Lab, School of Psychology, University of Sussex, Falmer, Brighton BN1 9QH, UK
| | - D Reby
- Mammal Vocal Communication and Cognition Research Group, MULTISENSE Lab, School of Psychology, University of Sussex, Falmer, Brighton BN1 9QH, UK.,Equipe Neuro-Ethologie Sensorielle, ENES/CRNL, CNRS UMR5292, INSERM UMR_S 1028, University of Lyon/Saint-Etienne, France
| |
Collapse
|
27
|
Dove G. Language as a disruptive technology: abstract concepts, embodiment and the flexible mind. Philos Trans R Soc Lond B Biol Sci 2019; 373:rstb.2017.0135. [PMID: 29915003 DOI: 10.1098/rstb.2017.0135] [Citation(s) in RCA: 37] [Impact Index Per Article: 7.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 12/12/2017] [Indexed: 11/12/2022] Open
Abstract
A growing body of evidence suggests that cognition is embodied and grounded. Abstract concepts, though, remain a significant theoretical challenge. A number of researchers have proposed that language makes an important contribution to our capacity to acquire and employ concepts, particularly abstract ones. In this essay, I critically examine this suggestion and ultimately defend a version of it. I argue that a successful account of how language augments cognition should emphasize its symbolic properties and incorporate a view of embodiment that recognizes the flexible, multimodal and task-related nature of action, emotion and perception systems. On this view, language is an ontogenetically disruptive cognitive technology that expands our conceptual reach.This article is part of the theme issue 'Varieties of abstract concepts: development, use and representation in the brain'.
Collapse
Affiliation(s)
- Guy Dove
- Department of Philosophy, University of Louisville, Louisville, KY, USA
| |
Collapse
|
28
|
Burzynska J, Wang QJ, Spence C, Bastian SEP. Taste the Bass: Low Frequencies Increase the Perception of Body and Aromatic Intensity in Red Wine. Multisens Res 2019; 32:429-454. [PMID: 31117049 DOI: 10.1163/22134808-20191406] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2019] [Accepted: 04/09/2019] [Indexed: 11/19/2022]
Abstract
Associations between heaviness and bass/low-pitched sounds reverberate throughout music, philosophy, literature, and language. Given that recent research into the field of cross-modal correspondences has revealed a number of robust relationships between sound and flavour, this exploratory study was designed to investigate the effects of lower frequency sound (10 Hz to 200 Hz) on the perception of the mouthfeel character of palate weight/body. This is supported by an overview of relevant cross-modal studies and cultural production. Wines were the tastants - a New Zealand Pinot Noir and a Spanish Garnacha - which were tasted in silence and with a 100 Hz (bass) and a higher 1000 Hz sine wave tone. Aromatic intensity was included as an additional character given suggestions that pitch may influence the perception of aromas, which might presumably affect the perception of wine body. Intensity of acidity and liking were also evaluated. The results revealed that the Pinot Noir wine was rated as significantly fuller-bodied when tasted with a bass frequency than in silence or with a higher frequency sound. The low frequency stimulus also resulted in the Garnacha wine being rated as significantly more aromatically intense than when tasted in the presence of the higher frequency auditory stimulus. Acidity was rated considerably higher with the higher frequency in both wines by those with high wine familiarity and the Pinot Noir significantly better liked than the Garnacha. Possible reasons as to why the tones used in this study affected perception of the two wines differently are discussed. Practical application of the findings are also proposed.
Collapse
Affiliation(s)
| | - Qian Janice Wang
- 2Department of Food Science, Aarhus University, Årslev, Denmark.,3Crossmodal Research Laboratory, Oxford University, UK
| | | | | |
Collapse
|
29
|
Spence C. On the Relative Nature of (Pitch-Based) Crossmodal Correspondences. Multisens Res 2019; 32:235-265. [DOI: 10.1163/22134808-20191407] [Citation(s) in RCA: 30] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/12/2019] [Accepted: 02/21/2019] [Indexed: 11/19/2022]
Abstract
Abstract
This review deals with the question of the relative vs absolute nature of crossmodal correspondences, with a specific focus on those correspondences involving the auditory dimension of pitch. Crossmodal correspondences have been defined as the often-surprising crossmodal associations that people experience between features, attributes, or dimensions of experience in different sensory modalities, when either physically present, or else merely imagined. In the literature, crossmodal correspondences have often been contrasted with synaesthesia in that the former are frequently said to be relative phenomena (e.g., it is the higher-pitched of two sounds that is matched with the smaller of two visual stimuli, say, rather than there being a specific one-to-one crossmodal mapping between a particular pitch of sound and size of object). By contrast, in the case of synaesthesia, the idiosyncratic mapping between inducer and concurrent tends to be absolute (e.g., it is a particular sonic inducer that elicits a specific colour concurrent). However, a closer analysis of the literature soon reveals that the distinction between relative and absolute in the case of crossmodal correspondences may not be as clear-cut as some commentators would have us believe. Furthermore, it is important to note that the relative vs absolute question may receive different answers depending on the particular (class of) correspondence under empirical investigation.
Collapse
Affiliation(s)
- Charles Spence
- Crossmodal Research Laboratory, Oxford University, Oxford, UK
| |
Collapse
|
30
|
Starr A, Srinivasan M. Spatial metaphor and the development of cross-domain mappings in early childhood. Dev Psychol 2018; 54:1822-1832. [PMID: 30234336 DOI: 10.1037/dev0000573] [Citation(s) in RCA: 10] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
Spatial language is often used metaphorically to describe other domains, including time (long sound) and pitch (high sound). How does experience with these metaphors shape the ability to associate space with other domains? Here, we tested 3- to 6-year-old English-speaking children and adults with a cross-domain matching task. We probed cross-domain relations that are expressed in English metaphors for time and pitch (length-time and height-pitch) as well as relations that are unconventional in English but expressed in other languages (size-time and thickness-pitch). Participants were tested with a perceptual matching task, in which they matched between spatial stimuli and sounds of different durations or pitches, and a linguistic matching task, in which they matched between a label denoting a spatial attribute, duration, or pitch and a picture or sound representing another dimension. Contrary to previous claims that experience with linguistic metaphors is necessary for children to make cross-domain mappings, children performed above chance for both familiar and unfamiliar relations in both tasks, as did adults. Children's performance was also better when a label was provided for one of the dimensions, but only when making length-time, size-time, and height-pitch mappings (not thickness-pitch mappings). These findings suggest that although experience with metaphorical language is not necessary to make cross-domain mappings, labels can promote these mappings, both when they have familiar metaphorical uses (e.g., English long denotes both length and duration) and when they describe dimensions that share a common ordinal reference frame (e.g., size and duration but not thickness and pitch). (PsycINFO Database Record
Collapse
Affiliation(s)
- Ariel Starr
- Department of Psychology, University of California, Berkeley
| | | |
Collapse
|
31
|
Tham DSY, Rees A, Bremner JG, Slater A, Johnson S. Auditory information for spatial location and pitch-height correspondence support young infants' perception of object persistence. J Exp Child Psychol 2018; 178:341-351. [PMID: 30139621 DOI: 10.1016/j.jecp.2018.05.017] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/06/2018] [Revised: 04/25/2018] [Accepted: 05/22/2018] [Indexed: 10/28/2022]
Abstract
Perception of object persistence across occlusion emerges at around 4 months of age for objects moving horizontally or vertically. In addition, congruent auditory information for movement enhances perception of persistence of an object moving horizontally. In two experiments, we examined the effect of presenting bimodal (visual and auditory) sensory information, both congruently and incongruently, for a vertical moving object occlusion event. A total of 68 4-month-old infants (34 girls) were tested for perception of persistence of an object moving up and down, passing at each translation behind a centrally placed occluder. Infants were exposed to these visual events accompanied by no sound, spatially colocated sound, or congruent or incongruent pitch-height correspondence sounds. Both spatially colocated and congruent pitch-height auditory information enhanced perception of trajectory continuity. However, no impairment occurred when pitch-height sound information was presented incongruently. These results highlight the importance of taking a multisensory approach to infant perceptual development.
Collapse
Affiliation(s)
| | | | | | | | - Scott Johnson
- University of California, Los Angeles (UCLA), Los Angeles, CA 90095, USA
| |
Collapse
|
32
|
Schmitz L, Vesper C, Sebanz N, Knoblich G. When Height Carries Weight: Communicating Hidden Object Properties for Joint Action. Cogn Sci 2018; 42:2021-2059. [PMID: 29936705 PMCID: PMC6120543 DOI: 10.1111/cogs.12638] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/11/2017] [Revised: 05/19/2019] [Accepted: 05/23/2018] [Indexed: 11/29/2022]
Abstract
In the absence of pre-established communicative conventions, people create novel communication systems to successfully coordinate their actions toward a joint goal. In this study, we address two types of such novel communication systems: sensorimotor communication, where the kinematics of instrumental actions are systematically modulated, versus symbolic communication. We ask which of the two systems co-actors preferentially create when aiming to communicate about hidden object properties such as weight. The results of three experiments consistently show that actors who knew the weight of an object transmitted this weight information to their uninformed co-actors by systematically modulating their instrumental actions, grasping objects of particular weights at particular heights. This preference for sensorimotor communication was reduced in a fourth experiment where co-actors could communicate with weight-related symbols. Our findings demonstrate that the use of sensorimotor communication extends beyond the communication of spatial locations to non-spatial, hidden object properties.
Collapse
Affiliation(s)
- Laura Schmitz
- Department of Cognitive ScienceCentral European University
| | - Cordula Vesper
- Department of Cognitive ScienceCentral European University
- School of Communication and CultureAarhus University
| | - Natalie Sebanz
- Department of Cognitive ScienceCentral European University
| | | |
Collapse
|
33
|
The Spatial Musical Association of Response Codes does not depend on a normal visual experience: A study with early blind individuals. Atten Percept Psychophys 2018; 80:813-821. [DOI: 10.3758/s13414-018-1495-x] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
|
34
|
McCormick K, Lacey S, Stilla R, Nygaard LC, Sathian K. Neural basis of the crossmodal correspondence between auditory pitch and visuospatial elevation. Neuropsychologia 2018; 112:19-30. [PMID: 29501792 DOI: 10.1016/j.neuropsychologia.2018.02.029] [Citation(s) in RCA: 19] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/25/2017] [Revised: 02/22/2018] [Accepted: 02/26/2018] [Indexed: 01/09/2023]
Abstract
Crossmodal correspondences refer to associations between otherwise unrelated stimulus features in different sensory modalities. For example, high and low auditory pitches are associated with high and low visuospatial elevation, respectively. The neural mechanisms underlying crossmodal correspondences are currently unknown. Here, we used functional magnetic resonance imaging (fMRI) to investigate the neural basis of the pitch-elevation correspondence. Pitch-elevation congruency effects were observed bilaterally in the inferior frontal and insular cortex, the right frontal eye field and right inferior parietal cortex. Independent functional localizers failed to provide strong evidence for any of three proposed mechanisms for crossmodal correspondences: semantic mediation, magnitude estimation, and multisensory integration. Instead, pitch-elevation congruency effects overlapped with areas selective for visually presented non-word strings relative to sentences, and with regions sensitive to audiovisual asynchrony. Taken together with the prior literature, the observed congruency effects are most consistent with mediation by multisensory attention.
Collapse
Affiliation(s)
- Kelly McCormick
- Depart ment of Neurology, Emory University, Atlanta, GA 30322, USA; Department of Psychology, Emory University, Atlanta, GA 30322, USA
| | - Simon Lacey
- Depart ment of Neurology, Emory University, Atlanta, GA 30322, USA
| | - Randall Stilla
- Depart ment of Neurology, Emory University, Atlanta, GA 30322, USA
| | - Lynne C Nygaard
- Department of Psychology, Emory University, Atlanta, GA 30322, USA
| | - K Sathian
- Depart ment of Neurology, Emory University, Atlanta, GA 30322, USA; Department of Rehabilitation Medicine, Emory University, Atlanta, GA 30322, USA; Department of Psychology, Emory University, Atlanta, GA 30322, USA; Center for Visual and Neurocognitive Rehabilitation, Atlanta VAMC, Decatur, GA 30033, USA.
| |
Collapse
|
35
|
Goubet N, Durand K, Schaal B, McCall DD. Seeing odors in color: Cross-modal associations in children and adults from two cultural environments. J Exp Child Psychol 2018; 166:380-399. [DOI: 10.1016/j.jecp.2017.09.007] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/17/2017] [Revised: 09/07/2017] [Accepted: 09/07/2017] [Indexed: 11/24/2022]
|
36
|
Nava E, Grassi M, Brenna V, Croci E, Turati C. Multisensory Motion Perception in 3-4 Month-Old Infants. Front Psychol 2017; 8:1994. [PMID: 29187829 PMCID: PMC5694769 DOI: 10.3389/fpsyg.2017.01994] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/20/2017] [Accepted: 10/31/2017] [Indexed: 11/13/2022] Open
Abstract
Human infants begin very early in life to take advantage of multisensory information by extracting the invariant amodal information that is conveyed redundantly by multiple senses. Here we addressed the question as to whether infants can bind multisensory moving stimuli, and whether this occurs even if the motion produced by the stimuli is only illusory. Three- to 4-month-old infants were presented with two bimodal pairings: visuo-tactile and audio-visual. Visuo-tactile pairings consisted of apparently vertically moving bars (the Barber Pole illusion) moving in either the same or opposite direction with a concurrent tactile stimulus consisting of strokes given on the infant's back. Audio-visual pairings consisted of the Barber Pole illusion in its visual and auditory version, the latter giving the impression of a continuous rising or ascending pitch. We found that infants were able to discriminate congruently (same direction) vs. incongruently moving (opposite direction) pairs irrespective of modality (Experiment 1). Importantly, we also found that congruently moving visuo-tactile and audio-visual stimuli were preferred over incongruently moving bimodal stimuli (Experiment 2). Our findings suggest that very young infants are able to extract motion as amodal component and use it to match stimuli that only apparently move in the same direction.
Collapse
Affiliation(s)
- Elena Nava
- Department of Psychology, Università degli Studi di Milano Bicocca, Milan, Italy
| | - Massimo Grassi
- Department of Psychology, University of Padua, Padua, Italy
| | - Viola Brenna
- Department of Psychology, Università degli Studi di Milano Bicocca, Milan, Italy
| | - Emanuela Croci
- Department of Psychology, Università degli Studi di Milano Bicocca, Milan, Italy
| | - Chiara Turati
- Department of Psychology, Università degli Studi di Milano Bicocca, Milan, Italy
| |
Collapse
|
37
|
Turoman N, Styles SJ. Glyph guessing for 'oo' and 'ee': spatial frequency information in sound symbolic matching for ancient and unfamiliar scripts. ROYAL SOCIETY OPEN SCIENCE 2017; 4:170882. [PMID: 28989784 PMCID: PMC5627124 DOI: 10.1098/rsos.170882] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 07/11/2017] [Accepted: 08/14/2017] [Indexed: 06/07/2023]
Abstract
In three experiments, we asked whether diverse scripts contain interpretable information about the speech sounds they represent. When presented with a pair of unfamiliar letters, adult readers correctly guess which is /i/ (the 'ee' sound in 'feet'), and which is /u/ (the 'oo' sound in 'shoe') at rates higher than expected by chance, as shown in a large sample of Singaporean university students (Experiment 1) and replicated in a larger sample of international Internet users (Experiment 2). To uncover what properties of the letters contribute to different scripts' 'guessability,' we analysed the visual spatial frequencies in each letter (Experiment 3). We predicted that the lower spectral frequencies in the formants of the vowel /u/ would pattern with lower spatial frequencies in the corresponding letters. Instead, we found that across all spatial frequencies, the letter with more black/white cycles (i.e. more ink) was more likely to be guessed as /u/, and the larger the difference between the glyphs in a pair, the higher the script's guessability. We propose that diverse groups of humans across historical time and geographical space tend to employ similar iconic strategies for representing speech in visual form, and provide norms for letter pairs from 56 diverse scripts.
Collapse
Affiliation(s)
- Nora Turoman
- Department of Radiology, Centre Hospitalier Universitaire Vaudois (CHUV) and University of Lausanne (UniL), Rue du Bugnon 46, 1011 Lausanne, Switzerland
- Department of Clinical Neurosciences, Centre Hospitalier Universitaire Vaudois (CHUV) and University of Lausanne (UniL), Rue du Bugnon 46, 1011 Lausanne, Switzerland
| | - Suzy J. Styles
- Division of Psychology, School of Humanities and Social Sciences, Nanyang Technological University, 14 Nanyang Drive, Singapore
| |
Collapse
|
38
|
Fernandez-Prieto I, Spence C, Pons F, Navarra J. Does Language Influence the Vertical Representation of Auditory Pitch and Loudness? Iperception 2017; 8:2041669517716183. [PMID: 28694959 PMCID: PMC5484432 DOI: 10.1177/2041669517716183] [Citation(s) in RCA: 11] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022] Open
Abstract
Higher frequency and louder sounds are associated with higher positions whereas lower frequency and quieter sounds are associated with lower locations. In English, "high" and "low" are used to label pitch, loudness, and spatial verticality. By contrast, different words are preferentially used, in Catalan and Spanish, for pitch (high: "agut/agudo"; low: "greu/grave") and for loudness/verticality (high: "alt/alto"; low: "baix/bajo"). Thus, English and Catalan/Spanish differ in the spatial connotations for pitch. To analyze the influence of language on these crossmodal associations, a task was conducted in which English and Spanish/Catalan speakers had to judge whether a tone was higher or lower (in pitch or loudness) than a reference tone. The response buttons were located at crossmodally congruent or incongruent positions with respect to the probe tone. Crossmodal correspondences were evidenced in both language groups. However, English speakers showed greater effects for pitch, suggesting an influence of linguistic background.
Collapse
Affiliation(s)
- Irune Fernandez-Prieto
- Fundació Sant Joan de Déu, Parc Sanitari de Sant Joan de Déu, Esplugues de Llobregat (Barcelona), Spain; Crossmodal Research Laboratory, Department of Experimental Psychology, University of Oxford, Oxford, UK; Institute of Neurosciences, University of Barcelona, Barcelona, Spain
| | - Charles Spence
- Crossmodal Research Laboratory, Department of Experimental Psychology, University of Oxford, Oxford, UK
| | - Ferran Pons
- Department of Cognition, Development and Educational Psychology, University of Barcelona, Barcelona, Spain; Institute of Neurosciences, University of Barcelona, Barcelona, Spain; Institut de Recerca Pediàtrica, Hospital Sant Joan de Déu, Barcelona, Spain
| | - Jordi Navarra
- Department of Cognition, Development and Educational Psychology, University of Barcelona, Barcelona, Spain; Fundació Sant Joan de Déu, Parc Sanitari de Sant Joan de Déu, Esplugues de Llobregat (Barcelona), Spain
| |
Collapse
|
39
|
Smith NA, Folland NA, Martinez DM, Trainor LJ. Multisensory object perception in infancy: 4-month-olds perceive a mistuned harmonic as a separate auditory and visual object. Cognition 2017; 164:1-7. [PMID: 28346869 PMCID: PMC5429982 DOI: 10.1016/j.cognition.2017.01.016] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2015] [Revised: 01/17/2017] [Accepted: 01/24/2017] [Indexed: 10/19/2022]
Abstract
Infants learn to use auditory and visual information to organize the sensory world into identifiable objects with particular locations. Here we use a behavioural method to examine infants' use of harmonicity cues to auditory object perception in a multisensory context. Sounds emitted by different objects sum in the air and the auditory system must figure out which parts of the complex waveform belong to different sources (auditory objects). One important cue to this source separation is that complex tones with pitch typically contain a fundamental frequency and harmonics at integer multiples of the fundamental. Consequently, adults hear a mistuned harmonic in a complex sound as a distinct auditory object (Alain, Theunissen, Chevalier, Batty, & Taylor, 2003). Previous work by our group demonstrated that 4-month-old infants are also sensitive to this cue. They behaviourally discriminate a complex tone with a mistuned harmonic from the same complex with in-tune harmonics, and show an object-related event-related potential (ERP) electrophysiological (EEG) response to the stimulus with mistuned harmonics. In the present study we use an audiovisual procedure to investigate whether infants perceive a complex tone with an 8% mistuned harmonic as emanating from two objects, rather than merely detecting the mistuned cue. We paired in-tune and mistuned complex tones with visual displays that contained either one or two bouncing balls. Four-month-old infants showed surprise at the incongruous pairings, looking longer at the display of two balls when paired with the in-tune complex and at the display of one ball when paired with the mistuned harmonic complex. We conclude that infants use harmonicity as a cue for source separation when integrating auditory and visual information in object perception.
Collapse
Affiliation(s)
- Nicholas A Smith
- Perceptual Development Laboratory, Boys Town National Research Hospital, 555 N. 30th Street, Omaha, NE 68131, United States
| | - Nicole A Folland
- Department of Psychology, Neuroscience and Behaviour, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada
| | - Diana M Martinez
- Department of Psychology, Neuroscience and Behaviour, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada
| | - Laurel J Trainor
- Department of Psychology, Neuroscience and Behaviour, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; McMaster Institute for Music and the Mind, McMaster University, 1280 Main Street West, Hamilton, Ontario L8S 4K1, Canada; Rotman Research Institute, Baycrest, University of Toronto, 3560 Bathurst Street, Toronto, Ontario M6A 2E1, Canada.
| |
Collapse
|
40
|
Pietraszewski D, Wertz AE, Bryant GA, Wynn K. Three-month-old human infants use vocal cues of body size. Proc Biol Sci 2017; 284:rspb.2017.0656. [PMID: 28592674 DOI: 10.1098/rspb.2017.0656] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/28/2017] [Accepted: 05/05/2017] [Indexed: 11/12/2022] Open
Abstract
Differences in vocal fundamental (F0) and average formant (Fn) frequencies covary with body size in most terrestrial mammals, such that larger organisms tend to produce lower frequency sounds than smaller organisms, both between species and also across different sex and life-stage morphs within species. Here we examined whether three-month-old human infants are sensitive to the relationship between body size and sound frequencies. Using a violation-of-expectation paradigm, we found that infants looked longer at stimuli inconsistent with the relationship-that is, a smaller organism producing lower frequency sounds, and a larger organism producing higher frequency sounds-than at stimuli that were consistent with it. This effect was stronger for fundamental frequency than it was for average formant frequency. These results suggest that by three months of age, human infants are already sensitive to the biologically relevant covariation between vocalization frequencies and visual cues to body size. This ability may be a consequence of developmental adaptations for building a phenotype capable of identifying and representing an organism's size, sex and life-stage.
Collapse
Affiliation(s)
- David Pietraszewski
- Center for Adaptive Rationality, Lentzeallee 94, 14195 Berlin, Germany .,Department of Psychology, Yale University, 2 Hillhouse Avenue, New Haven, CT 06520-8205, USA
| | - Annie E Wertz
- Max Planck Research Group Naturalistic Social Cognition, Lentzeallee 94, 14195 Berlin, Germany.,Department of Psychology, Yale University, 2 Hillhouse Avenue, New Haven, CT 06520-8205, USA
| | - Gregory A Bryant
- Department of Communication, Center for Behavior, Evolution, and Culture, University of California, Los Angeles, CA 90095, USA
| | - Karen Wynn
- Department of Psychology, Yale University, 2 Hillhouse Avenue, New Haven, CT 06520-8205, USA
| |
Collapse
|
41
|
Abstract
Crossmodal correspondences refer to the systematic associations often found across seemingly unrelated sensory features from different sensory modalities. Such phenomena constitute a universal trait of multisensory perception even in non-human species, and seem to result, at least in part, from the adaptation of sensory systems to natural scene statistics. Despite recent developments in the study of crossmodal correspondences, there are still a number of standing questions about their definition, their origins, their plasticity, and their underlying computational mechanisms. In this paper, I will review such questions in the light of current research on sensory cue integration, where crossmodal correspondences can be conceptualized in terms of natural mappings across different sensory cues that are present in the environment and learnt by the sensory systems. Finally, I will provide some practical guidelines for the design of experiments that might shed new light on crossmodal correspondences.
Collapse
|
42
|
Abstract
The renewed interest that has emerged around the topic of crossmodal correspondences in recent years has demonstrated that crossmodal matchings and mappings exist between the majority of sensory dimensions, and across all combinations of sensory modalities. This renewed interest also offers a rapidly-growing list of ways in which correspondences affect--or interact with--metaphorical understanding, feelings of 'knowing', behavioral tasks, learning, mental imagery, and perceptual experiences. Here we highlight why, more generally, crossmodal correspondences matter to theories of multisensory interactions.
Collapse
|
43
|
Nava E, Grassi M, Turati C. Audio-Visual, Visuo-Tactile and Audio-Tactile Correspondences in Preschoolers. Multisens Res 2017; 29:93-111. [PMID: 27311292 DOI: 10.1163/22134808-00002493] [Citation(s) in RCA: 25] [Impact Index Per Article: 3.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
Abstract
Interest in crossmodal correspondences has recently seen a renaissance thanks to numerous studies in human adults. Yet, still very little is known about crossmodal correspondences in children, particularly in sensory pairings other than audition and vision. In the current study, we investigated whether 4-5-year-old children match auditory pitch to the spatial motion of visual objects (audio-visual condition). In addition, we investigated whether this correspondence extends to touch, i.e., whether children also match auditory pitch to the spatial motion of touch (audio-tactile condition) and the spatial motion of visual objects to touch (visuo-tactile condition). In two experiments, two different groups of children were asked to indicate which of two stimuli fitted best with a centrally located third stimulus (Experiment 1), or to report whether two presented stimuli fitted together well (Experiment 2). We found sensitivity to the congruency of all of the sensory pairings only in Experiment 2, suggesting that only under specific circumstances can these correspondences be observed. Our results suggest that pitch-height correspondences for audio-visual and audio-tactile combinations may still be weak in preschool children, and speculate that this could be due to immature linguistic and auditory cues that are still developing at age five.
Collapse
|
44
|
Jamal Y, Lacey S, Nygaard L, Sathian K. Interactions Between Auditory Elevation, Auditory Pitch and Visual Elevation During Multisensory Perception. Multisens Res 2017; 30:287-306. [PMID: 31287081 DOI: 10.1163/22134808-00002553] [Citation(s) in RCA: 17] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2016] [Accepted: 02/07/2017] [Indexed: 11/19/2022]
Abstract
Cross-modal correspondences refer to associations between apparently unrelated stimulus features in different senses. For example, high and low auditory pitches are associated with high and low visual elevations, respectively. Here we examined how this crossmodal correspondence between visual elevation and auditory pitch relates to auditory elevation. We used audiovisual combinations of high- or low-frequency bursts of white noise and a visual stimulus comprising a white circle. Auditory and visual stimuli could each occur at high or low elevations. These multisensory stimuli could be congruent or incongruent for three correspondence types: cross-modal featural (auditory pitch/visual elevation), within-modal featural (auditory pitch/auditory elevation) and cross-modal spatial (auditory and visual elevation). Participants performed a 2AFC speeded classification (high or low) task while attending to auditory pitch, auditory elevation, or visual elevation. We tested for modulatory interactions between the three correspondence types. Modulatory interactions were absent when discriminating visual elevation. However, the within-modal featural correspondence affected the cross-modal featural correspondence during discrimination of auditory elevation and pitch, while the reverse modulation was observed only during discrimination of auditory pitch. The cross-modal spatial correspondence modulated the other two correspondences only when auditory elevation was being attended, was modulated by the cross-modal featural correspondence only during attention to auditory pitch, and was modulated by the within-modal featural correspondence while performing discrimination of either auditory elevation or pitch. We conclude that the cross-modal correspondence between auditory pitch and visual elevation interacts strongly with auditory elevation.
Collapse
Affiliation(s)
- Yaseen Jamal
- Department of Neurology, Emory University, Atlanta, GA 30322, USA
| | - Simon Lacey
- Department of Neurology, Emory University, Atlanta, GA 30322, USA
| | - Lynne Nygaard
- Department of Psychology, Emory University, Atlanta, GA 30322, USA
| | - K Sathian
- Department of Neurology, Emory University, Atlanta, GA 30322, USA.,Department of Psychology, Emory University, Atlanta, GA 30322, USA.,Department of Rehabilitation Medicine, Emory University, Atlanta, GA 30322, USA.,Center for Visual and Neurocognitive Rehabilitation, Atlanta VAMC, Decatur, GA 30033, USA
| |
Collapse
|
45
|
Abstract
Everyday language reveals how stimuli encoded in one sensory feature domain can possess qualities normally associated with a different domain (e.g., higher pitch sounds are bright, light in weight, sharp, and thin). Such cross-sensory associations appear to reflect crosstalk among aligned (corresponding) feature dimensions, including brightness, heaviness, and sharpness. Evidence for heaviness being one such dimension is very limited, with heaviness appearing primarily as a verbal associate of other feature contrasts (e.g., darker objects and lower pitch sounds are heavier than their opposites). Given the presumed bidirectionality of the crosstalk between corresponding dimensions, heaviness should itself induce the cross-sensory associations observed elsewhere, including with brightness and pitch. Taking care to dissociate effects arising from the size and mass of an object, this is confirmed. When hidden objects varying independently in size and mass are lifted, objects that feel heavier are judged to be darker and to make lower pitch sounds than objects feeling less heavy. These judgements track the changes in perceived heaviness induced by the size-weight illusion. The potential involvement of language, natural scene statistics, and Bayesian processes in correspondences, and the effects they induce, is considered.
Collapse
Affiliation(s)
- Peter Walker
- Department of Psychology, Lancaster University, UK; Department of Psychology, Sunway University, Malaysia
| | | | - Brian Francis
- Department of Mathematics and Statistics, Lancaster University, UK
| |
Collapse
|
46
|
Fernández-Prieto I, Caprile C, Tinoco-González D, Ristol-Orriols B, López-Sala A, Póo-Argüelles P, Pons F, Navarra J. Pitch perception deficits in nonverbal learning disability. RESEARCH IN DEVELOPMENTAL DISABILITIES 2016; 59:378-386. [PMID: 27710893 DOI: 10.1016/j.ridd.2016.09.011] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 02/01/2016] [Revised: 09/14/2016] [Accepted: 09/15/2016] [Indexed: 06/06/2023]
Abstract
The nonverbal learning disability (NLD) is a neurological dysfunction that affects cognitive functions predominantly related to the right hemisphere such as spatial and abstract reasoning. Previous evidence in healthy adults suggests that acoustic pitch (i.e., the relative difference in frequency between sounds) is, under certain conditions, encoded in specific areas of the right hemisphere that also encode the spatial elevation of external objects (e.g., high vs. low position). Taking this evidence into account, we explored the perception of pitch in preadolescents and adolescents with NLD and in a group of healthy participants matched by age, gender, musical knowledge and handedness. Participants performed four speeded tests: a stimulus detection test and three perceptual categorization tests based on colour, spatial position and pitch. Results revealed that both groups were equally fast at detecting visual targets and categorizing visual stimuli according to their colour. In contrast, the NLD group showed slower responses than the control group when categorizing space (direction of a visual object) and pitch (direction of a change in sound frequency). This pattern of results suggests the presence of a subtle deficit at judging pitch in NLD along with the traditionally-described difficulties in spatial processing.
Collapse
Affiliation(s)
- I Fernández-Prieto
- Fundació Sant Joan de Déu, Parc Sanitari de Sant Joan de Déu, Esplugues de Llobregat, Barcelona, Spain; Institute of Neurosciences, University of Barcelona, Barcelona, Spain.
| | - C Caprile
- Fundació Sant Joan de Déu, Parc Sanitari de Sant Joan de Déu, Esplugues de Llobregat, Barcelona, Spain
| | - D Tinoco-González
- Fundació Sant Joan de Déu, Parc Sanitari de Sant Joan de Déu, Esplugues de Llobregat, Barcelona, Spain
| | - B Ristol-Orriols
- Departments of Neuropediatrics, Hospital Sant Joan de Déu, Barcelona, Spain
| | - A López-Sala
- Departments of Neuropediatrics, Hospital Sant Joan de Déu, Barcelona, Spain
| | | | - F Pons
- Department of Cognition, Development and Educational Psychology, University of Barcelona, Barcelona, Spain; Institute of Neurosciences, University of Barcelona, Barcelona, Spain; Institut de Recerca Pediàtrica, Hospital Sant Joan de Déu, Barcelona, Spain
| | - J Navarra
- Fundació Sant Joan de Déu, Parc Sanitari de Sant Joan de Déu, Esplugues de Llobregat, Barcelona, Spain; Department of Cognition, Development and Educational Psychology, University of Barcelona, Barcelona, Spain
| |
Collapse
|
47
|
Wolter S, Dudschig C, Kaup B. Reading sentences describing high- or low-pitched auditory events: only pianists show evidence for a horizontal space-pitch association. PSYCHOLOGICAL RESEARCH 2016; 81:1213-1223. [PMID: 27734156 DOI: 10.1007/s00426-016-0812-z] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2016] [Accepted: 09/20/2016] [Indexed: 10/20/2022]
Abstract
This study explored differences between pianists and non-musicians during reading of sentences describing high- or low-pitched auditory events. Based on the embodied model of language comprehension, it was hypothesized that the experience of playing the piano encourages a corresponding association between high-pitched sounds and the right and low-pitched sounds and the left. This pitch-space association is assumed to become elicited during understanding of sentences describing either a high- or low-pitched auditory event. In this study, pianists and non-musicians were tested based on the hypothesis that only pianists show a compatibility effect between implied pitch height and horizontal space, because only pianists have the corresponding experience with the piano keyboard. Participants read pitch-related sentences (e.g., the bear growls deeply, the soprano singer sings an aria) and judged whether the sentence was sensible or not by pressing either a left or right response key. The results indicated that only the pianists showed the predicted compatibility effect between implied pitch height and response location. Based on the results, it can be inferred that the experience of playing the piano led to an association between horizontal space and pitch height in pianists, while no such spatial association was elicited in non-musicians.
Collapse
Affiliation(s)
- Sibylla Wolter
- Faculty of Science, Department of Psychology, University of Tübingen, Schleichstr. 4, 72076, Tübingen, Germany.
| | - Carolin Dudschig
- Faculty of Science, Department of Psychology, University of Tübingen, Schleichstr. 4, 72076, Tübingen, Germany
| | - Barbara Kaup
- Faculty of Science, Department of Psychology, University of Tübingen, Schleichstr. 4, 72076, Tübingen, Germany
| |
Collapse
|
48
|
Bender A, Beller S. Current Perspectives on Cognitive Diversity. Front Psychol 2016; 7:509. [PMID: 27148118 PMCID: PMC4828464 DOI: 10.3389/fpsyg.2016.00509] [Citation(s) in RCA: 49] [Impact Index Per Article: 6.1] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/10/2015] [Accepted: 03/24/2016] [Indexed: 01/29/2023] Open
Abstract
To what extent is cognition influenced by a person’s cultural background? This question has remained controversial in large fields of the cognitive sciences, including cognitive psychology, and is also underexplored in anthropology. In this perspective article, findings from a recent wave of cross-cultural studies will be outlined with respect to three aspects of cognition: perception and categorization, number representation and counting, and explanatory frameworks and beliefs. Identifying similarities and differences between these domains allows for general conclusions regarding cognitive diversity and helps to highlight the importance of culturally shaped content for a comprehensive understanding of cognition.
Collapse
Affiliation(s)
- Andrea Bender
- Department of Psychosocial Science, Faculty of Psychology, University of Bergen Bergen, Norway
| | - Sieghard Beller
- Department of Psychosocial Science, Faculty of Psychology, University of Bergen Bergen, Norway
| |
Collapse
|
49
|
Rabaglia CD, Maglio SJ, Krehm M, Seok JH, Trope Y. The sound of distance. Cognition 2016; 152:141-149. [PMID: 27062226 DOI: 10.1016/j.cognition.2016.04.001] [Citation(s) in RCA: 28] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2015] [Revised: 03/29/2016] [Accepted: 04/01/2016] [Indexed: 11/28/2022]
Abstract
Human languages may be more than completely arbitrary symbolic systems. A growing literature supports sound symbolism, or the existence of consistent, intuitive relationships between speech sounds and specific concepts. Prior work establishes that these sound-to-meaning mappings can shape language-related judgments and decisions, but do their effects generalize beyond merely the linguistic and truly color how we navigate our environment? We examine this possibility, relating a predominant sound symbolic distinction (vowel frontness) to a novel associate (spatial proximity) in five studies. We show that changing one vowel in a label can influence estimations of distance, impacting judgment, perception, and action. The results (1) provide the first experimental support for a relationship between vowels and spatial distance and (2) demonstrate that sound-to-meaning mappings have outcomes that extend beyond just language and can - through a single sound - influence how we perceive and behave toward objects in the world.
Collapse
Affiliation(s)
| | - Sam J Maglio
- University of Toronto Scarborough, Toronto, Ontario, Canada.
| | | | - Jin H Seok
- University of Missouri Columbia, Columbia, MO, United States
| | - Yaacov Trope
- New York University, New York, NY, United States
| |
Collapse
|
50
|
Möhring W, Ramsook KA, Hirsh-Pasek K, Golinkoff RM, Newcombe NS. Where music meets space: Children's sensitivity to pitch intervals is related to their mental spatial transformation skills. Cognition 2016; 151:1-5. [PMID: 26922894 DOI: 10.1016/j.cognition.2016.02.016] [Citation(s) in RCA: 7] [Impact Index Per Article: 0.9] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/12/2015] [Revised: 01/27/2016] [Accepted: 02/20/2016] [Indexed: 12/01/2022]
Abstract
Relations have been found among various continuous dimensions, including space and musical pitch. To probe the nature and development of space-pitch mappings, we tested 5- to 7-year-olds and adults (N=69), who heard pitch intervals and were asked to choose the corresponding spatial representation. Results showed that children and adults both mapped pitches continuously onto space, although effects were stronger in older than younger children. Additionally, children's spatial and numerical skills were tested, showing a relation between children's spatial and pitch-matching skills, and between their spatial and numerical skills. However, pitch and number were not related, suggesting spatial underpinnings for pitch and number.
Collapse
Affiliation(s)
- Wenke Möhring
- Department of Psychology, Temple University, Weiss Hall 318, 1701 North 13th Street, Philadelphia, PA 19122, United States.
| | - Kizzann Ashana Ramsook
- Department of Psychology, Temple University, Weiss Hall 318, 1701 North 13th Street, Philadelphia, PA 19122, United States.
| | - Kathryn Hirsh-Pasek
- Department of Psychology, Temple University, Weiss Hall 318, 1701 North 13th Street, Philadelphia, PA 19122, United States.
| | - Roberta M Golinkoff
- School of Education, University of Delaware, Willard Hall, Newark, DE 19716, United States.
| | - Nora S Newcombe
- Department of Psychology, Temple University, Weiss Hall 318, 1701 North 13th Street, Philadelphia, PA 19122, United States.
| |
Collapse
|