1
|
Morett LM. Observing gesture at learning enhances subsequent phonological and semantic processing of L2 words: An N400 study. BRAIN AND LANGUAGE 2023; 246:105327. [PMID: 37804717 DOI: 10.1016/j.bandl.2023.105327] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/28/2023] [Revised: 09/26/2023] [Accepted: 09/29/2023] [Indexed: 10/09/2023]
Abstract
This study employed the N400 event-related potential (ERP) to investigate how observing different types of gestures at learning affects the subsequent processing of L2 Mandarin words differing in lexical tone by L1 English speakers. The effects of pitch gestures conveying lexical tones (e.g., upwards diagonal movements for rising tone), semantic gestures conveying word meanings (e.g., waving goodbye for to wave), and no gesture were compared. In a lexical tone discrimination task, larger N400s for Mandarin target words mismatching vs. matching Mandarin prime words in lexical tone were observed for words learned with pitch gesture. In a meaning discrimination task, larger N400s for English target words mismatching vs. matching Mandarin prime words in meaning were observed for words learned with pitch and semantic gesture. These findings provide the first neural evidence that observing gestures during L2 word learning enhances subsequent phonological and semantic processing of learned L2 words.
Collapse
Affiliation(s)
- Laura M Morett
- Department of Speech, Language and Hearing Sciences, University of Missouri, 421 Lewis Hall, Columbia, MO 65211, United States; Department of Educational Studies in Psychology, Research Methodology, and Counseling, University of Alabama, United States.
| |
Collapse
|
2
|
Zeng Y, Leung KKW, Jongman A, Sereno JA, Wang Y. Multi-modal cross-linguistic perception of Mandarin tones in clear speech. Front Hum Neurosci 2023; 17:1247811. [PMID: 37829822 PMCID: PMC10565566 DOI: 10.3389/fnhum.2023.1247811] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2023] [Accepted: 09/08/2023] [Indexed: 10/14/2023] Open
Abstract
Clearly enunciated speech (relative to conversational, plain speech) involves articulatory and acoustic modifications that enhance auditory-visual (AV) segmental intelligibility. However, little research has explored clear-speech effects on the perception of suprasegmental properties such as lexical tone, particularly involving visual (facial) perception. Since tone production does not primarily rely on vocal tract configurations, tones may be less visually distinctive. Questions thus arise as to whether clear speech can enhance visual tone intelligibility, and if so, whether any intelligibility gain can be attributable to tone-specific category-enhancing (code-based) clear-speech cues or tone-general saliency-enhancing (signal-based) cues. The present study addresses these questions by examining the identification of clear and plain Mandarin tones with visual-only, auditory-only, and AV input modalities by native (Mandarin) and nonnative (English) perceivers. Results show that code-based visual and acoustic clear tone modifications, although limited, affect both native and nonnative intelligibility, with category-enhancing cues increasing intelligibility and category-blurring cues decreasing intelligibility. In contrast, signal-based cues, which are extensively available, do not benefit native intelligibility, although they contribute to nonnative intelligibility gain. These findings demonstrate that linguistically relevant visual tonal cues are existent. In clear speech, such tone category-enhancing cues are incorporated with saliency-enhancing cues across AV modalities for intelligibility improvements.
Collapse
Affiliation(s)
- Yuyu Zeng
- Department of Linguistics, University of Kansas, Lawrence, KS, United States
| | - Keith K. W. Leung
- Department of Linguistics, Simon Fraser University, Burnaby, BC, Canada
| | - Allard Jongman
- Department of Linguistics, University of Kansas, Lawrence, KS, United States
| | - Joan A. Sereno
- Department of Linguistics, University of Kansas, Lawrence, KS, United States
| | - Yue Wang
- Department of Linguistics, Simon Fraser University, Burnaby, BC, Canada
| |
Collapse
|
3
|
Hong S, Wang R, Zeng B. Incongruent visual cues affect the perception of Mandarin vowel but not tone. Front Psychol 2023; 13:971979. [PMID: 36687891 PMCID: PMC9846355 DOI: 10.3389/fpsyg.2022.971979] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/17/2022] [Accepted: 12/01/2022] [Indexed: 01/06/2023] Open
Abstract
Over the recent few decades, a large number of audiovisual speech studies have been focusing on the visual cues of consonants and vowels but neglecting those relating to lexical tones. In this study, we investigate whether incongruent audiovisual information interfered with the perception of lexical tones. We found that, for both Chinese and English speakers, incongruence between auditory and visemic mouth shape (i.e., visual form information) significantly interfered with reaction time and reduced the identification accuracy of vowels. However, incongruent lip movements (i.e., visual timing information) did not interfere with the perception of auditory lexical tone. We conclude that, in contrast to vowel perception, auditory tone perception seems relatively impervious to visual congruence cues, at least under these restricted laboratory conditions. The salience of visual form and timing information is discussed based on this finding.
Collapse
Affiliation(s)
- Shanhu Hong
- Institute of Foreign Language and Tourism, Quanzhou Preschool Education College, Quanzhou, China,Department of Psychology, Bournemouth University, Poole, United Kingdom
| | - Rui Wang
- School of Foreign Languages, Guangdong Pharmaceutical University, Guangzhou, China
| | - Biao Zeng
- Department of Psychology, Bournemouth University, Poole, United Kingdom,EEG Lab, Department of Psychology, University of South Wales, Newport, United Kingdom,*Correspondence: Biao Zeng ✉
| |
Collapse
|
4
|
Wei Y, Jia L, Gao F, Wang J. Visual-Auditory Integration and High-Variability Speech Can Facilitate Mandarin Chinese Tone Identification. JOURNAL OF SPEECH, LANGUAGE, AND HEARING RESEARCH : JSLHR 2022; 65:4096-4111. [PMID: 36279876 DOI: 10.1044/2022_jslhr-21-00691] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
PURPOSE Previous studies have demonstrated that tone identification can be facilitated when auditory tones are integrated with visual information that depicts the pitch contours of the auditory tones (hereafter, visual effect). This study investigates this visual effect in combined visual-auditory integration with high- and low-variability speech and examines whether one's prior tonal-language learning experience shapes the strength of this visual effect. METHOD Thirty Mandarin-naïve listeners, 25 Mandarin second language learners, and 30 native Mandarin listeners participated in a tone identification task in which participants judged whether an auditory tone was rising or falling in pitch. Moving arrows depicted the pitch contours of the auditory tones. A priming paradigm was used with the target auditory tones primed by four multimodal conditions: no stimuli (A-V-), visual-only stimuli (A-V+), auditory-only stimuli (A+V-), and both auditory and visual stimuli (A+V+). RESULTS For Mandarin naïve listeners, the visual effect in accuracy produced under the cross-modal integration (A+V+ vs. A+V-) was superior to a unimodal approach (A-V+ vs. A-V-), as evidenced by a higher d prime of A+V+ as opposed to A+V-. However, this was not the case in response time. Additionally, the visual effect in accuracy and response time under the unimodal approach only occurred for high-variability speech, not for low-variability speech. Across the three groups of listeners, we found that the less tonal-language learning experience one had, the stronger the visual effect. CONCLUSION Our study revealed the visual-auditory advantage and disadvantage of the visual effect and the joint contribution of visual-auditory integration and high-variability speech on facilitating tone perception via the process of speech symbolization and categorization. SUPPLEMENTAL MATERIAL https://doi.org/10.23641/asha.21357729.
Collapse
Affiliation(s)
- Yanjun Wei
- Center for Cognitive Science of Language, Beijing Language and Culture University, China
| | - Lin Jia
- Beijing Chinese Language and Culture College, China
| | - Fei Gao
- Faculty of Arts and Humanities, University of Macau, China
- Centre for Cognitive and Brain Sciences, University of Macau, China
| | - Jianqin Wang
- Center for Cognitive Science of Language, Beijing Language and Culture University, China
| |
Collapse
|
5
|
Xi X, Li P, Baills F, Prieto P. Hand Gestures Facilitate the Acquisition of Novel Phonemic Contrasts When They Appropriately Mimic Target Phonetic Features. JOURNAL OF SPEECH, LANGUAGE, AND HEARING RESEARCH : JSLHR 2020; 63:3571-3585. [PMID: 33090915 DOI: 10.1044/2020_jslhr-20-00084] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Purpose Research has shown that observing hand gestures mimicking pitch movements or rhythmic patterns can improve the learning of second language (L2) suprasegmental features. However, less is known about the effects of hand gestures on the learning of novel phonemic contrasts. This study examines (a) whether hand gestures mimicking phonetic features can boost L2 segment learning by naive learners and (b) whether a mismatch between the hand gesture form and the target phonetic feature influences the learning effect. Method Fifty Catalan native speakers undertook a short multimodal training session on two types of Mandarin Chinese consonants (plosives and affricates) in either of two conditions: Gesture and No Gesture. In the Gesture condition, a fist-to-open-hand gesture was used to mimic air burst, while the No Gesture condition included no such use of gestures. Crucially, while the hand gesture appropriately mimicked the air burst produced in plosives, this was not the case for affricates. Before and after training, participants were tested on two tasks, namely, the identification task and the imitation task. Participants' speech output was rated by five Chinese native speakers. Results The perception results showed that training with or without gestures yielded similar degrees of improvement for the identification of aspiration contrasts. By contrast, the production results showed that, while training without gestures did not help improve L2 pronunciation, training with gestures improved pronunciation, but only when the given gestures appropriately mimicked the phonetic properties they represented. Conclusions Results revealed that the efficacy of observing hand gestures on the learning of nonnative phonemes depends on the appropriateness of the form of those gestures relative to the target phonetic features. That is, hand gestures seem to be more useful when they appropriately mimic phonetic features. Supplemental Material https://doi.org/10.23641/asha.13105442.
Collapse
Affiliation(s)
- Xiaotong Xi
- Department of Translation and Language Sciences, Universitat Pompeu Fabra, Barcelona, Spain
| | - Peng Li
- Department of Translation and Language Sciences, Universitat Pompeu Fabra, Barcelona, Spain
| | - Florence Baills
- Department of Translation and Language Sciences, Universitat Pompeu Fabra, Barcelona, Spain
| | - Pilar Prieto
- Department of Translation and Language Sciences, Universitat Pompeu Fabra, Barcelona, Spain
- Institució Catalana de Recerca i Estudis Avançats, Barcelona, Spain
| |
Collapse
|
6
|
Billot-Vasquez K, Lian Z, Hirata Y, Kelly SD. Emblem Gestures Improve Perception and Evaluation of Non-native Speech. Front Psychol 2020; 11:574418. [PMID: 33071912 PMCID: PMC7536367 DOI: 10.3389/fpsyg.2020.574418] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/19/2020] [Accepted: 08/19/2020] [Indexed: 01/02/2023] Open
Abstract
Traditionally, much of the attention on the communicative effects of non-native accent has focused on the accent itself rather than how it functions within a more natural context. The present study explores how the bodily context of co-speech emblematic gestures affects perceptual and social evaluation of non-native accent. In two experiments in two different languages, Mandarin and Japanese, we filmed learners performing a short utterance in three different within-subjects conditions: speech alone, culturally familiar gesture, and culturally unfamiliar gesture. Native Mandarin participants watched videos of foreign-accented Mandarin speakers (Experiment 1), and native Japanese participants watched videos of foreign-accented Japanese speakers (Experiment 2). Following each video, native language participants were asked a set of questions targeting speech perception and social impressions of the learners. Results from both experiments demonstrate that familiar—and occasionally unfamiliar—emblems facilitated speech perception and enhanced social evaluations compared to the speech alone baseline. The variability in our findings suggests that gesture may serve varied functions in the perception and evaluation of non-native accent.
Collapse
Affiliation(s)
- Kiana Billot-Vasquez
- Department of Psychological and Brain Sciences, Colgate University, Hamilton, NY, United States.,Center for Language and Brain, Hamilton, NY, United States
| | - Zhongwen Lian
- Center for Language and Brain, Hamilton, NY, United States.,Linguistics Program, Colgate University, Hamilton, NY, United States
| | - Yukari Hirata
- Center for Language and Brain, Hamilton, NY, United States.,Linguistics Program, Colgate University, Hamilton, NY, United States.,Department of East Asian Languages, Colgate University, Hamilton, NY, United States
| | - Spencer D Kelly
- Department of Psychological and Brain Sciences, Colgate University, Hamilton, NY, United States.,Center for Language and Brain, Hamilton, NY, United States.,Linguistics Program, Colgate University, Hamilton, NY, United States
| |
Collapse
|
7
|
Drijvers L, Özyürek A. Non-native Listeners Benefit Less from Gestures and Visible Speech than Native Listeners During Degraded Speech Comprehension. LANGUAGE AND SPEECH 2020; 63:209-220. [PMID: 30795715 PMCID: PMC7254629 DOI: 10.1177/0023830919831311] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
Native listeners benefit from both visible speech and iconic gestures to enhance degraded speech comprehension (Drijvers & Ozyürek, 2017). We tested how highly proficient non-native listeners benefit from these visual articulators compared to native listeners. We presented videos of an actress uttering a verb in clear, moderately, or severely degraded speech, while her lips were blurred, visible, or visible and accompanied by a gesture. Our results revealed that unlike native listeners, non-native listeners were less likely to benefit from the combined enhancement of visible speech and gestures, especially since the benefit from visible speech was minimal when the signal quality was not sufficient.
Collapse
Affiliation(s)
- Linda Drijvers
- Linda Drijvers, Radboud University, Centre for Language Studies, Donders Institute for Brain, Cognition and Behaviour, Wundtlaan 1, Nijmegen, 6525 XD, The Netherlands.
| | - Asli Özyürek
- />Centre for Language Studies, Radboud University, Nijmegen, The Netherlands
- />Donders Institute for Brain, Cognition, and Behaviour, Radboud University, Nijmegen, The Netherlands
- />Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands
| |
Collapse
|
8
|
Cho S, Jongman A, Wang Y, Sereno JA. Multi-modal cross-linguistic perception of fricatives in clear speech. THE JOURNAL OF THE ACOUSTICAL SOCIETY OF AMERICA 2020; 147:2609. [PMID: 32359282 DOI: 10.1121/10.0001140] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/01/2019] [Accepted: 04/06/2020] [Indexed: 05/24/2023]
Abstract
Research shows that acoustic modifications in clearly enunciated fricative consonants (relative to the plain, conversational productions) facilitate auditory fricative perception, particularly for auditorily salient sibilant fricatives and for native perception. However, clear-speech effects on visual fricative perception have received less attention. A comparison of auditory and visual (facial) clear-fricative perception is particularly interesting since sibilant fricatives in English are more auditorily salient while non-sibilants are more visually salient. This study thus examines clear-speech effects on multi-modal perception of English sibilant and non-sibilant fricatives. Native English perceivers and non-native (Mandarin, Korean) perceivers with different fricative inventories in their native languages (L1s) identified clear and conversational fricative-vowel syllables in audio-only, visual-only, and audio-visual (AV) modes. The results reveal an overall positive clear-speech effect when visual information is involved. Considering the factor of AV saliency, clear speech benefits sibilants more in the auditory domain and non-sibilants more in the visual domain. With respect to language background, non-native (Mandarin and Korean) perceivers benefit from visual as well as auditory information, even for fricatives non-existent in their respective L1s, but the patterns of clear-speech gains are affected by the relative AV weighting and "nativeness" of the fricatives. These findings are discussed in terms of how saliency-enhancing and category-distinctive cues of speech sounds are adopted in AV perception to improve intelligibility.
Collapse
Affiliation(s)
- Sylvia Cho
- Language and Brain Lab, Department of Linguistics, Simon Fraser University, 8888 University Drive, Burnaby, British Columbia, V5A 1S6, Canada
| | - Allard Jongman
- The University of Kansas Phonetics and Psycholinguistics Lab, Department of Linguistics, The University of Kansas, Lawrence, Kansas 66044, USA
| | - Yue Wang
- Language and Brain Lab, Department of Linguistics, Simon Fraser University, 8888 University Drive, Burnaby, British Columbia, V5A 1S6, Canada
| | - Joan A Sereno
- The University of Kansas Phonetics and Psycholinguistics Lab, Department of Linguistics, The University of Kansas, Lawrence, Kansas 66044, USA
| |
Collapse
|
9
|
Drijvers L, Vaitonytė J, Özyürek A. Degree of Language Experience Modulates Visual Attention to Visible Speech and Iconic Gestures During Clear and Degraded Speech Comprehension. Cogn Sci 2019; 43:e12789. [PMID: 31621126 PMCID: PMC6790953 DOI: 10.1111/cogs.12789] [Citation(s) in RCA: 21] [Impact Index Per Article: 4.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/22/2018] [Revised: 07/12/2019] [Accepted: 08/19/2019] [Indexed: 11/27/2022]
Abstract
Visual information conveyed by iconic hand gestures and visible speech can enhance speech comprehension under adverse listening conditions for both native and non-native listeners. However, how a listener allocates visual attention to these articulators during speech comprehension is unknown. We used eye-tracking to investigate whether and how native and highly proficient non-native listeners of Dutch allocated overt eye gaze to visible speech and gestures during clear and degraded speech comprehension. Participants watched video clips of an actress uttering a clear or degraded (6-band noise-vocoded) action verb while performing a gesture or not, and were asked to indicate the word they heard in a cued-recall task. Gestural enhancement was the largest (i.e., a relative reduction in reaction time cost) when speech was degraded for all listeners, but it was stronger for native listeners. Both native and non-native listeners mostly gazed at the face during comprehension, but non-native listeners gazed more often at gestures than native listeners. However, only native but not non-native listeners' gaze allocation to gestures predicted gestural benefit during degraded speech comprehension. We conclude that non-native listeners might gaze at gesture more as it might be more challenging for non-native listeners to resolve the degraded auditory cues and couple those cues to phonological information that is conveyed by visible speech. This diminished phonological knowledge might hinder the use of semantic information that is conveyed by gestures for non-native compared to native listeners. Our results demonstrate that the degree of language experience impacts overt visual attention to visual articulators, resulting in different visual benefits for native versus non-native listeners.
Collapse
Affiliation(s)
- Linda Drijvers
- Donders Institute for Brain, Cognition, and BehaviourRadboud University
| | - Julija Vaitonytė
- Department of Cognitive and Artificial Intelligence (School of Humanities and Digital Sciences)Tilburg University
| | - Asli Özyürek
- Donders Institute for Brain, Cognition, and BehaviourRadboud University
- Centre for Language StudiesRadboud University
- Max Planck Institute for Psycholinguistics
| |
Collapse
|
10
|
Kager R. How Do Infants Disaggregate Referential and Affective Pitch? Front Psychol 2018; 9:2093. [PMID: 30429817 PMCID: PMC6220085 DOI: 10.3389/fpsyg.2018.02093] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2018] [Accepted: 10/10/2018] [Indexed: 11/20/2022] Open
Abstract
Infants are faced with a challenge of disaggregating functions of pitch in the ambient language into affective, pragmatic or referential (the latter in tone languages only). This mini review discusses several factors that might facilitate the disaggregation of referential and affective pitch in infancy: acoustic characteristics of infant-directed speech, recognition of vocal affect, facial cues accompanying affective prosody, and lateralization of affective and referential prosody in the brain. It proposes two hypotheses concerning the role of audiovisual cues and brain lateralization
Collapse
Affiliation(s)
- René Kager
- Utrecht Institute of Linguistics OTS, Utrecht University, Utrecht, Netherlands
| |
Collapse
|
11
|
Zheng A, Hirata Y, Kelly SD. Exploring the Effects of Imitating Hand Gestures and Head Nods on L1 and L2 Mandarin Tone Production. JOURNAL OF SPEECH, LANGUAGE, AND HEARING RESEARCH : JSLHR 2018; 61:2179-2195. [PMID: 30193334 DOI: 10.1044/2018_jslhr-s-17-0481] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/26/2017] [Accepted: 05/07/2018] [Indexed: 06/08/2023]
Abstract
PURPOSE This study investigated the impact of metaphoric actions-head nods and hand gestures-in producing Mandarin tones for first language (L1) and second language (L2) speakers. METHOD In 2 experiments, participants imitated videos of Mandarin tones produced under 3 conditions: (a) speech alone, (b) speech + head nods, and (c) speech + hand gestures. Fundamental frequency was recorded for both L1 (Experiment 1) and L2 (Experiment 2a) speakers, and the output of the L2 speakers was rated for tonal accuracy by 7 native Mandarin judges (Experiment 2b). RESULTS Experiment 1 showed that 12 L1 speakers' fundamental frequency spectral data did not differ among the 3 conditions. In Experiment 2a, the conditions did not affect the production of 24 English speakers for the most part, but there was some evidence that hand gestures helped Tone 4. In Experiment 2b, native Mandarin judges found limited conditional differences in L2 productions, with Tone 3 showing a slight head nods benefit in a subset of "correct" L2 tokens. CONCLUSION Results suggest that metaphoric bodily actions do not influence the lowest levels of L1 speech production in a tonal language and may play a very modest role during preliminary L2 learning.
Collapse
Affiliation(s)
- Annie Zheng
- Department of Neuroscience, Washington University, St. Louis, MO
- Center for Language and Brain, Colgate University, Hamilton, NY
| | - Yukari Hirata
- Department of East Asian Languages and Literatures, Colgate University, Hamilton, NY
- Center for Language and Brain, Colgate University, Hamilton, NY
| | - Spencer D Kelly
- Department of Psychological and Brain Sciences, Neuroscience Program, Colgate University, Hamilton, NY
- Center for Language and Brain, Colgate University, Hamilton, NY
| |
Collapse
|