1
|
Vannasing P, Dionne-Dostie E, Tremblay J, Paquette N, Collignon O, Gallagher A. Electrophysiological responses of audiovisual integration from infancy to adulthood. Brain Cogn 2024; 178:106180. [PMID: 38815526 DOI: 10.1016/j.bandc.2024.106180] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/25/2024] [Revised: 05/17/2024] [Accepted: 05/17/2024] [Indexed: 06/01/2024]
Abstract
Our ability to merge information from different senses into a unified percept is a crucial perceptual process for efficient interaction with our multisensory environment. Yet, the developmental process underlying how the brain implements multisensory integration (MSI) remains poorly known. This cross-sectional study aims to characterize the developmental patterns of audiovisual events in 131 individuals aged from 3 months to 30 years. Electroencephalography (EEG) was recorded during a passive task, including simple auditory, visual, and audiovisual stimuli. In addition to examining age-related variations in MSI responses, we investigated Event-Related Potentials (ERPs) linked with auditory and visual stimulation alone. This was done to depict the typical developmental trajectory of unisensory processing from infancy to adulthood within our sample and to contextualize the maturation effects of MSI in relation to unisensory development. Comparing the neural response to audiovisual stimuli to the sum of the unisensory responses revealed signs of MSI in the ERPs, more specifically between the P2 and N2 components (P2 effect). Furthermore, adult-like MSI responses emerge relatively late in the development, around 8 years old. The automatic integration of simple audiovisual stimuli is a long developmental process that emerges during childhood and continues to mature during adolescence with ERP latencies decreasing with age.
Collapse
Affiliation(s)
- Phetsamone Vannasing
- Neurodevelopmental Optical Imaging Laboratory (LION Lab), Sainte-Justine University Hospital Research Centre, Montreal, QC, Canada.
| | - Emmanuelle Dionne-Dostie
- Neurodevelopmental Optical Imaging Laboratory (LION Lab), Sainte-Justine University Hospital Research Centre, Montreal, QC, Canada.
| | - Julie Tremblay
- Neurodevelopmental Optical Imaging Laboratory (LION Lab), Sainte-Justine University Hospital Research Centre, Montreal, QC, Canada.
| | - Natacha Paquette
- Neurodevelopmental Optical Imaging Laboratory (LION Lab), Sainte-Justine University Hospital Research Centre, Montreal, QC, Canada.
| | - Olivier Collignon
- Institute of Psychology (IPSY) and Institute of Neuroscience (IoNS), Université Catholique de Louvain, Louvain-La-Neuve, Belgium; School of Health Sciences, HES-SO Valais-Wallis, The Sense Innovation and Research Center, Lausanne and Sion, Switzerland.
| | - Anne Gallagher
- Neurodevelopmental Optical Imaging Laboratory (LION Lab), Sainte-Justine University Hospital Research Centre, Montreal, QC, Canada; Cerebrum, Department of Psychology, University of Montreal, Montreal, Qc, Canada.
| |
Collapse
|
2
|
Bayne T, Frohlich J, Cusack R, Moser J, Naci L. Consciousness in the cradle: on the emergence of infant experience. Trends Cogn Sci 2023; 27:1135-1149. [PMID: 37838614 PMCID: PMC10660191 DOI: 10.1016/j.tics.2023.08.018] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/25/2023] [Revised: 08/28/2023] [Accepted: 08/29/2023] [Indexed: 10/16/2023]
Abstract
Although each of us was once a baby, infant consciousness remains mysterious and there is no received view about when, and in what form, consciousness first emerges. Some theorists defend a 'late-onset' view, suggesting that consciousness requires cognitive capacities which are unlikely to be in place before the child's first birthday at the very earliest. Other theorists defend an 'early-onset' account, suggesting that consciousness is likely to be in place at birth (or shortly after) and may even arise during the third trimester. Progress in this field has been difficult, not just because of the challenges associated with procuring the relevant behavioral and neural data, but also because of uncertainty about how best to study consciousness in the absence of the capacity for verbal report or intentional behavior. This review examines both the empirical and methodological progress in this field, arguing that recent research points in favor of early-onset accounts of the emergence of consciousness.
Collapse
Affiliation(s)
- Tim Bayne
- Monash University, Melbourne, VIC, Australia; Brain, Mind, and Consciousness Program, Canadian Institute for Advanced Research, Toronto, Canada.
| | - Joel Frohlich
- Institute for Neuromodulation and Neurotechnology, University Hospital and University of Tübingen, Tübingen, Germany; Institute for Advanced Consciousness Studies, Santa Monica, CA, USA
| | - Rhodri Cusack
- Thomas Mitchell Professor of Cognitive Neuroscience, Trinity College Institute of Neuroscience, Trinity College Dublin, Dublin, Ireland
| | - Julia Moser
- Masonic Institute for the Developing Brain, University of Minnesota, Minneapolis, MN, USA
| | - Lorina Naci
- Trinity College Institute of Neuroscience and Global Brain Health Institute, Trinity College, Dublin, Ireland
| |
Collapse
|
3
|
Choi I, Demir I, Oh S, Lee SH. Multisensory integration in the mammalian brain: diversity and flexibility in health and disease. Philos Trans R Soc Lond B Biol Sci 2023; 378:20220338. [PMID: 37545309 PMCID: PMC10404930 DOI: 10.1098/rstb.2022.0338] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/03/2023] [Accepted: 04/30/2023] [Indexed: 08/08/2023] Open
Abstract
Multisensory integration (MSI) occurs in a variety of brain areas, spanning cortical and subcortical regions. In traditional studies on sensory processing, the sensory cortices have been considered for processing sensory information in a modality-specific manner. The sensory cortices, however, send the information to other cortical and subcortical areas, including the higher association cortices and the other sensory cortices, where the multiple modality inputs converge and integrate to generate a meaningful percept. This integration process is neither simple nor fixed because these brain areas interact with each other via complicated circuits, which can be modulated by numerous internal and external conditions. As a result, dynamic MSI makes multisensory decisions flexible and adaptive in behaving animals. Impairments in MSI occur in many psychiatric disorders, which may result in an altered perception of the multisensory stimuli and an abnormal reaction to them. This review discusses the diversity and flexibility of MSI in mammals, including humans, primates and rodents, as well as the brain areas involved. It further explains how such flexibility influences perceptual experiences in behaving animals in both health and disease. This article is part of the theme issue 'Decision and control processes in multisensory perception'.
Collapse
Affiliation(s)
- Ilsong Choi
- Center for Synaptic Brain Dysfunctions, Institute for Basic Science (IBS), Daejeon 34141, Republic of Korea
| | - Ilayda Demir
- Department of biological sciences, KAIST, Daejeon 34141, Republic of Korea
| | - Seungmi Oh
- Department of biological sciences, KAIST, Daejeon 34141, Republic of Korea
| | - Seung-Hee Lee
- Center for Synaptic Brain Dysfunctions, Institute for Basic Science (IBS), Daejeon 34141, Republic of Korea
- Department of biological sciences, KAIST, Daejeon 34141, Republic of Korea
| |
Collapse
|
4
|
Birulés J, Goupil L, Josse J, Fort M. The Role of Talking Faces in Infant Language Learning: Mind the Gap between Screen-Based Settings and Real-Life Communicative Interactions. Brain Sci 2023; 13:1167. [PMID: 37626523 PMCID: PMC10452843 DOI: 10.3390/brainsci13081167] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/10/2023] [Revised: 07/28/2023] [Accepted: 08/01/2023] [Indexed: 08/27/2023] Open
Abstract
Over the last few decades, developmental (psycho) linguists have demonstrated that perceiving talking faces audio-visually is important for early language acquisition. Using mostly well-controlled and screen-based laboratory approaches, this line of research has shown that paying attention to talking faces is likely to be one of the powerful strategies infants use to learn their native(s) language(s). In this review, we combine evidence from these screen-based studies with another line of research that has studied how infants learn novel words and deploy their visual attention during naturalistic play. In our view, this is an important step toward developing an integrated account of how infants effectively extract audiovisual information from talkers' faces during early language learning. We identify three factors that have been understudied so far, despite the fact that they are likely to have an important impact on how infants deploy their attention (or not) toward talking faces during social interactions: social contingency, speaker characteristics, and task- dependencies. Last, we propose ideas to address these issues in future research, with the aim of reducing the existing knowledge gap between current experimental studies and the many ways infants can and do effectively rely upon the audiovisual information extracted from talking faces in their real-life language environment.
Collapse
Affiliation(s)
- Joan Birulés
- Laboratoire de Psychologie et NeuroCognition, CNRS UMR 5105, Université Grenoble Alpes, 38058 Grenoble, France; (L.G.); (J.J.); (M.F.)
| | - Louise Goupil
- Laboratoire de Psychologie et NeuroCognition, CNRS UMR 5105, Université Grenoble Alpes, 38058 Grenoble, France; (L.G.); (J.J.); (M.F.)
| | - Jérémie Josse
- Laboratoire de Psychologie et NeuroCognition, CNRS UMR 5105, Université Grenoble Alpes, 38058 Grenoble, France; (L.G.); (J.J.); (M.F.)
| | - Mathilde Fort
- Laboratoire de Psychologie et NeuroCognition, CNRS UMR 5105, Université Grenoble Alpes, 38058 Grenoble, France; (L.G.); (J.J.); (M.F.)
- Centre de Recherche en Neurosciences de Lyon, INSERM U1028-CNRS UMR 5292, Université Lyon 1, 69500 Bron, France
| |
Collapse
|
5
|
Dopierała AAW, Pérez DL, Mercure E, Pluta A, Malinowska-Korczak A, Evans S, Wolak T, Tomalski P. The Development of Cortical Responses to the Integration of Audiovisual Speech in Infancy. Brain Topogr 2023:10.1007/s10548-023-00959-8. [PMID: 37171657 PMCID: PMC10176292 DOI: 10.1007/s10548-023-00959-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2022] [Accepted: 04/11/2023] [Indexed: 05/13/2023]
Abstract
In adults, the integration of audiovisual speech elicits specific higher (super-additive) or lower (sub-additive) cortical responses when compared to the responses to unisensory stimuli. Although there is evidence that the fronto-temporal network is active during perception of audiovisual speech in infancy, the development of fronto-temporal responses to audiovisual integration remains unknown. In the current study, 5-month-olds and 10-month-olds watched bimodal (audiovisual) and alternating unimodal (auditory + visual) syllables. In this context we use alternating unimodal to denote alternating auditory and visual syllables that are perceived as separate syllables by adults. Using fNIRS we measured responses over large cortical areas including the inferior frontal and superior temporal regions. We identified channels showing different responses to bimodal than alternating unimodal condition and used multivariate pattern analysis (MVPA) to decode patterns of cortical responses to bimodal (audiovisual) and alternating unimodal (auditory + visual) speech. Results showed that in both age groups integration elicits cortical responses consistent with both super- and sub-additive responses in the fronto-temporal cortex. The univariate analyses revealed that between 5 and 10 months spatial distribution of these responses becomes increasingly focal. MVPA correctly classified responses at 5 months, with key input from channels located in the inferior frontal and superior temporal channels of the right hemisphere. However, MVPA classification was not successful at 10 months, suggesting a potential cortical re-organisation of audiovisual speech perception at this age. These results show the complex and non-gradual development of the cortical responses to integration of congruent audiovisual speech in infancy.
Collapse
Affiliation(s)
- Aleksandra A W Dopierała
- Faculty of Psychology, University of Warsaw, Warsaw, Poland.
- Department of Psychology, University of British Columbia, Vancouver, Canada.
| | | | | | - Agnieszka Pluta
- Faculty of Psychology, University of Warsaw, Warsaw, Poland
- University of Westminister, London, UK
| | | | - Samuel Evans
- Kings College London, London, UK
- University of Westminister, London, UK
| | - Tomasz Wolak
- Institute of Physiology and Pathology of Hearing, Bioimaging Research Center, World Hearing Centre, Warsaw, Poland
| | - Przemysław Tomalski
- Faculty of Psychology, University of Warsaw, Warsaw, Poland.
- Institute of Psychology, Polish Academy of Sciences, Warsaw, Poland.
| |
Collapse
|
6
|
Singh L, Quinn PC. Effects of face masks on language comprehension in bilingual children. INFANCY 2023. [PMID: 37186027 DOI: 10.1111/infa.12543] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/27/2022] [Revised: 03/19/2023] [Accepted: 04/05/2023] [Indexed: 05/17/2023]
Abstract
Due to the COVID-19 pandemic, many children receive language input through face coverings. The impact of face coverings for children's abilities to understand language remains unclear. Past research with monolingual children suggests that hearing words through surgical masks does not disrupt word recognition, but hearing words through transparent face shields proves more challenging. In this study, we investigated effects of different face coverings (surgical masks and transparent face shields) on language comprehension in bilingual children. Three-year-old English-Mandarin bilingual children (N = 28) heard familiar words in both English and Mandarin spoken through transparent face shields, surgical masks, and without masks. When tested in English, children recognized words presented without a mask and through a surgical mask, but did not recognize words presented with transparent face shields, replicating past findings with monolingual children. In contrast, when tested in Mandarin, children recognized words presented without a mask, through a surgical mask, and through a transparent face shield. Results are discussed in terms of specific properties of English and Mandarin that may elicit different effects for transparent face shields. Overall, the present findings suggest that face coverings, and in particular, surgical masks do not disrupt spoken word recognition in young bilingual children.
Collapse
Affiliation(s)
- Leher Singh
- Department of Psychology, National University of Singapore, Singapore, Singapore
| | - Paul C Quinn
- Department of Psychological and Brain Sciences, University of Delaware, Newark, Delaware, USA
| |
Collapse
|
7
|
Arias Sarah P, Hall L, Saitovitch A, Aucouturier JJ, Zilbovicius M, Johansson P. Pupil dilation reflects the dynamic integration of audiovisual emotional speech. Sci Rep 2023; 13:5507. [PMID: 37016041 PMCID: PMC10073148 DOI: 10.1038/s41598-023-32133-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2022] [Accepted: 03/22/2023] [Indexed: 04/06/2023] Open
Abstract
Emotional speech perception is a multisensory process. When speaking with an individual we concurrently integrate the information from their voice and face to decode e.g., their feelings, moods, and emotions. However, the physiological reactions-such as the reflexive dilation of the pupil-associated to these processes remain mostly unknown. That is the aim of the current article, to investigate whether pupillary reactions can index the processes underlying the audiovisual integration of emotional signals. To investigate this question, we used an algorithm able to increase or decrease the smiles seen in a person's face or heard in their voice, while preserving the temporal synchrony between visual and auditory channels. Using this algorithm, we created congruent and incongruent audiovisual smiles, and investigated participants' gaze and pupillary reactions to manipulated stimuli. We found that pupil reactions can reflect emotional information mismatch in audiovisual speech. In our data, when participants were explicitly asked to extract emotional information from stimuli, the first fixation within emotionally mismatching areas (i.e., the mouth) triggered pupil dilation. These results reveal that pupil dilation can reflect the dynamic integration of audiovisual emotional speech and provide insights on how these reactions are triggered during stimulus perception.
Collapse
Affiliation(s)
- Pablo Arias Sarah
- Lund University Cognitive Science, Lund University, Lund, Sweden.
- STMS Lab, UMR 9912 (IRCAM/CNRS/SU), Paris, France.
- School of Neuroscience and Psychology, Glasgow University, Glasgow, UK.
| | - Lars Hall
- STMS Lab, UMR 9912 (IRCAM/CNRS/SU), Paris, France
| | - Ana Saitovitch
- U1000 Brain Imaging in Psychiatry, INSERM-CEA, Pediatric Radiology Service, Necker Enfants Malades Hospital, Paris V René Descartes University, Paris, France
| | - Jean-Julien Aucouturier
- Department of Robotics and Automation FEMTO-ST Institute (CNRS/Université de Bourgogne Franche Comté), Besançon, France
| | - Monica Zilbovicius
- U1000 Brain Imaging in Psychiatry, INSERM-CEA, Pediatric Radiology Service, Necker Enfants Malades Hospital, Paris V René Descartes University, Paris, France
| | | |
Collapse
|
8
|
Chawarska K, Lewkowicz D, Feiner H, Macari S, Vernetti A. Attention to audiovisual speech does not facilitate language acquisition in infants with familial history of autism. J Child Psychol Psychiatry 2022; 63:1466-1476. [PMID: 35244219 DOI: 10.1111/jcpp.13595] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 06/30/2021] [Revised: 01/10/2022] [Accepted: 01/21/2022] [Indexed: 11/30/2022]
Abstract
BACKGROUND Due to familial liability, siblings of children with ASD exhibit elevated risk for language delays. The processes contributing to language delays in this population remain unclear. METHODS Considering well-established links between attention to dynamic audiovisual cues inherent in a speaker's face and speech processing, we investigated if attention to a speaker's face and mouth differs in 12-month-old infants at high familial risk for ASD but without ASD diagnosis (hr-sib; n = 91) and in infants at low familial risk (lr-sib; n = 62) for ASD and whether attention at 12 months predicts language outcomes at 18 months. RESULTS At 12 months, hr-sib and lr-sib infants did not differ in attention to face (p = .14), mouth preference (p = .30), or in receptive and expressive language scores (p = .36, p = .33). At 18 months, the hr-sib infants had lower receptive (p = .01) but not expressive (p = .84) language scores than the lr-sib infants. In the lr-sib infants, greater attention to the face (p = .022) and a mouth preference (p = .025) contributed to better language outcomes at 18 months. In the hr-sib infants, neither attention to the face nor a mouth preference was associated with language outcomes at 18 months. CONCLUSIONS Unlike low-risk infants, high-risk infants do not appear to benefit from audiovisual prosodic and speech cues in the service of language acquisition despite intact attention to these cues. We propose that impaired processing of audiovisual cues may constitute the link between genetic risk factors and poor language outcomes observed across the autism risk spectrum and may represent a promising endophenotype in autism.
Collapse
Affiliation(s)
- Katarzyna Chawarska
- Child Study Center, Yale University School of Medicine, New Haven, CT, USA
- Haskins Laboratories, New Haven, CT, USA
| | - David Lewkowicz
- Child Study Center, Yale University School of Medicine, New Haven, CT, USA
- Haskins Laboratories, New Haven, CT, USA
| | - Hannah Feiner
- Child Study Center, Yale University School of Medicine, New Haven, CT, USA
| | - Suzanne Macari
- Child Study Center, Yale University School of Medicine, New Haven, CT, USA
| | - Angelina Vernetti
- Child Study Center, Yale University School of Medicine, New Haven, CT, USA
| |
Collapse
|
9
|
Van Engen KJ, Dey A, Sommers MS, Peelle JE. Audiovisual speech perception: Moving beyond McGurk. THE JOURNAL OF THE ACOUSTICAL SOCIETY OF AMERICA 2022; 152:3216. [PMID: 36586857 PMCID: PMC9894660 DOI: 10.1121/10.0015262] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/28/2022] [Revised: 10/26/2022] [Accepted: 11/05/2022] [Indexed: 05/29/2023]
Abstract
Although it is clear that sighted listeners use both auditory and visual cues during speech perception, the manner in which multisensory information is combined is a matter of debate. One approach to measuring multisensory integration is to use variants of the McGurk illusion, in which discrepant auditory and visual cues produce auditory percepts that differ from those based on unimodal input. Not all listeners show the same degree of susceptibility to the McGurk illusion, and these individual differences are frequently used as a measure of audiovisual integration ability. However, despite their popularity, we join the voices of others in the field to argue that McGurk tasks are ill-suited for studying real-life multisensory speech perception: McGurk stimuli are often based on isolated syllables (which are rare in conversations) and necessarily rely on audiovisual incongruence that does not occur naturally. Furthermore, recent data show that susceptibility to McGurk tasks does not correlate with performance during natural audiovisual speech perception. Although the McGurk effect is a fascinating illusion, truly understanding the combined use of auditory and visual information during speech perception requires tasks that more closely resemble everyday communication: namely, words, sentences, and narratives with congruent auditory and visual speech cues.
Collapse
Affiliation(s)
- Kristin J Van Engen
- Department of Psychological and Brain Sciences, Washington University, St. Louis, Missouri 63130, USA
| | - Avanti Dey
- PLOS ONE, 1265 Battery Street, San Francisco, California 94111, USA
| | - Mitchell S Sommers
- Department of Psychological and Brain Sciences, Washington University, St. Louis, Missouri 63130, USA
| | - Jonathan E Peelle
- Department of Otolaryngology, Washington University, St. Louis, Missouri 63130, USA
| |
Collapse
|
10
|
Development of visual dominance in face-voice integration: Evidence from cross-modal compatibility effects in a gender categorization task. COGNITIVE DEVELOPMENT 2022. [DOI: 10.1016/j.cogdev.2022.101263] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022]
|
11
|
Tcherkassof A, Busquet M, Hurtig M, Thollon Behar M. Les effets du port du masque sanitaire sur les jeunes enfants en lieux d’accueil collectif. PSYCHOLOGIE FRANCAISE 2022. [DOI: 10.1016/j.psfr.2022.04.003] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Indexed: 10/16/2022]
|
12
|
Menn KH, Ward EK, Braukmann R, van den Boomen C, Buitelaar J, Hunnius S, Snijders TM. Neural Tracking in Infancy Predicts Language Development in Children With and Without Family History of Autism. NEUROBIOLOGY OF LANGUAGE (CAMBRIDGE, MASS.) 2022; 3:495-514. [PMID: 37216063 PMCID: PMC10158647 DOI: 10.1162/nol_a_00074] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/16/2021] [Accepted: 05/16/2022] [Indexed: 05/24/2023]
Abstract
During speech processing, neural activity in non-autistic adults and infants tracks the speech envelope. Recent research in adults indicates that this neural tracking relates to linguistic knowledge and may be reduced in autism. Such reduced tracking, if present already in infancy, could impede language development. In the current study, we focused on children with a family history of autism, who often show a delay in first language acquisition. We investigated whether differences in tracking of sung nursery rhymes during infancy relate to language development and autism symptoms in childhood. We assessed speech-brain coherence at either 10 or 14 months of age in a total of 22 infants with high likelihood of autism due to family history and 19 infants without family history of autism. We analyzed the relationship between speech-brain coherence in these infants and their vocabulary at 24 months as well as autism symptoms at 36 months. Our results showed significant speech-brain coherence in the 10- and 14-month-old infants. We found no evidence for a relationship between speech-brain coherence and later autism symptoms. Importantly, speech-brain coherence in the stressed syllable rate (1-3 Hz) predicted later vocabulary. Follow-up analyses showed evidence for a relationship between tracking and vocabulary only in 10-month-olds but not in 14-month-olds and indicated possible differences between the likelihood groups. Thus, early tracking of sung nursery rhymes is related to language development in childhood.
Collapse
Affiliation(s)
- Katharina H. Menn
- Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
- Department of Neuropsychology, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
- Research Group Language Cycles, Max Planck Institute for Human Cognitive and Brain Sciences, Leipzig, Germany
- International Max Planck Research School on Neuroscience of Communication: Function, Structure, and Plasticity, Leipzig, Germany
| | - Emma K. Ward
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
| | - Ricarda Braukmann
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
| | - Carlijn van den Boomen
- Department of Experimental Psychology, Helmholtz Institute, Utrecht University, Utrecht, The Netherlands
| | - Jan Buitelaar
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
- Department of Cognitive Neuroscience, Radboud University Medical Center, Nijmegen, The Netherlands
| | - Sabine Hunnius
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
| | - Tineke M. Snijders
- Max Planck Institute for Psycholinguistics, Nijmegen, The Netherlands
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, The Netherlands
- Cognitive Neuropsychology Department, Tilburg University
| |
Collapse
|
13
|
Goldenberg D, Tiede MK, Bennett RT, Whalen DH. Congruent aero-tactile stimuli bias perception of voicing continua. Front Hum Neurosci 2022; 16:879981. [PMID: 35911601 PMCID: PMC9334670 DOI: 10.3389/fnhum.2022.879981] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/20/2022] [Accepted: 06/28/2022] [Indexed: 11/13/2022] Open
Abstract
Multimodal integration is the formation of a coherent percept from different sensory inputs such as vision, audition, and somatosensation. Most research on multimodal integration in speech perception has focused on audio-visual integration. In recent years, audio-tactile integration has also been investigated, and it has been established that puffs of air applied to the skin and timed with listening tasks shift the perception of voicing by naive listeners. The current study has replicated and extended these findings by testing the effect of air puffs on gradations of voice onset time along a continuum rather than the voiced and voiceless endpoints of the original work. Three continua were tested: bilabial (“pa/ba”), velar (“ka/ga”), and a vowel continuum (“head/hid”) used as a control. The presence of air puffs was found to significantly increase the likelihood of choosing voiceless responses for the two VOT continua but had no effect on choices for the vowel continuum. Analysis of response times revealed that the presence of air puffs lengthened responses for intermediate (ambiguous) stimuli and shortened them for endpoint (non-ambiguous) stimuli. The slowest response times were observed for the intermediate steps for all three continua, but for the bilabial continuum this effect interacted with the presence of air puffs: responses were slower in the presence of air puffs, and faster in their absence. This suggests that during integration auditory and aero-tactile inputs are weighted differently by the perceptual system, with the latter exerting greater influence in those cases where the auditory cues for voicing are ambiguous.
Collapse
Affiliation(s)
| | - Mark K. Tiede
- Haskins Laboratories, New Haven, CT, United States
- *Correspondence: Mark K. Tiede,
| | - Ryan T. Bennett
- Department of Linguistics, University of California, Santa Cruz, Santa Cruz, CA, United States
| | - D. H. Whalen
- Haskins Laboratories, New Haven, CT, United States
- The Graduate Center, City University of New York (CUNY), New York, NY, United States
- Department of Linguistics, Yale University, New Haven, CT, United States
| |
Collapse
|
14
|
Mercure E, Bright P, Quiroz I, Filippi R. Effect of infant bilingualism on audiovisual integration in a McGurk task. J Exp Child Psychol 2022; 217:105351. [DOI: 10.1016/j.jecp.2021.105351] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/02/2021] [Revised: 11/12/2021] [Accepted: 12/10/2021] [Indexed: 10/19/2022]
|
15
|
Fiber tracing and microstructural characterization among audiovisual integration brain regions in neonates compared with young adults. Neuroimage 2022; 254:119141. [PMID: 35342006 DOI: 10.1016/j.neuroimage.2022.119141] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/09/2021] [Revised: 02/23/2022] [Accepted: 03/21/2022] [Indexed: 11/23/2022] Open
Abstract
Audiovisual integration has been related with cognitive-processing and behavioral advantages, as well as with various socio-cognitive disorders. While some studies have identified brain regions instantiating this ability shortly after birth, little is known about the structural pathways connecting them. The goal of the present study was to reconstruct fiber tracts linking AVI regions in the newborn in-vivo brain and assess their adult-likeness by comparing them with analogous fiber tracts of young adults. We performed probabilistic tractography and compared connective probabilities between a sample of term-born neonates (N = 311; the Developing Human Connectome Project (dHCP, http://www.developingconnectome.org) and young adults (N = 311 The Human Connectome Project; https://www.humanconnectome.org/) by means of a classification algorithm. Furthermore, we computed Dice coefficients to assess between-group spatial similarity of the reconstructed fibers and used diffusion metrics to characterize neonates' AVI brain network in terms of microstructural properties, interhemispheric differences and the association with perinatal covariates and biological sex. Overall, our results indicate that the AVI fiber bundles were successfully reconstructed in a vast majority of neonates, similarly to adults. Connective probability distributional similarities and spatial overlaps of AVI fibers between the two groups differed across the reconstructed fibers. There was a rank-order correspondence of the fibers' connective strengths across the groups. Additionally, the study revealed patterns of diffusion metrics in line with early white matter developmental trajectories and a developmental advantage for females. Altogether, these findings deliver evidence of meaningful structural connections among AVI regions in the newborn in-vivo brain.
Collapse
|
16
|
Riva V, Riboldi EM, Dondena C, Piazza C, Molteni M, Cantiani C. Atypical ERP responses to audiovisual speech integration and sensory responsiveness in infants at risk for autism spectrum disorder. INFANCY 2022; 27:369-388. [PMID: 35037381 DOI: 10.1111/infa.12456] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/09/2020] [Revised: 10/21/2021] [Accepted: 01/02/2022] [Indexed: 11/27/2022]
Abstract
Atypical sensory responses are included in the diagnostic criteria of autism spectrum disorder (ASD). Autistic individuals perform poorly during conditions that require integration across multiple sensory modalities such as audiovisual (AV) integration. Previous research investigated neural processing of AV integration in infancy. Yet, this has never been studied in infants at higher likelihood of later ASD (HR) using neurophysiological (EEG/ERP) techniques. In this study, we investigated whether ERP measures of AV integration differentiate HR infants from low-risk (LR) infants and whether early AV integration abilities are associated with clinical measures of sensory responsiveness. At age 12 months, AV integration in HR (n = 21) and LR infants (n = 19) was characterized in a novel ERP paradigm measuring the McGurk effect, and clinical measures of sensory responsiveness were evaluated. Different brain responses over the left temporal area emerge between HR and LR infants, specifically when AV stimuli cannot be integrated into a fusible percept. Furthermore, ERP responses related to integration of AV incongruent stimuli were found to be associated with sensory responsiveness, with reduced effects of AV incongruency being associated with reduced sensory reactivity. These data suggest that early identification of AV deficits may pave the way to innovative therapeutic strategies for the autistic symptomatology. Further replications in independent cohorts are needed for generalizability of findings.
Collapse
Affiliation(s)
- Valentina Riva
- Child Psychopathology Unit, Scientific Institute, IRCCS E. Medea, Lecco, Italy
| | - Elena Maria Riboldi
- Child Psychopathology Unit, Scientific Institute, IRCCS E. Medea, Lecco, Italy
| | - Chiara Dondena
- Child Psychopathology Unit, Scientific Institute, IRCCS E. Medea, Lecco, Italy
| | - Caterina Piazza
- Bioengineering Lab, Scientific Institute, IRCCS E. Medea, Lecco, Italy
| | - Massimo Molteni
- Child Psychopathology Unit, Scientific Institute, IRCCS E. Medea, Lecco, Italy
| | - Chiara Cantiani
- Child Psychopathology Unit, Scientific Institute, IRCCS E. Medea, Lecco, Italy
| |
Collapse
|
17
|
Development of multisensory integration following prolonged early-onset visual deprivation. Curr Biol 2021; 31:4879-4885.e6. [PMID: 34534443 DOI: 10.1016/j.cub.2021.08.060] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2021] [Revised: 07/12/2021] [Accepted: 08/23/2021] [Indexed: 11/23/2022]
Abstract
Adult humans make effortless use of multisensory signals and typically integrate them in an optimal fashion.1 This remarkable ability takes many years for normally sighted children to develop.2,3 Would individuals born blind or with extremely low vision still be able to develop multisensory integration later in life when surgically treated for sight restoration? Late acquisition of such capability would be a vivid example of the brain's ability to retain high levels of plasticity. We studied the development of multisensory integration in individuals suffering from congenital dense bilateral cataract, surgically treated years after birth. We assessed cataract-treated individuals' reliance on their restored visual abilities when estimating the size of an object simultaneously explored by touch. Within weeks to months after surgery, when combining information from vision and touch, they developed a multisensory weighting behavior similar to matched typically sighted controls. Next, we tested whether cataract-treated individuals benefited from integrating vision with touch by increasing the precision of size estimates, as it occurs when integrating signals in a statistically optimal fashion.1 For participants retested multiple times, such a benefit developed within months after surgery to levels of precision indistinguishable from optimal behavior. To summarize, the development of multisensory integration does not merely depend on age, but requires extensive multisensory experience with the world, rendered possible by the improved post-surgical visual acuity. We conclude that early exposure to multisensory signals is not essential for the development of multisensory integration, which can still be acquired even after many years of visual deprivation.
Collapse
|
18
|
Cox CMM, Keren-Portnoy T, Roepstorff A, Fusaroli R. A Bayesian meta-analysis of infants' ability to perceive audio-visual congruence for speech. INFANCY 2021; 27:67-96. [PMID: 34542230 DOI: 10.1111/infa.12436] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2020] [Revised: 08/25/2021] [Accepted: 09/02/2021] [Indexed: 11/29/2022]
Abstract
This paper quantifies the extent to which infants can perceive audio-visual congruence for speech information and assesses whether this ability changes with native language exposure over time. A hierarchical Bayesian robust regression model of 92 separate effect sizes extracted from 24 studies indicates a moderate effect size in a positive direction (0.35, CI [0.21: 0.50]). This result suggests that infants possess a robust ability to detect audio-visual congruence for speech. Moderator analyses, moreover, suggest that infants' audio-visual matching ability for speech emerges at an early point in the process of language acquisition and remains stable for both native and non-native speech throughout early development. A sensitivity analysis of the meta-analytic data, however, indicates that a moderate publication bias for significant results could shift the lower credible interval to include null effects. Based on these findings, we outline recommendations for new lines of enquiry and suggest ways to improve the replicability of results in future investigations.
Collapse
Affiliation(s)
- Christopher Martin Mikkelsen Cox
- School of Communication and Culture, Aarhus University, Aarhus, Denmark.,Interacting Minds Centre, Aarhus University, Aarhus, Denmark.,Department of Language and Linguistic Science, University of York, Heslington, UK
| | - Tamar Keren-Portnoy
- Department of Language and Linguistic Science, University of York, Heslington, UK
| | - Andreas Roepstorff
- School of Communication and Culture, Aarhus University, Aarhus, Denmark.,Interacting Minds Centre, Aarhus University, Aarhus, Denmark
| | - Riccardo Fusaroli
- School of Communication and Culture, Aarhus University, Aarhus, Denmark.,Interacting Minds Centre, Aarhus University, Aarhus, Denmark
| |
Collapse
|
19
|
Fecher N, Johnson EK. Revisiting the talker recognition advantage in bilingual infants. J Exp Child Psychol 2021; 214:105276. [PMID: 34507181 DOI: 10.1016/j.jecp.2021.105276] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2020] [Revised: 08/03/2021] [Accepted: 08/03/2021] [Indexed: 11/19/2022]
Abstract
Recent work has shown that exposure to multiple languages affects nonlinguistic processing of speech during infancy. Specifically, Fecher and Johnson found that bilingual 9-month-olds outperformed their monolingual peers in a face-voice matching task in an unfamiliar language [Developmental Science (2019a), 22(4), e12778]. What factors were driving this effect? That is, was this finding truly reflective of a bilingual advantage specific to talker processing, or did the study demonstrate a general cognitive advantage in bilingual infants? Here, we revisited this question by testing bilingual and monolingual 9-month-olds (N = 48) on their ability to associate previously unknown voices with animated cartoon characters. In comparison with earlier work, where infants were presented with characters speaking an unfamiliar language (Spanish), the characters in this study spoke a language familiar to both groups of infants (English). Critically, we found that the monolingual and bilingual infants learned the face-voice pairings equally well when they were tested on the familiar language. We conclude that whereas bilingual infants are skilled at recognizing talkers regardless of the language spoken by the talkers, monolingual infants succeed at talker recognition in a familiar language only. These results begin to clarify the underlying nature of the talker recognition benefit previously reported for bilingual infants.
Collapse
Affiliation(s)
- Natalie Fecher
- Department of Psychology, University of Toronto Mississauga, Mississauga, Ontario L5L 1C6, Canada
| | - Elizabeth K Johnson
- Department of Psychology, University of Toronto Mississauga, Mississauga, Ontario L5L 1C6, Canada.
| |
Collapse
|
20
|
The other-race effect on the McGurk effect in infancy. Atten Percept Psychophys 2021; 83:2924-2936. [PMID: 34386882 PMCID: PMC8460584 DOI: 10.3758/s13414-021-02342-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 06/14/2021] [Indexed: 11/30/2022]
Abstract
This study investigated the difference in the McGurk effect between own-race-face and other-race-face stimuli among Japanese infants from 5 to 9 months of age. The McGurk effect results from infants using information from a speaker’s face in audiovisual speech integration. We hypothesized that the McGurk effect varies with the speaker’s race because of the other-race effect, which indicates an advantage for own-race faces in our face processing system. Experiment 1 demonstrated the other-race effect on audiovisual speech integration such that the infants ages 5–6 months and 8–9 months are likely to perceive the McGurk effect when observing an own-race-face speaker, but not when observing an other-race-face speaker. Experiment 2 found the other-race effect on audiovisual speech integration regardless of irrelevant speech identity cues. Experiment 3 confirmed the infants’ ability to differentiate two auditory syllables. These results showed that infants are likely to integrate voice with an own-race-face, but not with an other-race-face. This implies the role of experiences with own-race-faces in the development of audiovisual speech integration. Our findings also contribute to the discussion of whether perceptual narrowing is a modality-general, pan-sensory process.
Collapse
|
21
|
Abstract
Visual speech cues play an important role in speech recognition, and the McGurk effect is a classic demonstration of this. In the original McGurk & Macdonald (Nature 264, 746-748 1976) experiment, 98% of participants reported an illusory "fusion" percept of /d/ when listening to the spoken syllable /b/ and watching the visual speech movements for /g/. However, more recent work shows that subject and task differences influence the proportion of fusion responses. In the current study, we varied task (forced-choice vs. open-ended), stimulus set (including /d/ exemplars vs. not), and data collection environment (lab vs. Mechanical Turk) to investigate the robustness of the McGurk effect. Across experiments, using the same stimuli to elicit the McGurk effect, we found fusion responses ranging from 10% to 60%, thus showing large variability in the likelihood of experiencing the McGurk effect across factors that are unrelated to the perceptual information provided by the stimuli. Rather than a robust perceptual illusion, we therefore argue that the McGurk effect exists only for some individuals under specific task situations.Significance: This series of studies re-evaluates the classic McGurk effect, which shows the relevance of visual cues on speech perception. We highlight the importance of taking into account subject variables and task differences, and challenge future researchers to think carefully about the perceptual basis of the McGurk effect, how it is defined, and what it can tell us about audiovisual integration in speech.
Collapse
|
22
|
Dorn K, Cauvet E, Weinert S. A cross‐linguistic study of multisensory perceptual narrowing in German and Swedish infants during the first year of life. INFANT AND CHILD DEVELOPMENT 2021. [DOI: 10.1002/icd.2217] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Affiliation(s)
- Katharina Dorn
- Department of Developmental Psychology Otto‐Friedrich University Bamberg Germany
| | - Elodie Cauvet
- Department of Women's and Children's health Karolinska Institute of Neurodevelopmental Disorders (KIND) Stockholm Sweden
| | - Sabine Weinert
- Department of Developmental Psychology Otto‐Friedrich University Bamberg Germany
| |
Collapse
|
23
|
Irwin J, Avery T, Kleinman D, Landi N. Audiovisual Speech Perception in Children with Autism Spectrum Disorders: Evidence from Visual Phonemic Restoration. J Autism Dev Disord 2021; 52:28-37. [DOI: 10.1007/s10803-021-04916-x] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 02/08/2021] [Indexed: 10/22/2022]
|
24
|
Weatherhead D, Arredondo MM, Nácar Garcia L, Werker JF. The Role of Audiovisual Speech in Fast-Mapping and Novel Word Retention in Monolingual and Bilingual 24-Month-Olds. Brain Sci 2021; 11:brainsci11010114. [PMID: 33467100 PMCID: PMC7830540 DOI: 10.3390/brainsci11010114] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/01/2020] [Revised: 01/12/2021] [Accepted: 01/14/2021] [Indexed: 11/19/2022] Open
Abstract
Three experiments examined the role of audiovisual speech on 24-month-old monolingual and bilinguals’ performance in a fast-mapping task. In all three experiments, toddlers were exposed to familiar trials which tested their knowledge of known word–referent pairs, disambiguation trials in which novel word–referent pairs were indirectly learned, and retention trials which probed their recognition of the newly-learned word–referent pairs. In Experiment 1 (n = 48), lip movements were present during familiar and disambiguation trials, but not retention trials. In Experiment 2 (n = 48), lip movements were present during all three trial types. In Experiment 3 (bilinguals only, n = 24), a still face with no lip movements was present in all three trial types. While toddlers succeeded in the familiar and disambiguation trials of every experiment, success in the retention trials was only found in Experiment 2. This work suggests that the extra-linguistic support provided by lip movements improved the learning and recognition of the novel words.
Collapse
Affiliation(s)
- Drew Weatherhead
- Department of Psychology and Neuroscience, Dalhousie University, Halifax, NS B3H 4R2, Canada
- Correspondence:
| | - Maria M. Arredondo
- Department of Human Development and Family Sciences, University of Texas at Austin, Austin, TX 78705, USA;
| | - Loreto Nácar Garcia
- Department of Psychology, University of British Columbia, Vancouver, BC V6T 1Z4, Canada; (L.N.G.); (J.F.W.)
| | - Janet F. Werker
- Department of Psychology, University of British Columbia, Vancouver, BC V6T 1Z4, Canada; (L.N.G.); (J.F.W.)
| |
Collapse
|
25
|
Lalonde K, Werner LA. Development of the Mechanisms Underlying Audiovisual Speech Perception Benefit. Brain Sci 2021; 11:49. [PMID: 33466253 PMCID: PMC7824772 DOI: 10.3390/brainsci11010049] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/05/2020] [Revised: 12/30/2020] [Accepted: 12/30/2020] [Indexed: 02/07/2023] Open
Abstract
The natural environments in which infants and children learn speech and language are noisy and multimodal. Adults rely on the multimodal nature of speech to compensate for noisy environments during speech communication. Multiple mechanisms underlie mature audiovisual benefit to speech perception, including reduced uncertainty as to when auditory speech will occur, use of correlations between the amplitude envelope of auditory and visual signals in fluent speech, and use of visual phonetic knowledge for lexical access. This paper reviews evidence regarding infants' and children's use of temporal and phonetic mechanisms in audiovisual speech perception benefit. The ability to use temporal cues for audiovisual speech perception benefit emerges in infancy. Although infants are sensitive to the correspondence between auditory and visual phonetic cues, the ability to use this correspondence for audiovisual benefit may not emerge until age four. A more cohesive account of the development of audiovisual speech perception may follow from a more thorough understanding of the development of sensitivity to and use of various temporal and phonetic cues.
Collapse
Affiliation(s)
- Kaylah Lalonde
- Center for Hearing Research, Boys Town National Research Hospital, Omaha, NE 68131, USA
| | - Lynne A. Werner
- Department of Speech and Hearing Sciences, University of Washington, Seattle, WA 98105, USA;
| |
Collapse
|
26
|
Atagi N, Johnson SP. Language Experience Is Associated with Infants' Visual Attention to Speakers. Brain Sci 2020; 10:E550. [PMID: 32823522 PMCID: PMC7463430 DOI: 10.3390/brainsci10080550] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/24/2020] [Revised: 08/04/2020] [Accepted: 08/07/2020] [Indexed: 11/17/2022] Open
Abstract
Early social-linguistic experience influences infants' attention to faces but little is known about how infants attend to the faces of speakers engaging in conversation. Here, we examine how monolingual and bilingual infants attended to speakers during a conversation, and we tested for the possibility that infants' visual attention may be modulated by familiarity with the language being spoken. We recorded eye movements in monolingual and bilingual 15-to-24-month-olds as they watched video clips of speakers using infant-directed speech while conversing in a familiar or unfamiliar language, with each other and to the infant. Overall, findings suggest that bilingual infants visually shift attention to a speaker prior to speech onset more when an unfamiliar, rather than a familiar, language is being spoken. However, this same effect was not found for monolingual infants. Thus, infants' familiarity with the language being spoken, and perhaps their language experiences, may modulate infants' visual attention to speakers.
Collapse
Affiliation(s)
- Natsuki Atagi
- Department of Child and Adolescent Studies, California State University, Fullerton, CA 92831, USA
| | - Scott P. Johnson
- Department of Psychology, University of California, Los Angeles, CA 90095, USA;
| |
Collapse
|
27
|
Kittler PM, Kim SY, Flory MJ, Phan HTT, Karmel BZ, Gardner JM. Effects of motion and audio-visual redundancy on upright and inverted face and feature preferences in 4-13-month old pre- and full-term NICU graduates. Infant Behav Dev 2020; 60:101439. [PMID: 32438215 PMCID: PMC7671943 DOI: 10.1016/j.infbeh.2020.101439] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/06/2019] [Revised: 02/05/2020] [Accepted: 03/17/2020] [Indexed: 01/01/2023]
Abstract
NICU infants are reported to have diminished social orientation and increased risk of socio-communicative disorders. In this eye tracking study, we used a preference for upright compared to inverted faces as a gauge of social interest in high medical risk full- and pre-term NICU infants. We examined the effects of facial motion and audio-visual redundancy on face and eye/mouth preferences across the first year. Upright and inverted baby faces were simultaneously presented in a paired-preference paradigm with motion and synchronized vocalization varied. NICU risk factors including birth weight, sex, and degree of CNS injury were examined. Overall, infants preferred the more socially salient upright faces, making this the first report, to our knowledge, of an upright compared to inverted face preference among high medical risk NICU infants. Infants with abnormalities on cranial ultrasound displayed lower social interest, i.e. less of a preferential interest in upright faces, when viewing static faces. However, motion selectively increased their upright face looking time to a level equal that of infants in other CNS injury groups. We also observed an age-related sex effect suggesting higher risk in NICU males. Females increased their attention to the mouth in upright faces across the first year, especially between 7-10 months, but males did not. Although vocalization increased diffuse attention toward the screen, contrary to our predictions, there was no evidence that the audio-visual redundancy embodied in a vocalizing face focused additional attention on upright faces or mouths. This unexpected result may suggest a vulnerability in response to talking faces among NICU infants that could potentially affect later verbal and socio-communicative development.
Collapse
Affiliation(s)
- P M Kittler
- Department of Infant Development, New York State Institute for Basic Research in Developmental Disabilities, United States.
| | - S-Y Kim
- Department of Infant Development, New York State Institute for Basic Research in Developmental Disabilities, United States
| | - M J Flory
- Department of Infant Development, New York State Institute for Basic Research in Developmental Disabilities, United States
| | - H T T Phan
- Department of Infant Development, New York State Institute for Basic Research in Developmental Disabilities, United States
| | - B Z Karmel
- Department of Infant Development, New York State Institute for Basic Research in Developmental Disabilities and Department of Pediatrics, Richmond University Medical Center, United States
| | - J M Gardner
- Department of Infant Development, New York State Institute for Basic Research in Developmental Disabilities and Department of Pediatrics, Richmond University Medical Center, United States
| |
Collapse
|
28
|
de la Cruz-Pavía I, Werker JF, Vatikiotis-Bateson E, Gervain J. Finding Phrases: The Interplay of Word Frequency, Phrasal Prosody and Co-speech Visual Information in Chunking Speech by Monolingual and Bilingual Adults. LANGUAGE AND SPEECH 2020; 63:264-291. [PMID: 31002280 PMCID: PMC7254630 DOI: 10.1177/0023830919842353] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Indexed: 06/09/2023]
Abstract
The audiovisual speech signal contains multimodal information to phrase boundaries. In three artificial language learning studies with 12 groups of adult participants we investigated whether English monolinguals and bilingual speakers of English and a language with opposite basic word order (i.e., in which objects precede verbs) can use word frequency, phrasal prosody and co-speech (facial) visual information, namely head nods, to parse unknown languages into phrase-like units. We showed that monolinguals and bilinguals used the auditory and visual sources of information to chunk "phrases" from the input. These results suggest that speech segmentation is a bimodal process, though the influence of co-speech facial gestures is rather limited and linked to the presence of auditory prosody. Importantly, a pragmatic factor, namely the language of the context, seems to determine the bilinguals' segmentation, overriding the auditory and visual cues and revealing a factor that begs further exploration.
Collapse
Affiliation(s)
- Irene de la Cruz-Pavía
- Irene de la Cruz-Pavía, Integrative Neuroscience and Cognition Center (INCC—UMR 8002), Université Paris Descartes-CNRS, 45 rue des Saints-Pères, Paris, 75006, France.
| | - Janet F. Werker
- Department of Psychology, University of British Columbia, Canada
| | | | - Judit Gervain
- Integrative Neuroscience and Cognition Center (INCC—UMR 8002), Université Paris Descartes (Sorbonne Paris Cité), France; Integrative Neuroscience and Cognition Center (INCC—UMR 8002), CNRS, France
| |
Collapse
|
29
|
Ujiie Y, Kanazawa S, Yamaguchi MK. The Other-Race-Effect on Audiovisual Speech Integration in Infants: A NIRS Study. Front Psychol 2020; 11:971. [PMID: 32499746 PMCID: PMC7243679 DOI: 10.3389/fpsyg.2020.00971] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2019] [Accepted: 04/20/2020] [Indexed: 11/21/2022] Open
Abstract
Previous studies have revealed perceptual narrowing for the own-race-face in face discrimination, but this phenomenon is poorly understood in face and voice integration. We focused on infants' brain responses to the McGurk effect to examine whether the other-race effect occurs in the activation patterns. In Experiment 1, we conducted fNIRS measurements to find the presence of a mapping of the McGurk effect in Japanese 8- to 9-month-old infants and to examine the difference between the activation patterns in response to own-race-face and other-race-face stimuli. We used two race-face conditions, own-race-face (East Asian) and other-race-face (Caucasian), each of which contained audiovisual-matched and McGurk-type stimuli. While the infants (N = 34) were observing each speech stimulus for each race, we measured cerebral hemoglobin concentrations in bilateral temporal brain regions. The results showed that in the own-race-face condition, audiovisual-matched stimuli induced the activation of the left temporal region, and the McGurk stimuli induced the activation of the bilateral temporal regions. No significant activations were found in the other-race-face condition. These results mean that the McGurk effect occurred only in the own-race-face condition. In Experiment 2, we used a familiarization/novelty preference procedure to confirm that the infants (N = 28) could perceive the McGurk effect in the own-race-face condition but not that of the other-race-face. The behavioral data supported the results of the fNIRS data, implying the presence of narrowing for the own-race face in the McGurk effect. These results suggest that narrowing of the McGurk effect may be involved in the development of relatively high-order processing, such as face-to-face communication with people surrounding the infant. We discuss the hypothesis that perceptual narrowing is a modality-general, pan-sensory process.
Collapse
Affiliation(s)
- Yuta Ujiie
- Graduate School of Psychology, Chukyo University, Aichi, Japan
- Research and Development Initiative, Chuo University, Tokyo, Japan
- Japan Society for the Promotion of Science, Tokyo, Japan
| | - So Kanazawa
- Department of Psychology, Japan Women’s University, Kawasaki, Japan
| | | |
Collapse
|
30
|
Abstract
What is a fundamental ability for cognitive development? Although many researchers have been addressing this question, no shared understanding has been acquired yet. We propose that predictive learning of sensorimotor signals plays a key role in early cognitive development. The human brain is known to represent sensorimotor signals in a predictive manner, i.e. it attempts to minimize prediction error between incoming sensory signals and top–down prediction. We extend this view and suggest that two mechanisms for minimizing prediction error lead to the development of cognitive abilities during early infancy. The first mechanism is to update an immature predictor. The predictor must be trained through sensorimotor experiences because it does not inherently have prediction ability. The second mechanism is to execute an action anticipated by the predictor. Interacting with other individuals often increases prediction error, which can be minimized by executing one's own action corresponding to others’ action. Our experiments using robotic systems replicated developmental dynamics observed in infants. The capabilities of self–other cognition and goal-directed action were acquired based on the first mechanism, whereas imitation and prosocial behaviours emerged based on the second mechanism. Our theory further provides a potential mechanism for autism spectrum condition. Atypical tolerance for prediction error is hypothesized to be a cause of perceptual and social difficulties. This article is part of the theme issue ‘From social brains to social robots: applying neurocognitive insights to human–robot interaction’.
Collapse
Affiliation(s)
- Yukie Nagai
- National Institute of Information and Communications Technology , Suita, Osaka 565-0871 , Japan
| |
Collapse
|
31
|
de la Cruz-Pavía I, Gervain J, Vatikiotis-Bateson E, Werker JF. Finding phrases: On the role of co-verbal facial information in learning word order in infancy. PLoS One 2019; 14:e0224786. [PMID: 31710615 PMCID: PMC6844464 DOI: 10.1371/journal.pone.0224786] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/27/2018] [Accepted: 10/22/2019] [Indexed: 11/23/2022] Open
Abstract
The input contains perceptually available cues, which might allow young infants to discover abstract properties of the target language. Thus, word frequency and prosodic prominence correlate systematically with basic word order in natural languages. Prelexical infants are sensitive to these frequency-based and prosodic cues, and use them to parse new input into phrases that follow the order characteristic of their native languages. Importantly, young infants readily integrate auditory and visual facial information while processing language. Here, we ask whether co-verbal visual information provided by talking faces also helps prelexical infants learn the word order of their native language in addition to word frequency and prosodic prominence. We created two structurally ambiguous artificial languages containing head nods produced by an animated avatar, aligned or misaligned with the frequency-based and prosodic information. During 4 minutes, two groups of 4- and 8-month-old infants were familiarized with the artificial language containing aligned auditory and visual cues, while two further groups were exposed to the misaligned language. Using a modified Headturn Preference Procedure, we tested infants’ preference for test items exhibiting the word order of the native language, French, vs. the opposite word order. At 4 months, infants had no preference, suggesting that 4-month-olds were not able to integrate the three available cues, or had not yet built a representation of word order. By contrast, 8-month-olds showed no preference when auditory and visual cues were aligned and a preference for the native word order when visual cues were misaligned. These results imply that infants at this age start to integrate the co-verbal visual and auditory cues.
Collapse
Affiliation(s)
- Irene de la Cruz-Pavía
- Integrative Neuroscience and Cognition Center (INCC–UMR 8002), Université Paris Descartes (Sorbonne Paris Cité), Paris, France
- Integrative Neuroscience and Cognition Center (INCC–UMR 8002), CNRS, Paris, France
- Department of Psychology, University of British Columbia, Vancouver, British Columbia, Canada
- * E-mail:
| | - Judit Gervain
- Integrative Neuroscience and Cognition Center (INCC–UMR 8002), Université Paris Descartes (Sorbonne Paris Cité), Paris, France
- Integrative Neuroscience and Cognition Center (INCC–UMR 8002), CNRS, Paris, France
| | - Eric Vatikiotis-Bateson
- Department of Linguistics, University of British Columbia, Vancouver, British Columbia, Canada
| | - Janet F. Werker
- Department of Psychology, University of British Columbia, Vancouver, British Columbia, Canada
| |
Collapse
|
32
|
Negen J, Bou Ali L, Chere B, Roome HE, Park Y, Nardini M. Coding Locations Relative to One or Many Landmarks in Childhood. PLoS Comput Biol 2019; 15:e1007380. [PMID: 31658253 PMCID: PMC6816551 DOI: 10.1371/journal.pcbi.1007380] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/15/2019] [Accepted: 09/04/2019] [Indexed: 11/19/2022] Open
Abstract
Cognitive development studies how information processing in the brain changes over the course of development. A key part of this question is how information is represented and stored in memory. This study examined allocentric (world-based) spatial memory, an important cognitive tool for planning routes and interacting with the space around us. This is typically theorized to use multiple landmarks all at once whenever it operates. In contrast, here we show that allocentric spatial memory frequently operates over a limited spatial window, much less than the full proximal scene, for children between 3.5 and 8.5 years old. The use of multiple landmarks increases gradually with age. Participants were asked to point to a remembered target location after a change of view in immersive virtual reality. A k-fold cross-validation model-comparison selected a model where young children usually use the target location's vector to the single nearest landmark and rarely take advantage of the vectors to other nearby landmarks. The comparison models, which attempt to explain the errors as generic forms of noise rather than encoding to a single spatial cue, did not capture the distribution of responses as well. Parameter fits of this new single- versus multi-cue model are also easily interpretable and related to other variables of interest in development (age, executive function). Based on this, we theorize that spatial memory in humans develops through three advancing levels (but not strict stages): most likely to encode locations egocentrically (relative to the self), then allocentrically (relative to the world) but using only one landmark, and finally, most likely to encode locations relative to multiple parts of the scene.
Collapse
Affiliation(s)
- James Negen
- Department of Psychology, Durham University, Durham, United Kingdom
| | - Linda Bou Ali
- Department of Psychiatry, American University of Beirut Medical Center, Beirut, Lebanon
| | - Brittney Chere
- Department of Psychological Sciences, Birkbeck, London, United Kingdom
| | - Hannah E. Roome
- Center for Learning and Memory, University of Texas at Austin, Austin, Texas, United States of America
| | - Yeachan Park
- Department of Psychology, University of Amsterdam, Amsterdam, Netherlands
| | - Marko Nardini
- Department of Psychology, Durham University, Durham, United Kingdom
| |
Collapse
|
33
|
Richards MD, Goltz HC, Wong AM. Audiovisual perception in amblyopia: A review and synthesis. Exp Eye Res 2019; 183:68-75. [DOI: 10.1016/j.exer.2018.04.017] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2018] [Revised: 04/27/2018] [Accepted: 04/28/2018] [Indexed: 11/15/2022]
|
34
|
Gong H, Chen L, Lei J. The effect of speech training experiences on speechreading skills of Chinese children with hearing impairment. CLINICAL LINGUISTICS & PHONETICS 2019; 33:1071-1085. [PMID: 31006281 DOI: 10.1080/02699206.2019.1604807] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/29/2019] [Revised: 04/01/2019] [Accepted: 04/03/2019] [Indexed: 06/09/2023]
Abstract
Speechreading contributes significantly to effective communication, and persons with hearing impairment (HI) may need to rely more on speechreading. However, whether they may benefit from training/practice to improve their speechreading performance remains unclear. In this study, we examine the effect of speech training on speechreading performance of children with HI in China, and how such effect, if any, may be influenced by age. Fifty-nine HI children with speech training experiences, and fifty-eight HI children without speech training experiences completed tasks measuring their ability to speechread Chinese at the linguistic levels of words, phrases, and sentences. The children ranged from 7 to 14 years old, and were divided into four age groups by two-year age intervals. Both accuracy rate and response time data were collected. Results revealed three findings: (1) HI children with speech training experiences speechread more accurately but more slowly than those without speech training experiences; (2) while speechreading performance generally improved with age, age didn't alter the relative performance between the two groups; and (3) speechreading performance was best for phrases, and worst for sentences. These findings suggest that HI children benefit from speech training to improve their speechreading performance, and their speechreading performance is influenced by age, linguistic level, and the specific measure used.
Collapse
Affiliation(s)
- Huina Gong
- Department of Special Education, Central China Normal University , Wuhan , China
| | - Liang Chen
- Communication Sciences and Special Education, University of Georgia , Athens , Georgia , USA
| | - Jianghua Lei
- Department of Special Education, Central China Normal University , Wuhan , China
| |
Collapse
|
35
|
Imafuku M, Kanakogi Y, Butler D, Myowa M. Demystifying infant vocal imitation: The roles of mouth looking and speaker's gaze. Dev Sci 2019; 22:e12825. [PMID: 30980494 DOI: 10.1111/desc.12825] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/31/2017] [Revised: 01/08/2019] [Accepted: 03/01/2019] [Indexed: 12/20/2022]
Abstract
Vocal imitation plays a fundamental role in human language acquisition from infancy. Little is known, however, about how infants imitate other's sounds. We focused on three factors: (a) whether infants receive information from upright faces, (b) the infant's observation of the speaker's mouth and (c) the speaker directing their gaze towards the infant. We recorded the eye movements of 6-month-olds who participated in experiments watching videos of a speaker producing vowel sounds. We found that an infants' tendency to vocally imitate such videos increased as a function of (a) seeing upright rather than inverted faces, (b) their increased looking towards the speaker's mouth and (c) whether the speaker directed their gaze towards, rather than away from infants. These latter findings are consistent with theories of motor resonance and natural pedagogy respectively. New light has been shed on the cues and underlying mechanisms linking infant speech perception and production.
Collapse
Affiliation(s)
- Masahiro Imafuku
- Graduate School of Education, Kyoto University, Kyoto, Japan.,Faculty of Education, Musashino University, Tokyo, Japan
| | | | - David Butler
- Graduate School of Education, Kyoto University, Kyoto, Japan.,The Institute for Social Neuroscience Psychology, Heidelberg, Victoria, Australia
| | - Masako Myowa
- Graduate School of Education, Kyoto University, Kyoto, Japan
| |
Collapse
|
36
|
Language in Preterm Born Children: Atypical Development and Effects of Early Interventions on Neuroplasticity. Neural Plast 2019; 2019:6873270. [PMID: 30930944 PMCID: PMC6410465 DOI: 10.1155/2019/6873270] [Citation(s) in RCA: 31] [Impact Index Per Article: 6.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/20/2018] [Accepted: 12/23/2018] [Indexed: 01/28/2023] Open
Abstract
Predicting language performances after preterm birth is challenging. It is described in the literature that early exposure to the extrauterine environment can be either detrimental or advantageous for neurodevelopment. However, the emphasis mostly lies on the fact that preterm birth may have an unfavorable effect on numerous aspects of development such as cognition, language, and behavior. Various studies reported atypical language development in preterm born children in the preschool years but also in school-aged children and adolescents. This review gives an overview of the course of language development and examines how prematurity can lead to atypical linguistic performances. In this paper, we mainly focus on environmental and neurophysiological factors influencing preterm infant neuroplasticity with potential short- and long-term effects on language development. Further research, however, should focus on examining the possible benefits that early exposure might entail.
Collapse
|
37
|
Pons F, Bosch L, Lewkowicz DJ. Twelve-month-old infants’ attention to the eyes of a talking face is associated with communication and social skills. Infant Behav Dev 2019; 54:80-84. [DOI: 10.1016/j.infbeh.2018.12.003] [Citation(s) in RCA: 46] [Impact Index Per Article: 9.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/15/2017] [Revised: 12/10/2018] [Accepted: 12/10/2018] [Indexed: 12/01/2022]
|
38
|
Mercure E, Kushnerenko E, Goldberg L, Bowden‐Howl H, Coulson K, Johnson MH, MacSweeney M. Language experience influences audiovisual speech integration in unimodal and bimodal bilingual infants. Dev Sci 2019; 22:e12701. [PMID: 30014580 PMCID: PMC6393757 DOI: 10.1111/desc.12701] [Citation(s) in RCA: 16] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2017] [Accepted: 05/17/2018] [Indexed: 11/29/2022]
Abstract
Infants as young as 2 months can integrate audio and visual aspects of speech articulation. A shift of attention from the eyes towards the mouth of talking faces occurs around 6 months of age in monolingual infants. However, it is unknown whether this pattern of attention during audiovisual speech processing is influenced by speech and language experience in infancy. The present study investigated this question by analysing audiovisual speech processing in three groups of 4- to 8-month-old infants who differed in their language experience: monolinguals, unimodal bilinguals (infants exposed to two or more spoken languages) and bimodal bilinguals (hearing infants with Deaf mothers). Eye-tracking was used to study patterns of face scanning while infants were viewing faces articulating syllables with congruent, incongruent and silent auditory tracks. Monolinguals and unimodal bilinguals increased their attention to the mouth of talking faces between 4 and 8 months, while bimodal bilinguals did not show any age difference in their scanning patterns. Moreover, older (6.6 to 8 months), but not younger, monolinguals (4 to 6.5 months) showed increased visual attention to the mouth of faces articulating audiovisually incongruent rather than congruent faces, indicating surprise or novelty. In contrast, no audiovisual congruency effect was found in unimodal or bimodal bilinguals. Results suggest that speech and language experience influences audiovisual integration in infancy. Specifically, reduced or more variable experience of audiovisual speech from the primary caregiver may lead to less sensitivity to the integration of audio and visual cues of speech articulation.
Collapse
Affiliation(s)
| | | | | | - Harriet Bowden‐Howl
- UCL Institute of Cognitive NeuroscienceLondonUK
- School of PsychologyUniversity of PlymouthPlymouthUK
| | - Kimberley Coulson
- UCL Institute of Cognitive NeuroscienceLondonUK
- Department of Psychology and Sports SciencesUniversity of HertfordshireHatfieldHertfordshireUK
| | - Mark H Johnson
- Centre for Brain and Cognitive DevelopmentBirkbeck – University of LondonLondonUK
- Department of PsychologyUniversity of CambridgeCambridgeUK
| | | |
Collapse
|
39
|
Imafuku M, Kawai M, Niwa F, Shinya Y, Myowa M. Audiovisual speech perception and language acquisition in preterm infants: A longitudinal study. Early Hum Dev 2019; 128:93-100. [PMID: 30541680 DOI: 10.1016/j.earlhumdev.2018.11.001] [Citation(s) in RCA: 8] [Impact Index Per Article: 1.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 02/06/2018] [Revised: 09/07/2018] [Accepted: 11/01/2018] [Indexed: 10/27/2022]
Abstract
BACKGROUND Preterm infants have a higher risk of language delay throughout childhood. The ability to integrate audiovisual speech information is associated with language acquisition in term infants; however, the relation is still unclear in preterm infant. AIM AND METHODS This study longitudinally investigated visual preference for audiovisual congruent and incongruent speech during a preferential looking task using eye-tracking in preterm and term infants at 6, 12, and 18 months of corrected age. The infants' receptive and expressive vocabulary at 12 and 18 months were obtained by parent report, using the Japanese MacArthur Communicative Development Inventory. RESULTS We found that preterm infants did not clearly show visual preference for the congruent audiovisual display at any age, whereas term infants looked at the congruent audiovisual display longer than the incongruent audiovisual display at 6 and 18 months. Preterm infants' receptive and expressive vocabulary scores were lower than those of term infants at 12 and 18 months. Furthermore, the proportion of looking time toward the congruent audiovisual display at 6 months was positively correlated with receptive vocabulary scores at 12 and 18 months for both groups. CONCLUSIONS These findings suggest that better audiovisual speech perception abilities are one factor that results in better language acquisition in preterm as well as term infants. Early identification of behaviors associated with later language in preterm infants may contribute to planning intervention for developmental problems.
Collapse
Affiliation(s)
- Masahiro Imafuku
- Graduate School of Education, Kyoto University, Kyoto, Japan; Faculty of Education, Musashino University, Tokyo, Japan.
| | - Masahiko Kawai
- Department of Pediatrics, Graduate School of Medicine, Kyoto University, Kyoto, Japan
| | - Fusako Niwa
- Department of Pediatrics, Graduate School of Medicine, Kyoto University, Kyoto, Japan
| | - Yuta Shinya
- Graduate School of Education, Kyoto University, Kyoto, Japan; Graduate School of Education, The University of Tokyo, Tokyo, Japan
| | - Masako Myowa
- Graduate School of Education, Kyoto University, Kyoto, Japan
| |
Collapse
|
40
|
Abstract
Speech research during recent years has moved progressively away from its traditional focus on audition toward a more multisensory approach. In addition to audition and vision, many somatosenses including proprioception, pressure, vibration and aerotactile sensation are all highly relevant modalities for experiencing and/or conveying speech. In this article, we review both long-standing cross-modal effects stemming from decades of audiovisual speech research as well as new findings related to somatosensory effects. Cross-modal effects in speech perception to date are found to be constrained by temporal congruence and signal relevance, but appear to be unconstrained by spatial congruence. Far from taking place in a one-, two- or even three-dimensional space, the literature reveals that speech occupies a highly multidimensional sensory space. We argue that future research in cross-modal effects should expand to consider each of these modalities both separately and in combination with other modalities in speech.
Collapse
Affiliation(s)
- Megan Keough
- Interdisciplinary Speech Research Lab, Department of Linguistics, University of British Columbia, Vancouver, British Columbia V6T 1Z4, Canada
| | - Donald Derrick
- New Zealand Institute of Brain and Behaviour, University of Canterbury, Christchurch 8140, New Zealand
- MARCS Institute for Brain, Behaviour and Development, Western Sydney University, Penrith, New South Wales 2751, Australia
| | - Bryan Gick
- Interdisciplinary Speech Research Lab, Department of Linguistics, University of British Columbia, Vancouver, British Columbia V6T 1Z4, Canada
- Haskins Laboratories, Yale University, New Haven, CT 06511, USA
| |
Collapse
|
41
|
Jo J, Ko ES. Korean Mothers Attune the Frequency and Acoustic Saliency of Sound Symbolic Words to the Linguistic Maturity of Their Children. Front Psychol 2018; 9:2225. [PMID: 30618893 PMCID: PMC6305434 DOI: 10.3389/fpsyg.2018.02225] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2018] [Accepted: 10/26/2018] [Indexed: 11/25/2022] Open
Abstract
The present study investigates Korean mothers' use of sound symbolism, in particular expressive lengthening and ideophones, in their speech directed to their children. Specifically, we explore whether the frequency and acoustic saliency of sound symbolic words are modulated by the maturity of children's linguistic ability. A total of 36 infant-mother dyads, 12 each belonging to the three groups of preverbal (M = 8-month-old), early speech (M = 13-month-old), and multiword (M = 27-month-old) stage, were recorded in a 40-min free-play session. The results were consistent with the findings in previous research that the ratio of sound symbolic words in mothers' speech decreases with child age and that they are acoustically more salient than conventional words in duration and pitch measures. We additionally found that mothers weaken the prominence for ideophones for older children in mean pitch, suggesting that such prominence of these iconic words might bootstrap infants' word learning especially when they are younger. Interestingly, however, we found that mothers maintain the acoustic saliency of expressive lengthening consistently across children's ages in all acoustic measures. There is some indication that children at age 2 are not likely to have mastered the fine details of scalar properties in certain words. Thus, it could be that they still benefit from the enhanced prosody of expressive lengthening in learning the semantic attributes of scalar adjectives, and, accordingly, mothers continue to provide redundant acoustic cues longer for expressive lengthening than ideophones.
Collapse
Affiliation(s)
- Jinyoung Jo
- Department of English Language and Literature, Seoul National University, Seoul, South Korea
| | - Eon-Suk Ko
- Department of English Language and Literature, Chosun University, Gwangju, South Korea
| |
Collapse
|
42
|
Birulés J, Bosch L, Brieke R, Pons F, Lewkowicz DJ. Inside bilingualism: Language background modulates selective attention to a talker's mouth. Dev Sci 2018; 22:e12755. [PMID: 30251757 DOI: 10.1111/desc.12755] [Citation(s) in RCA: 18] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/12/2018] [Revised: 08/08/2018] [Accepted: 09/21/2018] [Indexed: 11/28/2022]
Abstract
Previous findings indicate that bilingual Catalan/Spanish-learning infants attend more to the highly salient audiovisual redundancy cues normally available in a talker's mouth than do monolingual infants. Presumably, greater attention to such cues renders the challenge of learning two languages easier. Spanish and Catalan are, however, rhythmically and phonologically close languages. This raises the possibility that bilinguals only rely on redundant audiovisual cues when their languages are close. To test this possibility, we exposed 15-month-old and 4- to 6-year-old close-language bilinguals (Spanish/Catalan) and distant-language bilinguals (Spanish/"other") to videos of a talker uttering Spanish or Catalan (native) and English (non-native) monologues and recorded eye-gaze to the talker's eyes and mouth. At both ages, the close-language bilinguals attended more to the talker's mouth than the distant-language bilinguals. This indicates that language proximity modulates selective attention to a talker's mouth during early childhood and suggests that reliance on the greater salience of audiovisual speech cues depends on the difficulty of the speech-processing task.
Collapse
Affiliation(s)
- Joan Birulés
- Department of Cognition, Development and Educational Psychology, Universitat de Barcelona, Barcelona, Spain
| | - Laura Bosch
- Department of Cognition, Development and Educational Psychology, Universitat de Barcelona, Barcelona, Spain
| | - Ricarda Brieke
- Department of Cognition, Development and Educational Psychology, Universitat de Barcelona, Barcelona, Spain
| | - Ferran Pons
- Department of Cognition, Development and Educational Psychology, Universitat de Barcelona, Barcelona, Spain
| | - David J Lewkowicz
- Department of Communication Sciences and Disorders, Northeastern University, Boston, Massachusetts
| |
Collapse
|
43
|
Kasisopa B, El-Khoury Antonios L, Jongman A, Sereno JA, Burnham D. Training Children to Perceive Non-native Lexical Tones: Tone Language Background, Bilingualism, and Auditory-Visual Information. Front Psychol 2018; 9:1508. [PMID: 30233446 PMCID: PMC6131621 DOI: 10.3389/fpsyg.2018.01508] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2017] [Accepted: 07/31/2018] [Indexed: 11/13/2022] Open
Abstract
This study investigates the role of language background and bilingual status in the perception of foreign lexical tones. Eight groups of participants, consisting of children of 6 and 8 years from one of four language background (tone or non-tone) × bilingual status (monolingual or bilingual)-Thai monolingual, English monolingual, English-Thai bilingual, and English-Arabic bilingual were trained to perceive the four Mandarin lexical tones. Half the children in each of these eight groups were given auditory-only (AO) training and half auditory-visual (AV) training. In each group Mandarin tone identification was tested before and after (pre- and post-) training with both auditory-only test (ao-test) and auditory-visual test (av test). The effect of training on Mandarin tone identification was minimal for 6-year-olds. On the other hand, 8-year-olds, particularly those with tone language experience showed greater pre- to post-training improvement, and this was best indexed by ao-test trials. Bilingual vs. monolingual background did not facilitate overall improvement due to training, but it did modulate the efficacy of the Training mode: for bilinguals both AO and AV training, and especially AO, resulted in performance gain; but for monolinguals training was most effective with AV stimuli. Again this effect was best indexed by ao-test trials. These results suggest that tone language experience, be it monolingual or bilingual, is a strong predictor of learning unfamiliar tones; that monolinguals learn best from AV training trials and bilinguals from AO training trials; and that there is no metalinguistic advantage due to bilingualism in learning to perceive lexical tones.
Collapse
Affiliation(s)
- Benjawan Kasisopa
- MARCS Institute for Brain, Behaviour and Development, Western Sydney University, Penrith, NSW, Australia
| | - Lamya El-Khoury Antonios
- MARCS Institute for Brain, Behaviour and Development, Western Sydney University, Penrith, NSW, Australia
| | - Allard Jongman
- Department of Linguistics, College of Liberal Arts & Sciences, University of Kansas, Lawrence, KS, United States.,Phonetics & Psycholinguistics Laboratory, Department of Linguistics, College of Liberal Arts & Sciences, University of Kansas, Lawrence, KS, United States
| | - Joan A Sereno
- Department of Linguistics, College of Liberal Arts & Sciences, University of Kansas, Lawrence, KS, United States.,Phonetics & Psycholinguistics Laboratory, Department of Linguistics, College of Liberal Arts & Sciences, University of Kansas, Lawrence, KS, United States
| | - Denis Burnham
- MARCS Institute for Brain, Behaviour and Development, Western Sydney University, Penrith, NSW, Australia
| |
Collapse
|
44
|
Chen L, Lei J, Gong H. The effect of hearing status on speechreading performance of Chinese adolescents. CLINICAL LINGUISTICS & PHONETICS 2018; 32:1090-1102. [PMID: 30183411 DOI: 10.1080/02699206.2018.1510986] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
The effect of hearing status on the ability to speechread is poorly understood, and current findings are inconclusive regarding differences in speechreading performance between children and adults with hearing impairment and those with normal hearing. In this study, we investigated the effect of hearing status on speechreading skills in Chinese adolescents. Thirty seven severely deaf students with a mean pure-tone average of 93 dB hearing threshold level and 21 hearing controls aged 16 completed tasks measuring their speechreading of simplex finals (monophthongs), complex finals (diphthongs or vowel + nasal constellations) and initials (consonants) in Chinese. Both accuracy rate and response time data were collected. Results showed no significant difference in accuracy between groups. By contrast, deaf individuals were significantly faster at speechreading than their hearing controls. In addition, for both groups, performance on speechreading simplex finals was faster and more accurate than complex finals, which in turn was better than initial consonants. We conclude that speechreading skills in Chinese adolescents are influenced by hearing status, characteristics of sounds to be identified, as well as the measures used.
Collapse
Affiliation(s)
- Liang Chen
- a Communication Sciences and Special Education , University of Georgia , Athens , GA , USA
| | - Jianghua Lei
- b Department of Special Education , Central China Normal University , Wuhan , China
| | - Huina Gong
- b Department of Special Education , Central China Normal University , Wuhan , China
| |
Collapse
|
45
|
Masapollo M, Polka L, Ménard L, Franklin L, Tiede M, Morgan J. Asymmetries in unimodal visual vowel perception: The roles of oral-facial kinematics, orientation, and configuration. J Exp Psychol Hum Percept Perform 2018; 44:1103-1118. [PMID: 29517257 PMCID: PMC6037555 DOI: 10.1037/xhp0000518] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
Masapollo, Polka, and Ménard (2017) recently reported a robust directional asymmetry in unimodal visual vowel perception: Adult perceivers discriminate a change from an English /u/ viseme to a French /u/ viseme significantly better than a change in the reverse direction. This asymmetry replicates a frequent pattern found in unimodal auditory vowel perception that points to a universal bias favoring more extreme vocalic articulations, which lead to acoustic signals with increased formant convergence. In the present article, the authors report 5 experiments designed to investigate whether this asymmetry in the visual realm reflects a speech-specific or general processing bias. They successfully replicated the directional effect using Masapollo et al.'s dynamically articulating faces but failed to replicate the effect when the faces were shown under static conditions. Asymmetries also emerged during discrimination of canonically oriented point-light stimuli that retained the kinematics and configuration of the articulating mouth. In contrast, no asymmetries emerged during discrimination of rotated point-light stimuli or Lissajou patterns that retained the kinematics, but not the canonical orientation or spatial configuration, of the labial gestures. These findings suggest that the perceptual processes underlying asymmetries in unimodal visual vowel discrimination are sensitive to speech-specific motion and configural properties and raise foundational questions concerning the role of specialized and general processes in vowel perception. (PsycINFO Database Record
Collapse
Affiliation(s)
- Matthew Masapollo
- Brown University
- McGill University
- Centre for Research on Brain, Language, and Music
| | - Linda Polka
- McGill University
- Centre for Research on Brain, Language, and Music
| | - Lucie Ménard
- Centre for Research on Brain, Language, and Music
- University of Quebec at Montreal
| | | | | | | |
Collapse
|
46
|
Tsang T, Atagi N, Johnson SP. Selective attention to the mouth is associated with expressive language skills in monolingual and bilingual infants. J Exp Child Psychol 2018; 169:93-109. [PMID: 29406126 PMCID: PMC5933852 DOI: 10.1016/j.jecp.2018.01.002] [Citation(s) in RCA: 50] [Impact Index Per Article: 8.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/04/2017] [Revised: 12/06/2017] [Accepted: 01/03/2018] [Indexed: 12/27/2022]
Abstract
Infants increasingly attend to the mouths of others during the latter half of the first postnatal year, and individual differences in selective attention to talking mouths during infancy predict verbal skills during toddlerhood. There is some evidence suggesting that trajectories in mouth-looking vary by early language environment, in particular monolingual or bilingual language exposure, which may have differential consequences in developing sensitivity to the communicative and social affordances of the face. Here, we evaluated whether 6- to 12-month-olds' mouth-looking is related to skills associated with concurrent social communicative development-including early language functioning and emotion discriminability. We found that attention to the mouth of a talking face increased with age but that mouth-looking was more strongly associated with concurrent expressive language skills than chronological age for both monolingual and bilingual infants. Mouth-looking was not related to emotion discrimination. These data suggest that selective attention to a talking mouth may be one important mechanism by which infants learn language regardless of home language environment.
Collapse
Affiliation(s)
- Tawny Tsang
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA 90095, USA.
| | - Natsuki Atagi
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA 90095, USA
| | - Scott P Johnson
- Department of Psychology, University of California, Los Angeles, Los Angeles, CA 90095, USA
| |
Collapse
|
47
|
Abstract
It is a matter of debate, whether and how improved auditory discrimination abilities enable speeded speech comprehension in congenitally blind adults. Previous research has concentrated on semantic and syntactic aspects of processing. Here we investigated phonologically mediated spoken word access processes by means of word onset priming. Blind adults and age- and gender-matched sighted adults listened to spoken word onsets (primes) followed by complete words (targets). Phonological overlap between primes and targets varied. Blind participants made faster lexical decision responses than sighted participants, yet their speeded responses were not restricted to phonologically overlapping trials. Furthermore, timing of Event Related Potential (ERP) results did not differ between blind and sighted participants. Together these results suggest that blind and sighted listeners are equally fast in implicit phonological encoding and lexical matching mechanisms. It appears that blind adults' speeded speech processing emerges when phonological analysis makes promising word candidates available for further processing. As one possible interpretation, we speculate that lexical selection processes in blind adults do not need to wait for information from the visual domain, while auditory-visual integration mechanisms are mandatorily implemented in speech recognition routines of sighted adults.
Collapse
|
48
|
Erdener D, Burnham D. Auditory-visual speech perception in three- and four-year-olds and its relationship to perceptual attunement and receptive vocabulary. JOURNAL OF CHILD LANGUAGE 2018; 45:273-289. [PMID: 28585512 DOI: 10.1017/s0305000917000174] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/07/2023]
Abstract
Despite the body of research on auditory-visual speech perception in infants and schoolchildren, development in the early childhood period remains relatively uncharted. In this study, English-speaking children between three and four years of age were investigated for: (i) the development of visual speech perception - lip-reading and visual influence in auditory-visual integration; (ii) the development of auditory speech perception and native language perceptual attunement; and (iii) the relationship between these and a language skill relevant at this age, receptive vocabulary. Visual speech perception skills improved even over this relatively short time period. However, regression analyses revealed that vocabulary was predicted by auditory-only speech perception, and native language attunement, but not by visual speech perception ability. The results suggest that, in contrast to infants and schoolchildren, in three- to four-year-olds the relationship between speech perception and language ability is based on auditory and not visual or auditory-visual speech perception ability. Adding these results to existing findings allows elaboration of a more complete account of the developmental course of auditory-visual speech perception.
Collapse
Affiliation(s)
- Doğu Erdener
- Psychology Program,Middle East Technical University,Northern Cyprus Campus,Güzelyurt/Morphou,Northern Cyprus
| | - Denis Burnham
- The MARCS Institute for Brain,Behaviour and Development,Western Sydney University,Australia
| |
Collapse
|
49
|
Jerger S, Damian MF, McAlpine RP, Abdi H. Visual speech fills in both discrimination and identification of non-intact auditory speech in children. JOURNAL OF CHILD LANGUAGE 2018; 45:392-414. [PMID: 28724465 PMCID: PMC5775942 DOI: 10.1017/s0305000917000265] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/07/2023]
Abstract
To communicate, children must discriminate and identify speech sounds. Because visual speech plays an important role in this process, we explored how visual speech influences phoneme discrimination and identification by children. Critical items had intact visual speech (e.g. bæz) coupled to non-intact (excised onsets) auditory speech (signified by /-b/æz). Children discriminated syllable pairs that differed in intactness (i.e. bæz:/-b/æz) and identified non-intact nonwords (/-b/æz). We predicted that visual speech would cause children to perceive the non-intact onsets as intact, resulting in more same responses for discrimination and more intact (i.e. bæz) responses for identification in the audiovisual than auditory mode. Visual speech for the easy-to-speechread /b/ but not for the difficult-to-speechread /g/ boosted discrimination and identification (about 35-45%) in children from four to fourteen years. The influence of visual speech on discrimination was uniquely associated with the influence of visual speech on identification and receptive vocabulary skills.
Collapse
Affiliation(s)
- Susan Jerger
- School of Behavioral and Brain Sciences, GR4.1, University of Texas at Dallas, 800 W. Campbell Rd, Richardson, TX 75080
- Callier Center for Communication Disorders, 811 Synergy Park Blvd., Richardson, TX 75080
| | - Markus F. Damian
- University of Bristol, School of Experimental Psychology, 12a Priory Road, Room 1D20, Bristol BS8 1TU, United Kingdom
| | - Rachel P. McAlpine
- School of Behavioral and Brain Sciences, GR4.1, University of Texas at Dallas, 800 W. Campbell Rd, Richardson, TX 75080
- Callier Center for Communication Disorders, 811 Synergy Park Blvd., Richardson, TX 75080
| | - Hervé Abdi
- School of Behavioral and Brain Sciences, GR4.1, University of Texas at Dallas, 800 W. Campbell Rd, Richardson, TX 75080
| |
Collapse
|
50
|
Individual differences and the effect of face configuration information in the McGurk effect. Exp Brain Res 2018; 236:973-984. [PMID: 29383400 DOI: 10.1007/s00221-018-5188-4] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/08/2017] [Accepted: 01/23/2018] [Indexed: 10/18/2022]
Abstract
The McGurk effect, which denotes the influence of visual information on audiovisual speech perception, is less frequently observed in individuals with autism spectrum disorder (ASD) compared to those without it; the reason for this remains unclear. Several studies have suggested that facial configuration context might play a role in this difference. More specifically, people with ASD show a local processing bias for faces-that is, they process global face information to a lesser extent. This study examined the role of facial configuration context in the McGurk effect in 46 healthy students. Adopting an analogue approach using the Autism-Spectrum Quotient (AQ), we sought to determine whether this facial configuration context is crucial to previously observed reductions in the McGurk effect in people with ASD. Lip-reading and audiovisual syllable identification tasks were assessed via presentation of upright normal, inverted normal, upright Thatcher-type, and inverted Thatcher-type faces. When the Thatcher-type face was presented, perceivers were found to be sensitive to the misoriented facial characteristics, causing them to perceive a weaker McGurk effect than when the normal face was presented (this is known as the McThatcher effect). Additionally, the McGurk effect was weaker in individuals with high AQ scores than in those with low AQ scores in the incongruent audiovisual condition, regardless of their ability to read lips or process facial configuration contexts. Our findings, therefore, do not support the assumption that individuals with ASD show a weaker McGurk effect due to a difficulty in processing facial configuration context.
Collapse
|