Effect of Simultaneous Bilingualism on Speech Intelligibility across Different Masker Types, Modalities, and Signal-to-Noise Ratios in School-Age Children

Development of the Children's English and Spanish Speech Recognition Test: Psychometric Properties, Feasibility, Reliability, and Normative Data

OBJECTIVES

The Children's English and Spanish Speech Recognition (ChEgSS) test is a computer-based tool for assessing closed-set word recognition in English and in Spanish, with a masker that is either speech-shaped noise or competing speech. The present study was conducted to (1) characterize the psychometric properties of the ChEgSS test, (2) evaluate feasibility and reliability for a large cohort of Spanish/English bilingual children with normal hearing, and (3) establish normative data.

DESIGN

Three experiments were conducted to evaluate speech perception in children (4-17 years) and adults (19-40 years) with normal hearing using the ChEgSS test. In Experiment 1, data were collected from Spanish/English bilingual and English monolingual adults at multiple, fixed signal-to-noise ratios. Psychometric functions were fitted to the word-level data to characterize variability across target words in each language and in each masker condition. In Experiment 2, Spanish/English bilingual adults were tested using an adaptive tracking procedure to evaluate the influence of different target-word normalization approaches on the reliability of estimates of masked-speech recognition thresholds corresponding to 70.7% correct word recognition and to determine the optimal number of reversals needed to obtain reliable estimates. In Experiment 3, Spanish/English bilingual and English monolingual children completed speech perception testing using the ChEgSS test to (1) characterize feasibility across age and language group, (2) evaluate test-retest reliability, and (3) establish normative data.

RESULTS

Experiments 1 and 2 yielded data that are essential for stimulus normalization, optimizing threshold estimation procedures, and interpreting threshold data across test language and masker type. Findings obtained from Spanish/English bilingual and English monolingual children with normal hearing in Experiment 3 support feasibility and demonstrate reliability for use with children as young as 4 years of age. Equivalent results for testing in English and Spanish were observed for Spanish/English bilingual children, contingent on adequate proficiency in the target language. Regression-based threshold norms were established for Spanish/English bilingual and English monolingual children between 4 and 17 years of age.

CONCLUSIONS

The present findings indicate the ChEgSS test is appropriate for testing a wide age range of children with normal hearing in either Spanish, English, or both languages. The ChEgSS test is currently being evaluated in a large cohort of patients with hearing loss at pediatric audiology clinics across the United States. Results will be compared with normative data established in the present study and with established clinical measures used to evaluate English- and Spanish-speaking children. Questionnaire data from parents and clinician feedback will be used to further improve test procedures.

Exploring audiovisual speech perception in monolingual and bilingual children in Uzbekistan

This study aimed to investigate the development of audiovisual speech perception in monolingual Uzbek-speaking and bilingual Uzbek-Russian-speaking children, focusing on the impact of language experience on audiovisual speech perception and the role of visual phonetic (i.e., mouth movements corresponding to phonetic/lexical information) and temporal (i.e., timing of speech signals) cues. A total of 321 children aged 4 to 10 years in Tashkent, Uzbekistan, discriminated /ba/ and /da/ syllables across three conditions: auditory-only, audiovisual phonetic (i.e., sound accompanied by mouth movements), and audiovisual temporal (i.e., sound onset/offset accompanied by mouth opening/closing). Effects of modality (audiovisual phonetic, audiovisual temporal, or audio-only cues), age, group (monolingual or bilingual), and their interactions were tested using a Bayesian regression model. Overall, older participants performed better than younger participants. Participants performed better in the audiovisual phonetic modality compared with the auditory modality. However, no significant difference between monolingual and bilingual children was observed across all modalities. This finding stands in contrast to earlier studies. We attribute the contrasting findings of our study and the existing literature to the cross-linguistic similarity of the language pairs involved. When the languages spoken by bilinguals exhibit substantial linguistic similarity, there may be an increased necessity to disambiguate speech signals, leading to a greater reliance on audiovisual cues. The limited phonological similarity between Uzbek and Russian might have minimized bilinguals' need to rely on visual speech cues, contributing to the lack of group differences in our study.

The Involvement of Listening Effort in Explaining Bilingual Listening Under Adverse Listening Conditions

The current review examines listening effort to uncover how it is implicated in bilingual performance under adverse listening conditions. Various measures of listening effort, including physiological, behavioral, and subjective measures, have been employed to examine listening effort in bilingual children and adults. Adverse listening conditions, stemming from environmental factors, as well as factors related to the speaker or listener, have been examined. The existing literature, although relatively limited to date, points to increased listening effort among bilinguals in their nondominant second language (L2) compared to their dominant first language (L1) and relative to monolinguals. Interestingly, increased effort is often observed even when speech intelligibility remains unaffected. These findings emphasize the importance of considering listening effort alongside speech intelligibility. Building upon the insights gained from the current review, we propose that various factors may modulate the observed effects. These include the particular measure selected to examine listening effort, the characteristics of the adverse condition, as well as factors related to the particular linguistic background of the bilingual speaker. Critically, further research is needed to better understand the impact of these factors on listening effort. The review outlines avenues for future research that would promote a comprehensive understanding of listening effort in bilingual individuals.

Masked-Speech Recognition for Linguistically Diverse Populations: A Focused Review and Suggestions for the Future

PURPOSE

Twenty years ago, von Hapsburg and Peña (2002) wrote a tutorial that reviewed the literature on speech audiometry and bilingualism and outlined valuable recommendations to increase the rigor of the evidence base. This review article returns to that seminal tutorial to reflect on how that advice was applied over the last 20 years and to provide updated recommendations for future inquiry.

METHOD

We conducted a focused review of the literature on masked-speech recognition for bilingual children and adults. First, we evaluated how studies published since 2002 described bilingual participants. Second, we reviewed the literature on native language masked-speech recognition. Third, we discussed theoretically motivated experimental work. Fourth, we outlined how recent research in bilingual speech recognition can be used to improve clinical practice.

RESULTS

Research conducted since 2002 commonly describes bilingual samples in terms of their language status, competency, and history. Bilingualism was not consistently associated with poor masked-speech recognition. For example, bilinguals who were exposed to English prior to age 7 years and who were dominant in English performed comparably to monolinguals for masked-sentence recognition tasks. To the best of our knowledge, there are no data to document the masked-speech recognition ability of these bilinguals in their other language compared to a second monolingual group, which is an important next step. Nonetheless, individual factors that commonly vary within bilingual populations were associated with masked-speech recognition and included language dominance, competency, and age of acquisition. We identified methodological issues in sampling strategies that could, in part, be responsible for inconsistent findings between studies. For instance, disparities in socioeconomic status (SES) between recruited bilingual and monolingual groups could cause confounding bias within the research design.

CONCLUSIONS

Dimensions of the bilingual linguistic profile should be considered in clinical practice to inform counseling and (re)habilitation strategies since susceptibility to masking is elevated in at least one language for most bilinguals. Future research should continue to report language status, competency, and history but should also report language stability and demand for use data. In addition, potential confounds (e.g., SES, educational attainment) when making group comparisons between monolinguals and bilinguals must be considered.

Effects of language experience on selective auditory attention and speech-in-noise perception among English second language learners: Preliminary findings

OBJECTIVE

The purpose of the study was to examine the effects of language experience on selective auditory attention and speech-in-noise perception in English Second Language (ESL) learners aged seven to eight years.

METHOD

A quantitative, descriptive, comparative cross-sectional research design was used to determine the effect of age of exposure to English on the selective auditory attention abilities and speech-in-noise perception skills of 40 children with normal hearing in first or second grade (aged seven to eight years). The control group comprised of 20 English first language (EFL) learners (mean age = 7.35 years ±0.49) and the research group included 20 s language learners (mean age = 7.70 years ±0.47). In order to compare the control and research groups with respect to the age of exposure to English through various sources, the Mann Whitney test was used. Information regarding the age of exposure was gathered by a case history questionnaire, completed by the parents/guardians of the participants. The Selective Auditory Attention Test (SAAT) and Digits-in-Noise (DIN) test were performed in one sitting.

RESULTS

No statistically significant differences between the EFL and ESL groups were found for the SAAT and DIN. However, a statistically significant difference was obtained between the SAAT lists 1 and 3 & the DIN: diotic listening condition for the ESL group only (r_s = -0.623; p = 0.003). The difference between the EFL and ESL groups in the mean age of exposure to English was statistically significant (p = 0,019), with mean age of exposure to English in the ESL group (mean age = 2.82 ± 0.53) being higher than the mean age of exposure in the EFL group (mean age = 1.81 ± 1.53). However, this difference did not influence the results of the SAAT and DIN significantly.

CONCLUSION

The main finding was that selective auditory attention and speech-in-noise perception were not significantly affected in the ESL learners who participated in the study - learners who were recruited from private schools located in an urban area and thus from higher socio-economic status (SES) households. There is a need for additional research with a larger sample size to determine the selective auditory attention abilities and speech-in-noise perception skills of ESL learners in government-funded schools located in rural areas and from various socio-economic backgrounds.

Bilinguals Show Proportionally Greater Benefit From Visual Speech Cues and Sentence Context in Their Second Compared to Their First Language

OBJECTIVES

Speech perception in noise is challenging, but evidence suggests that it may be facilitated by visual speech cues (e.g., lip movements) and supportive sentence context in native speakers. Comparatively few studies have investigated speech perception in noise in bilinguals, and little is known about the impact of visual speech cues and supportive sentence context in a first language compared to a second language within the same individual. The current study addresses this gap by directly investigating the extent to which bilinguals benefit from visual speech cues and supportive sentence context under similarly noisy conditions in their first and second language.

DESIGN

Thirty young adult English-French/French-English bilinguals were recruited from the undergraduate psychology program at Concordia University and from the Montreal community. They completed a speech perception in noise task during which they were presented with video-recorded sentences and instructed to repeat the last word of each sentence out loud. Sentences were presented in three different modalities: visual-only, auditory-only, and audiovisual. Additionally, sentences had one of two levels of context: moderate (e.g., "In the woods, the hiker saw a bear.") and low (e.g., "I had not thought about that bear."). Each participant completed this task in both their first and second language; crucially, the level of background noise was calibrated individually for each participant and was the same throughout the first language and second language (L2) portions of the experimental task.

RESULTS

Overall, speech perception in noise was more accurate in bilinguals' first language compared to the second. However, participants benefited from visual speech cues and supportive sentence context to a proportionally greater extent in their second language compared to their first. At the individual level, performance during the speech perception in noise task was related to aspects of bilinguals' experience in their second language (i.e., age of acquisition, relative balance between the first and the second language).

CONCLUSIONS

Bilinguals benefit from visual speech cues and sentence context in their second language during speech in noise and do so to a greater extent than in their first language given the same level of background noise. Together, this indicates that L2 speech perception can be conceptualized within an inverse effectiveness hypothesis framework with a complex interplay of sensory factors (i.e., the quality of the auditory speech signal and visual speech cues) and linguistic factors (i.e., presence or absence of supportive context and L2 experience of the listener).

A comparison of monolingual and bilingual toddlers' word recognition in noise

AIMS AND OBJECTIVES

The purpose of this study was to examine whether differences in language exposure (i.e., being raised in a bilingual versus a monolingual environment) influence young children's ability to comprehend words when speech is heard in the presence of background noise.

METHODOLOGY

Forty-four children (22 monolinguals and 22 bilinguals) between the ages of 29 and 31 months completed a preferential looking task where they saw picture-pairs of familiar objects (e.g., balloon and apple) on a screen and simultaneously heard sentences instructing them to locate one of the objects (e.g., look at the apple!). Speech was heard in quiet and in the presence of competing white noise.

DATA AND ANALYSES

Children's eye-movements were coded off-line to identify the proportion of time they fixated on the correct object on the screen and performance across groups was compared using a 2 × 3 mixed analysis of variance.

FINDINGS

Bilingual toddlers performed worse than monolinguals during the task. This group difference in performance was particularly clear when the listening condition contained background noise.

ORIGINALITY

There are clear differences in how infants and adults process speech in noise. To date, developmental work on this topic has mainly been carried out with monolingual infants. This study is one of the first to examine how background noise might influence word identification in young bilingual children who are just starting to acquire their languages.

SIGNIFICANCE

High noise levels are often reported in daycares and classrooms where bilingual children are present. Therefore, this work has important implications for learning and education practices with young bilinguals.

A Simplified Approach to Quantifying a Child's Bilingual Language Experience

Purpose Bilingual children's linguistic experience can vary markedly from child to child. For appropriate audiological assessment and intervention, audiologists need accurate and efficient ways to describe and understand a bilingual child's dynamic linguistic experience. This report documents an approach for quantitatively capturing a child's language exposure and usage in a time-efficient manner. Method A well-known pediatric bilingual language survey was administered to 83 parents of bilingual children, obtaining information about the child's exposure to (input) and usage of (output) Spanish and English for seventeen 1-hr intervals during a typical weekday and weekend day. Results A factor analysis indicated that capturing linguistic exposure and usage over three grouped-time intervals during a typical weekday and weekend day accounted for ≥ 74% of the total variance of the linguistic information captured with the full-length survey. Conclusions Although further confirmation is required, these results suggest that collecting language exposure and usage data from parents of bilingual children for three grouped-time intervals provides similar information as a comprehensive hour-by-hour approach. A time-efficient method of capturing the dynamic bilingual linguistic experience of a child would benefit pediatric audiologists and speech-language pathologists alike.

Behavioral Pattern Analysis between Bilingual and Monolingual Listeners’ Natural Speech Perception on Foreign-Accented English Language Using Different Machine Learning Approaches

Speech perception in an adverse background/noisy environment is a complex and challenging human process, which is made even more complicated in foreign-accented language for bilingual and monolingual individuals. Listeners who have difficulties in hearing are affected most by such a situation. Despite considerable efforts, the increase in speech intelligibility in noise remains elusive. Considering this opportunity, this study investigates Bengali–English bilinguals and native American English monolinguals’ behavioral patterns on foreign-accented English language considering bubble noise, gaussian or white noise, and quiet sound level. Twelve regular hearing participants (Six Bengali–English bilinguals and Six Native American English monolinguals) joined in this study. Statistical computation shows that speech with different noise has a significant effect (p = 0.009) on listening for both bilingual and monolingual under different sound levels (e.g., 55 dB, 65 dB, and 75 dB). Here, six different machine learning approaches (Logistic Regression (LR), Linear Discriminant Analysis (LDA), K-nearest neighbors (KNN), Naïve Bayes (NB), Classification and regression trees (CART), and Support vector machine (SVM)) are tested and evaluated to differentiate between bilingual and monolingual individuals from their behavioral patterns in both noisy and quiet environments. Results show that most optimal performances were observed using LDA by successfully differentiating between bilingual and monolingual 60% of the time. A deep neural network-based model is proposed to improve this measure further and achieved an accuracy of nearly 100% in successfully differentiating between bilingual and monolingual individuals.

Second-language word recognition in noise: Interdependent neuromodulatory effects of semantic context and crosslinguistic interactions driven by word form similarity

Spoken language comprehension is a fundamental component of our cognitive skills. We are quite proficient at deciphering words from the auditory input despite the fact that the speech we hear is often masked by noise such as background babble originating from talkers other than the one we are attending to. To perceive spoken language as intended, we rely on prior linguistic knowledge and context. Prior knowledge includes all sounds and words that are familiar to a listener and depends on linguistic experience. For bilinguals, the phonetic and lexical repertoire encompasses two languages, and the degree of overlap between word forms across languages affects the degree to which they influence one another during auditory word recognition. To support spoken word recognition, listeners often rely on semantic information (i.e., the words we hear are usually related in a meaningful way). Although the number of multilinguals across the globe is increasing, little is known about how crosslinguistic effects (i.e., word overlap) interact with semantic context and affect the flexible neural systems that support accurate word recognition. The current multi-echo functional magnetic resonance imaging (fMRI) study addresses this question by examining how prime-target word pair semantic relationships interact with the target word's form similarity (cognate status) to the translation equivalent in the dominant language (L1) during accurate word recognition of a non-dominant (L2) language. We tested 26 early-proficient Spanish-Basque (L1-L2) bilinguals. When L2 targets matching L1 translation-equivalent phonological word forms were preceded by unrelated semantic contexts that drive lexical competition, a flexible language control (fronto-parietal-subcortical) network was upregulated, whereas when they were preceded by related semantic contexts that reduce lexical competition, it was downregulated. We conclude that an interplay between semantic and crosslinguistic effects regulates flexible control mechanisms of speech processing to facilitate L2 word recognition, in noise.

Noise Exposure May Diminish the Musician Advantage for Perceiving Speech in Noise

OBJECTIVE

Although numerous studies have shown that musicians have better speech perception in noise (SPIN) compared to nonmusicians, other studies have not replicated the "musician advantage for SPIN." One factor that has not been adequately addressed in previous studies is how musicians' SPIN is affected by routine exposure to high levels of sound. We hypothesized that such exposure diminishes the musician advantage for SPIN.

DESIGN

Environmental sound levels were measured continuously for 1 week via body-worn noise dosimeters in 56 college students with diverse musical backgrounds and clinically normal pure-tone audiometric averages. SPIN was measured using the Quick Speech in Noise Test (QuickSIN). Multiple linear regression modeling was used to examine how music practice (years of playing a musical instrument) and routine noise exposure predict QuickSIN scores.

RESULTS

Noise exposure and music practice were both significant predictors of QuickSIN, but they had opposing influences, with more years of music practice predicting better QuickSIN scores and greater routine noise exposure predicting worse QuickSIN scores. Moreover, mediation analysis suggests that noise exposure suppresses the relationship between music practice and QuickSIN scores.

CONCLUSIONS

Our findings suggest a beneficial relationship between music practice and SPIN that is suppressed by noise exposure.

Masked English Speech Recognition Performance in Younger and Older Spanish-English Bilingual and English Monolingual Children

Purpose The purpose of this study was to compare masked English speech recognition thresholds between Spanish-English bilingual and English monolingual children and to evaluate effects of age, maternal education, and English receptive language abilities on individual differences in masked speech recognition. Method Forty-three Spanish-English bilingual children and 42 English monolingual children completed an English sentence recognition task in 2 masker conditions: (a) speech-shaped noise and (b) 2-talker English speech. Two age groups of children, younger (5-6 years) and older (9-10 years), were tested. The predictors of masked speech recognition performance were evaluated using 2 mixed-effects regression models. In the 1st model, fixed effects were age group (younger children vs. older children), language group (bilingual vs. monolingual), and masker type (speech-shaped noise vs. 2-talker speech). In the 2nd model, the fixed effects of receptive English vocabulary scores and maternal education level were also included. Results Younger children performed more poorly than older children, but no significant difference in masked speech recognition was observed between bilingual and monolingual children for either age group when English proficiency and maternal education were also included in the model. English language abilities fell within age-appropriate norms for both groups, but individual children with larger receptive vocabularies in English tended to show better recognition; this effect was stronger for younger children than for older children. Speech reception thresholds for all children were lower in the speech-shaped noise masker than in the 2-talker speech masker. Conclusions Regardless of age, similar masked speech recognition was observed for Spanish-English bilingual and English monolingual children tested in this study when receptive English language abilities were accounted for. Receptive English vocabulary scores were associated with better masked speech recognition performance for both bilinguals and monolinguals, with a stronger relationship observed for younger children than older children. Further investigation involving a Spanish-dominant bilingual sample is warranted given the high English language proficiency of children included in this study.

Effects of Language History on Sentence Recognition in Noise or Two-Talker Speech: Monolingual, Early Bilingual, and Late Bilingual Speakers of English

Purpose Language history is an important factor in masked speech recognition. Listeners who acquire the target language later in life perform more poorly than native speakers. However, there are inconsistencies in the literature regarding performance of bilingual speakers who begin learning the target language early in life. The purpose of this experiment was to evaluate speech-in-noise and speech-in-speech recognition for highly proficient early bilingual listeners compared to monolingual and late bilingual listeners. Method Three groups of young adults participated: native monolingual English speakers, bilingual Mandarin-English speakers who learned English from birth (early bilinguals), and native Mandarin speakers who learned English later in life (late bilinguals). All participants had normal hearing and were full-time college students. Recognition was assessed for English sentences in speech-shaped noise and two-talker English speech. Participants provided linguistic and demographic information, and late bilinguals completed the Versant test of spoken English abilities. Results All listeners performed better in speech-shaped noise than two-talker speech. Performance was similar for monolingual and early bilinguals. Late bilinguals performed more poorly overall. There was evidence for a stronger association between masked speech recognition and English dominance for late bilinguals compared to early bilinguals. Conclusion These results support the conclusion that bilingualism itself does not necessarily result in a disadvantage when recognizing masked speech in noise and speech in speech. For populations similar to those studied here (highly proficient early bilinguals), it would be appropriate to evaluate masked speech recognition using the same simple stimuli and normative data used for monolingual speakers of English.

Intelligibility of Children With Speech Sound Disorders Evaluated by Listeners With Swedish as a Second Language

Purpose This study aimed to investigate the intelligibility of children's atypical speech in relation to listeners' language background. Method Forty-eight participants listened to and transcribed isolated words repeated by children with speech sound disorders. Participants were divided into, on the one hand, a multilingual group (n = 29) that was further divided into subgroups based on age of acquisition (early, 0-3 years; intermediate, 4-12 years; and late, > 12 years) and, on the other hand, a monolingual comparison group (n = 19). Results The monolingual listeners obtained higher intelligibility scores than the multilingual listeners; this difference was statistically significant. Participants who acquired Swedish at an older age (> 4 years) were found to have lower scores than other listeners. The later the age of acquisition, the less of the atypical speech was decoded correctly. A further analysis of the transcriptions also revealed a higher level of nonwords among the incorrect transcriptions of the multilinguals than that of the monolinguals who used more real words, whereas both groups were equally prone to using blanks when they did not perceive a word. Conclusions This indicates a higher risk of communicative problems between late acquirers of Swedish and children with speech sound disorders. Clinical implications, such as involving communication partners in the intervention process, are discussed as well as possible linguistic explanations to the findings. This study could be seen as a starting point in the field of research regarding the relations between the language background of the listener and the ability to perceive atypical speech.

Language learning experience and mastering the challenges of perceiving speech in noise

Given the ubiquity of noisy environments and increasing globalization, the necessity to perceive speech in noise in a non-native language is common and necessary for successful communication. In the current investigation, bilingual individuals who learned their non-native language at different ages underwent magnetic resonance imaging while listening to sentences in both of their languages, in quiet and in noise. Sentence context was varied such that the final word could be of high or low predictability. Results show that early non-native language learning is associated with superior ability to benefit from contextual information behaviourally, and a pattern of neural recruitment in the left inferior frontal gyrus that suggests easier processing when perceiving non-native speech in noise. These findings have implications for our understanding of speech processing in non-optimal listening conditions and shed light on how individuals navigate every day complex communicative environments, in a native and non-native language.

Impact of depression on speech perception in noise

Effective speech communication is critical to everyday quality of life and social well-being. In addition to the well-studied deficits in cognitive and motor function, depression also impacts communication. Here, we examined speech perception in individuals who were clinically diagnosed with major depressive disorder (MDD) relative to neurotypical controls. Forty-two normal-hearing (NH) individuals with MDD and 41 NH neurotypical controls performed sentence recognition tasks across three conditions with maskers varying in the extent of linguistic content (high, low, and none): 1-talker masker (1T), reversed 1-talker masker (1T_tr), and speech-shaped noise (SSN). Individuals with MDD, relative to neurotypical controls, demonstrated lower recognition accuracy in the 1T condition but not in the 1T_tr or SSN condition. To examine the nature of the listening condition-specific speech perception deficit, we analyzed speech recognition errors. Errors as a result of interference from masker sentences were higher for individuals with MDD (vs. neurotypical controls) in the 1T condition. This depression-related listening condition-specific pattern in recognition errors was not observed for other error types. We posit that this depression-related listening condition-specific deficit in speech perception may be related to heightened distractibility due to linguistic interference from background talkers.

Psychobiological Responses Reveal Audiovisual Noise Differentially Challenges Speech Recognition

OBJECTIVES

In noisy environments, listeners benefit from both hearing and seeing a talker, demonstrating audiovisual (AV) cues enhance speech-in-noise (SIN) recognition. Here, we examined the relative contribution of auditory and visual cues to SIN perception and the strategies used by listeners to decipher speech in noise interference(s).

DESIGN

Normal-hearing listeners (n = 22) performed an open-set speech recognition task while viewing audiovisual TIMIT sentences presented under different combinations of signal degradation including visual (AVn), audio (AnV), or multimodal (AnVn) noise. Acoustic and visual noises were matched in physical signal-to-noise ratio. Eyetracking monitored participants' gaze to different parts of a talker's face during SIN perception.

RESULTS

As expected, behavioral performance for clean sentence recognition was better for A-only and AV compared to V-only speech. Similarly, with noise in the auditory channel (AnV and AnVn speech), performance was aided by the addition of visual cues of the talker regardless of whether the visual channel contained noise, confirming a multimodal benefit to SIN recognition. The addition of visual noise (AVn) obscuring the talker's face had little effect on speech recognition by itself. Listeners' eye gaze fixations were biased toward the eyes (decreased at the mouth) whenever the auditory channel was compromised. Fixating on the eyes was negatively associated with SIN recognition performance. Eye gazes on the mouth versus eyes of the face also depended on the gender of the talker.

CONCLUSIONS

Collectively, results suggest listeners (1) depend heavily on the auditory over visual channel when seeing and hearing speech and (2) alter their visual strategy from viewing the mouth to viewing the eyes of a talker with signal degradations, which negatively affects speech perception.

Acoustic noise and vision differentially warp the auditory categorization of speech

Speech perception requires grouping acoustic information into meaningful linguistic-phonetic units via categorical perception (CP). Beyond shrinking observers' perceptual space, CP might aid degraded speech perception if categories are more resistant to noise than surface acoustic features. Combining audiovisual (AV) cues also enhances speech recognition, particularly in noisy environments. This study investigated the degree to which visual cues from a talker (i.e., mouth movements) aid speech categorization amidst noise interference by measuring participants' identification of clear and noisy speech (0 dB signal-to-noise ratio) presented in auditory-only or combined AV modalities (i.e., A, A+noise, AV, AV+noise conditions). Auditory noise expectedly weakened (i.e., shallower identification slopes) and slowed speech categorization. Interestingly, additional viseme cues largely counteracted noise-related decrements in performance and stabilized classification speeds in both clear and noise conditions suggesting more precise acoustic-phonetic representations with multisensory information. Results are parsimoniously described under a signal detection theory framework and by a reduction (visual cues) and increase (noise) in the precision of perceptual object representation, which were not due to lapses of attention or guessing. Collectively, findings show that (i) mapping sounds to categories aids speech perception in "cocktail party" environments; (ii) visual cues help lattice formation of auditory-phonetic categories to enhance and refine speech identification.

The Music-In-Noise Task (MINT): A Tool for Dissecting Complex Auditory Perception

The ability to segregate target sounds in noisy backgrounds is relevant both to neuroscience and to clinical applications. Recent research suggests that hearing-in-noise (HIN) problems are solved using combinations of sub-skills that are applied according to task demand and information availability. While evidence is accumulating for a musician advantage in HIN, the exact nature of the reported training effect is not fully understood. Existing HIN tests focus on tasks requiring understanding of speech in the presence of competing sound. Because visual, spatial and predictive cues are not systematically considered in these tasks, few tools exist to investigate the most relevant components of cognitive processes involved in stream segregation. We present the Music-In-Noise Task (MINT) as a flexible tool to expand HIN measures beyond speech perception, and for addressing research questions pertaining to the relative contributions of HIN sub-skills, inter-individual differences in their use, and their neural correlates. The MINT uses a match-mismatch trial design: in four conditions (Baseline, Rhythm, Spatial, and Visual) subjects first hear a short instrumental musical excerpt embedded in an informational masker of "multi-music" noise, followed by either a matching or scrambled repetition of the target musical excerpt presented in silence; the four conditions differ according to the presence or absence of additional cues. In a fifth condition (Prediction), subjects hear the excerpt in silence as a target first, which helps to anticipate incoming information when the target is embedded in masking sound. Data from samples of young adults show that the MINT has good reliability and internal consistency, and demonstrate selective benefits of musicianship in the Prediction, Rhythm, and Visual subtasks. We also report a performance benefit of multilingualism that is separable from that of musicianship. Average MINT scores were correlated with scores on a sentence-in-noise perception task, but only accounted for a relatively small percentage of the variance, indicating that the MINT is sensitive to additional factors and can provide a complement and extension of speech-based tests for studying stream segregation. A customizable version of the MINT is made available for use and extension by the scientific community.

Neural Correlates of Enhanced Audiovisual Processing in the Bilingual Brain

Bilingualism is associated with enhancements in perceptual and cognitive processing necessary for juggling multiple languages. Recent psychophysical studies demonstrate bilinguals also show enhanced multisensory processing and more restricted temporal binding windows for integrating audiovisual information. Here, we probed the neural mechanisms of bilinguals' audiovisual benefits. We recorded neuroelectric responses in mono- and bi-lingual listeners to the double-flash paradigm in which auditory beeps concurrent with a single visual flash induces the perceptual illusion of multiple flashes. Relative to monolinguals, bilinguals showed less susceptibility to the illusion (fewer false perceptual reports) coupled with stronger and faster event-related potentials to audiovisual information. Source analyses of EEG data revealed monolinguals' increased propensity for erroneously perceiving audiovisual stimuli was attributed to increased activity in primary visual (V1) and auditory cortex (PAC), increases in multisensory association areas (BA 37), but reduced frontal activity (BA 10). Regional activations were associated with an opposite pattern of behaviors: whereas stronger V1 and PAC activity predicted slower behavioral responses, stronger frontal BA10 responses elicited faster judgments. Our results suggest bilinguals' higher precision in audiovisual perception reflects more veridical sensory coding of physical cues coupled with superior top-down gating of sensory information to suppress the generation of false percepts. Findings underscore that the plasticity afforded by speaking multiple languages shapes extra-linguistic brain regions and can enhance audiovisual brain processing in a domain-general manner.