Use of audiovisual information in speech perception by prelingually deaf children with cochlear implants: a first report

Cross-modal digit span and vocabulary proficiency in deaf or hard-of-hearing children

This study aims to examine the relationship between vocabulary proficiency and short-term memory capacity in deaf or hard-of-hearing (DHH) children. We test the hypothesis that the relationship between vocabulary skills and digit span performance could be strengthened when the digit span task encompasses cross-modal integration processes. A group of DHH children performed two types of auditory digit span tasks. Furthermore, they participated in a standardized vocabulary proficiency test, comprising two subtests: Receptive Vocabulary and Expressive Vocabulary. The verbal digit span served as a significant predictor of Expressive Vocabulary among the DHH children. Simultaneously, the auditory-pointing digit span accounted for a substantial portion of performance variation in both Receptive and Expressive Vocabulary. After considering the impact of the duration of auditory-verbal intervention through regression models, likelihood ratio tests demonstrated that the auditory-pointing digit span persisted as a significant determinant of both receptive and expressive vocabulary skills. A positive influence of the intervention was also confirmed by the present results. This study provides evidence that memory span and the ability to integrate cross-modal information could serve as significant cognitive correlates of vocabulary proficiency in DHH children.

Prosodic and Visual Cues Facilitate Irony Comprehension by Mandarin-Speaking Children With Cochlear Implants

PURPOSE

This study investigated irony comprehension by Mandarin-speaking children with cochlear implants, focusing on how prosodic and visual cues contribute to their comprehension, and whether second-order Theory of Mind is required for using these cues.

METHOD

We tested 52 Mandarin-speaking children with cochlear implants (aged 3-7 years) and 52 age- and gender-matched children with normal hearing. All children completed a Theory of Mind test and a story comprehension test. Ironic stories were presented in three conditions, each providing different cues: (a) context-only, (b) context and prosody, and (c) context, prosody, and visual cues. Comparisons were conducted on the accuracy of story understanding across the three conditions to examine the role of prosodic and visual cues.

RESULTS

The results showed that, compared to the context-only condition, the additional prosodic and visual cues both improved the accuracy of irony comprehension for children with cochlear implants, similar to their normal-hearing peers. Furthermore, such improvements were observed for all children, regardless of whether they passed the second-order Theory of Mind test or not.

CONCLUSIONS

This study is the first to demonstrate the benefits of prosodic and visual cues in irony comprehension, without reliance on second-order Theory of Mind, for Mandarin-speaking children with cochlear implants. It implies potential insights for utilizing prosodic and visual cues in intervention strategies to promote irony comprehension.

SLPs' perceptions of language learning myths about children who are DHH

This article reports on speech-language pathologists' (SLPs') knowledge related to myths about spoken language learning of children who are deaf and hard of hearing (DHH). The broader study was designed as a step toward narrowing the research-practice gap and providing effective, evidence-based language services to children. In the broader study, SLPs (n = 106) reported their agreement/disagreement with myth statements and true statements (n = 52) about 7 clinical topics related to speech and language development. For the current report, participant responses to 7 statements within the DHH topic were analyzed. Participants exhibited a relative strength in bilingualism knowledge for spoken languages and a relative weakness in audiovisual integration knowledge. Much individual variation was observed. Participants' responses were more likely to align with current evidence about bilingualism if the participants had less experience as an SLP. The findings provide guidance on prioritizing topics for speech-language pathology preservice and professional development.

Orthographic Learning in French-Speaking Deaf and Hard of Hearing Children

PURPOSE

Children are assumed to acquire orthographic representations during autonomous reading by decoding new written words. The present study investigates how deaf and hard of hearing (DHH) children build new orthographic representations compared to typically hearing (TH) children.

METHOD

Twenty-nine DHH children, from 7.8 to 13.5 years old, with moderate-to-profound hearing loss, matched for reading level and chronological age to TH controls, were exposed to 10 pseudowords (novel words) in written stories. Then, they performed a spelling task and an orthographic recognition task on these new words.

RESULTS

In the spelling task, we found no difference in accuracy, but a difference in errors emerged between the two groups: Phonologically plausible errors were less common in DHH children than in TH children. In the recognition task, DHH children were better than TH children at recognizing target pseudowords. Phonological strategies seemed to be used less by DHH than by TH children who very often chose phonological distractors.

CONCLUSIONS

Both groups created sufficiently detailed orthographic representations to complete the tasks, which support the self-teaching hypothesis. DHH children used phonological information in both tasks but could use more orthographic cues than TH children to build up orthographic representations. Using the combination of a spelling task and a recognition task, as well as analyzing the nature of errors, in this study, provides a methodological implication for further understanding of underlying cognitive processes.

Voice Onset Time of Greek Stops Productions by Greek Children with Cochlear Implants and Normal Hearing

INTRODUCTION

Research on voice onset time (VOT) production of stops in children with CI versus NH has reported conflicting results. Effects of age and place of articulation on VOT have not been examined for children with CI. The purpose of this study was to examine VOT production by Greek-speaking children with CI in comparison to NH controls, with a focus on the effects of age, type of stimuli, and place of articulation.

METHODS

Participants were 24 children with CI aged from 2;8 to 13;3 years and 24 age- and gender-matched children with NH. Words were elicited via a picture-naming task, and nonwords were elicited via a fast mapping procedure.

RESULTS

For voiced stops, children with CI showed longer VOT than children with NH, whereas VOT for voiceless stops was similar to that of NH peers. Also, in both voiced and voiceless stops, the VOT differed as a function of age and place of articulation across groups. Differences as a function of stimulus type were only noted for voiced stops across groups.

CONCLUSIONS

For the voiced stop consonants, which demand more articulatory effort, VOT production in children with CI was longer than in children with NH. For the voiceless stop consonants, VOT production in children with CI is acquired at a young age.

Event-Related Potentials in Assessing Visual Speech Cues in the Broader Autism Phenotype: Evidence from a Phonemic Restoration Paradigm

Audiovisual speech perception includes the simultaneous processing of auditory and visual speech. Deficits in audiovisual speech perception are reported in autistic individuals; however, less is known regarding audiovisual speech perception within the broader autism phenotype (BAP), which includes individuals with elevated, yet subclinical, levels of autistic traits. We investigate the neural indices of audiovisual speech perception in adults exhibiting a range of autism-like traits using event-related potentials (ERPs) in a phonemic restoration paradigm. In this paradigm, we consider conditions where speech articulators (mouth and jaw) are present (AV condition) and obscured by a pixelated mask (PX condition). These two face conditions were included in both passive (simply viewing a speaking face) and active (participants were required to press a button for a specific consonant-vowel stimulus) experiments. The results revealed an N100 ERP component which was present for all listening contexts and conditions; however, it was attenuated in the active AV condition where participants were able to view the speaker's face, including the mouth and jaw. The P300 ERP component was present within the active experiment only, and significantly greater within the AV condition compared to the PX condition. This suggests increased neural effort for detecting deviant stimuli when visible articulation was present and visual influence on perception. Finally, the P300 response was negatively correlated with autism-like traits, suggesting that higher autistic traits were associated with generally smaller P300 responses in the active AV and PX conditions. The conclusions support the finding that atypical audiovisual processing may be characteristic of the BAP in adults.

Development of Sound Localization in Infants and Young Children with Cochlear Implants

Cochlear implantation as a treatment for severe-to-profound hearing loss allows children to develop hearing, speech, and language in many cases. However, cochlear implants are generally provided beyond the infant period and outcomes are assessed after years of implant use, making comparison with normal development difficult. The aim was to study whether the rate of improvement of horizontal localization accuracy in children with bilateral implants is similar to children with normal hearing. A convenience sample of 20 children with a median age at simultaneous bilateral implantation = 0.58 years (0.42−2.3 years) participated in this cohort study. Longitudinal follow-up of sound localization accuracy for an average of ≈1 year generated 42 observations at a mean age = 1.5 years (0.58−3.6 years). The rate of development was compared to historical control groups including children with normal hearing and with relatively late bilateral implantation (≈4 years of age). There was a significant main effect of time with bilateral implants on localization accuracy (slope = 0.21/year, R2 = 0.25, F = 13.6, p < 0.001, n = 42). No differences between slopes (F = 0.30, p = 0.58) or correlation coefficients (Cohen’s q = 0.28, p = 0.45) existed when comparing children with implants and normal hearing (slope = 0.16/year since birth, p = 0.015, n = 12). The rate of development was identical to children implanted late. Results suggest that early bilateral implantation in children with severe-to-profound hearing loss allows development of sound localization at a similar age to children with normal hearing. Similar rates in children with early and late implantation and normal hearing suggest an intrinsic mechanism for the development of horizontal sound localization abilities.

Congruent aero-tactile stimuli bias perception of voicing continua

Multimodal integration is the formation of a coherent percept from different sensory inputs such as vision, audition, and somatosensation. Most research on multimodal integration in speech perception has focused on audio-visual integration. In recent years, audio-tactile integration has also been investigated, and it has been established that puffs of air applied to the skin and timed with listening tasks shift the perception of voicing by naive listeners. The current study has replicated and extended these findings by testing the effect of air puffs on gradations of voice onset time along a continuum rather than the voiced and voiceless endpoints of the original work. Three continua were tested: bilabial (“pa/ba”), velar (“ka/ga”), and a vowel continuum (“head/hid”) used as a control. The presence of air puffs was found to significantly increase the likelihood of choosing voiceless responses for the two VOT continua but had no effect on choices for the vowel continuum. Analysis of response times revealed that the presence of air puffs lengthened responses for intermediate (ambiguous) stimuli and shortened them for endpoint (non-ambiguous) stimuli. The slowest response times were observed for the intermediate steps for all three continua, but for the bilabial continuum this effect interacted with the presence of air puffs: responses were slower in the presence of air puffs, and faster in their absence. This suggests that during integration auditory and aero-tactile inputs are weighted differently by the perceptual system, with the latter exerting greater influence in those cases where the auditory cues for voicing are ambiguous.

Beyond Recognition: Visual Contributions to Verbal Working Memory

PURPOSE

It is well recognized that adding the visual to the acoustic speech signal improves recognition when the acoustic signal is degraded, but how that visual signal affects postrecognition processes is not so well understood. This study was designed to further elucidate the relationships among auditory and visual codes in working memory, a postrecognition process.

DESIGN

In a main experiment, 80 young adults with normal hearing were tested using an immediate serial recall paradigm. Three types of signals were presented (unprocessed speech, vocoded speech, and environmental sounds) in three conditions (audio-only, audio-video with dynamic visual signals, and audio-picture with static visual signals). Three dependent measures were analyzed: (a) magnitude of the recency effect, (b) overall recall accuracy, and (c) response times, to assess cognitive effort. In a follow-up experiment, 30 young adults with normal hearing were tested largely using the same procedures, but with a slight change in order of stimulus presentation.

RESULTS

The main experiment produced three major findings: (a) unprocessed speech evoked a recency effect of consistent magnitude across conditions; vocoded speech evoked a recency effect of similar magnitude to unprocessed speech only with dynamic visual (lipread) signals; environmental sounds never showed a recency effect. (b) Dynamic and static visual signals enhanced overall recall accuracy to a similar extent, and this enhancement was greater for vocoded speech and environmental sounds than for unprocessed speech. (c) All visual signals reduced cognitive load, except for dynamic visual signals with environmental sounds. The follow-up experiment revealed that dynamic visual (lipread) signals exerted their effect on the vocoded stimuli by enhancing phonological quality.

CONCLUSIONS

Acoustic and visual signals can combine to enhance working memory operations, but the source of these effects differs for phonological and nonphonological signals. Nonetheless, visual information can support better postrecognition processes for patients with hearing loss.

Estimating the reduced benefit of infant-directed speech in cochlear implant-related speech processing

Caregivers modify their speech when talking to infants, a specific type of speech known as infant-directed speech (IDS). This speaking style facilitates language learning compared to adult-directed speech (ADS) in infants with normal hearing (NH). While infants with NH and those with cochlear implants (CIs) prefer listening to IDS over ADS, it is yet unknown how CI processing may affect the acoustic distinctiveness between ADS and IDS, as well as the degree of intelligibility of these. This study analyzed speech of seven female adult talkers to model the effects of simulated CI processing on (1) acoustic distinctiveness between ADS and IDS, (2) estimates of intelligibility of caregivers' speech in ADS and IDS, and (3) individual differences in caregivers' ADS-to-IDS modification and estimated speech intelligibility. Results suggest that CI processing is substantially detrimental to the acoustic distinctiveness between ADS and IDS, as well as to the intelligibility benefit derived from ADS-to-IDS modifications. Moreover, the observed variability across individual talkers in acoustic implementation of ADS-to-IDS modification and the estimated speech intelligibility was significantly reduced due to CI processing. The findings are discussed in the context of the link between IDS and language learning in infants with CIs.

Feasibility of Virtual Reality Audiological Testing: Prospective Study

Background

It has been noted in the literature that there is a gap between clinical assessment and real-world performance. Real-world conversations entail visual and audio information, yet there are not any audiological assessment tools that include visual information. Virtual reality (VR) technology has been applied to various areas, including audiology. However, the use of VR in speech-in-noise perception has not yet been investigated.

Objective

The purpose of this study was to investigate the impact of virtual space (VS) on speech performance and its feasibility to be used as a speech test instrument. We hypothesized that individuals’ ability to recognize speech would improve when visual cues were provided.

Methods

A total of 30 individuals with normal hearing and 25 individuals with hearing loss completed pure-tone audiometry and the Korean version of the Hearing in Noise Test (K-HINT) under three conditions—conventional K-HINT (cK-HINT), VS on PC (VSPC), and VS head-mounted display (VSHMD)—at –10 dB, –5 dB, 0 dB, and +5 dB signal-to-noise ratios (SNRs). Participants listened to target speech and repeated it back to the tester for all conditions. Hearing aid users in the hearing loss group completed testing under unaided and aided conditions. A questionnaire was administered after testing to gather subjective opinions on the headset, the VSHMD condition, and test preference.

Results

Provision of visual information had a significant impact on speech performance between the normal hearing and hearing impaired groups. The Mann-Whitney U test showed statistical significance (P<.05) between the two groups under all test conditions. Hearing aid use led to better integration of audio and visual cues. Statistical significance through the Mann-Whitney U test was observed for –5 dB (P=.04) and 0 dB (P=.02) SNRs under the cK-HINT condition, as well as for –10 dB (P=.007) and 0 dB (P=.04) SNRs under the VSPC condition, between hearing aid and non–hearing aid users. Participants reported positive responses across almost all items on the questionnaire except for the weight of the headset. Participants preferred a test method with visual imagery, but found the headset to be heavy.

Conclusions

Findings are in line with previous literature that showed that visual cues were beneficial for communication. This is the first study to include hearing aid users with a more naturalistic stimulus and a relatively simple test environment, suggesting the feasibility of VR audiological testing in clinical practice.

Audiovisual Enhancement of Speech Perception in Noise by School-Age Children Who Are Hard of Hearing

OBJECTIVES

The purpose of this study was to examine age- and hearing-related differences in school-age children's benefit from visual speech cues. The study addressed three questions: (1) Do age and hearing loss affect degree of audiovisual (AV) speech enhancement in school-age children? (2) Are there age- and hearing-related differences in the mechanisms underlying AV speech enhancement in school-age children? (3) What cognitive and linguistic variables predict individual differences in AV benefit among school-age children?

DESIGN

Forty-eight children between 6 and 13 years of age (19 with mild to severe sensorineural hearing loss; 29 with normal hearing) and 14 adults with normal hearing completed measures of auditory and AV syllable detection and/or sentence recognition in a two-talker masker type and a spectrally matched noise. Children also completed standardized behavioral measures of receptive vocabulary, visuospatial working memory, and executive attention. Mixed linear modeling was used to examine effects of modality, listener group, and masker on sentence recognition accuracy and syllable detection thresholds. Pearson correlations were used to examine the relationship between individual differences in children's AV enhancement (AV-auditory-only) and age, vocabulary, working memory, executive attention, and degree of hearing loss.

RESULTS

Significant AV enhancement was observed across all tasks, masker types, and listener groups. AV enhancement of sentence recognition was similar across maskers, but children with normal hearing exhibited less AV enhancement of sentence recognition than adults with normal hearing and children with hearing loss. AV enhancement of syllable detection was greater in the two-talker masker than the noise masker, but did not vary significantly across listener groups. Degree of hearing loss positively correlated with individual differences in AV benefit on the sentence recognition task in noise, but not on the detection task. None of the cognitive and linguistic variables correlated with individual differences in AV enhancement of syllable detection or sentence recognition.

CONCLUSIONS

Although AV benefit to syllable detection results from the use of visual speech to increase temporal expectancy, AV benefit to sentence recognition requires that an observer extracts phonetic information from the visual speech signal. The findings from this study suggest that all listener groups were equally good at using temporal cues in visual speech to detect auditory speech, but that adults with normal hearing and children with hearing loss were better than children with normal hearing at extracting phonetic information from the visual signal and/or using visual speech information to access phonetic/lexical representations in long-term memory. These results suggest that standard, auditory-only clinical speech recognition measures likely underestimate real-world speech recognition skills of children with mild to severe hearing loss.

Listening Difficulties of Children With Cochlear Implants in Mainstream Secondary Education

OBJECTIVES

Previous research has shown that children with cochlear implants (CIs) encounter more communication difficulties than their normal-hearing (NH) peers in kindergarten and elementary schools. Yet, little is known about the potential listening difficulties that children with CIs may experience during secondary education. The aim of this study was to investigate the listening difficulties of children with a CI in mainstream secondary education and to compare these results to the difficulties of their NH peers and the difficulties observed by their teachers.

DESIGN

The Dutch version of the Listening Inventory for Education Revised (LIFE-R) was administered to 19 children (mean age = 13 years 9 months; SD = 9 months) who received a CI early in life, to their NH classmates (n = 239), and to their teachers (n = 18). All participants were enrolled in mainstream secondary education in Flanders (first to fourth grades). The Listening Inventory for Secondary Education consists of 15 typical listening situations as experienced by students (LIFEstudent) during class activities (LIFEclass) and during social activities at school (LIFEsocial). The teachers completed a separate version of the Listening Inventory for Secondary Education (LIFEteacher) and Screening Instrument for Targeting Educational Risk.

RESULTS

Participants with CIs reported significantly more listening difficulties than their NH peers. A regression model estimated that 75% of the participants with CIs were at risk of experiencing listening difficulties. The chances of experiencing listening difficulties were significantly higher in participants with CIs for 7 out of 15 listening situations. The 3 listening situations that had the highest chance of resulting in listening difficulties were (1) listening during group work, (2) listening to multimedia, and (3) listening in large-sized classrooms. Results of the teacher's questionnaires (LIFEteacher and Screening Instrument for Targeting Educational Risk) did not show a similar significant difference in listening difficulties between participants with a CI and their NH peers. According to teachers, NH participants even obtained significantly lower scores for staying on task and for participation in class than participants with a CI.

CONCLUSIONS

Although children with a CI seemingly fit in well in mainstream schools, they still experience significantly more listening difficulties than their NH peers. Low signal to noise ratios (SNRs), distortions of the speech signal (multimedia, reverberation), distance, lack of visual support, and directivity effects of the microphones were identified as difficulties for children with a CI in the classroom. As teachers may not always notice these listening difficulties, a list of practical recommendations was provided in this study, to raise awareness among teachers and to minimize the difficulties.

Sound localization latency in normal hearing and simulated unilateral hearing loss

Directing gaze towards auditory events is a natural behavior. In addition to the well-known accuracy of auditory elicited gaze responses for normal binaural listening, their latency is a measure of possible clinical interest and methodological importance. The aim was to develop a clinically feasible method to assess sound localization latency (SLL), and to study SLL as a function of simulated unilateral hearing loss (SUHL) and the relationship with accuracy. Eight healthy and normal-hearing adults (18-40 years) participated in this study. Horizontal gaze responses, recorded by non-invasive corneal reflection eye-tracking, were obtained during azimuthal shifts (24 trials) of a 3-min continuous auditory stimulus. In each trial, a sigmoid function was fitted to gaze samples. Latency was estimated by the abscissa corresponding to 50% of the arctangent amplitude. SLL was defined as the mean latency across trials. SLL was measured in normal-hearing and simulated SUHL conditions (SUHL₃₀ and SUHL₄₃: mean threshold of 30 dB HL and 43 dB HL across 0.5, 1, 2, and 4 kHz). In the normal-hearing condition, the mean ± SD SLL was 280 ± 40 ms (n = 8) with a test-retest SD = 20 ms. A linear mixed model showed a statistically significant effect of listening condition on SLL. The SUHL₃₀ and SUHL₄₃ conditions revealed a mean SLL of 370 ± 49 ms and 540 ± 120 ms, respectively. Repeated measures correlation analysis showed a clear relationship between SLL and the average sound localization accuracy (R² = 0.94). The rapid and reliable method to obtain SLL may be an important clinical tool for evaluation of binaural processing. Future studies in clinical cohorts are needed to assess whether SLL may reveal information about binaural processing abilities beyond that afforded by sound localization accuracy.

Effectiveness of Rehabilitation Approaches Proposed to Children With Severe-to-Profound Prelinguistic Deafness on the Development of Auditory, Speech, and Language Skills: A Systematic Review

Purpose The purpose of this systematic review is to identify and evaluate the available scientific evidence on the effectiveness of rehabilitation approaches proposed to children with severe-to-profound prelinguistic deafness on the hearing, speech, and language skills development. Method Databases (PubMed, CINHAL, PsycInfo, Cochrane, ERIC, and EMBASE) were searched with relevant key words (children, deafness, rehabilitation approach, auditory, speech, and language). Studies published between 2000 and 2017 were included. The methodological quality of the studies was evaluated with the Quality Assessment Tool for Quantitative Studies, and the level of evidence was evaluated with the Oxford Centre for Evidence-Based Medicine Levels of Evidence. Every step of the selection and analysis was made by 2 independent judges. Results Of 1,739 articles listed in different databases, 38 met the inclusion criteria and were selected for analysis. The majority of included articles present a relatively low level of evidence. Rehabilitation approaches that do not include signs appear more frequently associated with a better auditory, speech, and language development, except for receptive language, than approaches that included any form of signs. Conclusion More robust studies are needed to decide on the approach to prioritize with severe-to-profound deaf children.

Cued Speech Enhances Speech-in-Noise Perception

Speech perception in noise remains challenging for Deaf/Hard of Hearing people (D/HH), even fitted with hearing aids or cochlear implants. The perception of sentences in noise by 20 implanted or aided D/HH subjects mastering Cued Speech (CS), a system of hand gestures complementing lip movements, was compared with the perception of 15 typically hearing (TH) controls in three conditions: audio only, audiovisual, and audiovisual + CS. Similar audiovisual scores were obtained for signal-to-noise ratios (SNRs) 11 dB higher in D/HH participants compared with TH ones. Adding CS information enabled D/HH participants to reach a mean score of 83% in the audiovisual + CS condition at a mean SNR of 0 dB, similar to the usual audio score for TH participants at this SNR. This confirms that the combination of lipreading and Cued Speech system remains extremely important for persons with hearing loss, particularly in adverse hearing conditions.

Comparing Auditory-Only and Audiovisual Word Learning for Children With Hearing Loss

Although reducing visual input to emphasize auditory cues is a common practice in pediatric auditory (re)habilitation, the extant literature offers minimal empirical evidence for whether unisensory auditory-only (AO) or multisensory audiovisual (AV) input is more beneficial to children with hearing loss for developing spoken language skills. Using an adapted alternating treatments single case research design, we evaluated the effectiveness and efficiency of a receptive word learning intervention with and without access to visual speechreading cues. Four preschool children with prelingual hearing loss participated. Based on probes without visual cues, three participants demonstrated strong evidence for learning in the AO and AV conditions relative to a control (no-teaching) condition. No participants demonstrated a differential rate of learning between AO and AV conditions. Neither an inhibitory effect predicted by a unisensory theory nor a beneficial effect predicted by a multisensory theory for providing visual cues was identified. Clinical implications are discussed.

Audiovisual Temporal Processing in Postlingually Deafened Adults with Cochlear Implants

For many cochlear implant (CI) users, visual cues are vitally important for interpreting the impoverished auditory speech information that an implant conveys. Although the temporal relationship between auditory and visual stimuli is crucial for how this information is integrated, audiovisual temporal processing in CI users is poorly understood. In this study, we tested unisensory (auditory alone, visual alone) and multisensory (audiovisual) temporal processing in postlingually deafened CI users (n = 48) and normal-hearing controls (n = 54) using simultaneity judgment (SJ) and temporal order judgment (TOJ) tasks. We varied the timing onsets between the auditory and visual components of either a syllable/viseme or a simple flash/beep pairing, and participants indicated either which stimulus appeared first (TOJ) or if the pair occurred simultaneously (SJ). Results indicate that temporal binding windows-the interval within which stimuli are likely to be perceptually 'bound'-are not significantly different between groups for either speech or non-speech stimuli. However, the point of subjective simultaneity for speech was less visually leading in CI users, who interestingly, also had improved visual-only TOJ thresholds. Further signal detection analysis suggests that this SJ shift may be due to greater visual bias within the CI group, perhaps reflecting heightened attentional allocation to visual cues.

Multisensory Integration in Cochlear Implant Recipients

Speech perception is inherently a multisensory process involving integration of auditory and visual cues. Multisensory integration in cochlear implant (CI) recipients is a unique circumstance in that the integration occurs after auditory deprivation and the provision of hearing via the CI. Despite the clear importance of multisensory cues for perception, in general, and for speech intelligibility, specifically, the topic of multisensory perceptual benefits in CI users has only recently begun to emerge as an area of inquiry. We review the research that has been conducted on multisensory integration in CI users to date and suggest a number of areas needing further research. The overall pattern of results indicates that many CI recipients show at least some perceptual gain that can be attributable to multisensory integration. The extent of this gain, however, varies based on a number of factors, including age of implantation and specific task being assessed (e.g., stimulus detection, phoneme perception, word recognition). Although both children and adults with CIs obtain audiovisual benefits for phoneme, word, and sentence stimuli, neither group shows demonstrable gain for suprasegmental feature perception. Additionally, only early-implanted children and the highest performing adults obtain audiovisual integration benefits similar to individuals with normal hearing. Increasing age of implantation in children is associated with poorer gains resultant from audiovisual integration, suggesting a sensitive period in development for the brain networks that subserve these integrative functions, as well as length of auditory experience. This finding highlights the need for early detection of and intervention for hearing loss, not only in terms of auditory perception, but also in terms of the behavioral and perceptual benefits of audiovisual processing. Importantly, patterns of auditory, visual, and audiovisual responses suggest that underlying integrative processes may be fundamentally different between CI users and typical-hearing listeners. Future research, particularly in low-level processing tasks such as signal detection will help to further assess mechanisms of multisensory integration for individuals with hearing loss, both with and without CIs.

Visual Temporal Acuity Is Related to Auditory Speech Perception Abilities in Cochlear Implant Users

OBJECTIVES

Despite significant improvements in speech perception abilities following cochlear implantation, many prelingually deafened cochlear implant (CI) recipients continue to rely heavily on visual information to develop speech and language. Increased reliance on visual cues for understanding spoken language could lead to the development of unique audiovisual integration and visual-only processing abilities in these individuals. Brain imaging studies have demonstrated that good CI performers, as indexed by auditory-only speech perception abilities, have different patterns of visual cortex activation in response to visual and auditory stimuli as compared with poor CI performers. However, no studies have examined whether speech perception performance is related to any type of visual processing abilities following cochlear implantation. The purpose of the present study was to provide a preliminary examination of the relationship between clinical, auditory-only speech perception tests, and visual temporal acuity in prelingually deafened adult CI users. It was hypothesized that prelingually deafened CI users, who exhibit better (i.e., more acute) visual temporal processing abilities would demonstrate better auditory-only speech perception performance than those with poorer visual temporal acuity.

DESIGN

Ten prelingually deafened adult CI users were recruited for this study. Participants completed a visual temporal order judgment task to quantify visual temporal acuity. To assess auditory-only speech perception abilities, participants completed the consonant-nucleus-consonant word recognition test and the AzBio sentence recognition test. Results were analyzed using two-tailed partial Pearson correlations, Spearman's rho correlations, and independent samples t tests.

RESULTS

Visual temporal acuity was significantly correlated with auditory-only word and sentence recognition abilities. In addition, proficient CI users, as assessed via auditory-only speech perception performance, demonstrated significantly better visual temporal acuity than nonproficient CI users.

CONCLUSIONS

These findings provide the first behavioral evidence that visual temporal acuity is related to post implantation CI proficiency as indexed by auditory-only speech perception performance. These preliminary data bring to light the possible future role of visual temporal acuity in predicting CI outcomes before implantation, as well as the possible utility of visual training methods in improving CI outcomes.

Electrophysiological Indices of Audiovisual Speech Perception: Beyond the McGurk Effect and Speech in Noise

Visual information on a talker's face can influence what a listener hears. Commonly used approaches to study this include mismatched audiovisual stimuli (e.g., McGurk type stimuli) or visual speech in auditory noise. In this paper we discuss potential limitations of these approaches and introduce a novel visual phonemic restoration method. This method always presents the same visual stimulus (e.g., /ba/) dubbed with a matched auditory stimulus (/ba/) or one that has weakened consonantal information and sounds more /a/-like). When this reduced auditory stimulus (or /a/) is dubbed with the visual /ba/, a visual influence will result in effectively 'restoring' the weakened auditory cues so that the stimulus is perceived as a /ba/. An oddball design in which participants are asked to detect the /a/ among a stream of more frequently occurring /ba/s while either a speaking face or face with no visual speech was used. In addition, the same paradigm was presented for a second contrast in which participants detected /pa/ among /ba/s, a contrast which should be unaltered by the presence of visual speech. Behavioral and some ERP findings reflect the expected phonemic restoration for the /ba/ vs. /a/ contrast; specifically, we observed reduced accuracy and P300 response in the presence of visual speech. Further, we report an unexpected finding of reduced accuracy and P300 response for both speech contrasts in the presence of visual speech, suggesting overall modulation of the auditory signal in the presence of visual speech. Consistent with this, we observed a mismatch negativity (MMN) effect for the /ba/ vs. /pa/ contrast only that was larger in absence of visual speech. We discuss the potential utility for this paradigm for listeners who cannot respond actively, such as infants and individuals with developmental disabilities.

Relating quality of life to outcomes and predictors in adult cochlear implant users: Are we measuring the right things?

OBJECTIVE

Current clinical outcome measures for adults receiving cochlear implants (CIs) consist of word and sentence recognition, primarily under quiet conditions. However, these measures may not adequately reflect patients' CI-specific quality of life (QOL). This study first examined traditional auditory-only speech recognition measures and other potentially relevant auditory measures as correlates of QOL in CI users. Second, scores on nonauditory tasks of language and cognition were examined as potential predictors of QOL.

STUDY DESIGN

Twenty-five postlingually deafened adults with CIs were assessed.

METHODS

Participants completed a validated CI-specific QOL measure (the Nijmegen Cochlear Implant Questionnaire) and were tested for word and sentence recognition in quiet, as well as sentence recognition in speech-shaped noise. Participants also completed assessments of audiovisual speech recognition, environmental sound identification, and a task of complex auditory verbal processing. Several nonauditory language and cognitive tasks were examined as potential predictors of QOL.

RESULTS

Quality-of-life scores significantly correlated with scores for audiovisual speech recognition and recognition of complex sentences in quiet but not sentences in noise or isolated words. No significant correlations were obtained between QOL and environmental sound identification or complex auditory verbal processing. Quality-of-life subdomain scores were predicted by several nonauditory language and cognitive tasks as well as some patient characteristics.

CONCLUSION

Postoperative measures of recognition of sentences in quiet and audiovisual sentence recognition correlate with CI-related QOL. Findings suggest that sentence recognition tasks are QOL-relevant outcomes but only explain a small fraction of the variability in QOL outcomes for this patient population.

LEVEL OF EVIDENCE

4. Laryngoscope, 128:959-966, 2018.

Early Sign Language Exposure and Cochlear Implantation Benefits

BACKGROUND

Most children with hearing loss who receive cochlear implants (CI) learn spoken language, and parents must choose early on whether to use sign language to accompany speech at home. We address whether parents' use of sign language before and after CI positively influences auditory-only speech recognition, speech intelligibility, spoken language, and reading outcomes.

METHODS

Three groups of children with CIs from a nationwide database who differed in the duration of early sign language exposure provided in their homes were compared in their progress through elementary grades. The groups did not differ in demographic, auditory, or linguistic characteristics before implantation.

RESULTS

Children without early sign language exposure achieved better speech recognition skills over the first 3 years postimplant and exhibited a statistically significant advantage in spoken language and reading near the end of elementary grades over children exposed to sign language. Over 70% of children without sign language exposure achieved age-appropriate spoken language compared with only 39% of those exposed for 3 or more years. Early speech perception predicted speech intelligibility in middle elementary grades. Children without sign language exposure produced speech that was more intelligible (mean = 70%) than those exposed to sign language (mean = 51%).

CONCLUSIONS

This study provides the most compelling support yet available in CI literature for the benefits of spoken language input for promoting verbal development in children implanted by 3 years of age. Contrary to earlier published assertions, there was no advantage to parents' use of sign language either before or after CI.

Electrophysiological Indices of Audiovisual Speech Perception in the Broader Autism Phenotype

When a speaker talks, the consequences of this can both be heard (audio) and seen (visual). A novel visual phonemic restoration task was used to assess behavioral discrimination and neural signatures (event-related potentials, or ERP) of audiovisual processing in typically developing children with a range of social and communicative skills assessed using the social responsiveness scale, a measure of traits associated with autism. An auditory oddball design presented two types of stimuli to the listener, a clear exemplar of an auditory consonant-vowel syllable /ba/ (the more frequently occurring standard stimulus), and a syllable in which the auditory cues for the consonant were substantially weakened, creating a stimulus which is more like /a/ (the infrequently presented deviant stimulus). All speech tokens were paired with a face producing /ba/ or a face with a pixelated mouth containing motion but no visual speech. In this paradigm, the visual /ba/ should cause the auditory /a/ to be perceived as /ba/, creating an attenuated oddball response; in contrast, a pixelated video (without articulatory information) should not have this effect. Behaviorally, participants showed visual phonemic restoration (reduced accuracy in detecting deviant /a/) in the presence of a speaking face. In addition, ERPs were observed in both an early time window (N100) and a later time window (P300) that were sensitive to speech context (/ba/ or /a/) and modulated by face context (speaking face with visible articulation or with pixelated mouth). Specifically, the oddball responses for the N100 and P300 were attenuated in the presence of a face producing /ba/ relative to a pixelated face, representing a possible neural correlate of the phonemic restoration effect. Notably, those individuals with more traits associated with autism (yet still in the non-clinical range) had smaller P300 responses overall, regardless of face context, suggesting generally reduced phonemic discrimination.

Audiovisual speech perception: A new approach and implications for clinical populations

This selected overview of audiovisual (AV) speech perception examines the influence of visible articulatory information on what is heard. Thought to be a cross-cultural phenomenon that emerges early in typical language development, variables that influence AV speech perception include properties of the visual and the auditory signal, attentional demands, and individual differences. A brief review of the existing neurobiological evidence on how visual information influences heard speech indicates potential loci, timing, and facilitatory effects of AV over auditory only speech. The current literature on AV speech in certain clinical populations (individuals with an autism spectrum disorder, developmental language disorder, or hearing loss) reveals differences in processing that may inform interventions. Finally, a new method of assessing AV speech that does not require obvious cross-category mismatch or auditory noise was presented as a novel approach for investigators.

Plasticity in bilateral superior temporal cortex: Effects of deafness and cochlear implantation on auditory and visual speech processing

While many individuals can benefit substantially from cochlear implantation, the ability to perceive and understand auditory speech with a cochlear implant (CI) remains highly variable amongst adult recipients. Importantly, auditory performance with a CI cannot be reliably predicted based solely on routinely obtained information regarding clinical characteristics of the CI candidate. This review argues that central factors, notably cortical function and plasticity, should also be considered as important contributors to the observed individual variability in CI outcome. Superior temporal cortex (STC), including auditory association areas, plays a crucial role in the processing of auditory and visual speech information. The current review considers evidence of cortical plasticity within bilateral STC, and how these effects may explain variability in CI outcome. Furthermore, evidence of audio-visual interactions in temporal and occipital cortices is examined, and relation to CI outcome is discussed. To date, longitudinal examination of changes in cortical function and plasticity over the period of rehabilitation with a CI has been restricted by methodological challenges. The application of functional near-infrared spectroscopy (fNIRS) in studying cortical function in CI users is becoming increasingly recognised as a potential solution to these problems. Here we suggest that fNIRS offers a powerful neuroimaging tool to elucidate the relationship between audio-visual interactions, cortical plasticity during deafness and following cochlear implantation, and individual variability in auditory performance with a CI.

Phonological Priming in Children with Hearing Loss: Effect of Speech Mode, Fidelity, and Lexical Status

OBJECTIVES

This research determined (1) how phonological priming of picture naming was affected by the mode (auditory-visual [AV] versus auditory), fidelity (intact versus nonintact auditory onsets), and lexical status (words versus nonwords) of speech stimuli in children with prelingual sensorineural hearing impairment (CHI) versus children with normal hearing (CNH) and (2) how the degree of HI, auditory word recognition, and age influenced results in CHI. Note that the AV stimuli were not the traditional bimodal input but instead they consisted of an intact consonant/rhyme in the visual track coupled to a nonintact onset/rhyme in the auditory track. Example stimuli for the word bag are (1) AV: intact visual (b/ag) coupled to nonintact auditory (-b/ag) and 2) auditory: static face coupled to the same nonintact auditory (-b/ag). The question was whether the intact visual speech would "restore or fill-in" the nonintact auditory speech in which case performance for the same auditory stimulus would differ depending on the presence/absence of visual speech.

DESIGN

Participants were 62 CHI and 62 CNH whose ages had a group mean and group distribution akin to that in the CHI group. Ages ranged from 4 to 14 years. All participants met the following criteria: (1) spoke English as a native language, (2) communicated successfully aurally/orally, and (3) had no diagnosed or suspected disabilities other than HI and its accompanying verbal problems. The phonological priming of picture naming was assessed with the multimodal picture word task.

RESULTS

Both CHI and CNH showed greater phonological priming from high than low-fidelity stimuli and from AV than auditory speech. These overall fidelity and mode effects did not differ in the CHI versus CNH-thus these CHI appeared to have sufficiently well-specified phonological onset representations to support priming, and visual speech did not appear to be a disproportionately important source of the CHI's phonological knowledge. Two exceptions occurred, however. First-with regard to lexical status-both the CHI and CNH showed significantly greater phonological priming from the nonwords than words, a pattern consistent with the prediction that children are more aware of phonetics-phonology content for nonwords. This overall pattern of similarity between the groups was qualified by the finding that CHI showed more nearly equal priming by the high- versus low-fidelity nonwords than the CNH; in other words, the CHI were less affected by the fidelity of the auditory input for nonwords. Second, auditory word recognition-but not degree of HI or age-uniquely influenced phonological priming by the AV nonwords.

CONCLUSIONS

With minor exceptions, phonological priming in CHI and CNH showed more similarities than differences. Importantly, this research documented that the addition of visual speech significantly increased phonological priming in both groups. Clinically these data support intervention programs that view visual speech as a powerful asset for developing spoken language in CHI.

From Mimicry to Language: A Neuroanatomically Based Evolutionary Model of the Emergence of Vocal Language

The auditory cortex communicates with the frontal lobe via the middle temporal gyrus (auditory ventral stream; AVS) or the inferior parietal lobule (auditory dorsal stream; ADS). Whereas the AVS is ascribed only with sound recognition, the ADS is ascribed with sound localization, voice detection, prosodic perception/production, lip-speech integration, phoneme discrimination, articulation, repetition, phonological long-term memory and working memory. Previously, I interpreted the juxtaposition of sound localization, voice detection, audio-visual integration and prosodic analysis, as evidence that the behavioral precursor to human speech is the exchange of contact calls in non-human primates. Herein, I interpret the remaining ADS functions as evidence of additional stages in language evolution. According to this model, the role of the ADS in vocal control enabled early Homo (Hominans) to name objects using monosyllabic calls, and allowed children to learn their parents' calls by imitating their lip movements. Initially, the calls were forgotten quickly but gradually were remembered for longer periods. Once the representations of the calls became permanent, mimicry was limited to infancy, and older individuals encoded in the ADS a lexicon for the names of objects (phonological lexicon). Consequently, sound recognition in the AVS was sufficient for activating the phonological representations in the ADS and mimicry became independent of lip-reading. Later, by developing inhibitory connections between acoustic-syllabic representations in the AVS and phonological representations of subsequent syllables in the ADS, Hominans became capable of concatenating the monosyllabic calls for repeating polysyllabic words (i.e., developed working memory). Finally, due to strengthening of connections between phonological representations in the ADS, Hominans became capable of encoding several syllables as a single representation (chunking). Consequently, Hominans began vocalizing and mimicking/rehearsing lists of words (sentences).

Audiovisual spoken word recognition as a clinical criterion for sensory aids efficiency in Persian-language children with hearing loss

OBJECTIVES

The aim of this study was to examine the role of audiovisual speech recognition as a clinical criterion of cochlear implant or hearing aid efficiency in Persian-language children with severe-to-profound hearing loss.

DESIGN

This research was administered as a cross-sectional study. The sample size was 60 Persian 5-7 year old children. The assessment tool was one of subtests of Persian version of the Test of Language Development-Primary 3. The study included two experiments: auditory-only and audiovisual presentation conditions. The test was a closed-set including 30 words which were orally presented by a speech-language pathologist.

RESULTS

The scores of audiovisual word perception were significantly higher than auditory-only condition in the children with normal hearing (P<0.01) and cochlear implant (P<0.05); however, in the children with hearing aid, there was no significant difference between word perception score in auditory-only and audiovisual presentation conditions (P>0.05).

CONCLUSIONS

The audiovisual spoken word recognition can be applied as a clinical criterion to assess the children with severe to profound hearing loss in order to find whether cochlear implant or hearing aid has been efficient for them or not; i.e. if a child with hearing impairment who using CI or HA can obtain higher scores in audiovisual spoken word recognition than auditory-only condition, his/her auditory skills have appropriately developed due to effective CI or HA as one of the main factors of auditory habilitation.

Audiovisual integration in children listening to spectrally degraded speech

PURPOSE

The study explored whether visual information improves speech identification in typically developing children with normal hearing when the auditory signal is spectrally degraded.

METHOD

Children (n=69) and adults (n=15) were presented with noise-vocoded sentences from the Children's Co-ordinate Response Measure (Rosen, 2011) in auditory-only or audiovisual conditions. The number of bands was adaptively varied to modulate the degradation of the auditory signal, with the number of bands required for approximately 79% correct identification calculated as the threshold.

RESULTS

The youngest children (4- to 5-year-olds) did not benefit from accompanying visual information, in comparison to 6- to 11-year-old children and adults. Audiovisual gain also increased with age in the child sample.

CONCLUSIONS

The current data suggest that children younger than 6 years of age do not fully utilize visual speech cues to enhance speech perception when the auditory signal is degraded. This evidence not only has implications for understanding the development of speech perception skills in children with normal hearing but may also inform the development of new treatment and intervention strategies that aim to remediate speech perception difficulties in pediatric cochlear implant users.

Talker variability in audio-visual speech perception

A change in talker is a change in the context for the phonetic interpretation of acoustic patterns of speech. Different talkers have different mappings between acoustic patterns and phonetic categories and listeners need to adapt to these differences. Despite this complexity, listeners are adept at comprehending speech in multiple-talker contexts, albeit at a slight but measurable performance cost (e.g., slower recognition). So far, this talker variability cost has been demonstrated only in audio-only speech. Other research in single-talker contexts have shown, however, that when listeners are able to see a talker's face, speech recognition is improved under adverse listening (e.g., noise or distortion) conditions that can increase uncertainty in the mapping between acoustic patterns and phonetic categories. Does seeing a talker's face reduce the cost of word recognition in multiple-talker contexts? We used a speeded word-monitoring task in which listeners make quick judgments about target word recognition in single- and multiple-talker contexts. Results show faster recognition performance in single-talker conditions compared to multiple-talker conditions for both audio-only and audio-visual speech. However, recognition time in a multiple-talker context was slower in the audio-visual condition compared to audio-only condition. These results suggest that seeing a talker's face during speech perception may slow recognition by increasing the importance of talker identification, signaling to the listener a change in talker has occurred.

How is the McGurk effect modulated by Cued Speech in deaf and hearing adults?

Speech perception for both hearing and deaf people involves an integrative process between auditory and lip-reading information. In order to disambiguate information from lips, manual cues from Cued Speech may be added. Cued Speech (CS) is a system of manual aids developed to help deaf people to clearly and completely understand speech visually (Cornett, 1967). Within this system, both labial and manual information, as lone input sources, remain ambiguous. Perceivers, therefore, have to combine both types of information in order to get one coherent percept. In this study, we examined how audio-visual (AV) integration is affected by the presence of manual cues and on which form of information (auditory, labial or manual) the CS receptors primarily rely. To address this issue, we designed a unique experiment that implemented the use of AV McGurk stimuli (audio /pa/ and lip-reading /ka/) which were produced with or without manual cues. The manual cue was congruent with either auditory information, lip information or the expected fusion. Participants were asked to repeat the perceived syllable aloud. Their responses were then classified into four categories: audio (when the response was /pa/), lip-reading (when the response was /ka/), fusion (when the response was /ta/) and other (when the response was something other than /pa/, /ka/ or /ta/). Data were collected from hearing impaired individuals who were experts in CS (all of which had either cochlear implants or binaural hearing aids; N = 8), hearing-individuals who were experts in CS (N = 14) and hearing-individuals who were completely naïve of CS (N = 15). Results confirmed that, like hearing-people, deaf people can merge auditory and lip-reading information into a single unified percept. Without manual cues, McGurk stimuli induced the same percentage of fusion responses in both groups. Results also suggest that manual cues can modify the AV integration and that their impact differs between hearing and deaf people.

Seeing to hear? Patterns of gaze to speaking faces in children with autism spectrum disorders

Using eye-tracking methodology, gaze to a speaking face was compared in a group of children with autism spectrum disorders (ASD) and a group with typical development (TD). Patterns of gaze were observed under three conditions: audiovisual (AV) speech in auditory noise, visual only speech and an AV non-face, non-speech control. Children with ASD looked less to the face of the speaker and fixated less on the speakers’ mouth than TD controls. No differences in gaze were reported for the non-face, non-speech control task. Since the mouth holds much of the articulatory information available on the face, these findings suggest that children with ASD may have reduced access to critical linguistic information. This reduced access to visible articulatory information could be a contributor to the communication and language problems exhibited by children with ASD.

Seeing to hear? Patterns of gaze to speaking faces in children with autism spectrum disorders

Using eye-tracking methodology, gaze to a speaking face was compared in a group of children with autism spectrum disorders (ASD) and a group with typical development (TD). Patterns of gaze were observed under three conditions: audiovisual (AV) speech in auditory noise, visual only speech and an AV non-face, non-speech control. Children with ASD looked less to the face of the speaker and fixated less on the speakers' mouth than TD controls. No differences in gaze were reported for the non-face, non-speech control task. Since the mouth holds much of the articulatory information available on the face, these findings suggest that children with ASD may have reduced access to critical linguistic information. This reduced access to visible articulatory information could be a contributor to the communication and language problems exhibited by children with ASD.

The use of auditory and visual context in speech perception by listeners with normal hearing and listeners with cochlear implants

There is a wide range of acoustic and visual variability across different talkers and different speaking contexts. Listeners with normal hearing (NH) accommodate that variability in ways that facilitate efficient perception, but it is not known whether listeners with cochlear implants (CIs) can do the same. In this study, listeners with NH and listeners with CIs were tested for accommodation to auditory and visual phonetic contexts created by gender-driven speech differences as well as vowel coarticulation and lip rounding in both consonants and vowels. Accommodation was measured as the shifting of perceptual boundaries between /s/ and /∫/ sounds in various contexts, as modeled by mixed-effects logistic regression. Owing to the spectral contrasts thought to underlie these context effects, CI listeners were predicted to perform poorly, but showed considerable success. Listeners with CIs not only showed sensitivity to auditory cues to gender, they were also able to use visual cues to gender (i.e., faces) as a supplement or proxy for information in the acoustic domain, in a pattern that was not observed for listeners with NH. Spectrally-degraded stimuli heard by listeners with NH generally did not elicit strong context effects, underscoring the limitations of noise vocoders and/or the importance of experience with electric hearing. Visual cues for consonant lip rounding and vowel lip rounding were perceived in a manner consistent with coarticulation and were generally used more heavily by listeners with CIs. Results suggest that listeners with CIs are able to accommodate various sources of acoustic variability either by attending to appropriate acoustic cues or by inferring them via the visual signal.

Assessing spoken word recognition in children who are deaf or hard of hearing: a translational approach

Under natural conditions, listeners use both auditory and visual speech cues to extract meaning from speech signals containing many sources of variability. However, traditional clinical tests of spoken word recognition routinely employ isolated words or sentences produced by a single talker in an auditory-only presentation format. The more central cognitive processes used during multimodal integration, perceptual normalization, and lexical discrimination that may contribute to individual variation in spoken word recognition performance are not assessed in conventional tests of this kind. In this article, we review our past and current research activities aimed at developing a series of new assessment tools designed to evaluate spoken word recognition in children who are deaf or hard of hearing. These measures are theoretically motivated by a current model of spoken word recognition and also incorporate "real-world" stimulus variability in the form of multiple talkers and presentation formats. The goal of this research is to enhance our ability to estimate real-world listening skills and to predict benefit from sensory aid use in children with varying degrees of hearing loss.

Perception of speech features by French-speaking children with cochlear implants

PURPOSE

The present study investigates the perception of phonological features in French-speaking children with cochlear implants (CIs) compared with normal-hearing (NH) children matched for listening age.

METHOD

Scores for discrimination and identification of minimal pairs for all features defining consonants (e.g., place, voicing, manner, nasality) and vowels (e.g., frontness, nasality, aperture) were measured in each listener.

RESULTS

The results indicated no differences in "categorical perception," specified as a similar difference between discrimination and identification between CI children and controls. However, CI children demonstrated a lower level of "categorical precision," that is, lesser accuracy in both feature identification and discrimination, than NH children, with the magnitude of the deficit depending on the feature.

CONCLUSIONS

If sensitive periods of language development extend well beyond the moment of implantation, the consequences of hearing deprivation for the acquisition of categorical perception should be fairly important in comparison to categorical precision because categorical precision develops more slowly than categorical perception in NH children. These results do not support the idea that the sensitive period for development of categorical perception is restricted to the first 1-2 years of life. The sensitive period may be significantly longer. Differences in precision may reflect the acoustic limitations of the cochlear implant, such as coding for temporal fine structure and frequency resolution.

Objective eye-gaze behaviour during face-to-face communication with proficient alaryngeal speakers: a preliminary study

BACKGROUND

There is a large body of research demonstrating the impact of visual information on speaker intelligibility in both normal and disordered speaker populations. However, there is minimal information on which specific visual features listeners find salient during conversational discourse.

AIMS

To investigate listeners' eye-gaze behaviour during face-to-face conversation with normal, laryngeal and proficient alaryngeal speakers.

METHODS & PROCEDURES

Sixty participants individually participated in a 10-min conversation with one of four speakers (typical laryngeal, tracheoesophageal, oesophageal, electrolaryngeal; 15 participants randomly assigned to one mode of speech). All speakers were > 85% intelligible and were judged to be 'proficient' by two certified speech-language pathologists. Participants were fitted with a head-mounted eye-gaze tracking device (Mobile Eye, ASL) that calculated the region of interest and mean duration of eye-gaze. Self-reported gaze behaviour was also obtained following the conversation using a 10 cm visual analogue scale.

OUTCOMES & RESULTS

While listening, participants viewed the lower facial region of the oesophageal speaker more than the normal or tracheoesophageal speaker. Results of non-hierarchical cluster analyses showed that while listening, the pattern of eye-gaze was predominantly directed at the lower face of the oesophageal and electrolaryngeal speaker and more evenly dispersed among the background, lower face, and eyes of the normal and tracheoesophageal speakers. Finally, results show a low correlation between self-reported eye-gaze behaviour and objective regions of interest data.

CONCLUSIONS & IMPLICATIONS

Overall, results suggest similar eye-gaze behaviour when healthy controls converse with normal and tracheoesophageal speakers and that participants had significantly different eye-gaze patterns when conversing with an oesophageal speaker. Results are discussed in terms of existing eye-gaze data and its potential implications on auditory-visual speech perception.

Reading and reading-related skills in children using cochlear implants: prospects for the influence of cued speech

We assessed the reading and reading-related skills (phonemic awareness and phonological short-term memory) of deaf children fitted with cochlear implants (CI), either exposed to cued speech early (before 2 years old) (CS+) or never (CS-). Their performance was compared to that of 2 hearing control groups, 1 matched for reading level (RL), and 1 matched for chronological age (CA). Phonemic awareness and phonological short-term memory were assessed respectively through a phonemic similarity judgment task and through a word span task measuring phonological similarity effects. To assess the use of sublexical and lexical reading procedures, children read pseudowords and irregular words aloud. Results showed that cued speech improved performance on both the phonemic awareness and the reading tasks but not on the phonological short-term memory task. In phonemic awareness and reading, CS+ children obtained accuracy and rapidity scores similar to CA controls, whereas CS- children obtained lower scores than hearing controls. Nevertheless, in phonological short-term memory task, the phonological similarity effect of both CI groups was similar. Overall, these results support the use of cued speech to improve phonemic awareness and reading skills in CI children.

Assessing multimodal spoken word-in-sentence recognition in children with normal hearing and children with cochlear implants

PURPOSE

To examine multimodal spoken word-in-sentence recognition in children.

METHOD

Two experiments were undertaken. In Experiment 1, the youngest age with which the multimodal sentence recognition materials could be used was evaluated. In Experiment 2, lexical difficulty and presentation modality effects were examined, along with test-retest reliability and validity in normal-hearing children and those with cochlear implants.

RESULTS

Normal-hearing children as young as 3.25 years and those with cochlear implants just under 4 years who have used their device for at least 1 year were able to complete the multimodal sentence testing. Both groups identified lexically easy words in sentences more accurately than lexically hard words across modalities, although the largest effects occurred in the auditory-only modality. Both groups displayed audiovisual integration with the highest scores achieved in the audiovisual modality, followed sequentially by auditory-only and visual-only modalities. Recognition of words in sentences was correlated with recognition of words in isolation. Preliminary results suggest fair-to-good test-retest reliability.

CONCLUSIONS

The results suggest that children's audiovisual word-in-sentence recognition can be assessed using the materials developed for this investigation. With further development, the materials hold promise for becoming a test of multimodal sentence recognition for children with hearing loss.

Learning to use an artificial visual cue in speech identification

Visual information from a speaker's face profoundly influences auditory perception of speech. However, relatively little is known about the extent to which visual influences may depend on experience, and extent to which new sources of visual speech information can be incorporated in speech perception. In the current study, participants were trained on completely novel visual cues for phonetic categories. Participants learned to accurately identify phonetic categories based on novel visual cues. These newly-learned visual cues influenced identification responses to auditory speech stimuli, but not to the same extent as visual cues from a speaker's face. The novel methods and results of the current study raise theoretical questions about the nature of information integration in speech perception, and open up possibilities for further research on learning in multimodal perception, which may have applications in improving speech comprehension among the hearing-impaired.

An acoustic analysis of the vowel space in young and old cochlear-implant speakers

The main purpose of this study was to compare acoustically the vowel spaces of two groups of cochlear implantees (CI) with two age-matched normal hearing groups. Five young test persons (15-25 years) and five older test persons (55-70 years) with CI and two control groups of the same age with normal hearing were recorded. The speech material consisted of five German vowels V = /a, e, i, o, u/ in bilabial and alveolar contexts. The results showed no differences between the two groups on Euclidean distances for the first formant frequency. In contrast, Euclidean distances for F2 of the CI group were shorter than those of the control group, causing their overall vowel space to be compressed. The main differences between the groups are interpreted in terms of the extent to which the formants are associated with visual cues to the vowels. Further results were partially longer vowel durations for the CI speakers.

Neural development of networks for audiovisual speech comprehension

Everyday conversation is both an auditory and a visual phenomenon. While visual speech information enhances comprehension for the listener, evidence suggests that the ability to benefit from this information improves with development. A number of brain regions have been implicated in audiovisual speech comprehension, but the extent to which the neurobiological substrate in the child compares to the adult is unknown. In particular, developmental differences in the network for audiovisual speech comprehension could manifest through the incorporation of additional brain regions, or through different patterns of effective connectivity. In the present study we used functional magnetic resonance imaging and structural equation modeling (SEM) to characterize the developmental changes in network interactions for audiovisual speech comprehension. The brain response was recorded while children 8- to 11-years-old and adults passively listened to stories under audiovisual (AV) and auditory-only (A) conditions. Results showed that in children and adults, AV comprehension activated the same fronto-temporo-parietal network of regions known for their contribution to speech production and perception. However, the SEM network analysis revealed age-related differences in the functional interactions among these regions. In particular, the influence of the posterior inferior frontal gyrus/ventral premotor cortex on supramarginal gyrus differed across age groups during AV, but not A speech. This functional pathway might be important for relating motor and sensory information used by the listener to identify speech sounds. Further, its development might reflect changes in the mechanisms that relate visual speech information to articulatory speech representations through experience producing and perceiving speech.

Speech perception and production

Until recently, research in speech perception and speech production has largely focused on the search for psychological and phonetic evidence of discrete, abstract, context-free symbolic units corresponding to phonological segments or phonemes. Despite this common conceptual goal and intimately related objects of study, however, research in these two domains of speech communication has progressed more or less independently for more than 60 years. In this article, we present an overview of the foundational works and current trends in the two fields, specifically discussing the progress made in both lines of inquiry as well as the basic fundamental issues that neither has been able to resolve satisfactorily so far. We then discuss theoretical models and recent experimental evidence that point to the deep, pervasive connections between speech perception and production. We conclude that although research focusing on each domain individually has been vital in increasing our basic understanding of spoken language processing, the human capacity for speech communication is so complex that gaining a full understanding will not be possible until speech perception and production are conceptually reunited in a joint approach to problems shared by both modes. Copyright © 2010 John Wiley & Sons, Ltd. For further resources related to this article, please visit the WIREs website.

Primary and multisensory cortical activity is correlated with audiovisual percepts

Incongruent auditory and visual stimuli can elicit audiovisual illusions such as the McGurk effect where visual /ka/ and auditory /pa/ fuse into another percept such as/ta/. In the present study, human brain activity was measured with adaptation functional magnetic resonance imaging to investigate which brain areas support such audiovisual illusions. Subjects viewed trains of four movies beginning with three congruent /pa/ stimuli to induce adaptation. The fourth stimulus could be (i) another congruent /pa/, (ii) a congruent /ka/, (iii) an incongruent stimulus that evokes the McGurk effect in susceptible individuals (lips /ka/ voice /pa/), or (iv) the converse combination that does not cause the McGurk effect (lips /pa/ voice/ ka/). This paradigm was predicted to show increased release from adaptation (i.e. stronger brain activation) when the fourth movie and the related percept was increasingly different from the three previous movies. A stimulus change in either the auditory or the visual stimulus from /pa/ to /ka/ (iii, iv) produced within-modality and cross-modal responses in primary auditory and visual areas. A greater release from adaptation was observed for incongruent non-McGurk (iv) compared to incongruent McGurk (iii) trials. A network including the primary auditory and visual cortices, nonprimary auditory cortex, and several multisensory areas (superior temporal sulcus, intraparietal sulcus, insula, and pre-central cortex) showed a correlation between perceiving the McGurk effect and the fMRI signal, suggesting that these areas support the audiovisual illusion.