Processing of Visual Speech Cues in Speech-in-Noise Comprehension Depends on Working Memory Capacity and Enhances Neural Speech Tracking in Older Adults With Hearing Impairment

Comprehending speech in noise (SiN) poses a challenge for older hearing-impaired listeners, requiring auditory and working memory resources. Visual speech cues provide additional sensory information supporting speech understanding, while the extent of such visual benefit is characterized by large variability, which might be accounted for by individual differences in working memory capacity (WMC). In the current study, we investigated behavioral and neurofunctional (i.e., neural speech tracking) correlates of auditory and audio-visual speech comprehension in babble noise and the associations with WMC. Healthy older adults with hearing impairment quantified by pure-tone hearing loss (threshold average: 31.85-57 dB, N = 67) listened to sentences in babble noise in audio-only, visual-only and audio-visual speech modality and performed a pattern matching and a comprehension task, while electroencephalography (EEG) was recorded. Behaviorally, no significant difference in task performance was observed across modalities. However, we did find a significant association between individual working memory capacity and task performance, suggesting a more complex interplay between audio-visual speech cues, working memory capacity and real-world listening tasks. Furthermore, we found that the visual speech presentation was accompanied by increased cortical tracking of the speech envelope, particularly in a right-hemispheric auditory topographical cluster. Post-hoc, we investigated the potential relationships between the behavioral performance and neural speech tracking but were not able to establish a significant association. Overall, our results show an increase in neurofunctional correlates of speech associated with congruent visual speech cues, specifically in a right auditory cluster, suggesting multisensory integration.

Cortical hemisphere preference and brainstem ear asymmetry reflect experience-dependent functional modulation of pitch

Temporal attributes of pitch processing at cortical and subcortical levels are differentially weighted and well-coordinated. The question is whether language experience induces functional modulation of hemispheric preference complemented by brainstem ear symmetry for pitch processing. Brainstem frequency-following and cortical pitch responses were recorded concurrently from Mandarin and English participants. A Mandarin syllable with a rising pitch contour was presented to both ears with monaural stimulation. At the cortical level, left ear stimulation in the Chinese group revealed an experience-dependent response for pitch processing in the right hemisphere, consistent with a functionalaccount. The English group revealed a contralateral hemisphere preference consistent with a structuralaccount. At the brainstem level, Chinese participants showed a functional leftward ear asymmetry, whereas English were consistent with a structural account. Overall, language experience modulates both cortical hemispheric preference and brainstem ear asymmetry in a complementary manner to optimize processing of temporal attributes of pitch.

Language related differences of the sustained response evoked by natural speech sounds

In tonal languages, such as Mandarin Chinese, the pitch contour of vowels discriminates lexical meaning, which is not the case in non-tonal languages such as German. Recent data provide evidence that pitch processing is influenced by language experience. However, there are still many open questions concerning the representation of such phonological and language-related differences at the level of the auditory cortex (AC). Using magnetoencephalography (MEG), we recorded transient and sustained auditory evoked fields (AEF) in native Chinese and German speakers to investigate language related phonological and semantic aspects in the processing of acoustic stimuli. AEF were elicited by spoken meaningful and meaningless syllables, by vowels, and by a French horn tone. Speech sounds were recorded from a native speaker and showed frequency-modulations according to the pitch-contours of Mandarin. The sustained field (SF) evoked by natural speech signals was significantly larger for Chinese than for German listeners. In contrast, the SF elicited by a horn tone was not significantly different between groups. Furthermore, the SF of Chinese subjects was larger when evoked by meaningful syllables compared to meaningless ones, but there was no significant difference regarding whether vowels were part of the Chinese phonological system or not. Moreover, the N100m gave subtle but clear evidence that for Chinese listeners other factors than purely physical properties play a role in processing meaningful signals. These findings show that the N100 and the SF generated in Heschl’s gyrus are influenced by language experience, which suggests that AC activity related to specific pitch contours of vowels is influenced in a top-down fashion by higher, language related areas. Such interactions are in line with anatomical findings and neuroimaging data, as well as with the dual-stream model of language of Hickok and Poeppel that highlights the close and reciprocal interaction between superior temporal gyrus and sulcus.

Language-dependent changes in pitch-relevant neural activity in the auditory cortex reflect differential weighting of temporal attributes of pitch contours

There remains a gap in our knowledge base about neural representation of pitch attributes that occur between onset and offset of dynamic, curvilinear pitch contours. The aim is to evaluate how language experience shapes processing of pitch contours as reflected in the amplitude of cortical pitch-specific response components. Responses were elicited from three nonspeech, bidirectional (falling-rising) pitch contours representative of Mandarin Tone 2 varying in location of the turning point with fixed onset and offset. At the frontocentral Fz electrode site, Na-Pb and Pb-Nb amplitude of the Chinese group was larger than the English group for pitch contours exhibiting later location of the turning point relative to the one with the earliest location. Chinese listeners' amplitude was also greater than that of English in response to those same pitch contours with later turning points. At lateral temporal sites (T7/T8), Na-Pb amplitude was larger in Chinese listeners relative to English over the right temporal site. In addition, Pb-Nb amplitude of the Chinese group showed a rightward asymmetry. The pitch contour with its turning point located about halfway of total duration evoked a rightward asymmetry regardless of group. These findings suggest that neural mechanisms processing pitch in the right auditory cortex reflect experience-dependent modulation of sensitivity to weighted integration of changes in acceleration rates of rising and falling sections and the location of the turning point.

Changes in pitch height elicit both language-universal and language-dependent changes in neural representation of pitch in the brainstem and auditory cortex

Language experience shapes encoding of pitch-relevant information at both brainstem and cortical levels of processing. Pitch height is a salient dimension that orders pitch from low to high. Herein we investigate the effects of language experience (Chinese, English) in the brainstem and cortex on (i) neural responses to variations in pitch height, (ii) presence of asymmetry in cortical pitch representation, and (iii) patterns of relative changes in magnitude of pitch height between these two levels of brain structure. Stimuli were three nonspeech homologs of Mandarin Tone 2 varying in pitch height only. The frequency-following response (FFR) and the cortical pitch-specific response (CPR) were recorded concurrently. At the Fz-linked T7/T8 site, peak latency of Na, Pb, and Nb decreased with increasing pitch height for both groups. Peak-to-peak amplitude of Na-Pb and Pb-Nb increased with increasing pitch height across groups. A language-dependent effect was restricted to Na-Pb; the Chinese had larger amplitude than the English group. At temporal sites (T7/T8), the Chinese group had larger amplitude, as compared to English, across stimuli, but also limited to the Na-Pb component and right temporal site. In the brainstem, F0 magnitude decreased with increasing pitch height; Chinese had larger magnitude across stimuli. A comparison of CPR and FFR responses revealed distinct patterns of relative changes in magnitude common to both groups. CPR amplitude increased and FFR amplitude decreased with increasing pitch height. Experience-dependent effects on CPR components vary as a function of neural sensitivity to pitch height within a particular temporal window (Na-Pb). Differences between the auditory brainstem and cortex imply distinct neural mechanisms for pitch extraction at both levels of brain structure.

Language-experience plasticity in neural representation of changes in pitch salience

Neural representation of pitch-relevant information at the brainstem and cortical levels of processing is influenced by language experience. A well-known attribute of pitch is its salience. Brainstem frequency following responses and cortical pitch specific responses, recorded concurrently, were elicited by a pitch salience continuum spanning weak to strong pitch of a dynamic, iterated rippled noise pitch contour-homolog of a Mandarin tone. Our aims were to assess how language experience (Chinese, English) affects i) enhancement of neural activity associated with pitch salience at brainstem and cortical levels, ii) the presence of asymmetry in cortical pitch representation, and iii) patterns of relative changes in magnitude along the pitch salience continuum. Peak latency (Fz: Na, Pb, and Nb) was shorter in the Chinese than the English group across the continuum. Peak-to-peak amplitude (Fz: Na-Pb, Pb-Nb) of the Chinese group grew larger with increasing pitch salience, but an experience-dependent advantage was limited to the Na-Pb component. At temporal sites (T7/T8), the larger amplitude of the Chinese group across the continuum was both limited to the Na-Pb component and the right temporal site. At the brainstem level, F0 magnitude gets larger as you increase pitch salience, and it too reveals Chinese superiority. A direct comparison of cortical and brainstem responses for the Chinese group reveals different patterns of relative changes in magnitude along the pitch salience continuum. Such differences may point to a transformation in pitch processing at the cortical level presumably mediated by local sensory and/or extrasensory influence overlaid on the brainstem output.

Experience-dependent enhancement of pitch-specific responses in the auditory cortex is limited to acceleration rates in normal voice range

The aim of this study is to determine how pitch acceleration rates within and outside the normal pitch range may influence latency and amplitude of cortical pitch-specific responses (CPR) as a function of language experience (Chinese, English). Responses were elicited from a set of four pitch stimuli chosen to represent a range of acceleration rates (two each inside and outside the normal voice range) imposed on the high rising Mandarin Tone 2. Pitch-relevant neural activity, as reflected in the latency and amplitude of scalp-recorded CPR components, varied depending on language-experience and pitch acceleration of dynamic, time-varying pitch contours. Peak latencies of CPR components were shorter in the Chinese than the English group across stimuli. Chinese participants showed greater amplitude than English for CPR components at both frontocentral and temporal electrode sites in response to pitch contours with acceleration rates inside the normal voice pitch range as compared to pitch contours with acceleration rates that exceed the normal range. As indexed by CPR amplitude at the temporal sites, a rightward asymmetry was observed for the Chinese group only. Only over the right temporal site was amplitude greater in the Chinese group relative to the English. These findings may suggest that the neural mechanism(s) underlying processing of pitch in the right auditory cortex reflect experience-dependent modulation of sensitivity to acceleration in just those rising pitch contours that fall within the bounds of one's native language. More broadly, enhancement of native pitch stimuli and stronger rightward asymmetry of CPR components in the Chinese group is consistent with the notion that long-term experience shapes adaptive, distributed hierarchical pitch processing in the auditory cortex, and reflects an interaction with higher order, extrasensory processes beyond the sensory memory trace.

Modulation of pre-attentive spectro-temporal feature processing in the human auditory system by HD-tDCS

The present study examined the functional lateralization of the human auditory cortex (AC) for pre-attentive spectro-temporal feature processing. By using high-definition transcranial direct current stimulation (HD-tDCS), we systematically modulated neuronal activity of the bilateral AC. We assessed the influence of anodal and cathodal HD-tDCS delivered over the left or right AC on auditory mismatch negativity (MMN) in response to temporal as well as spectral deviants in 12 healthy subjects. The results showed that MMN to temporal deviants was significantly enhanced by anodal HD-tDCS applied over the left AC only. Our data indicate a left hemispheric dominance for the pre-attentive processing of low-level temporal information.

Pitch processing of dynamic lexical tones in the auditory cortex is influenced by sensory and extrasensory processes

The aim is to evaluate how language experience (Chinese, English) shapes processing of pitch contours as reflected in the amplitude of cortical pitch response components. Responses were elicited from three dynamic curvilinear nonspeech stimuli varying in pitch direction and location of peak acceleration: Mandarin lexical Tone 2 (rising) and Tone 4 (falling), and a flipped variant of Tone 2, Tone 2' (nonnative). At temporal sites (T7/T8), Chinese listeners' Na-Pb response amplitudes to Tones 2 and 4 were greater than those of English listeners in the right hemisphere only; a rightward asymmetry for Tones 2 and 4 was restricted to the Chinese group. In common to both Fz-to-linked T7/T8 and T7/T8 electrode sites, the stimulus pattern (Tones 2 and 4 > Tone 2') was found in the Chinese group only. As reflected by Pb-Nb at Fz, Chinese subjects' amplitudes were larger than those of English subjects in response to Tones 2 and 4, and Tones 2 and 4 were larger than Tone 2', whereas for English subjects, Tone 2 was larger than Tone 2' and Tone 4. At frontal electrode sites (F3/F4), regardless of component or hemisphere, Chinese subjects' responses were larger in amplitude than those of English subjects across stimuli. For either group, responses to Tones 2 and 4 were larger than Tone 2'. No hemispheric asymmetry was observed at the frontal electrode sites. These findings demonstrate that cortical pitch response components are differentially modulated by experience-dependent, temporally distinct but functionally overlapping, weighting of sensory and extrasensory effects on pitch processing of lexical tones in the right temporal lobe and, more broadly, are consistent with a distributed hierarchical predictive coding process.

Language experience enhances early cortical pitch-dependent responses

Pitch processing at cortical and subcortical stages of processing is shaped by language experience. We recently demonstrated that specific components of the cortical pitch response (CPR) index the more rapidly-changing portions of the high rising Tone 2 of Mandarin Chinese, in addition to marking pitch onset and sound offset. In this study, we examine how language experience (Mandarin vs. English) shapes the processing of different temporal attributes of pitch reflected in the CPR components using stimuli representative of within-category variants of Tone 2. Results showed that the magnitude of CPR components (Na-Pb and Pb-Nb) and the correlation between these two components and pitch acceleration were stronger for the Chinese listeners compared to English listeners for stimuli that fell within the range of Tone 2 citation forms. Discriminant function analysis revealed that the Na-Pb component was more than twice as important as Pb-Nb in grouping listeners by language affiliation. In addition, a stronger stimulus-dependent, rightward asymmetry was observed for the Chinese group at the temporal, but not frontal, electrode sites. This finding may reflect selective recruitment of experience-dependent, pitch-specific mechanisms in right auditory cortex to extract more complex, time-varying pitch patterns. Taken together, these findings suggest that long-term language experience shapes early sensory level processing of pitch in the auditory cortex, and that the sensitivity of the CPR may vary depending on the relative linguistic importance of specific temporal attributes of dynamic pitch.

Cortical pitch response components show differential sensitivity to native and nonnative pitch contours

The aim of this study is to evaluate how nonspeech pitch contours of varying shape influence latency and amplitude of cortical pitch-specific response (CPR) components differentially as a function of language experience. Stimuli included time-varying, high rising Mandarin Tone 2 (T2) and linear rising ramp (Linear), and steady-state (Flat). Both the latency and magnitude of CPR components were differentially modulated by (i) the overall trajectory of pitch contours (time-varying vs. steady-state), (ii) their pitch acceleration rates (changing vs. constant), and (iii) their linguistic status (lexical vs. non-lexical). T2 elicited larger amplitude than Linear in both language groups, but size of the effect was larger in Chinese than English. The magnitude of CPR components elicited by T2 were larger for Chinese than English at the right temporal electrode site. Using the CPR, we provide evidence in support of experience-dependent modulation of dynamic pitch contours at an early stage of sensory processing.

Cortical pitch response components index stimulus onset/offset and dynamic features of pitch contours

Voice pitch is an important information-bearing component of language that is subject to experience dependent plasticity at both early cortical and subcortical stages of processing. We have already demonstrated that pitch onset component (Na) of the cortical pitch response (CPR) is sensitive to flat pitch and its salience … CPR responses from Chinese listeners were elicited by three citation forms varying in pitch acceleration and duration. Results showed that the pitch onset component (Na) was invariant to changes in acceleration. In contrast, Na–Pb and Pb–Nb showed a systematic decrease in the interpeak latency and decrease in amplitude with increase in pitch acceleration that followed the time course of pitch change across the three stimuli. A strong correlation with pitch acceleration was observed for these two components only – a putative index of pitch-relevant neural activity associated with the more rapidly-changing portions of the pitch contour. Pc–Nc marks unambiguously the stimulus offset … and their functional roles as related to sensory and cognitive properties of the stimulus. [Corrected]

Right and left perisylvian cortex and left inferior frontal cortex mediate sentence-level rhyme detection in spoken language as revealed by sparse fMRI

In this study, we used functional magnetic resonance imaging to investigate the neural basis of auditory rhyme processing at the sentence level in healthy adults. In an explicit rhyme detection task, participants were required to decide whether the ending syllable of a metrically spoken pseudosentence rhymed or not. Participants performing this task revealed bilateral activation in posterior-superior temporal gyri with a much more extended cluster of activation in the right hemisphere. These findings suggest that the right hemisphere primarily supports suprasegmental tasks, such as the segmentation of speech into syllables; thus, our findings are in line with the "asymmetric sampling in time" model suggested by Poeppel (: Speech Commun 41:245-255). The direct contrast between rhymed and nonrhymed trials revealed a stronger BOLD response for rhymed trials in the frontal operculum and the anterior insula of the left hemisphere. Our results suggest an involvement of these frontal regions not only in articulatory rehearsal processes, but especially in the detection of a matching syllable, as well as in the execution of rhyme judgment.

Functional ear (a)symmetry in brainstem neural activity relevant to encoding of voice pitch: a precursor for hemispheric specialization?

Pitch processing is lateralized to the right hemisphere; linguistic pitch is further mediated by left cortical areas. This experiment investigates whether ear asymmetries vary in brainstem representation of pitch depending on linguistic status. Brainstem frequency-following responses (FFRs) were elicited by monaural stimulation of the left and right ear of 15 native speakers of Mandarin Chinese using two synthetic speech stimuli that differ in linguistic status of tone. One represented a native lexical tone (Tone 2: T2); the other, T2', a nonnative variant in which the pitch contour was a mirror image of T2 with the same starting and ending frequencies. Two 40-ms portions of f(0) contours were selected in order to compare two regions (R1, early; R2 late) differing in pitch acceleration rate and perceptual saliency. In R2, linguistic status effects revealed that T2 exhibited a larger degree of FFR rightward ear asymmetry as reflected in f(0) amplitude relative to T2'. Relative to midline (ear asymmetry=0), the only ear asymmetry reaching significance was that favoring left ear stimulation elicited by T2'. By left- and right-ear stimulation separately, FFRs elicited by T2 were larger than T2' in the right ear only. Within T2', FFRs elicited by the earlier region were larger than the later in both ears. Within T2, no significant differences in FFRS were observed between regions in either ear. Collectively, these findings support the idea that origins of cortical processing preferences for perceptually-salient portions of pitch are rooted in early, preattentive stages of processing in the brainstem.