Listening effort and accented speech

Neural correlates of listening to nonnative-accented speech in multi-talker background noise

We examined the neural correlates underlying the semantic processing of native- and nonnative-accented sentences, presented in quiet or embedded in multi-talker noise. Implementing a semantic violation paradigm, 36 English monolingual young adults listened to American-accented (native) and Chinese-accented (nonnative) English sentences with or without semantic anomalies, presented in quiet or embedded in multi-talker noise, while EEG was recorded. After hearing each sentence, participants verbally repeated the sentence, which was coded and scored as an offline comprehension accuracy measure. In line with earlier behavioral studies, the negative impact of background noise on sentence repetition accuracy was higher for nonnative-accented than for native-accented sentences. At the neural level, the N400 effect for semantic anomaly was larger for native-accented than for nonnative-accented sentences, and was also larger for sentences presented in quiet than in noise, indicating impaired lexical-semantic access when listening to nonnative-accented speech or sentences embedded in noise. No semantic N400 effect was observed for nonnative-accented sentences presented in noise. Furthermore, the frequency of neural oscillations in the alpha frequency band (an index of online cognitive listening effort) was higher when listening to sentences in noise versus in quiet, but no difference was observed across the accent conditions. Semantic anomalies presented in background noise also elicited higher theta activity, whereas processing nonnative-accented anomalies was associated with decreased theta activity. Taken together, we found that listening to nonnative accents or background noise is associated with processing challenges during online semantic access, leading to decreased comprehension accuracy. However, the underlying cognitive mechanism (e.g., associated listening efforts) might manifest differently across accented speech processing and speech in noise processing.

Social Priming: Exploring the Effects of Speaker Race and Ethnicity on Perception of Second Language Accents

Listeners use more than just acoustic information when processing speech. Social information, such as a speaker's perceived race or ethnicity, can also affect the processing of the speech signal, in some cases facilitating perception ("social priming"). We aimed to replicate and extend this line of inquiry, examining effects of multiple social primes (i.e., a Middle Eastern, White, or East Asian face, or a control silhouette image) on the perception of Mandarin Chinese-accented English and Arabic-accented English. By including uncommon priming combinations (e.g., a Middle Eastern prime for a Mandarin accent), we aimed to test the specificity of social primes: For example, can a Middle Eastern face facilitate perception of both Arabic-accented English and Mandarin-accented English? Contrary to our predictions, our results indicated no facilitative social priming effects for either of the second language (L2) accents. Results for our examination of specificity were mixed. Trends in the data indicated that the combination of an East Asian prime with Arabic accent resulted in lower accuracy as compared with a White prime, but the combination of a Middle Eastern prime with a Mandarin accent did not (and may have actually benefited listeners to some degree). We conclude that the specificity of priming effects may depend on listeners' level of familiarity with a given accent and/or racial/ethnic group and that the mixed outcomes in the current work motivate further inquiries to determine whether social priming effects for L2-accented speech may be smaller than previously hypothesized and/or highly dependent on listener experience.

Accent Attitudes: A Review Through Affective, Behavioral, and Cognitive Perspectives

Interacting with speakers of different accents is a prevalent global phenomenon. Given the considerable influence of accents in daily life, it is important to conduct a comprehensive review of listeners' accent attitudes. This paper provides an integrative summary of research on accent attitudes, drawing from the Affective, Behavioral, and Cognitive (ABC) perspectives. We begin by outlining the social meaning of accents and laying out the theoretical foundations of the ABC approach. Then, we organize and integrate existing research findings using the ABC framework. Next, we illustrate how the perspectives intersect by discussing pertinent research findings. Drawing from various sociocultural contexts over many years, this review underscores the significant impact of accents on people's thoughts, feelings, and behaviors. The review concludes by discussing limitations, proposing future directions, highlighting real-world relevance, and suggesting areas for research expansion.

Intelligibility of Natively and Nonnatively Produced English Speech Presented in Noise to a Large Cohort of United States Service Members

PURPOSE

A corpus of English matrix sentences produced by 60 native and nonnative speakers of English was developed as part of a multinational coalition task group. This corpus was tested on a large cohort of U.S. Service members in order to examine the effects of talker nativeness, listener nativeness, masker type, and hearing sensitivity on speech recognition performance in this population.

METHOD

A total of 1,939 U.S. Service members (ages 18-68 years) completed this closed-set listening task, including 430 women and 110 nonnative English speakers. Stimuli were produced by native and nonnative speakers of English and were presented in speech-shaped noise and multitalker babble. Keyword recognition accuracy and response times were analyzed.

RESULTS

General(ized) linear mixed-effects regression models found that, on the whole, speech recognition performance was lower for listeners who identified as nonnative speakers of English and when listening to speech produced by nonnative speakers of English. Talker and listener effects were more pronounced when listening in a babble masker than in a speech-shaped noise masker. Response times varied as a function of recognition score, with longest response times found for intermediate levels of performance.

CONCLUSIONS

This study found additive effects of talker and listener nonnativeness when listening to speech in background noise. These effects were present in both accuracy and response time measures. No multiplicative effects of talker and listener language background were found. There was little evidence of a negative interaction between talker nonnativeness and hearing impairment, suggesting that these factors may have redundant effects on speech recognition.

SUPPLEMENTAL MATERIAL

https://doi.org/10.23641/asha.26060191.

The Effects of Acoustic and Semantic Enhancements on Perception of Native and Non-Native Speech

Previous research has shown that native listeners benefit from clearly produced speech, as well as from predictable semantic context when these enhancements are delivered in native speech. However, it is unclear whether native listeners benefit from acoustic and semantic enhancements differently when listening to other varieties of speech, including non-native speech. The current study examines to what extent native English listeners benefit from acoustic and semantic cues present in native and non-native English speech. Native English listeners transcribed sentence final words that were of different levels of semantic predictability, produced in plain- or clear-speaking styles by Native English talkers and by native Mandarin talkers of higher- and lower-proficiency in English. The perception results demonstrated that listeners benefited from semantic cues in higher- and lower-proficiency talkers' speech (i.e., transcribed speech more accurately), but not from acoustic cues, even though higher-proficiency talkers did make substantial acoustic enhancements from plain to clear speech. The current results suggest that native listeners benefit more robustly from semantic cues than from acoustic cues when those cues are embedded in non-native speech.

The role of speech style, frequency, and density in recognition memory for spoken words

What determines whether listeners remember a spoken word? The Effortfulness Hypothesis claims that memory is modulated by a word's intelligibility during real-time processing, while the Distinctiveness Hypothesis claims that it is modulated by a word's distinguishing characteristics. We tested these differing predictions using American English words that varied along three dimensions known to affect both intelligibility and distinctiveness: speech style (clear versus casual), frequency (high versus low), and neighborhood density (high versus low). In a recognition memory experiment, participants (n = 66) listened to a set of study words, and then gave yes/no judgments to indicate whether or not they had heard the word earlier. Results showed that those words which exhibited distinctive characteristics - whether due to clear speech style, low frequency, or low density - were remembered better. The finding supports the Distinctiveness Hypothesis, suggesting that our capacity for remembering words relies on their distinctiveness, rather than on our capacity for recognizing them in real time.

Attention modulates perceptual learning of non-native-accented speech

Listeners readily adapt to variation in non-native-accented speech, learning to disambiguate between talker-specific and accent-based variation. We asked (1) which linguistic and indexical features of the spoken utterance are relevant for this learning to occur and (2) whether task-driven attention to these features affects the extent to which learning generalizes to novel utterances and voices. In two experiments, listeners heard English sentences (Experiment 1) or words (Experiment 2) produced by Spanish-accented talkers during an exposure phase. Listeners' attention was directed to lexical content (transcription), indexical cues (talker identification), or both (transcription + talker identification). In Experiment 1, listeners' test transcription of novel English sentences spoken by Spanish-accented talkers showed generalized perceptual learning to previously unheard voices and utterances for all training conditions. In Experiment 2, generalized learning occurred only in the transcription + talker identification condition, suggesting that attention to both linguistic and indexical cues optimizes listeners' ability to distinguish between individual talker- and group-based variation, especially with the reduced availability of sentence-length prosodic information. Collectively, these findings highlight the role of attentional processes in the encoding of speech input and underscore the interdependency of indexical and lexical characteristics in spoken language processing.

Exploring effects of social information on talker-independent accent adaptation

The present study examined whether race information about speakers can promote rapid and generalizable perceptual adaptation to second-language accent. First-language English listeners were presented with Cantonese-accented English sentences in speech-shaped noise during a training session with three intermixed talkers, followed by a test session with a novel (i.e., fourth) talker. Participants were assigned to view either three East Asian or three White faces during training, corresponding to each speaker. Results indicated no effect of the social priming manipulation on the training or test sessions, although both groups performed better at test than a control group.

Dual-Task Paradigm Measures of Listening Effort: To Include or Not to Include Secondary Task Responses with Incorrect Primary Task Responses

Response time–based dual-task paradigms are commonly adopted to measure behavioral listening effort. Most extant studies used an all-response approach that included secondary task responses under both correct and incorrect primary task responses during analysis. However, evidence supporting this strategy is limited. Therefore, the current study investigated the potential differences between including all responses versus only including correct responses. Data from two previous studies were reanalyzed. Experiment 1 included 16 listeners and used a dual-task paradigm to examine the effect of introducing background noise on listening effort. Experiment 2 included 19 participants and used a different dual-task paradigm to examine the effect of reverberation and loudspeaker-to-listener distance on listening effort. ANOVA results obtained using both analysis approaches were compared. The all-response and correct-only approaches revealed similar results. However, larger effect sizes and an additional main effect were found with the all-response approach. The current study supports the use of an all-response approach due to its greater sensitivity to changes in behavioral listening effort. However, a correct-only approach could be utilized to suit specific study purposes.

Exposure to nonnative-accented speech reduces listening effort and improves social judgments of the speaker

Prior research has revealed a native-accent advantage, whereby nonnative-accented speech is more difficult to process than native-accented speech. Nonnative-accented speakers also experience more negative social judgments. In the current study, we asked three questions. First, does exposure to nonnative-accented speech increase speech intelligibility or decrease listening effort, thereby narrowing the native-accent advantage? Second, does lower intelligibility or higher listening effort contribute to listeners' negative social judgments of speakers? Third and finally, does increased intelligibility or decreased listening effort with exposure to speech bring about more positive social judgments of speakers? To address these questions, normal-hearing adults listened to a block of English sentences with a native accent and a block with nonnative accent. We found that once participants were accustomed to the task, intelligibility was greater for nonnative-accented speech and increased similarly with exposure for both accents. However, listening effort decreased only for nonnative-accented speech, soon reaching the level of native-accented speech. In addition, lower intelligibility and higher listening effort was associated with lower ratings of speaker warmth, speaker competence, and willingness to interact with the speaker. Finally, competence ratings increased over time to a similar extent for both accents, with this relationship fully mediated by intelligibility and listening effort. These results offer insight into how listeners process and judge unfamiliar speakers.

Listen up! ADHD slows spoken-word processing in adverse listening conditions: Evidence from eye movements

BACKGROUND

Cognitive skills such as sustained attention, inhibition and working memory are essential for speech processing, yet are often impaired in people with ADHD. Offline measures have indicated difficulties in speech recognition on multi-talker babble (MTB) background for young adults with ADHD (yaADHD). However, to-date no study has directly tested online speech processing in adverse conditions for yaADHD.

AIMS

Gauging the effects of ADHD on segregating the spoken target-word from its sound-sharing competitor, in MTB and working-memory (WM) load.

METHODS AND PROCEDURES

Twenty-four yaADHD and 22 matched controls that differ in sustained attention (SA) but not in WM were asked to follow spoken instructions presented on MTB to touch a named object, while retaining one (low-load) or four (high-load) digit/s for later recall. Their eye fixations were tracked.

OUTCOMES AND RESULTS

In the high-load condition, speech processing was less accurate and slowed by 140ms for yaADHD. In the low-load condition, the processing advantage shifted from early perceptual to later cognitive stages. Fixation transitions (hesitations) were inflated for yaADHD.

CONCLUSIONS AND IMPLICATIONS

ADHD slows speech processing in adverse listening conditions and increases hesitation, as speech unfolds in time. These effects, detected only by online eyetracking, relate to attentional difficulties. We suggest online speech processing as a novel purview on ADHD. WHAT THIS PAPER ADDS?: We suggest speech processing in adverse listening conditions as a novel vantage point on ADHD. Successful speech recognition in noise is essential for performance across daily settings: academic, employment and social interactions. It involves several executive functions, such as inhibition and sustained attention. Impaired performance in these functions is characteristic of ADHD. However, to date there is only scant research on speech processing in ADHD. The current study is the first to investigate online speech processing as the word unfolds in time using eyetracking for young adults with ADHD (yaADHD). This method uncovered slower speech processing in multi-talker babble noise for yaADHD compared to matched controls. The performance of yaADHD indicated increased hesitation between the spoken word and sound-sharing alternatives (e.g., CANdle-CANdy). These delays and hesitations, on the single word level, could accumulate in continuous speech to significantly impair communication in ADHD, with severe implications on their quality of life and academic success. Interestingly, whereas yaADHD and controls were matched on WM standardized tests, WM load appears to affect speech processing for yaADHD more than for controls. This suggests that ADHD may lead to inefficient deployment of WM resources that may not be detected when WM is tested alone. Note that these intricate differences could not be detected using traditional offline accuracy measures, further supporting the use of eyetracking in speech tasks. Finally, communication is vital for active living and wellbeing. We suggest paying attention to speech processing in ADHD in treatment and when considering accessibility and inclusion.

Short Implicit Voice Training Affects Listening Effort During a Voice Cue Sensitivity Task With Vocoder-Degraded Speech

OBJECTIVES

Understanding speech in real life can be challenging and effortful, such as in multiple-talker listening conditions. Fundamental frequency (fo) and vocal-tract length (vtl) voice cues can help listeners segregate between talkers, enhancing speech perception in adverse listening conditions. Previous research showed lower sensitivity to fo and vtl voice cues when speech signal was degraded, such as in cochlear implant hearing and vocoder-listening compared to normal hearing, likely contributing to difficulties in understanding speech in adverse listening. Nevertheless, when multiple talkers are present, familiarity with a talker's voice, via training or exposure, could provide a speech intelligibility benefit. In this study, the objective was to assess how an implicit short-term voice training could affect perceptual discrimination of voice cues (fo+vtl), measured in sensitivity and listening effort, with or without vocoder degradations.

DESIGN

Voice training was provided via listening to a recording of a book segment for approximately 30 min, and answering text-related questions, to ensure engagement. Just-noticeable differences (JNDs) for fo+vtl were measured with an odd-one-out task implemented as a 3-alternative forced-choice adaptive paradigm, while simultaneously collecting pupil data. The reference voice either belonged to the trained voice or an untrained voice. Effects of voice training (trained and untrained voice), vocoding (non-vocoded and vocoded), and item variability (fixed or variable consonant-vowel triplets presented across three items) on voice cue sensitivity (fo+vtl JNDs) and listening effort (pupillometry measurements) were analyzed.

RESULTS

Results showed that voice training did not have a significant effect on voice cue discrimination. As expected, fo+vtl JNDs were significantly larger for vocoded conditions than for non-vocoded conditions and with variable item presentations than fixed item presentations. Generalized additive mixed models analysis of pupil dilation over the time course of stimulus presentation showed that pupil dilation was significantly larger during fo+vtl discrimination while listening to untrained voices compared to trained voices, but only for vocoder-degraded speech. Peak pupil dilation was significantly larger for vocoded conditions compared to non-vocoded conditions and variable items increased the pupil baseline relative to fixed items, which could suggest a higher anticipated task difficulty.

CONCLUSIONS

In this study, even though short voice training did not lead to improved sensitivity to small fo+vtl voice cue differences at the discrimination threshold level, voice training still resulted in reduced listening effort for discrimination among vocoded voice cues.

Chinese comprehenders' interpretation of underinformativeness in L1 and L2 accented speech narratives

Second language (L2) speakers with foreign accents are well-known to face disadvantages in terms of language processing; however, recent research has demonstrated possible social benefits for foreign-accented L2 speakers. While previous research has focused on the ways in which first language (L1) speakers of English comprehend L2 speech, the present article contributes to this line of research by exploring the ways in which comprehenders from a different culture and linguistic background perceive L2 speech narratives. This study investigates this issue by exploring how comprehenders with Mandarin Chinese as the first language interpret underinformative utterances containing scalar and ad hoc implicature in L1, accent-free L2, and foreign-accented L2 speech narratives. The sentence judgment task with a guise design used written sentences rather than oral utterances as stimuli in order to isolate the role of intelligibility factors. The results indicate that foreign accent confers social benefits on L2 speakers in that their omission of information in communication is tolerated and they are viewed as more likely to possess positive attributes. More importantly, we find that the bilingual characteristics of Chinese participants, as well as the different linguistic complexity of deriving scalar and ad hoc implicature, affect Chinese participants' explanations of underinformative sentences of L2 speakers. This study contributes to our understanding of L2 language processing.

Intelligibility as a measure of speech perception: Current approaches, challenges, and recommendations

Intelligibility measures, which assess the number of words or phonemes a listener correctly transcribes or repeats, are commonly used metrics for speech perception research. While these measures have many benefits for researchers, they also come with a number of limitations. By pointing out the strengths and limitations of this approach, including how it fails to capture aspects of perception such as listening effort, this article argues that the role of intelligibility measures must be reconsidered in fields such as linguistics, communication disorders, and psychology. Recommendations for future work in this area are presented.

Psychoacoustic abilities in pediatric cochlear implant recipients: The relation with short-term memory and working memory capacity

OBJECTIVE

The aim was to investigate school-age children with cochlear implants (CIs) and their typically developing peers in terms of auditory short-term memory (ASTM), auditory working memory (AWM), visuospatial short-term memory (VSTM), visuospatial working memory (VWM), spectral resolution and monosyllabic word recognition in noise.

METHODS

Twenty-three prelingually deaf CI users and twenty-three typically developing (TD) peers aged 7-10 years participated. Twelve children with CI were earlier-implanted (i.e., age at implantation ≤24 months). Children with CIs were compared to typically developing peers and correlations between cognitive and psychoacoustic abilities were computed separately for the groups. Besides, regression analyses were conducted to develop models that could predict SMRT (spectral-temporally modulated ripple test) and speech recognition scores.

RESULTS

The AWM scores of the later-implanted group were significantly lower than both earlier-implanted and TD groups. ASTM scores of TD children were significantly higher than both earlier-implanted and later-implanted participants. There was no statistically significant difference between groups in terms of VSTM and VWM. AWM performance was positively correlated with ASTM, SMRT scores, and speech recognition under noisy conditions for pediatric CI recipients. The AWM was a statistically significant predictor of the SMRT score and the SMRT score was an indicator of speech recognition score under 0 dB SNR condition.

CONCLUSION

Most of children using CI are at risk for clinically remarkable deficits across cognitive abilities such as AWM and ASTM. While evaluating cognitive and psychoacoustic abilities in the clinic routine, it should be kept in mind that they can be influenced by each other.

Adding noise is a confounded nuisance

A wide variety of research and clinical assessments involve presenting speech stimuli in the presence of some kind of noise. Here, I selectively review two theoretical perspectives and discuss ways in which these perspectives may help researchers understand the consequences for listeners of adding noise to a speech signal. I argue that adding noise changes more about the listening task than merely making the signal more difficult to perceive. To fully understand the effects of an added noise on speech perception, we must consider not just how much the noise affects task difficulty, but also how it affects all of the systems involved in understanding speech: increasing message uncertainty, modifying attentional demand, altering affective response, and changing motivation to perform the task.

Neural correlates of multisensory enhancement in audiovisual narrative speech perception: a fMRI investigation

This fMRI study investigated the effect of seeing articulatory movements of a speaker while listening to a naturalistic narrative stimulus. It had the goal to identify regions of the language network showing multisensory enhancement under synchronous audiovisual conditions. We expected this enhancement to emerge in regions known to underlie the integration of auditory and visual information such as the posterior superior temporal gyrus as well as parts of the broader language network, including the semantic system. To this end we presented 53 participants with a continuous narration of a story in auditory alone, visual alone, and both synchronous and asynchronous audiovisual speech conditions while recording brain activity using BOLD fMRI. We found multisensory enhancement in an extensive network of regions underlying multisensory integration and parts of the semantic network as well as extralinguistic regions not usually associated with multisensory integration, namely the primary visual cortex and the bilateral amygdala. Analysis also revealed involvement of thalamic brain regions along the visual and auditory pathways more commonly associated with early sensory processing. We conclude that under natural listening conditions, multisensory enhancement not only involves sites of multisensory integration but many regions of the wider semantic network and includes regions associated with extralinguistic sensory, perceptual and cognitive processing.

Processing emotional prosody in a foreign language: the case of German and Hebrew

This study investigated the universality of emotional prosody in perception of discrete emotions when semantics is not available. In two experiments the perception of emotional prosody in Hebrew and German by listeners who speak one of the languages but not the other was investigated. Having a parallel tool in both languages allowed to conduct controlled comparisons. In Experiment 1, 39 native German speakers with no knowledge of Hebrew and 80 native Israeli speakers rated Hebrew sentences spoken with four different emotional prosodies (anger, fear, happiness, sadness) or neutral. The Hebrew version of the Test for Rating of Emotions in Speech (T-RES) was used for this purpose. Ratings indicated participants’ agreement on how much the sentence conveyed each of four discrete emotions (anger, fear, happiness and sadness). In Experient 2, 30 native speakers of German, and 24 Israeli native speakers of Hebrew who had no knowledge of German rated sentences of the German version of the T-RES. Based only on the prosody, German-speaking participants were able to accurately identify the emotions in the Hebrew sentences and Hebrew-speaking participants were able to identify the emotions in the German sentences. In both experiments ratings between the groups were similar. These findings show that individuals are able to identify emotions in a foreign language even if they do not have access to semantics. This ability goes beyond identification of target emotion; similarities between languages exist even for “wrong” perception. This adds to accumulating evidence in the literature on the universality of emotional prosody.

Aberrant attentional modulation of the auditory steady state response (ASSR) is related to auditory hallucination severity in the first-episode schizophrenia-spectrum

The 40-Hz auditory steady state response (ASSR) is reduced early in schizophrenia, with differences evident even at the first episode of schizophrenia-spectrum psychosis (FESz). Although robust, there is high variability in effect size across studies, possibly due to differences in experimental control of attention and heterogeneity of symptom profiles across studies, both of which may affect the ASSR. We investigated the relationships among ASSR deficits, attention-mediated sensory gain, and auditory hallucinations in 25 FESz (15 male; 23.3 ± 4.5 years) and 32 matched healthy comparison subjects (HC, 22 male; 24.7 ± 5.8 years). ASSR was measured to 40-Hz click trains at three intensities (75, 80, and 85 dB) while participants attended or ignored stimuli. ASSR evoked power and inter-trial phase coherence (ITPC) were measured using the Morlet wavelet transform. FESz did not show overall ASSR power reduction (p > 0.1), but power was significantly increased with attention in HC (p < 0.01), but not in FESz (p > 0.1). Likewise, FESz did not evince overall ASSR ITPC reduction (p > 0.1), and ITPC was significantly increased with attention in HC (p < 0.01), but not in FESz (p > 0.09). Attention-related change in ASSR correlated with auditory hallucination severity for power (r = -0.49, p < 0.05) and ITPC (r = -0.58, p < 0.01). FESz with auditory hallucinations may have pathologically increased basal excitability of auditory cortex and consequent reduced ability to further increase auditory cortex sensory gain with focused attention. These findings indicate hallucination-related pathophysiology early in schizophrenia and may guide novel intervention strategies aimed to modulate basal activity levels.

Does race impact speech perception? An account of accented speech in two different multilingual locales

Upon hearing someone's speech, a listener can access information such as the speaker's age, gender identity, socioeconomic status, and their linguistic background. However, an open question is whether living in different locales modulates how listeners use these factors to assess speakers' speech. Here, an audio-visual test was used to measure whether listeners' accentedness judgments and intelligibility (i.e., speech perception) can be modulated depending on racial information in faces that they see. American, British, and Indian English were used as three different English varieties of speech. These speech samples were presented with either a white female face or a South Asian female face. Two experiments were completed in two locales: Gainesville, Florida (USA) and Montreal, Quebec (Canada). Overall, Montreal listeners were more accurate in their transcription of sentences (i.e., intelligibility) compared to Gainesville listeners. Moreover, Gainesville listeners' ability to transcribe the same spoken sentences decreased for all varieties when listening to speech paired with South Asian faces. However, seeing a white or a South Asian face did not impact speech intelligibility for the same spoken sentences for Montreal listeners. Finally, listeners' accentedness judgments increased for American English and Indian English when the visual information changed from a white face to a South Asian face in Gainesville, but not in Montreal. These findings suggest that visual cues for race impact speech perception to a greater degree in locales with greater ecological diversity.

Semantics-Prosody Stroop Effect on English Emotion Word Processing in Chinese College Students With Trait Depression

This study explored the performance of Chinese college students with different severity of trait depression to process English emotional speech under a complete semantics-prosody Stroop effect paradigm in quiet and noisy conditions. A total of 24 college students with high-trait depression and 24 students with low-trait depression participated in this study. They were required to selectively attend to either the prosodic emotion (happy, sad) or semantic valence (positive and negative) of the English words they heard and then respond quickly. Both prosody task and semantic task were performed in quiet and noisy listening conditions. Results showed that the high-trait group reacted slower than the low-trait group in the prosody task due to their bluntness and insensitivity toward emotional processing. Besides, both groups reacted faster under the consistent situation, showing a clear congruency-induced facilitation effect and the wide existence of the Stroop effect in both tasks. Only the Stroop effect played a bigger role during emotional prosody identification in quiet condition, and the noise eliminated such an effect. For the sake of experimental design, both groups spent less time on the prosody task than the semantic task regardless of consistency in all listening conditions, indicating the friendliness of basic emotion identification and the difficulty for second language learners in face of semantic judgment. These findings suggest the unneglectable effects of college students' mood conditions and noise outside on emotion word processing.

Reading aloud in clear speech reduces sentence recognition memory and recall for native and non-native talkers

Speaking style variation plays a role in how listeners remember speech. Compared to conversational sentences, clearly spoken sentences were better recalled and identified as previously heard by native and non-native listeners. The present study investigated whether speaking style variation also plays a role in how talkers remember speech that they produce. Although distinctive forms of production (e.g., singing, speaking loudly) can enhance memory, the cognitive and articulatory efforts required to plan and produce listener-oriented hyper-articulated clear speech could detrimentally affect encoding and subsequent retrieval. Native and non-native English talkers' memories for sentences that they read aloud in clear and conversational speaking styles were assessed through a sentence recognition memory task (experiment 1; N = 90) and a recall task (experiment 2; N = 75). The results showed enhanced recognition memory and recall for sentences read aloud conversationally rather than clearly for both talker groups. In line with the "effortfulness" hypothesis, producing clear speech may increase the processing load diverting resources from memory encoding. Implications for the relationship between speech perception and production are discussed.

Extended High-frequency Hearing Impairment Despite a Normal Audiogram: Relation to Early Aging, Speech-in-noise Perception, Cochlear Function, and Routine Earphone Use

OBJECTIVES

Humans can hear up to 20 kHz. Emerging evidence suggests that hearing in the extended high frequencies (EHFs; >8 kHz) contributes to speech perception in noise. The objective of the present study was to describe the features of EHF hearing impairment in young adults with normal standard audiograms (0.25-8 kHz). Specifically, the study goals were to: (1) characterize the EHF hearing impairment and identify potential risk factors; (2) elucidate the age-related changes in EHF hearing; (3) determine the effect of EHF hearing impairment on speech-in-noise recognition; and (4) examine the extent to which EHF hearing impairment influences cochlear functioning in the standard frequencies.

DESIGN

Hearing thresholds at standard frequencies and EHFs (10, 12.5, 14, and 16 kHz), and speech recognition thresholds (SRTs) using digit triplets in multi-talker babble were measured in both ears from 222 participants (19-38 years; n = 444 ears) with normal audiograms (≤20 dB HL at standard frequencies). Test-retest measurement of hearing thresholds was obtained in a subset of 50 participants (100 ears), and clinical distortion product otoacoustic emissions (f2 frequency = 2, 3, 4, and 5 kHz) were recorded in 49 participants (98 ears).

RESULTS

Forty-two of 222 participants had EHF hearing impairment (>20 dB HL for at least one EHF in either ear). Only seven individuals with EHF impairment had significant case history and/or listening-in-noise complaints. A breakpoint in the threshold-age function was observed for the EHFs for males but not for females. Linear mixed models revealed a significant effect of age, pure-tone averages for speech frequencies (0.5, 1, 2, and 4 kHz), and EHFs and group (NH versus EHF hearing impairment) independent of each other on the SRTs. Individuals with EHF hearing impairment had less measurable emissions and when present, had a lower magnitude of otoacoustic emissions relative to NH controls. There was no difference in hearing thresholds, SRTs, or otoacoustic emissions between earphone users and nonusers.

CONCLUSIONS

The hearing thresholds for the EHFs exhibit signs of early auditory aging. Age-related deterioration in auditory function can be observed in the third decade of human life. A breakpoint in the threshold-age function suggests that rapid aging processes are operational at a relatively younger age (21 years) for males. The audibility of EHFs contributes to speech-in-noise recognition. EHF hearing impairment independent of age and speech frequencies can affect speech-in-noise recognition. Reduced distortion product otoacoustic emissions in the standard frequencies may suggest preclinical cochlear degeneration in individuals with EHF hearing impairment.

The cortical organization of listening effort: New insight from functional near-infrared spectroscopy

Everyday challenges impact our ability to hear and comprehend spoken language with ease, such as accented speech (source factors), spectral degradation (transmission factors), complex or unfamiliar language use (message factors), and predictability (context factors). Auditory degradation and linguistic complexity in the brain and behavior have been well investigated, and several computational models have emerged. The work here provides a novel test of the hypotheses that listening effort is partially reliant on higher cognitive auditory attention and working memory mechanisms in the frontal lobe, and partially reliant on hierarchical linguistic computation in the brain's left hemisphere. We specifically hypothesize that these models are robust and can be applied in ecologically relevant and coarse-grain contexts that rigorously control for acoustic and linguistic listening challenges. Using functional near-infrared spectroscopy during an auditory plausibility judgment task, we show the hierarchical cortical organization for listening effort in the frontal and left temporal-parietal brain regions. In response to increasing levels of cognitive demand, we found (i) poorer comprehension, (ii) slower reaction times, (iii) increasing levels of perceived mental effort, (iv) increasing levels of brain activity in the prefrontal cortex, (v) hierarchical modulation of core language processing regions that reflect increasingly higher-order auditory-linguistic processing, and (vi) a correlation between participants' mental effort ratings and their performance on the task. Our results demonstrate that listening effort is partly reliant on higher cognitive auditory attention and working memory mechanisms in the frontal lobe and partly reliant on hierarchical linguistic computation in the brain's left hemisphere. Further, listening effort is driven by a voluntary, motivation-based attention system for which our results validate the use of a single-item post-task questionnaire for measuring perceived levels of mental effort and predicting listening performance. We anticipate our study to be a starting point for more sophisticated models of listening effort and even cognitive neuroplasticity in hearing aid and cochlear implant users.

How much COVID-19 face protections influence speech intelligibility in classrooms?

The ongoing pandemic caused by the COVID-19 virus is challenging many aspects of daily life. Several personal protective devices have become essential in our lives. Face protections are mostly used in order to stop the air aerosol coming out of our mouths. Nevertheless, this fact may also have a negative effect on speech transmission both in outdoor and indoor spaces. After a severe lockdown, classes have now started again. The adoption of face protection by teachers is either recommended or mandatory even though this is affecting speech intelligibility and thus students' comprehension. This study aims to understand how protections may affect the speech transmission in classrooms and how this could be influenced by the several typologies of face protections. An experimental campaign was conducted in a classroom in two different reverberant conditions, measuring and comparing the variation in speech transmission and sound pressure level at different receiver positions. Furthermore, a microphone array was used to investigate the distribution of the indoor sound field, depending on the sound source. Results clearly show how different types of personal protection equipment do affect speech transmission and sound pressure level especially at mid-high frequency and that the source emission lobes vary when wearing certain types of personal devices.

The Other Accent Effect in Talker Recognition: Now You See It, Now You Don't

The existence of the Language Familiarity Effect (LFE), where talkers of a familiar language are easier to identify than talkers of an unfamiliar language, is well-documented and uncontroversial. However, a closely related phenomenon known as the Other Accent Effect (OAE), where accented talkers are more difficult to recognize, is less well understood. There are several possible explanations for why the OAE exists, but to date, little data exist to adjudicate differences between them. Here, we begin to address this issue by directly comparing listeners' recognition of talkers who speak in different types of accents, and by examining both the LFE and OAE in the same set of listeners. Specifically, Canadian English listeners were tested on their ability to recognize talkers within four types of voice line-ups: Canadian English talkers, Australian English talkers, Mandarin-accented English talkers, and Mandarin talkers. We predicted that the OAE would be present for talkers of Mandarin-accented English but not for talkers of Australian English-which is precisely what we observed. We also observed a disconnect between listeners' confidence and performance across different types of accents; that is, listeners performed equally poorly with Mandarin and Mandarin-accented talkers, but they were more confident with their performance with the latter group of talkers. The present findings set the stage for further investigation into the nature of the OAE by exploring a range of potential explanations for the effect, and introducing important implications for forensic scientists' evaluation of ear witness testimony.

Development of an Australian behavioural method for assessing listening task difficulty at high speech intelligibility levels

OBJECTIVE

To develop and validate an Australian version of a behavioural test for assessing listening task difficulty at high speech intelligibility levels.

DESIGN

In the SWIR-Aus test, listeners perform two tasks: identify the last word of each of seven sentences in a list and recall the identified words after each list. First, the test material was developed by creating seven-sentence lists with similar final-word features. Then, for the validation, participant's performance on the SWIR-Aus test was compared when a binary mask noise reduction algorithm was on and off.

STUDY SAMPLE

All participants in this study had normal hearing thresholds. Nine participants (23.8-56.0 years) participated in the characterisation of the speech material. Another thirteen participants (18.4-59.1 years) participated in a pilot test to determine the SNR to use at the validation stage. Finally, twenty-four new participants (20.0-56.9 years) participated in the validation of the test.

RESULTS

The results of the validation of the test showed that recall and identification scores were significantly better when the binary mask noise reduction algorithm was on compared to off.

CONCLUSIONS

The SWIR-Aus test was developed using Australian speech material and can be used for assessing task difficulty at high speech intelligibility levels.

Face masks and speaking style affect audio-visual word recognition and memory of native and non-native speech

Though necessary, protective mask wearing in response to the COVID-19 pandemic presents communication challenges. The present study examines how signal degradation and loss of visual information due to masks affects intelligibility and memory for native and non-native speech. We also test whether clear speech can alleviate perceptual difficulty for masked speech. One native and one non-native speaker of English recorded video clips in conversational speech without a mask and conversational and clear speech with a mask. Native English listeners watched video clips presented in quiet or mixed with competing speech. The results showed that word recognition and recall of speech produced with a mask can be as accurate as without a mask in optimal listening conditions. Masks affected non-native speech processing at easier noise levels than native speech. Clear speech with a mask significantly improved accuracy in all listening conditions. Speaking clearly, reducing noise, and using surgical masks as well as good signal amplification can help compensate for the loss of intelligibility due to background noise, lack of visual cues, physical distancing, or non-native speech. The findings have implications for communication in classrooms and hospitals where listeners interact with teachers and healthcare providers, oftentimes non-native speakers, through their protective barriers.

Listener characteristics differentially affect self-reported and physiological measures of effort associated with two challenging listening conditions

Listeners vary in their ability to understand speech in adverse conditions. Differences in both cognitive and linguistic capacities play a role, but increasing evidence suggests that such factors may contribute differentially depending on the listening challenge. Here, we used multilevel modeling to evaluate contributions of individual differences in age, hearing thresholds, vocabulary, selective attention, working memory capacity, personality traits, and noise sensitivity to variability in measures of comprehension and listening effort in two listening conditions. A total of 35 participants completed a battery of cognitive and linguistic tests as well as a spoken story comprehension task using (1) native-accented English speech masked by speech-shaped noise and (2) nonnative accented English speech without masking. Masker levels were adjusted individually to ensure each participant would show (close to) equivalent word recognition performance across the two conditions. Dependent measures included comprehension tests results, self-rated effort, and electrodermal, cardiovascular, and facial electromyographic measures associated with listening effort. Results showed varied patterns of responsivity across different dependent measures as well as across listening conditions. In particular, results suggested that working memory capacity may play a greater role in the comprehension of nonnative accented speech than noise-masked speech, while hearing acuity and personality may have a stronger influence on physiological responses affected by demands of understanding speech in noise. Furthermore, electrodermal measures may be more strongly affected by affective response to noise-related interference while cardiovascular responses may be more strongly affected by demands on working memory and lexical access.

General adaptation to accented English: Speech intelligibility unaffected by perceived source of non-native accent

Foreign-accented speech commonly incurs a processing cost, but this cost can be offset when listeners are given informative cues to the speaker's purported ethnicity and/or language background. This study investigates the mechanism that underlies this facilitatory effect of top-down expectation, evaluating between general adaptation (an across-the-board relaxation of phonetic categorization criteria) and targeted adaptation (tuning in to accent-specific phonetics). In experiment 1, native speakers of American English completed a transcription-in-noise task with Chinese-accented English sentences. All listeners heard the same voice but were randomly assigned to one of four visual conditions: a blank silhouette, a European face, an East Asian face, or a South Asian face. Results showed that although there was no significant effect of visual condition, listeners who believed the speaker to be non-natively accented enjoyed significantly improved performance compared to those who reported hearing a native accent. Crucially, however, listeners who correctly perceived the speaker as Chinese-accented showed no additional benefit over those who heard some other foreign accent. This basic pattern held even when listeners were primed to expect congruent face-accent pairings (experiment 2). Overall, these results provide evidence for a general adaptation mechanism, rather than a targeted mechanism involving accent-specific phonetic adjustments.

Adaptation to mis-pronounced speech: evidence for a prefrontal-cortex repair mechanism

Speech is a complex and ambiguous acoustic signal that varies significantly within and across speakers. Despite the processing challenge that such variability poses, humans adapt to systematic variations in pronunciation rapidly. The goal of this study is to uncover the neurobiological bases of the attunement process that enables such fluent comprehension. Twenty-four native English participants listened to words spoken by a “canonical” American speaker and two non-canonical speakers, and performed a word-picture matching task, while magnetoencephalography was recorded. Non-canonical speech was created by including systematic phonological substitutions within the word (e.g. [s] → [sh]). Activity in the auditory cortex (superior temporal gyrus) was greater in response to substituted phonemes, and, critically, this was not attenuated by exposure. By contrast, prefrontal regions showed an interaction between the presence of a substitution and the amount of exposure: activity decreased for canonical speech over time, whereas responses to non-canonical speech remained consistently elevated. Grainger causality analyses further revealed that prefrontal responses serve to modulate activity in auditory regions, suggesting the recruitment of top-down processing to decode non-canonical pronunciations. In sum, our results suggest that the behavioural deficit in processing mispronounced phonemes may be due to a disruption to the typical exchange of information between the prefrontal and auditory cortices as observed for canonical speech.

Rapid adaptation to non-native speech is impaired in cochlear implant users

To examine difficulties experienced by cochlear implant (CI) users when perceiving non-native speech, intelligibility of non-native speech was compared in conditions with single and multiple alternating talkers. Compared to listeners with normal hearing, no rapid talker-dependent adaptation was observed and performance was approximately 40% lower for CI users following increased exposure in both talker conditions. Results suggest that lower performance for CI users may stem from combined effects of limited spectral resolution, which diminishes perceptible differences across accents, and limited access to talker-specific acoustic features of speech, which reduces the ability to adapt to non-native speech in a talker-dependent manner.

Listening Effort in Native and Nonnative English-Speaking Children Using Low Linguistic Single- and Dual-Task Paradigms

Purpose It is not clear if behavioral indices of listening effort are sensitive to changes in signal-to-noise ratio (SNR) for young children (7-12 years old) from multilingual backgrounds. The purpose of this study was to explore the effects of SNR on listening effort in multilingual school-aged children (native English, nonnative English) as measured with a single- and a dual-task paradigm with low-linguistic speech stimuli (digits). The study also aimed to explore age effects on digit triplet recognition and response times (RTs). Method Sixty children with normal hearing participated, 30 per language group. Participants completed single and dual tasks in three SNRs (quiet, -10 dB, and -15 dB). Speech stimuli for both tasks were digit triplets. Verbal RTs were the listening effort measure during the single-task paradigm. A visual monitoring task was the secondary task during the dual-task paradigm. Results Significant effects of SNR on RTs were evident during both single- and dual-task paradigms. As expected, language background did not affect the pattern of RTs. The data also demonstrate a maturation effect for triplet recognition during both tasks and for RTs during the dual-task only. Conclusions Both single- and dual-task paradigms were sensitive to changes in SNR for school-aged children between 7 and 12 years of age. Language background (English as native language vs. English as nonnative language) had no significant effect on triplet recognition or RTs, demonstrating practical utility of low-linguistic stimuli for testing children from multilingual backgrounds.

Rapid adaptation to fully intelligible nonnative-accented speech reduces listening effort

In noisy settings or when listening to an unfamiliar talker or accent, it can be difficult to understand spoken language. This difficulty typically results in reductions in speech intelligibility, but may also increase the effort necessary to process the speech even when intelligibility is unaffected. In this study, we used a dual-task paradigm and pupillometry to assess the cognitive costs associated with processing fully intelligible accented speech, predicting that rapid perceptual adaptation to an accent would result in decreased listening effort over time. The behavioural and physiological paradigms provided converging evidence that listeners expend greater effort when processing nonnative- relative to native-accented speech, and both experiments also revealed an overall reduction in listening effort over the course of the experiment. Only the pupillometry experiment, however, revealed greater adaptation to nonnative- relative to native-accented speech. An exploratory analysis of the dual-task data that attempted to minimise practice effects revealed weak evidence for greater adaptation to the nonnative accent. These results suggest that even when speech is fully intelligible, resolving deviations between the acoustic input and stored lexical representations incurs a processing cost, and adaptation may attenuate this cost.

Predicting Intelligibility Deficits in Parkinson's Disease With Perceptual Speech Ratings

Purpose This study investigated whether perceptual ratings of speech parameters were predictive of transcription intelligibility in quiet and in noise for speakers with Parkinson's disease (PD). Method Ten speakers with PD and five healthy controls read 56 sentences. One group of 60 listeners orthographically transcribed the sentences in quiet, and another group of 60 listeners transcribed in noise. An additional 23 listeners judged a variety of speech parameters, including articulation, prosody, resonance, voice quality, and ease of understanding on a visual analog scale. Scores of the visual analog scale ratings were regressed against transcription intelligibility in quiet and in noise. Results Perceptual ratings of all the speech parameters were lower for speakers with PD. Global speech understanding, indexed by ease of understanding ratings, was associated with transcription intelligibility in quiet and in noise with a stronger effect in noise. Among the rated speech parameters, ease of understanding and voice quality ratings were significant predictors of speech intelligibility in noise. Conclusions Speech in individuals with PD was more difficult for listeners to understand and was characterized by deficits in articulation, prosody, resonance, and voice quality compared to normal speech produced by healthy older adults. Ease of understanding ratings, even when performed in quiet, predicted intelligibility in noise. Poor voice quality ratings in PD, a sign of phonatory deficit, had a negative impact on intelligibility in noise for speakers with PD.

Task-evoked pupil response for accurately recognized accented speech

Unfamiliar second-language (L2) accents present a common challenge to speech understanding. However, the extent to which accurately recognized unfamiliar L2-accented speech imposes a greater cognitive load than native speech remains unclear. The current study used pupillometry to assess cognitive load for native English listeners during the perception of intelligible Mandarin Chinese-accented English and American-accented English. Results showed greater pupil response (indicating greater cognitive load) for the unfamiliar L2-accented speech. These findings indicate that the mismatches between unfamiliar L2-accented speech and native listeners' linguistic representations impose greater cognitive load even when recognition accuracy is at ceiling.

Listening Effort Among Adults With and Without Attention-Deficit/Hyperactivity Disorder

Purpose Few studies have assessed listening effort (LE)-the cognitive resources required to perceive speech-among populations with intact hearing but reduced availability of cognitive resources. Attention/deficit/hyperactivity disorder (ADHD) is theorized to restrict attention span, possibly making speech perception in adverse conditions more challenging. This study examined the effect of ADHD on LE among adults using a behavioral dual-task paradigm (DTP). Method Thirty-nine normal-hearing adults (aged 21-27 years) participated: 19 with ADHD (ADHD group) and 20 without ADHD (control group). Baseline group differences were measured in visual and auditory attention as well as speech perception. LE using DTP was assessed as the performance difference on a visual-motor task versus a simultaneous auditory and visual-motor task. Results Group differences in attention were confirmed by differences in visual attention (larger reaction times between congruent and incongruent conditions) and auditory attention (lower accuracy in the presence of distractors) among the ADHD group, compared to the controls. LE was greater among the ADHD group than the control group. Nevertheless, no group differences were found in speech perception. Conclusions LE is increased among those with ADHD. As a DTP assumes limited cognitive capacity to allocate attentional resources, LE among those with ADHD may be increased because higher level cognitive processes are more taxed in this population. Studies on LE using a DTP should take into consideration mechanisms of selective and divided attention. Among young adults who need to continuously process great volumes of auditory and visual information, much more effort may be expended by those with ADHD than those without it. As a result, those with ADHD may be more prone to fatigue and irritability, similar to those who are engaged in more outwardly demanding tasks.

Clear speech improves listeners' recall

The present study examined the effect of intelligibility-enhancing clear speech on listeners' recall. Native (n = 57) and non-native (n = 31) English listeners heard meaningful sentences produced in clear and conversational speech, and then completed a cued-recall task. Results showed that listeners recalled more words from clearly produced sentences. Sentence-level analysis revealed that listening to clear speech increased the odds of recalling whole sentences and decreased the odds of erroneous and omitted responses. This study showed that the clear speech benefit extends beyond word- and sentence-level recognition memory to include deeper linguistic encoding at the level of syntactic and semantic information.

Easy-to-hard effects in perceptual learning depend upon the degree to which initial trials are "easy"

Starting perceptual training at easy levels before progressing to difficult levels generally produces better learning outcomes than constantly difficult training does. However, little is known about how "easy" these initial levels should be in order to yield easy-to-hard effects. We compared five levels of initial training block difficulty varying from very easy to hard in two auditory-discrimination learning tasks-a frequency modulation rate discrimination (Experiment 1) and a frequency range discrimination (Experiment 2). The degree of difficulty was based on individualized pretraining ~71% correct discrimination thresholds. Both experiments revealed a sweet spot for easy-to-hard effects. Conditions where initial blocks were either too easy or too difficult produced less benefit than did blocks of intermediate difficulty. Results challenge assumptions that sequencing effects in learning are related to attentional spotlighting of task-relevant dimensions. Rather, they support incremental learning models that account for easy-to-hard effects. Further, the results have implications for how perceptual training regimens should be designed to maximize the benefits of rehabilitative perceptual training.

Intelligibility and Listening Effort of Spanish Oesophageal Speech

Communication is a huge challenge for oesophageal speakers, be it for interactions with fellow humans or with digital voice assistants. We aim to quantify these communication challenges (both human–human and human–machine interactions) by measuring intelligibility and Listening Effort (LE) of Oesophageal Speech (OS) in comparison to Healthy Laryngeal Speech (HS). We conducted two listening tests (one web-based, the other in laboratory settings) to collect these measurements. Participants performed a sentence recognition and LE rating task in each test. Intelligibility, calculated as Word Error Rate, showed significant correlation with self-reported LE ratings. Speaker type (healthy or oesophageal) had a major effect on intelligibility and effort. More LE was reported for OS compared to HS even when OS intelligibility was close to HS. Listeners familiar with OS reported less effort when listening to OS compared to nonfamiliar listeners. However, such advantage of familiarity was not observed for intelligibility. Automatic speech recognition scores were higher for OS compared to HS.

To believe or not to believe? How voice and accent information in speech alter listener impressions of trust

Our decision to believe what another person says can be influenced by vocally expressed confidence in speech and by whether the speaker–listener are members of the same social group. The dynamic effects of these two information sources on neurocognitive processes that promote believability impressions from vocal cues are unclear. Here, English Canadian listeners were presented personal statements ( She has access to the building) produced in a confident or doubtful voice by speakers of their own dialect (in-group) or speakers from two different “out-groups” (regional or foreign-accented English). Participants rated how believable the speaker is for each statement and event-related potentials (ERPs) were analysed from utterance onset. Believability decisions were modulated by both the speaker’s vocal confidence level and their perceived in-group status. For in-group speakers, ERP effects revealed an early differentiation of vocally expressed confidence (i.e., N100, P200), highlighting the motivational significance of doubtful voices for drawing believability inferences. These early effects on vocal confidence perception were qualitatively different or absent when speakers had an accent; evaluating out-group voices was associated with increased demands on contextual integration and re-analysis of a non-native representation of believability (i.e., increased N400, late negativity response). Accent intelligibility and experience with particular out-group accents each influenced how vocal confidence was processed for out-group speakers. The N100 amplitude was sensitive to out-group attitudes and predicted actual believability decisions for certain out-group speakers. We propose a neurocognitive model in which vocal identity information (social categorization) dynamically influences how vocal expressions are decoded and used to derive social inferences during person perception.

Listening effort: Are we measuring cognition or affect, or both?

Listening effort is increasingly recognized as a factor in communication, particularly for and with nonnative speakers, for the elderly, for individuals with hearing impairment and/or for those working in noise. However, as highlighted by McGarrigle et al., International Journal of Audiology, 2014, 53, 433-445, the term "listening effort" encompasses a wide variety of concepts, including the engagement and control of multiple possibly distinct neural systems for information processing, and the affective response to the expenditure of those resources in a given context. Thus, experimental or clinical methods intended to objectively quantify listening effort may ultimately reflect a complex interaction between the operations of one or more of those information processing systems, and/or the affective and motivational response to the demand on those systems. Here we examine theoretical, behavioral, and psychophysiological factors related to resolving the question of what we are measuring, and why, when we measure "listening effort." This article is categorized under: Linguistics > Language in Mind and Brain Psychology > Theory and Methods Psychology > Attention Psychology > Emotion and Motivation.

Expectations about the source of a speaker's accent affect accent adaptation

When encountering speakers whose accents differ from the listener's own, listeners initially show a processing cost, but that cost can be attenuated after short term exposure. The extent to which processing foreign accents (L2-accents) and within-language accents (L1-accents) is similar is still an open question. This study considers whether listeners' expectations about the source of a speaker's accent-whether the speaker is purported to be an L1 or an L2 speaker-affect intelligibility. Prior work has indirectly manipulated expectations about a speaker's accent through photographs, but the present study primes listeners with a description of the speaker's accent itself. In experiment 1, native English listeners transcribed Spanish-accented English sentences in noise under three different conditions (speaker's accent: monolingual L1 Latinx English, L1-Spanish/L2-English, no information given). Results indicate that, by the end of the experiment, listeners given some information about the accent outperformed listeners given no information, and listeners told the speaker was L1-accented outperformed listeners told to expect L2-accented speech. Findings are interpreted in terms of listeners' expectations about task difficulty, and a follow-up experiment (experiment 2) found that priming listeners to expect that their ability to understand L2-accented speech can improve does in fact improve intelligibility.

Listening Effort by Native and Nonnative Listeners Due to Noise, Reverberation, and Talker Foreign Accent During English Speech Perception

Purpose Understanding speech in complex realistic acoustic environments requires effort. In everyday listening situations, speech quality is often degraded due to adverse acoustics, such as excessive background noise level (BNL) and reverberation time (RT), or talker characteristics such as foreign accent ( Mattys, Davis, Bradlow, & Scott, 2012 ). In addition to factors affecting the quality of the input acoustic signals, listeners' individual characteristics such as language abilities can also make it more difficult and effortful to understand speech. Based on the Framework for Understanding Effortful Listening ( Pichora-Fuller et al., 2016 ), factors such as adverse acoustics, talker accent, and listener language abilities can all contribute to increasing listening effort. In this study, using both a dual-task paradigm and a self-report questionnaire, we seek to understand listening effort in a wide range of realistic classroom acoustic conditions as well as varying talker accent and listener English proficiency. Method One hundred fifteen native and nonnative adult listeners with normal hearing were tested in a dual task of speech comprehension and adaptive pursuit rotor (APR) under 15 acoustic conditions from combinations of BNLs and RTs. Listeners provided responses on the NASA Task Load Index (TLX) questionnaire immediately after completing the dual task under each acoustic condition. The NASA TLX surveyed 6 dimensions of perceived listening effort: mental demand, physical demand, temporal demand, effort, frustration, and perceived performance. Fifty-six listeners were tested with speech produced by native American English talkers; the other 59 listeners, with speech from native Mandarin Chinese talkers. Based on their 1st language learned during childhood, 3 groups of listeners were recruited: listeners who were native English speakers, native Mandarin Chinese speakers, and native speakers of other languages (e.g., Hindu, Korean, and Portuguese). Results Listening effort was measured objectively through the APR task performance and subjectively using the NASA TLX questionnaire. Performance on the APR task did not vary with changing acoustic conditions, but it did suggest increased listening effort for native listeners of other languages compared to the 2 other listener groups. From the NASA TLX, listeners reported feeling more frustrated and less successful in understanding Chinese-accented speech. Nonnative listeners reported more listening effort (i.e., physical demand, temporal demand, and effort) than native listeners in speech comprehension under adverse acoustics. When listeners' English proficiency was controlled, higher BNL was strongly related to a decrease in perceived performance, whereas such relationship with RT was much weaker. Nonnative listeners who shared the foreign talkers' accent reported no change in listening effort, whereas other listeners reported more difficulty in understanding the accented speech. Conclusions Adverse acoustics required more effortful listening as measured subjectively with a self-report NASA TLX. This subjective scale was more sensitive than a dual task that involved speech comprehension, which was beyond sentence recall. It was better at capturing the negative impacts on listening effort from acoustic factors (i.e., both BNL and RT), talker accent, and listener language abilities.

Listening Effort: How the Cognitive Consequences of Acoustic Challenge Are Reflected in Brain and Behavior

Everyday conversation frequently includes challenges to the clarity of the acoustic speech signal, including hearing impairment, background noise, and foreign accents. Although an obvious problem is the increased risk of making word identification errors, extracting meaning from a degraded acoustic signal is also cognitively demanding, which contributes to increased listening effort. The concepts of cognitive demand and listening effort are critical in understanding the challenges listeners face in comprehension, which are not fully predicted by audiometric measures. In this article, the authors review converging behavioral, pupillometric, and neuroimaging evidence that understanding acoustically degraded speech requires additional cognitive support and that this cognitive load can interfere with other operations such as language processing and memory for what has been heard. Behaviorally, acoustic challenge is associated with increased errors in speech understanding, poorer performance on concurrent secondary tasks, more difficulty processing linguistically complex sentences, and reduced memory for verbal material. Measures of pupil dilation support the challenge associated with processing a degraded acoustic signal, indirectly reflecting an increase in neural activity. Finally, functional brain imaging reveals that the neural resources required to understand degraded speech extend beyond traditional perisylvian language networks, most commonly including regions of prefrontal cortex, premotor cortex, and the cingulo-opercular network. Far from being exclusively an auditory problem, acoustic degradation presents listeners with a systems-level challenge that requires the allocation of executive cognitive resources. An important point is that a number of dissociable processes can be engaged to understand degraded speech, including verbal working memory and attention-based performance monitoring. The specific resources required likely differ as a function of the acoustic, linguistic, and cognitive demands of the task, as well as individual differences in listeners' abilities. A greater appreciation of cognitive contributions to processing degraded speech is critical in understanding individual differences in comprehension ability, variability in the efficacy of assistive devices, and guiding rehabilitation approaches to reducing listening effort and facilitating communication.

Linguistic Context Versus Semantic Competition in Word Recognition by Younger and Older Adults With Cochlear Implants

OBJECTIVES

The increasing numbers of older adults now receiving cochlear implants raises the question of how the novel signal produced by cochlear implants may interact with cognitive aging in the recognition of words heard spoken within a linguistic context. The objective of this study was to pit the facilitative effects of a constraining linguistic context against a potential age-sensitive negative effect of response competition on effectiveness of word recognition.

DESIGN

Younger (n = 8; mean age = 22.5 years) and older (n = 8; mean age = 67.5 years) adult implant recipients heard 20 target words as the final words in sentences that manipulated the target word's probability of occurrence within the sentence context. Data from published norms were also used to measure response entropy, calculated as the total number of different responses and the probability distribution of the responses suggested by the sentence context. Sentence-final words were presented to participants using a word-onset gating paradigm, in which a target word was presented with increasing amounts of its onset duration in 50 msec increments until the word was correctly identified.

RESULTS

Results showed that for both younger and older adult implant users, the amount of word-onset information needed for correct recognition of sentence-final words was inversely proportional to their likelihood of occurrence within the sentence context, with older adults gaining differential advantage from the contextual constraints offered by a sentence context. On the negative side, older adults' word recognition was differentially hampered by high response entropy, with this effect being driven primarily by the number of competing responses that might also fit the sentence context.

CONCLUSIONS

Consistent with previous research with normal-hearing younger and older adults, the present results showed older adult implant users' recognition of spoken words to be highly sensitive to linguistic context. This sensitivity, however, also resulted in a greater degree of interference from other words that might also be activated by the context, with negative effects on ease of word recognition. These results are consistent with an age-related inhibition deficit extending to the domain of semantic constraints on word recognition.

Sentence Context Facilitation for Children's and Adults' Recognition of Native- and Nonnative-Accented Speech

Purpose Supportive semantic and syntactic information can increase children's and adults' word recognition accuracy in adverse listening conditions. However, there are inconsistent findings regarding how a talker's accent or dialect modulates these context effects. Here, we compare children's and adults' abilities to capitalize on sentence context to overcome misleading acoustic-phonetic cues in nonnative-accented speech. Method Monolingual American English-speaking 5- to 7-year-old children ( n = 90) and 18- to 35-year-old adults ( n = 30) were presented with full sentences or the excised final word from each of the sentences and repeated what they heard. Participants were randomly assigned to 1 of 2 conditions: native-accented (Midland American English) or nonnative-accented (Spanish- and Japanese-accented English) speech. Participants also completed the NIH Toolbox Picture Vocabulary Test. Results Children and adults benefited from sentence context for both native- and nonnative-accent talkers, but the benefit was greater for nonnative than native talkers. Furthermore, adults showed a greater context benefit than children for nonnative talkers, but the 2 age groups showed a similar benefit for native talkers. Children's age and vocabulary scores both correlated with context benefit. Conclusions The cognitive-linguistic development that occurs between the early school-age years and adulthood may increase listeners' abilities to capitalize on top-down cues for lexical identification with nonnative-accented speech. These results have implications for the perception of speech with source degradation, including speech sound disorders, hearing loss, or signal processing that does not faithfully represent the original signal.

A Tipping Point in Listening Effort: Effects of Linguistic Complexity and Age-Related Hearing Loss on Sentence Comprehension

In recent years, there has been a growing interest in the relationship between effort and performance. Early formulations implied that, as the challenge of a task increases, individuals will exert more effort, with resultant maintenance of stable performance. We report an experiment in which normal-hearing young adults, normal-hearing older adults, and older adults with age-related mild-to-moderate hearing loss were tested for comprehension of recorded sentences that varied the comprehension challenge in two ways. First, sentences were constructed that expressed their meaning either with a simpler subject-relative syntactic structure or a more computationally demanding object-relative structure. Second, for each sentence type, an adjectival phrase was inserted that created either a short or long gap in the sentence between the agent performing an action and the action being performed. The measurement of pupil dilation as an index of processing effort showed effort to increase with task difficulty until a difficulty tipping point was reached. Beyond this point, the measurement of pupil size revealed a commitment of effort by the two groups of older adults who failed to keep pace with task demands as evidenced by reduced comprehension accuracy. We take these pupillometry data as revealing a complex relationship between task difficulty, effort, and performance that might not otherwise appear from task performance alone.