Effortful listening under the microscope: Examining relations between pupillometric and subjective markers of effort and tiredness from listening

Effect of background noise and memory load on listening effort of young adults with and without hearing loss

OBJECTIVE

People with hearing loss often encounter difficulties in hearing under adverse conditions, such as listening in the presence of noise. Listening effort is an indicator used to assess listening difficulties in daily life. Although many studies on listening effort have been conducted in recent years, there is a notable gap in the exploration of how task load influences listening effort in young adults. This study compared the effects of background noise and memory load on task performance and subjective listening effort in young adults with and without hearing loss.

METHODS

The study included a group of 8 adults with hearing loss (mean age: 24.1 ± 6.0 years) and a group of 16 individuals with normal hearing (mean age: 27.9 ± 4.9 years). A number memorizing task was conducted, involving two types of auditory digits (either three or seven digits) presented under multi-talker babble noise conditions of signal-to-noise ratio of -5 dB [SN -5 dB] or SN +5 dB. Participants determined whether the number presented in the encoding interval matched the one presented in the retrieval interval. Subsequently, they were asked to complete a questionnaire using a Visual Analog Scale (VAS) to assess their subjective listening effort. Percentage of correct responses, reaction times, and VAS ratings were compared between adults with and without hearing loss.

RESULTS

Our results showed significant differences between the two groups in the percentage of correct responses and the reaction time under the SN -5 dB conditions, regardless of the memory load. Under the SN +5 dB conditions, a significant difference was found only in the percentage of correct responses for seven digits. In the normal hearing group, the percentage of correct responses and VAS ratings tended to decrease as the memory load increased, even under the same noise condition. Conversely, in the hearing loss group, a consistent trend could not be identified in the effects of noise and memory load on the percentage of correct responses and VAS ratings.

CONCLUSION

These results suggest that in conditions of high noise load, young adults with hearing loss show a higher tendency for listening effort to be affected by other loads. We confirmed that for some participants with hearing loss, the task exceeded a certain level of difficulty in the SN -5 dB and seven digits condition, leading to a change in their motivation and strategy used. Future research should examine ways to control for participants' motivations.

Mood shapes the impact of reward on perceived fatigue from listening

Knowledge of the underlying mechanisms of effortful listening could help to reduce cases of social withdrawal and mitigate fatigue, especially in older adults. However, the relationship between transient effort and longer term fatigue is likely to be more complex than originally thought. Here, we manipulated the presence/absence of monetary reward to examine the role of motivation and mood state in governing changes in perceived effort and fatigue from listening. In an online study, 185 participants were randomly assigned to either a "reward" (n = 91) or "no-reward" (n = 94) group and completed a dichotic listening task along with a series of questionnaires assessing changes over time in perceived effort, mood, and fatigue. Effort ratings were higher overall in the reward group, yet fatigue ratings in that group showed a shallower linear increase over time. Mediation analysis revealed an indirect effect of reward on fatigue ratings via perceived mood state; reward induced a more positive mood state which was associated with reduced fatigue. These results suggest that: (1) listening conditions rated as more "effortful" may be less fatiguing if the effort is deemed worthwhile, and (2) alterations to one's mood state represent a potential mechanism by which fatigue may be elicited during unrewarding listening situations.

Dual-tasking while using two languages: Examining the cognitive resource demands of cued and voluntary language production in bilinguals

The way bilinguals switch languages can differ depending on the context. In cued dual-language environments, bilinguals select a language in response to environmental cues (e.g., a monolingual conversation partner). In voluntary dual-language environments, bilinguals communicating with people who speak the same languages can use their languages more freely. The control demands of these types of language-production contexts, and the costs of language switches, have been argued to differ (Adaptive Control Hypothesis). Here, we used a dual-task paradigm to examine how cued and voluntary bilingual production differ in cognitive resources used. Forty Mandarin-English bilinguals completed two language-switching paradigms as the primary task; one in response to cues and one while using two languages freely. At the same time, they also had to respond to the pitch of tones (secondary task). Response times (RTs) on the secondary task, as well as naming times on the primary task, were shorter under the voluntary- than cued-naming condition. Task workload ratings were also higher under the cued- than voluntary-naming condition. This suggests more attentional resources are needed in a cued-naming context to monitor cues and select languages accordingly. However, the costs associated with switching from one language to the other were similar in both voluntary- and cued-naming contexts. Thus, while cued-naming might be more effortful overall, cued and voluntary switching recruited similar levels of cognitive resources.

Phonological discrimination and contrast detection in pupillometry

Introduction

The perception of phonemes is guided by both low-level acoustic cues and high-level linguistic context. However, differentiating between these two types of processing can be challenging. In this study, we explore the utility of pupillometry as a tool to investigate both low- and high-level processing of phonological stimuli, with a particular focus on its ability to capture novelty detection and cognitive processing during speech perception.

Methods

Pupillometric traces were recorded from a sample of 22 Danish-speaking adults, with self-reported normal hearing, while performing two phonological-contrast perception tasks: a nonword discrimination task, which included minimal-pair combinations specific to the Danish language, and a nonword detection task involving the detection of phonologically modified words within sentences. The study explored the perception of contrasts in both unprocessed speech and degraded speech input, processed with a vocoder.

Results

No difference in peak pupil dilation was observed when the contrast occurred between two isolated nonwords in the nonword discrimination task. For unprocessed speech, higher peak pupil dilations were measured when phonologically modified words were detected within a sentence compared to sentences without the nonwords. For vocoded speech, higher peak pupil dilation was observed for sentence stimuli, but not for the isolated nonwords, although performance decreased similarly for both tasks.

Conclusion

Our findings demonstrate the complexity of pupil dynamics in the presence of acoustic and phonological manipulation. Pupil responses seemed to reflect higher-level cognitive and lexical processing related to phonological perception rather than low-level perception of acoustic cues. However, the incorporation of multiple talkers in the stimuli, coupled with the relatively low task complexity, may have affected the pupil dilation.

The impact of speech type on listening effort and intelligibility for native and non-native listeners

Listeners are routinely exposed to many different types of speech, including artificially-enhanced and synthetic speech, styles which deviate to a greater or lesser extent from naturally-spoken exemplars. While the impact of differing speech types on intelligibility is well-studied, it is less clear how such types affect cognitive processing demands, and in particular whether those speech forms with the greatest intelligibility in noise have a commensurately lower listening effort. The current study measured intelligibility, self-reported listening effort, and a pupillometry-based measure of cognitive load for four distinct types of speech: (i) plain i.e. natural unmodified speech; (ii) Lombard speech, a naturally-enhanced form which occurs when speaking in the presence of noise; (iii) artificially-enhanced speech which involves spectral shaping and dynamic range compression; and (iv) speech synthesized from text. In the first experiment a cohort of 26 native listeners responded to the four speech types in three levels of speech-shaped noise. In a second experiment, 31 non-native listeners underwent the same procedure at more favorable signal-to-noise ratios, chosen since second language listening in noise has a more detrimental effect on intelligibility than listening in a first language. For both native and non-native listeners, artificially-enhanced speech was the most intelligible and led to the lowest subjective effort ratings, while the reverse was true for synthetic speech. However, pupil data suggested that Lombard speech elicited the lowest processing demands overall. These outcomes indicate that the relationship between intelligibility and cognitive processing demands is not a simple inverse, but is mediated by speech type. The findings of the current study motivate the search for speech modification algorithms that are optimized for both intelligibility and listening effort.

Listening efficiency in adult cochlear-implant users compared with normally-hearing controls at ecologically relevant signal-to-noise ratios

Introduction

Due to having to work with an impoverished auditory signal, cochlear-implant (CI) users may experience reduced speech intelligibility and/or increased listening effort in real-world listening situations, compared to their normally-hearing (NH) peers. These two challenges to perception may be usefully integrated in a measure of listening efficiency: conceptually, the amount of accuracy achieved for a certain amount of effort expended.

Methods

We describe a novel approach to quantifying listening efficiency based on the rate of evidence accumulation toward a correct response in a linear ballistic accumulator (LBA) model of choice decision-making. Estimation of this objective measure within a hierarchical Bayesian framework confers further benefits, including full quantification of uncertainty in parameter estimates. We applied this approach to examine the speech-in-noise performance of a group of 24 CI users (M age: 60.3, range: 20-84 years) and a group of 25 approximately age-matched NH controls (M age: 55.8, range: 20-79 years). In a laboratory experiment, participants listened to reverberant target sentences in cafeteria noise at ecologically relevant signal-to-noise ratios (SNRs) of +20, +10, and +4 dB SNR. Individual differences in cognition and self-reported listening experiences were also characterised by means of cognitive tests and hearing questionnaires.

Results

At the group level, the CI group showed much lower listening efficiency than the NH group, even in favourable acoustic conditions. At the individual level, within the CI group (but not the NH group), higher listening efficiency was associated with better cognition (i.e., working-memory and linguistic-closure) and with more positive self-reported listening experiences, both in the laboratory and in daily life.

Discussion

We argue that listening efficiency, measured using the approach described here, is: (i) conceptually well-motivated, in that it is theoretically impervious to differences in how individuals approach the speed-accuracy trade-off that is inherent to all perceptual decision making; and (ii) of practical utility, in that it is sensitive to differences in task demand, and to differences between groups, even when speech intelligibility remains at or near ceiling level. Further research is needed to explore the sensitivity and practical utility of this metric across diverse listening situations.

Be Quiet! Effects of Competing Speakers and Individual Characteristics on Listening Comprehension for Primary School Students

Students learn in noisy classrooms, where the main sources of noise are their own voices. In this sound environment, students are not equally at risk from background noise interference during lessons, due to the moderation effect of the individual characteristics on the listening conditions. This study investigates the effect of the number of competing speakers on listening comprehension and whether this is modulated by selective attention skills, working memory, and noise sensitivity. Seventy-one primary school students aged 10 to 13 years completed a sentence comprehension task in three listening conditions: quiet, two competing speakers, and four competing speakers. Outcome measures were accuracy, listening effort (response times and self-reported), motivation, and confidence in completing the task. Individual characteristics were assessed in quiet. Results showed that the number of competing speakers has no direct effects on the task, whilst the individual characteristics were found to moderate the effect of the listening conditions. Selective attention moderated the effects on accuracy and response times, working memory on motivation, and noise sensitivity on both perceived effort and confidence. Students with low cognitive abilities and high noise sensitivity were found to be particularly at risk in the condition with two competing speakers.

The Involvement of Listening Effort in Explaining Bilingual Listening Under Adverse Listening Conditions

The current review examines listening effort to uncover how it is implicated in bilingual performance under adverse listening conditions. Various measures of listening effort, including physiological, behavioral, and subjective measures, have been employed to examine listening effort in bilingual children and adults. Adverse listening conditions, stemming from environmental factors, as well as factors related to the speaker or listener, have been examined. The existing literature, although relatively limited to date, points to increased listening effort among bilinguals in their nondominant second language (L2) compared to their dominant first language (L1) and relative to monolinguals. Interestingly, increased effort is often observed even when speech intelligibility remains unaffected. These findings emphasize the importance of considering listening effort alongside speech intelligibility. Building upon the insights gained from the current review, we propose that various factors may modulate the observed effects. These include the particular measure selected to examine listening effort, the characteristics of the adverse condition, as well as factors related to the particular linguistic background of the bilingual speaker. Critically, further research is needed to better understand the impact of these factors on listening effort. The review outlines avenues for future research that would promote a comprehensive understanding of listening effort in bilingual individuals.

Beyond Speech Intelligibility: Quantifying Behavioral and Perceived Listening Effort in Response to Dysarthric Speech

PURPOSE

This study investigated whether listener processing of dysarthric speech requires the recruitment of more cognitive resources (i.e., higher levels of listening effort) than neurotypical speech. We also explored relationships between behavioral listening effort, perceived listening effort, and objective measures of word transcription accuracy.

METHOD

A word recall paradigm was used to index behavioral listening effort. The primary task involved word transcription, whereas a memory task involved recalling words from previous sentences. Nineteen listeners completed the paradigm twice, once while transcribing dysarthric speech and once while transcribing neurotypical speech. Perceived listening effort was rated using a visual analog scale.

RESULTS

Results revealed significant effects of dysarthria on the likelihood of correct word recall, indicating that the transcription of dysarthric speech required higher levels of behavioral listening effort relative to neurotypical speech. There was also a significant relationship between transcription accuracy and measures of behavioral listening effort, such that listeners who were more accurate in understanding dysarthric speech exhibited smaller changes in word recall when listening to dysarthria. The subjective measure of perceived listening effort did not have a statistically significant correlation with measures of behavioral listening effort or transcription accuracy.

CONCLUSIONS

Results suggest that cognitive resources, particularly listeners' working memory capacity, are more taxed when deciphering dysarthric versus neurotypical speech. An increased demand on these resources may affect a listener's ability to remember aspects of their conversations with people with dysarthria, even when the speaker is fully intelligible.

Speech cortical activation and connectivity in typically developing children and those with listening difficulties

Listening difficulties (LiD) in people who have normal audiometry are a widespread but poorly understood form of hearing impairment. Recent research suggests that childhood LiD are cognitive rather than auditory in origin. We examined decoding of sentences using a novel combination of behavioral testing and fMRI with 43 typically developing children and 42 age matched (6-13 years old) children with LiD, categorized by caregiver report (ECLiPS). Both groups had clinically normal hearing. For sentence listening tasks, we found no group differences in fMRI brain cortical activation by increasingly complex speech stimuli that progressed in emphasis from phonology to intelligibility to semantics. Using resting state fMRI, we examined the temporal connectivity of cortical auditory and related speech perception networks. We found significant group differences only in cortical connections engaged when processing more complex speech stimuli. The strength of the affected connections was related to the children's performance on tests of dichotic listening, speech-in-noise, attention, memory and verbal vocabulary. Together, these results support the novel hypothesis that childhood LiD reflects difficulties in language rather than in auditory or phonological processing.

Predictors of Listening-Related Fatigue Across the Adult Life Span

Listening-related fatigue is a potentially serious negative consequence of an aging auditory and cognitive system. However, the impact of age on listening-related fatigue and the factors underpinning any such effect remain unexplored. Using data from a large sample of adults (N = 281), we conducted a conditional process analysis to examine potential mediators and moderators of age-related changes in listening-related fatigue. Mediation analyses revealed opposing effects of age on listening-related fatigue: Older adults with greater perceived hearing impairment tended to report increased listening-related fatigue. However, aging was otherwise associated with decreased listening-related fatigue via reductions in both mood disturbance and sensory-processing sensitivity. Results suggested that the effect of auditory attention ability on listening-related fatigue was moderated by sensory-processing sensitivity; for individuals with high sensory-processing sensitivity, better auditory attention ability was associated with increased fatigue. These findings shed light on the perceptual, cognitive, and psychological factors underlying age-related changes in listening-related fatigue.

A comparison of simultaneously-obtained measures of listening effort: pupil dilation, verbal response time and self-rating

OBJECTIVE

The aim of this study was to assess to what extent simultaneously-obtained measures of listening effort (task-evoked pupil dilation, verbal response time [RT], and self-rating) could be sensitive to auditory and cognitive manipulations in a speech perception task. The study also aimed to explore the possible relationship between RT and pupil dilation.

DESIGN

A within-group design was adopted. All participants were administered the Matrix Sentence Test in 12 conditions (signal-to-noise ratios [SNR] of -3, -6, -9 dB; attentional resources focussed vs divided; spatial priors present vs absent).

STUDY SAMPLE

Twenty-four normal-hearing adults, 20-41 years old (M = 23.5), were recruited in the study.

RESULTS

A significant effect of the SNR was found for all measures. However, pupil dilation discriminated only partially between the SNRs. Neither of the cognitive manipulations were effective in modulating the measures. No relationship emerged between pupil dilation, RT and self-ratings.

CONCLUSIONS

RT, pupil dilation, and self-ratings can be obtained simultaneously when administering speech perception tasks, even though some limitations remain related to the absence of a retention period after the listening phase. The sensitivity of the three measures to changes in the auditory environment differs. RTs and self-ratings proved most sensitive to changes in SNR.

Psychophysiological responses to potentially annoying heating, ventilation, and air conditioning noise during mentally demanding work

Exposure to noise-or unwanted sound-is considered a major public health issue in the United States and internationally. Previous work has shown that even acute noise exposure can influence physiological response in humans and that individuals differ markedly in their susceptibility to noise. Recent research also suggests that specific acoustic properties of noise may have distinct effects on human physiological response. Much of the existing research on physiological response to noise consists of laboratory studies using very simple acoustic stimuli-like white noise or tone bursts-or field studies of longer-term workplace noise exposure that may neglect acoustic properties of the noise entirely. By using laboratory exposure to realistic heating, ventilation, and air conditioning (HVAC) noise, the current study explores the interaction between acoustic properties of annoying noise and individual response to working in occupational noise. This study assessed autonomic response to two acoustically distinct noises while participants performed cognitively demanding work. Results showed that the two HVAC noises affected physiological arousal in different ways. Individual differences in physiological response to noise as a function of noise sensitivity were also observed. Further research is necessary to link specific acoustic characteristics with differential physiological responses in humans.

Listening effort and fatigue in native and non-native primary school children

Background noise makes listening effortful and may lead to fatigue. This may compromise classroom learning, especially for children with a non-native background. In the current study, we used pupillometry to investigate listening effort and fatigue during listening comprehension under typical (0 dB signal-to-noise ratio [SNR]) and favorable (+10 dB SNR) listening conditions in 63 Swedish primary school children (7-9 years of age) performing a narrative speech-picture verification task. Our sample comprised both native (n = 25) and non-native (n = 38) speakers of Swedish. Results revealed greater pupil dilation, indicating more listening effort, in the typical listening condition compared with the favorable listening condition, and it was primarily the non-native speakers who contributed to this effect (and who also had lower performance accuracy than the native speakers). Furthermore, the native speakers had greater pupil dilation during successful trials, whereas the non-native speakers showed greatest pupil dilation during unsuccessful trials, especially in the typical listening condition. This set of results indicates that whereas native speakers can apply listening effort to good effect, non-native speakers may have reached their effort ceiling, resulting in poorer listening comprehension. Finally, we found that baseline pupil size decreased over trials, which potentially indicates more listening-related fatigue, and this effect was greater in the typical listening condition compared with the favorable listening condition. Collectively, these results provide novel insight into the underlying dynamics of listening effort, fatigue, and listening comprehension in typical classroom conditions compared with favorable classroom conditions, and they demonstrate for the first time how sensitive this interplay is to language experience.