Effect of Auditory Task Type on Physiological and Subjective Measures of Listening Effort in Individuals With Normal Hearing

A Multimodal Approach to Measuring Listening Effort: A Systematic Review on the Effects of Auditory Task Demand on Physiological Measures and Their Relationship

OBJECTIVES

Listening effort involves the mental effort required to perceive an auditory stimulus, for example in noisy environments. Prolonged increased listening effort, for example due to impaired hearing ability, may increase risk of health complications. It is therefore important to identify valid and sensitive measures of listening effort. Physiological measures have been shown to be sensitive to auditory task demand manipulations and are considered to reflect changes in listening effort. Such measures include pupil dilation, alpha power, skin conductance level, and heart rate variability. The aim of the current systematic review was to provide an overview of studies to listening effort that used multiple physiological measures. The two main questions were: (1) what is the effect of changes in auditory task demand on simultaneously acquired physiological measures from various modalities? and (2) what is the relationship between the responses in these physiological measures?

DESIGN

Following Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, relevant articles were sought in PubMed, PsycInfo, and Web of Science and by examining the references of included articles. Search iterations with different combinations of psychophysiological measures were performed in conjunction with listening effort-related search terms. Quality was assessed using the Appraisal Tool for Cross-Sectional Studies.

RESULTS

A total of 297 articles were identified from three databases, of which 27 were included. One additional article was identified from reference lists. Of the total 28 included articles, 16 included an analysis regarding the relationship between the physiological measures. The overall quality of the included studies was reasonable.

CONCLUSIONS

The included studies showed that most of the physiological measures either show no effect to auditory task demand manipulations or a consistent effect in the expected direction. For example, pupil dilation increased, pre-ejection period decreased, and skin conductance level increased with increasing auditory task demand. Most of the relationships between the responses of these physiological measures were nonsignificant or weak. The physiological measures varied in their sensitivity to auditory task demand manipulations. One of the identified knowledge gaps was that the included studies mostly used tasks with high-performance levels, resulting in an underrepresentation of the physiological changes at lower performance levels. This makes it difficult to capture how the physiological responses behave across the full psychometric curve. Our results support the Framework for Understanding Effortful Listening and the need for a multimodal approach to listening effort. We furthermore discuss focus points for future studies.

The P300 Auditory Evoked Potential: A Physiological Measure of the Engagement of Cognitive Systems Contributing to Listening Effort?

OBJECTIVES

This study aimed to explore the potential of the P300 (P3b) as a physiological measure of the engagement of cognitive systems contributing to listening effort.

DESIGN

Nineteen right-handed young adults (mean age: 24.79 years) and 20 right-handed older adults (mean age: 58.90 years) with age-appropriate hearing were included. The P300 was recorded at Fz, Cz, and Pz using a two-stimulus oddball paradigm with the Flemish monosyllabic numbers "one" and "three" as standard and deviant stimuli, respectively. This oddball paradigm was conducted in three listening conditions, varying in listening demand: one quiet and two noisy listening conditions (+4 and -2 dB signal to noise ratio [SNR]). At each listening condition, physiological, behavioral, and subjective tests of listening effort were administered. P300 amplitude and latency served as a potential physiological measure of the engagement of cognitive systems contributing to listening effort. In addition, the mean reaction time to respond to the deviant stimuli was used as a behavioral listening effort measurement. Last, subjective listening effort was administered through a visual analog scale. To assess the effects of listening condition and age group on each of these measures, linear mixed models were conducted. Correlation coefficients were calculated to determine the relationship between the physiological, behavioral, and subjective measures.

RESULTS

P300 amplitude and latency, mean reaction time, and subjective scores significantly increased as the listening condition became more taxing. Moreover, a significant group effect was found for all physiological, behavioral, and subjective measures, favoring young adults. Last, no clear relationships between the physiological, behavioral, and subjective measures were found.

CONCLUSIONS

The P300 was considered a physiological measure of the engagement of cognitive systems contributing to listening effort. Because advancing age is associated with hearing loss and cognitive decline, more research is needed on the effects of all these variables on the P300 to further explore its usefulness as a listening effort measurement for research and clinical purposes.

Listening efficiency in adult cochlear-implant users compared with normally-hearing controls at ecologically relevant signal-to-noise ratios

Introduction

Due to having to work with an impoverished auditory signal, cochlear-implant (CI) users may experience reduced speech intelligibility and/or increased listening effort in real-world listening situations, compared to their normally-hearing (NH) peers. These two challenges to perception may be usefully integrated in a measure of listening efficiency: conceptually, the amount of accuracy achieved for a certain amount of effort expended.

Methods

We describe a novel approach to quantifying listening efficiency based on the rate of evidence accumulation toward a correct response in a linear ballistic accumulator (LBA) model of choice decision-making. Estimation of this objective measure within a hierarchical Bayesian framework confers further benefits, including full quantification of uncertainty in parameter estimates. We applied this approach to examine the speech-in-noise performance of a group of 24 CI users (M age: 60.3, range: 20-84 years) and a group of 25 approximately age-matched NH controls (M age: 55.8, range: 20-79 years). In a laboratory experiment, participants listened to reverberant target sentences in cafeteria noise at ecologically relevant signal-to-noise ratios (SNRs) of +20, +10, and +4 dB SNR. Individual differences in cognition and self-reported listening experiences were also characterised by means of cognitive tests and hearing questionnaires.

Results

At the group level, the CI group showed much lower listening efficiency than the NH group, even in favourable acoustic conditions. At the individual level, within the CI group (but not the NH group), higher listening efficiency was associated with better cognition (i.e., working-memory and linguistic-closure) and with more positive self-reported listening experiences, both in the laboratory and in daily life.

Discussion

We argue that listening efficiency, measured using the approach described here, is: (i) conceptually well-motivated, in that it is theoretically impervious to differences in how individuals approach the speed-accuracy trade-off that is inherent to all perceptual decision making; and (ii) of practical utility, in that it is sensitive to differences in task demand, and to differences between groups, even when speech intelligibility remains at or near ceiling level. Further research is needed to explore the sensitivity and practical utility of this metric across diverse listening situations.

Exploring the Correlations Between Measures of Listening Effort in Adults and Children: A Systematic Review with Narrative Synthesis

Listening effort (LE) describes the cognitive resources needed to process an auditory message. Our understanding of this notion remains in its infancy, hindering our ability to appreciate how it impacts individuals with hearing impairment effectively. Despite the myriad of proposed measurement tools, a validated method remains elusive. This is complicated by the seeming lack of association between tools demonstrated via correlational analyses. This review aims to systematically review the literature relating to the correlational analyses between different measures of LE. Five databases were used- PubMed, Cochrane, EMBASE, PsychINFO, and CINAHL. The quality of the evidence was assessed using the GRADE criteria and risk of bias with ROBINS-I/GRADE tools. Each statistically significant analysis was classified using an approved system for medical correlations. The final analyses included 48 papers, equating to 274 correlational analyses, of which 99 reached statistical significance (36.1%). Within these results, the most prevalent classifications were poor or fair. Moreover, when moderate or very strong correlations were observed, they tended to be dependent on experimental conditions. The quality of evidence was graded as very low. These results show that measures of LE are poorly correlated and supports the multi-dimensional concept of LE. The lack of association may be explained by considering where each measure operates along the effort perception pathway. Moreover, the fragility of significant correlations to specific conditions further diminishes the hope of finding an all-encompassing tool. Therefore, it may be prudent to focus on capturing the consequences of LE rather than the notion itself.

Linking Affective and Hearing Sciences-Affective Audiology

A growing number of health-related sciences, including audiology, have increasingly recognized the importance of affective phenomena. However, in audiology, affective phenomena are mostly studied as a consequence of hearing status. This review first addresses anatomical and functional bidirectional connections between auditory and affective systems that support a reciprocal affect-hearing relationship. We then postulate, by focusing on four practical examples (hearing public campaigns, hearing intervention uptake, thorough hearing evaluation, and tinnitus), that some important challenges in audiology are likely affect-related and that potential solutions could be developed by inspiration from affective science advances. We continue by introducing useful resources from affective science that could help audiology professionals learn about the wide range of affective constructs and integrate them into hearing research and clinical practice in structured and applicable ways. Six important considerations for good quality affective audiology research are summarized. We conclude that it is worthwhile and feasible to explore the explanatory power of emotions, feelings, motivations, attitudes, moods, and other affective processes in depth when trying to understand and predict how people with hearing difficulties perceive, react, and adapt to their environment.

Beyond Speech Intelligibility: Quantifying Behavioral and Perceived Listening Effort in Response to Dysarthric Speech

PURPOSE

This study investigated whether listener processing of dysarthric speech requires the recruitment of more cognitive resources (i.e., higher levels of listening effort) than neurotypical speech. We also explored relationships between behavioral listening effort, perceived listening effort, and objective measures of word transcription accuracy.

METHOD

A word recall paradigm was used to index behavioral listening effort. The primary task involved word transcription, whereas a memory task involved recalling words from previous sentences. Nineteen listeners completed the paradigm twice, once while transcribing dysarthric speech and once while transcribing neurotypical speech. Perceived listening effort was rated using a visual analog scale.

RESULTS

Results revealed significant effects of dysarthria on the likelihood of correct word recall, indicating that the transcription of dysarthric speech required higher levels of behavioral listening effort relative to neurotypical speech. There was also a significant relationship between transcription accuracy and measures of behavioral listening effort, such that listeners who were more accurate in understanding dysarthric speech exhibited smaller changes in word recall when listening to dysarthria. The subjective measure of perceived listening effort did not have a statistically significant correlation with measures of behavioral listening effort or transcription accuracy.

CONCLUSIONS

Results suggest that cognitive resources, particularly listeners' working memory capacity, are more taxed when deciphering dysarthric versus neurotypical speech. An increased demand on these resources may affect a listener's ability to remember aspects of their conversations with people with dysarthria, even when the speaker is fully intelligible.

Quantifying the Effect of Noise on Cognitive Processes: A Review of Psychophysiological Correlates of Workload

Noise is present in most work environments, including emissions from machines and devices, irrelevant speech from colleagues, and traffic noise. Although it is generally accepted that noise below the permissible exposure limits does not pose a considerable risk for auditory effects like hearing impairments. Yet, noise can have a direct adverse effect on cognitive performance (non-auditory effects like workload or stress). Under certain circumstances, the observable performance for a task carried out in silence compared to noisy surroundings may not differ. One possible explanation for this phenomenon needs further investigation: individuals may invest additional cognitive resources to overcome the distraction from irrelevant auditory stimulation. Recent developments in measurements of psychophysiological correlates and analysis methods of load-related parameters can shed light on this complex interaction. These objective measurements complement subjective self-report of perceived effort by quantifying unnoticed noise-related cognitive workload. In this review, literature databases were searched for peer-reviewed journal articles that deal with an at least partially irrelevant "auditory stimulation" during an ongoing "cognitive task" that is accompanied by "psychophysiological correlates" to quantify the "momentary workload." The spectrum of assessed types of "auditory stimulations" extended from speech stimuli (varying intelligibility), oddball sounds (repeating short tone sequences), and auditory stressors (white noise, task-irrelevant real-life sounds). The type of "auditory stimulation" was related (speech stimuli) or unrelated (oddball, auditory stressor) to the type of primary "cognitive task." The types of "cognitive tasks" include speech-related tasks, fundamental psychological assessment tasks, and real-world/simulated tasks. The "psychophysiological correlates" include pupillometry and eye-tracking, recordings of brain activity (hemodynamic, potentials), cardiovascular markers, skin conductance, endocrinological markers, and behavioral markers. The prevention of negative effects on health by unexpected stressful soundscapes during mental work starts with the continuous estimation of cognitive workload triggered by auditory noise. This review gives a comprehensive overview of methods that were tested for their sensitivity as markers of workload in various auditory settings during cognitive processing.

A comparison of simultaneously-obtained measures of listening effort: pupil dilation, verbal response time and self-rating

OBJECTIVE

The aim of this study was to assess to what extent simultaneously-obtained measures of listening effort (task-evoked pupil dilation, verbal response time [RT], and self-rating) could be sensitive to auditory and cognitive manipulations in a speech perception task. The study also aimed to explore the possible relationship between RT and pupil dilation.

DESIGN

A within-group design was adopted. All participants were administered the Matrix Sentence Test in 12 conditions (signal-to-noise ratios [SNR] of -3, -6, -9 dB; attentional resources focussed vs divided; spatial priors present vs absent).

STUDY SAMPLE

Twenty-four normal-hearing adults, 20-41 years old (M = 23.5), were recruited in the study.

RESULTS

A significant effect of the SNR was found for all measures. However, pupil dilation discriminated only partially between the SNRs. Neither of the cognitive manipulations were effective in modulating the measures. No relationship emerged between pupil dilation, RT and self-ratings.

CONCLUSIONS

RT, pupil dilation, and self-ratings can be obtained simultaneously when administering speech perception tasks, even though some limitations remain related to the absence of a retention period after the listening phase. The sensitivity of the three measures to changes in the auditory environment differs. RTs and self-ratings proved most sensitive to changes in SNR.

Reduced Semantic Context and Signal-to-Noise Ratio Increase Listening Effort As Measured Using Functional Near-Infrared Spectroscopy

OBJECTIVES

Understanding speech-in-noise can be highly effortful. Decreasing the signal-to-noise ratio (SNR) of speech increases listening effort, but it is relatively unclear if decreasing the level of semantic context does as well. The current study used functional near-infrared spectroscopy to evaluate two primary hypotheses: (1) listening effort (operationalized as oxygenation of the left lateral PFC) increases as the SNR decreases and (2) listening effort increases as context decreases.

DESIGN

Twenty-eight younger adults with normal hearing completed the Revised Speech Perception in Noise Test, in which they listened to sentences and reported the final word. These sentences either had an easy SNR (+4 dB) or a hard SNR (-2 dB), and were either low in semantic context (e.g., "Tom could have thought about the sport") or high in context (e.g., "She had to vacuum the rug"). PFC oxygenation was measured throughout using functional near-infrared spectroscopy.

RESULTS

Accuracy on the Revised Speech Perception in Noise Test was worse when the SNR was hard than when it was easy, and worse for sentences low in semantic context than high in context. Similarly, oxygenation across the entire PFC (including the left lateral PFC) was greater when the SNR was hard, and left lateral PFC oxygenation was greater when context was low.

CONCLUSIONS

These results suggest that activation of the left lateral PFC (interpreted here as reflecting listening effort) increases to compensate for acoustic and linguistic challenges. This may reflect the increased engagement of domain-general and domain-specific processes subserved by the dorsolateral prefrontal cortex (e.g., cognitive control) and inferior frontal gyrus (e.g., predicting the sensory consequences of articulatory gestures), respectively.

Cognitive and Physiological Measures of Listening Effort During Degraded Speech Perception: Relating Dual-Task and Pupillometry Paradigms

Purpose Listening effort is quickly becoming an important metric for assessing speech perception in less-than-ideal situations. However, the relationship between the construct of listening effort and the measures used to assess it remains unclear. We compared two measures of listening effort: a cognitive dual task and a physiological pupillometry task. We sought to investigate the relationship between these measures of effort and whether engaging effort impacts speech accuracy. Method In Experiment 1, 30 participants completed a dual task and a pupillometry task that were carefully matched in stimuli and design. The dual task consisted of a spoken word recognition task and a visual match-to-sample task. In the pupillometry task, pupil size was monitored while participants completed a spoken word recognition task. Both tasks presented words at three levels of listening difficulty (unmodified, eight-channel vocoding, and four-channel vocoding) and provided response feedback on every trial. We refined the pupillometry task in Experiment 2 (n = 31); crucially, participants no longer received response feedback. Finally, we ran a new group of subjects on both tasks in Experiment 3 (n = 30). Results In Experiment 1, accuracy in the visual task decreased with increased signal degradation in the dual task, but pupil size was sensitive to accuracy and not vocoding condition. After removing feedback in Experiment 2, changes in pupil size were predicted by listening condition, suggesting the task was now sensitive to engaged effort. Both tasks were sensitive to listening difficulty in Experiment 3, but there was no relationship between the tasks and neither task predicted speech accuracy. Conclusions Consistent with previous work, we found little evidence for a relationship between different measures of listening effort. We also found no evidence that effort predicts speech accuracy, suggesting that engaging more effort does not lead to improved speech recognition. Cognitive and physiological measures of listening effort are likely sensitive to different aspects of the construct of listening effort. Supplemental Material https://doi.org/10.23641/asha.16455900.

Listener characteristics differentially affect self-reported and physiological measures of effort associated with two challenging listening conditions

Listeners vary in their ability to understand speech in adverse conditions. Differences in both cognitive and linguistic capacities play a role, but increasing evidence suggests that such factors may contribute differentially depending on the listening challenge. Here, we used multilevel modeling to evaluate contributions of individual differences in age, hearing thresholds, vocabulary, selective attention, working memory capacity, personality traits, and noise sensitivity to variability in measures of comprehension and listening effort in two listening conditions. A total of 35 participants completed a battery of cognitive and linguistic tests as well as a spoken story comprehension task using (1) native-accented English speech masked by speech-shaped noise and (2) nonnative accented English speech without masking. Masker levels were adjusted individually to ensure each participant would show (close to) equivalent word recognition performance across the two conditions. Dependent measures included comprehension tests results, self-rated effort, and electrodermal, cardiovascular, and facial electromyographic measures associated with listening effort. Results showed varied patterns of responsivity across different dependent measures as well as across listening conditions. In particular, results suggested that working memory capacity may play a greater role in the comprehension of nonnative accented speech than noise-masked speech, while hearing acuity and personality may have a stronger influence on physiological responses affected by demands of understanding speech in noise. Furthermore, electrodermal measures may be more strongly affected by affective response to noise-related interference while cardiovascular responses may be more strongly affected by demands on working memory and lexical access.

Comparisons of the Sensitivity and Reliability of Multiple Measures of Listening Effort

OBJECTIVES

The objective of this study was to evaluate the sensitivity and reliability of one subjective (rating scale) and three objective (dual-task paradigm, pupillometry, and skin conductance response amplitude) measures of listening effort across multiple signal to noise ratios (SNRs).

DESIGN

Twenty adults with normal hearing attended two sessions and listened to sentences presented in quiet and in stationary noise at three different SNRs: 0, -3, and -5 dB. Listening effort was assessed by examining change in reaction time (dual-task paradigm), change in peak to peak pupil diameter (pupillometry), and change in mean skin conductance response amplitude; self-reported listening effort on a scale from 0 to 100 was also evaluated. Responses were averaged within each SNR and based on three word recognition ability categories (≤50%, 51% to 71%, and >71%) across all SNRs. Measures were considered reliable if there were no significant changes between sessions, and intraclass correlation coefficients were a minimum of 0.40. Effect sizes were calculated to compare the sensitivity of the measures.

RESULTS

Intraclass correlation coefficient values indicated fair-to-moderate reliability for all measures while individual measurement sensitivity was variable. Self-reports were sensitive to listening effort but were less reliable, given that subjective effort was greater during the dual task than either of the physiologic measures. The dual task was sensitive to a narrow range of word recognition abilities but was less reliable as it exhibited a global decrease in reaction time across sessions. Pupillometry was consistently sensitive and reliable to changes in listening effort. Skin conductance response amplitude was not sensitive or reliable while the participants listened to the sentences. Skin conductance response amplitude during the verbal response was sensitive to poor (≤50%) speech recognition abilities; however, it was less reliable as there was a significant change in amplitude across sessions.

CONCLUSIONS

In this study, pupillometry was the most sensitive and reliable objective measure of listening effort. Intersession variability significantly influenced the other objective measures of listening effort, which suggests challenges for cross-study comparability. Therefore, intraclass correlation coefficients combined with other statistical tests more fully describe the reliability of measures of listening effort across multiple difficulties. Minimizing intersession variability will increase measurement sensitivity. Further work toward standardized methods and analysis will strengthen our understanding of the reliability and sensitivity of measures of listening effort and better facilitate cross-modal and cross-study comparisons.

Listening effort: Are we measuring cognition or affect, or both?

Listening effort is increasingly recognized as a factor in communication, particularly for and with nonnative speakers, for the elderly, for individuals with hearing impairment and/or for those working in noise. However, as highlighted by McGarrigle et al., International Journal of Audiology, 2014, 53, 433-445, the term "listening effort" encompasses a wide variety of concepts, including the engagement and control of multiple possibly distinct neural systems for information processing, and the affective response to the expenditure of those resources in a given context. Thus, experimental or clinical methods intended to objectively quantify listening effort may ultimately reflect a complex interaction between the operations of one or more of those information processing systems, and/or the affective and motivational response to the demand on those systems. Here we examine theoretical, behavioral, and psychophysiological factors related to resolving the question of what we are measuring, and why, when we measure "listening effort." This article is categorized under: Linguistics > Language in Mind and Brain Psychology > Theory and Methods Psychology > Attention Psychology > Emotion and Motivation.