The eye as a window to the listening brain: Neural correlates of pupil size as a measure of cognitive listening load

EmotionMeter: A Multimodal Framework for Recognizing Human Emotions

In this paper, we present a multimodal emotion recognition framework called EmotionMeter that combines brain waves and eye movements. To increase the feasibility and wearability of EmotionMeter in real-world applications, we design a six-electrode placement above the ears to collect electroencephalography (EEG) signals. We combine EEG and eye movements for integrating the internal cognitive states and external subconscious behaviors of users to improve the recognition accuracy of EmotionMeter. The experimental results demonstrate that modality fusion with multimodal deep neural networks can significantly enhance the performance compared with a single modality, and the best mean accuracy of 85.11% is achieved for four emotions (happy, sad, fear, and neutral). We explore the complementary characteristics of EEG and eye movements for their representational capacities and identify that EEG has the advantage of classifying happy emotion, whereas eye movements outperform EEG in recognizing fear emotion. To investigate the stability of EmotionMeter over time, each subject performs the experiments three times on different days. EmotionMeter obtains a mean recognition accuracy of 72.39% across sessions with the six-electrode EEG and eye movement features. These experimental results demonstrate the effectiveness of EmotionMeter within and between sessions.

Effects of Task Difficulty on Neural Processes Underlying Semantics: An Event-Related Potentials Study

Purpose Improving the ability to listen efficiently in noisy environments is a critical goal for hearing rehabilitation. However, understanding of the impact of difficult listening conditions on language processing is limited. The current study evaluated the neural processes underlying semantics in challenging listening conditions. Method Thirty adults with normal hearing completed an auditory sentence processing task in 4-talker babble. Event-related brain potentials were elicited by the final word in high- or low-context sentences, where the final word was either highly expected or not expected, followed by a 4-alternative forced-choice response with either longer (1,000 ms), middle (700 ms), or shorter (400 ms) response time deadlines (RTDs). Results Behavioral accuracy was reduced, and reactions times were faster for shorter RTDs. N400 amplitudes, reflecting ease of lexical access, were larger when elicited by target words in low-context sentences followed by shorter compared with longer RTDs. Conclusions These results reveal that more neural resources are allocated for semantic processing/lexical access when listening difficulty increases. Differences between RTDs may reflect increased attentional allocation for shorter RTDs. These findings suggest that situational listening demands can impact the demands for cognitive resources engaged in language processing, which could significantly impact listener experiences across environments.

The Pupil Dilation Response to Auditory Stimuli: Current State of Knowledge

The measurement of cognitive resource allocation during listening, or listening effort, provides valuable insight in the factors influencing auditory processing. In recent years, many studies inside and outside the field of hearing science have measured the pupil response evoked by auditory stimuli. The aim of the current review was to provide an exhaustive overview of these studies. The 146 studies included in this review originated from multiple domains, including hearing science and linguistics, but the review also covers research into motivation, memory, and emotion. The present review provides a unique overview of these studies and is organized according to the components of the Framework for Understanding Effortful Listening. A summary table presents the sample characteristics, an outline of the study design, stimuli, the pupil parameters analyzed, and the main findings of each study. The results indicate that the pupil response is sensitive to various task manipulations as well as interindividual differences. Many of the findings have been replicated. Frequent interactions between the independent factors affecting the pupil response have been reported, which indicates complex processes underlying cognitive resource allocation. This complexity should be taken into account in future studies that should focus more on interindividual differences, also including older participants. This review facilitates the careful design of new studies by indicating the factors that should be controlled for. In conclusion, measuring the pupil dilation response to auditory stimuli has been demonstrated to be sensitive method applicable to numerous research questions. The sensitivity of the measure calls for carefully designed stimuli.

Midcingulate somatomotor and autonomic functions

The midcingulate cortex (MCC) is viewed as a central node within a large-scale system devoted to adjusting behavior in the face of changing environments. Whereas the role of the MCC in interfacing action and cognition is well established, its role in regulating the autonomic nervous system is poorly understood. Yet, adaptive reactions to novel or threatening situations induce coordinated changes in the sympathetic and the parasympathetic systems. The somatomotor maps in the MCC are organized dorsoventrally. A meta-analysis of the literature reveals that the dorsoventral organization might also concern connections with the autonomic nervous system. Activation of the dorsal and ventral parts of the MCC correlate with recruitments of the sympathetic and the parasympathetic systems, respectively. Data also suggest that, in the MCC, projections toward the sympathetic system are mapped along the sensory-motor system following the same cervico-sacral organization as projections on the spinal cord for skeletal motor control.

Pupillary Measures of the Cognitive Effort in Auditory Novel Word Processing and Short-Term Retention

The use of the task-evoked pupillary responses (TEPRs) methodology is emerging in the psycholinguistics literature, as a sensitive, reliable and dynamic psychophysiological measure of the cognitive effort produced by various aspects of language processing. This preliminary study aimed to assess the functionality and effectiveness of a TEPRs design for measuring the cognitive effort required for the processing and spontaneous (non-explicitly prompted) short-term retention of novel phonological forms presented auditorily. Twenty-four young adult participants (aged 19-28 years, M = 20.3, SD = 2.13) were auditorily presented with a series of pseudowords differing in their number of syllables and their syllabic complexity. Then, they were asked to produce a response to a delayed pseudoword-color matching task aimed to induce the short-term retention of the novel forms. Results on the size and timing of the TEPRs reveal a significant pupillary activation, starting immediately after the presentation of the auditory stimuli, peaking at 1080 ms and not subsiding significantly during the protracted retention period. Moreover, the differential complexity of the novel words phonology significantly affected pupillary activation. Overall, these preliminary results point to the effectiveness of pupillometry as a technique for capturing the cognitive effort entailed in the short-term maintenance of novel word forms in the phonological loop, a process deemed crucial in the everyday novel word learning process. Results are discussed in view of future research that could establish and extend their implications.

Oxytocin increases eye-gaze towards novel social and non-social stimuli

Research on oxytocin (OT) has revealed a substantial involvement of this neuropeptide in social cognition processes and attachment behavior. The rationale of the present project was to decipher the differential role of OT in basic social cognition processes towards non-erotic attachment stimuli vs. reproduction-related stimuli in human subjects. In a randomized double-blind repeated-measures cross-over design, N = 82 participants were investigated twice and received either intranasal OT or placebo at the first assessment followed by placebo or OT at second assessment. Participants were presented with standardized pictures of parent-child dyads, romantic couples engaging in non-erotic or explicit sexual activities, and non-social pictures while we assessed pupil dilation and eye focus on specific pre-defined areas of interest. Multilevel analyses suggest that during the initial presentation, OT increased pupil dilation towards all categories of stimuli and led the eye focus towards the eyes and body regions, followed by a strong decrease in pupil dilation and fixations at the second session. These carry-over effects indicate that hormonal treatment at an initial contact to social stimuli can determine how these stimuli are processed later. These results might have implications for OT as a treatment in interventions with repeated exposure to social material.

Effects of pupillary responses to luminance and attention on visual spatial discrimination

The optic quality of the eyes is, at least in part, determined by pupil size. Large pupils let more light enter the eyes, but degrade the point spread function, and thus the spatial resolution that can be achieved (Campbell & Gregory, 1960). In natural conditions, the pupil is mainly driven by the luminance (and possibly the color and contrast) at the gazed location, but is also modulated by attention and cognitive factors. Whether changes in eyes' optics related to pupil size modulation by luminance and attention impacts visual processing was assessed in two experiments. In Experiment 1, we measured pupil size using a constantly visible display made of four disks with different luminance levels, with no other task than fixating the disks in succession. The results confirmed that pupil size depends on the luminance of the gazed stimulus. Experiment 2, using similar settings as Experiment 1, used a two-interval forced-choice design to test whether discriminating high spatial frequencies that requires covert attention to parafoveal stimuli is better during the fixation of bright disks that entails a small pupil size, and hence better eyes' optics, as compared to fixating dark disks that entails a large pupil size, and hence poorer eyes' optics. As in Experiment 1, we observed large modulations of pupil size depending on the luminance of the gazed stimulus, but pupil dynamics was more variable, with marked pupil dilation during stimulus encoding, presumably because the demanding spatial frequency discrimination task engaged attention. However, discrimination performance and mean pupil size were not correlated. Despite this lack of correlation, the slopes of pupil dilation during stimulus encoding were correlated to performance, while the slopes of pupil dilation during decision-making were not. We discuss these results regarding the possible functional roles of pupil size modulations.

An information-theoretic perspective on the costs of cognition

In statistics and machine learning, model accuracy is traded off with complexity, which can be viewed as the amount of information extracted from the data. Here, we discuss how cognitive costs can be expressed in terms of similar information costs, i.e. as a function of the amount of information required to update a person's prior knowledge (or internal model) to effectively solve a task. We then examine the theoretical consequences that ensue from this assumption. This framework naturally explains why some tasks - for example, unfamiliar or dual tasks - are costly and permits to quantify these costs using information-theoretic measures. Finally, we discuss brain implementation of this principle and show that subjective cognitive costs can originate either from local or global capacity limitations on information processing or from increased rate of metabolic alterations. These views shed light on the potential adaptive value of cost-avoidance mechanisms.

Insights From Pupil Size to Mental Workload of Surgical Residents: Feasibility of an Educational Computer-Based Surgical Simulation Environment (ECE) Considering the Hand Condition

The advantage of simulation environments is that they present various insights into real situations, where experimental research opportunities are very limited—for example, in endoscopic surgery. These operations require simultaneous use of both hands. For this reason, surgical residents need to develop several motor skills, such as eye-hand coordination and left-right hand coordination. While performing these tasks, the hand condition (dominant, nondominant, both hands) creates different degrees of mental workload, which can be assessed through mental physiological measures—namely, pupil size. Studies show that pupil size grows in direct proportion to mental workload. However, in the literature, there are very limited studies exploring this workload through the pupil sizes of the surgical residents under different hand conditions. Therefore, in this study, we present a computer-based simulation of a surgical task using eye-tracking technology to better understand the influence of the hand condition on the performance of skill-based surgical tasks in a computer-based simulated environment. The results show that under the both-hand condition, the pupil size of the surgical residents is larger than the one under the dominant and nondominant hand conditions. This indicates that when the computer-simulated surgical task is performed with both hands, it is considered more difficult than in the dominant and nondominant hand conditions. In conclusion, this study shows that pupil size measurements are sufficiently feasible to estimate the mental workload of the participants while performing surgical tasks. The results of this study can be used as a guide by instructional system designers of skill-based training programs.

Verbal Response Times as a Potential Indicator of Cognitive Load During Conventional Speech Audiometry With Matrix Sentences

This study examined verbal response times-that is, the duration from stimulus offset to voice onset-as a potential measure of cognitive load during conventional testing of speech-in-noise understanding. Response times were compared with a measure of perceived effort as assessed by listening effort scaling. Three listener groups differing in age and hearing status participated in the study. Testing was done at two target intelligibility levels (80%, 95%) and with two noise types (stationary and fluctuating). Verbal response times reflected effects of intelligibility level, noise type, and listener group. Response times were shorter for 95% compared with 80% target intelligibility, shorter for fluctuating compared with stationary noise, and shorter for young listeners compared with older listeners. Responses were also faster for the older listeners with near normal hearing compared with the older hearing-aid users. In contrast, subjective listening effort scaling predominantly revealed effects of target intelligibility level but did not show consistent noise-type or listener-group effects. These findings show that verbal response times and effort scalings tap into different domains of listening effort. Verbal response times can be easily assessed during conventional speech audiometry and have the potential to show effects beyond performance measures and subjective effort estimates.

Physiological methods as indexes of listening effort measurement: an integrative literature review

ABSTRACT Purpose: to review the scientific literature and present existing instruments and methods for the objective assessment of the listening effort in normal hearing individuals worldwide. Methods: a literature integrative review whose purpose was to gather and summarize the scientific knowledge regarding the objective methods theme for measuring the listening effort, developed through the search of articles in specialized national and international journals, in the English and Portuguese languages, available in the databases: PUBMED, Cochrane Library, LILACS and SCIELO. Results: 18 articles which used physiological methods to measure the listening effort in individuals with normal hearing were reviewed. The main findings described in those articles refer to the author(s) and purpose(s) of the research, country where the research was conducted, casuistry, physiological method used and results. Conclusion: there is no consensus among the researchers about the best physiological method to measure this parameter, that is, this effort in the speech perception tasks, although the level of skin conductance is considered the most accurate measure to date.

Effects of Second Language Proficiency and Linguistic Uncertainty on Recognition of Speech in Native and Nonnative Competing Speech

PURPOSE

The purpose of this study was to investigate the effects of 2nd language proficiency and linguistic uncertainty on performance and listening effort in mixed language contexts.

METHOD

Thirteen native speakers of Dutch with varying degrees of fluency in English listened to and repeated sentences produced in both Dutch and English and presented in the presence of single-talker competing speech in both Dutch and English. Target and masker language combinations were presented in both blocked and mixed (unpredictable) conditions. In the blocked condition, in each block of trials the target-masker language combination remained constant, and the listeners were informed of both prior to beginning the block. In the mixed condition, target and masker language varied randomly from trial to trial. All listeners participated in all conditions. Performance was assessed in terms of speech reception thresholds, whereas listening effort was quantified in terms of pupil dilation.

RESULTS

Performance (speech reception thresholds) and listening effort (pupil dilation) were both affected by 2nd language proficiency (English test score) and target and masker language: Performance was better in blocked as compared to mixed conditions, with Dutch as compared to English targets, and with English as compared to Dutch maskers. English proficiency was correlated with listening performance. Listeners also exhibited greater peak pupil dilation in mixed as compared to blocked conditions for trials with Dutch maskers, whereas pupil dilation during preparation for speaking was higher for English targets as compared to Dutch ones in almost all conditions.

CONCLUSIONS

Both listener's proficiency in a 2nd language and uncertainty about the target language on a given trial play a significant role in how bilingual listeners attend to speech in the presence of competing speech in different languages, but precise effects also depend on which language is serving as target and which as masker.

Measuring Listening Effort: Convergent Validity, Sensitivity, and Links With Cognitive and Personality Measures

PURPOSE

Listening effort (LE) describes the attentional or cognitive requirements for successful listening. Despite substantial theoretical and clinical interest in LE, inconsistent operationalization makes it difficult to make generalizations across studies. The aims of this large-scale validation study were to evaluate the convergent validity and sensitivity of commonly used measures of LE and assess how scores on those tasks relate to cognitive and personality variables.

METHOD

Young adults with normal hearing (N = 111) completed 7 tasks designed to measure LE, 5 tests of cognitive ability, and 2 personality measures.

RESULTS

Scores on some behavioral LE tasks were moderately intercorrelated but were generally not correlated with subjective and physiological measures of LE, suggesting that these tasks may not be tapping into the same underlying construct. LE measures differed in their sensitivity to changes in signal-to-noise ratio and the extent to which they correlated with cognitive and personality variables.

CONCLUSIONS

Given that LE measures do not show consistent, strong intercorrelations and differ in their relationships with cognitive and personality predictors, these findings suggest caution in generalizing across studies that use different measures of LE. The results also indicate that people with greater cognitive ability appear to use their resources more efficiently, thereby diminishing the detrimental effects associated with increased background noise during language processing.

Eyes and ears: Using eye tracking and pupillometry to understand challenges to speech recognition

Although human speech recognition is often experienced as relatively effortless, a number of common challenges can render the task more difficult. Such challenges may originate in talkers (e.g., unfamiliar accents, varying speech styles), the environment (e.g. noise), or in listeners themselves (e.g., hearing loss, aging, different native language backgrounds). Each of these challenges can reduce the intelligibility of spoken language, but even when intelligibility remains high, they can place greater processing demands on listeners. Noisy conditions, for example, can lead to poorer recall for speech, even when it has been correctly understood. Speech intelligibility measures, memory tasks, and subjective reports of listener difficulty all provide critical information about the effects of such challenges on speech recognition. Eye tracking and pupillometry complement these methods by providing objective physiological measures of online cognitive processing during listening. Eye tracking records the moment-to-moment direction of listeners' visual attention, which is closely time-locked to unfolding speech signals, and pupillometry measures the moment-to-moment size of listeners' pupils, which dilate in response to increased cognitive load. In this paper, we review the uses of these two methods for studying challenges to speech recognition.

Perceived listener effort as an outcome measure for disordered speech

PURPOSE

Perceived listening effort is a perceptual dimension used to identify the amount of work necessary to understand disordered speech. The purpose of this study was to investigate the utility of perceived listening effort to provide unique information about disordered speech. The relationships between perceived listening effort and two current outcome measures (speech acceptability, intelligibility) were examined for listeners rating electrolaryngeal speech, along with their reliability and intra-rater agreement.

METHODS

Ten healthy male speakers read low-context sentences using an electrolarynx. Twenty-five inexperienced listeners orthographically transcribed and rated the stimuli for perceived listening effort and speech acceptability using a visual analog scale. Strict reliability and agreement criteria were set.

RESULTS

Perceived listening effort was moderately to strongly correlated with intelligibility (r = -0.76) and acceptability (r = -0.80), each of which contributed uniquely to ratings of perceived listening effort. However, only 17 listeners met stringent reliability and agreement criteria.

CONCLUSIONS

Ratings of perceived listening effort may provide unique information about the communicative success of individuals with communication disorders. There is great variability, however, among inexperienced listeners' perceptual ratings of electrolaryngeal speech. Future research should investigate variables that may affect perceived listening effort specifically and auditory-perceptual ratings in general.

Objective Assessment of Listening Effort: Coregistration of Pupillometry and EEG

Listening to speech in noise is effortful, particularly for people with hearing impairment. While it is known that effort is related to a complex interplay between bottom-up and top-down processes, the cognitive and neurophysiological mechanisms contributing to effortful listening remain unknown. Therefore, a reliable physiological measure to assess effort remains elusive. This study aimed to determine whether pupil dilation and alpha power change, two physiological measures suggested to index listening effort, assess similar processes. Listening effort was manipulated by parametrically varying spectral resolution (16- and 6-channel noise vocoding) and speech reception thresholds (SRT; 50% and 80%) while 19 young, normal-hearing adults performed a speech recognition task in noise. Results of off-line sentence scoring showed discrepancies between the target SRTs and the true performance obtained during the speech recognition task. For example, in the SRT80% condition, participants scored an average of 64.7%. Participants’ true performance levels were therefore used for subsequent statistical modelling. Results showed that both measures appeared to be sensitive to changes in spectral resolution (channel vocoding), while pupil dilation only was also significantly related to their true performance levels (%) and task accuracy (i.e., whether the response was correctly or partially recalled). The two measures were not correlated, suggesting they each may reflect different cognitive processes involved in listening effort. This combination of findings contributes to a growing body of research aiming to develop an objective measure of listening effort.

Listening under difficult conditions: An activation likelihood estimation meta-analysis

The brain networks supporting speech identification and comprehension under difficult listening conditions are not well specified. The networks hypothesized to underlie effortful listening include regions responsible for executive control. We conducted meta-analyses of auditory neuroimaging studies to determine whether a common activation pattern of the frontal lobe supports effortful listening under different speech manipulations. Fifty-three functional neuroimaging studies investigating speech perception were divided into three independent Activation Likelihood Estimate analyses based on the type of speech manipulation paradigm used: Speech-in-noise (SIN, 16 studies, involving 224 participants); spectrally degraded speech using filtering techniques (15 studies involving 270 participants); and linguistic complexity (i.e., levels of syntactic, lexical and semantic intricacy/density, 22 studies, involving 348 participants). Meta-analysis of the SIN studies revealed higher effort was associated with activation in left inferior frontal gyrus (IFG), left inferior parietal lobule, and right insula. Studies using spectrally degraded speech demonstrated increased activation of the insula bilaterally and the left superior temporal gyrus (STG). Studies manipulating linguistic complexity showed activation in the left IFG, right middle frontal gyrus, left middle temporal gyrus and bilateral STG. Planned contrasts revealed left IFG activation in linguistic complexity studies, which differed from activation patterns observed in SIN or spectral degradation studies. Although there were no significant overlap in prefrontal activation across these three speech manipulation paradigms, SIN and spectral degradation showed overlapping regions in left and right insula. These findings provide evidence that there is regional specialization within the left IFG and differential executive networks underlie effortful listening.

Listening Effort During Sentence Processing Is Increased for Non-native Listeners: A Pupillometry Study

Current evidence demonstrates that even though some non-native listeners can achieve native-like performance for speech perception tasks in quiet, the presence of a background noise is much more detrimental to speech intelligibility for non-native compared to native listeners. Even when performance is equated across groups, it is likely that greater listening effort is required for non-native listeners. Importantly, the added listening effort might result in increased fatigue and a reduced ability to successfully perform multiple tasks simultaneously. Task-evoked pupil responses have been demonstrated to be a reliable measure of cognitive effort and can be useful in clarifying those aspects. In this study we compared the pupil response for 23 native English speakers and 27 Italian speakers of English as a second language. Speech intelligibility was tested for sentences presented in quiet and in background noise at two performance levels that were matched across groups. Signal-to-noise levels corresponding to these sentence intelligibility levels were pre-determined using an adaptive intelligibility task. Pupil response was significantly greater in non-native compared to native participants across both intelligibility levels. Therefore, for a given intelligibility level, a greater listening effort is required when listening in a second language in order to understand speech in noise. Results also confirmed that pupil response is sensitive to speech intelligibility during language comprehension, in line with previous research. However, contrary to our predictions, pupil response was not differentially modulated by intelligibility levels for native and non-native listeners. The present study corroborates that pupillometry can be deemed as a valid measure to be used in speech perception investigation, because it is sensitive to differences both across participants, such as listener type, and across conditions, such as variations in the level of speech intelligibility. Importantly, pupillometry offers us the possibility to uncover differences in listening effort even when those do not emerge in the performance level of individuals.

Task-induced deactivation in diverse brain systems correlates with interindividual differences in distinct autonomic indices

Neuroimaging research has shown that different cognitive tasks induce relatively specific activation patterns, as well as less task-specific deactivation patterns. Here we examined whether individual differences in Autonomic Nervous System (ANS) activity during task performance correlate with the magnitude of task-induced deactivation. In an fMRI study, participants performed a continuous mental arithmetic task in a task/rest block design, while undergoing combined fMRI and heart/respiration rate acquisitions using photoplethysmograph and respiration belt. As expected, task performance increased heart-rate and reduced the RMSSD, a cardiac index related to vagal tone. Across participants, higher heart rate during task was linked to increased activation in fronto-parietal regions, as well as to stronger deactivation in ventromedial prefrontal regions. Respiration frequency during task was associated with similar patterns, but in different regions than those identified for heart-rate. Finally, in a large set of regions, almost exclusively limited to the Default Mode Network, lower RMSSD was associated with greater deactivation, and furthermore, the vast majority of these regions were task-deactivated at the group level. Together, our findings show that inter-individual differences in ANS activity are strongly linked to task-induced deactivation. Importantly, our findings suggest that deactivation is a multifaceted construct potentially linked to ANS control, because distinct ANS measures correlate with deactivation in different regions. We discuss the implications for current theories of cortical control of the ANS and for accounts of deactivation, with particular reference to studies documenting a "failure to deactivate" in multiple clinical states.

Is Listening in Noise Worth It? The Neurobiology of Speech Recognition in Challenging Listening Conditions

This review examines findings from functional neuroimaging studies of speech recognition in noise to provide a neural systems level explanation for the effort and fatigue that can be experienced during speech recognition in challenging listening conditions. Neuroimaging studies of speech recognition consistently demonstrate that challenging listening conditions engage neural systems that are used to monitor and optimize performance across a wide range of tasks. These systems appear to improve speech recognition in younger and older adults, but sustained engagement of these systems also appears to produce an experience of effort and fatigue that may affect the value of communication. When considered in the broader context of the neuroimaging and decision making literature, the speech recognition findings from functional imaging studies indicate that the expected value, or expected level of speech recognition given the difficulty of listening conditions, should be considered when measuring effort and fatigue. The authors propose that the behavioral economics or neuroeconomics of listening can provide a conceptual and experimental framework for understanding effort and fatigue that may have clinical significance.

Cortisol, Chromogranin A, and Pupillary Responses Evoked by Speech Recognition Tasks in Normally Hearing and Hard-of-Hearing Listeners: A Pilot Study

Pupillometry is one method that has been used to measure processing load expended during speech understanding. Notably, speech perception (in noise) tasks can evoke a pupil response. It is not known if there is concurrent activation of the sympathetic nervous system as indexed by salivary cortisol and chromogranin A (CgA) and whether such activation differs between normally hearing (NH) and hard-of-hearing (HH) adults. Ten NH and 10 adults with mild-to-moderate hearing loss (mean age 52 years) participated. Two speech perception tests were administered in random order: one in quiet targeting 100% correct performance and one in noise targeting 50% correct performance. Pupil responses and salivary samples for cortisol and CgA analyses were collected four times: before testing, after the two speech perception tests, and at the end of the session. Participants rated their perceived accuracy, effort, and motivation. Effects were examined using repeated-measures analyses of variance. Correlations between outcomes were calculated. HH listeners had smaller peak pupil dilations (PPDs) than NH listeners in the speech in noise condition only. No group or condition effects were observed for the cortisol data, but HH listeners tended to have higher cortisol levels across conditions. CgA levels were larger at the pretesting time than at the three other test times. Hearing impairment did not affect CgA. Self-rated motivation correlated most often with cortisol or PPD values. The three physiological indicators of cognitive load and stress (PPD, cortisol, and CgA) are not equally affected by speech testing or hearing impairment. Each of them seem to capture a different dimension of sympathetic nervous system activity.

Listening in Naturalistic Scenes: What Can Functional Near-Infrared Spectroscopy and Intersubject Correlation Analysis Tell Us About the Underlying Brain Activity?

Listening to speech in the noisy conditions of everyday life can be effortful, reflecting the increased cognitive workload involved in extracting meaning from a degraded acoustic signal. Studying the underlying neural processes has the potential to provide mechanistic insight into why listening is effortful under certain conditions. In a move toward studying listening effort under ecologically relevant conditions, we used the silent and flexible neuroimaging technique functional near-infrared spectroscopy (fNIRS) to examine brain activity during attentive listening to speech in naturalistic scenes. Thirty normally hearing participants listened to a series of narratives continuously varying in acoustic difficulty while undergoing fNIRS imaging. Participants then listened to another set of closely matched narratives and rated perceived effort and intelligibility for each scene. As expected, self-reported effort generally increased with worsening signal-to-noise ratio. After controlling for better-ear signal-to-noise ratio, perceived effort was greater in scenes that contained competing speech than in those that did not, potentially reflecting an additional cognitive cost of overcoming informational masking. We analyzed the fNIRS data using intersubject correlation, a data-driven approach suitable for analyzing data collected under naturalistic conditions. Significant intersubject correlation was seen in the bilateral auditory cortices and in a range of channels across the prefrontal cortex. The involvement of prefrontal regions is consistent with the notion that higher order cognitive processes are engaged during attentive listening to speech in complex real-world conditions. However, further research is needed to elucidate the relationship between perceived listening effort and activity in these extended cortical networks.

Best Practices and Advice for Using Pupillometry to Measure Listening Effort: An Introduction for Those Who Want to Get Started

Within the field of hearing science, pupillometry is a widely used method for quantifying listening effort. Its use in research is growing exponentially, and many labs are (considering) applying pupillometry for the first time. Hence, there is a growing need for a methods paper on pupillometry covering topics spanning from experiment logistics and timing to data cleaning and what parameters to analyze. This article contains the basic information and considerations needed to plan, set up, and interpret a pupillometry experiment, as well as commentary about how to interpret the response. Included are practicalities like minimal system requirements for recording a pupil response and specifications for peripheral, equipment, experiment logistics and constraints, and different kinds of data processing. Additional details include participant inclusion and exclusion criteria and some methodological considerations that might not be necessary in other auditory experiments. We discuss what data should be recorded and how to monitor the data quality during recording in order to minimize artifacts. Data processing and analysis are considered as well. Finally, we share insights from the collective experience of the authors and discuss some of the challenges that still lie ahead.

Theta- and alpha-power enhancements in the electroencephalogram as an auditory delayed match-to-sample task becomes impossibly difficult

Recent studies have related enhancements of theta- (∼4-8 Hz) and alpha-power (∼8-13 Hz) to listening effort based on parallels between enhancement and task difficulty. In contrast, nonauditory works demonstrate that, although increases in difficulty are initially accompanied by increases in effort, effort decreases when a task becomes so difficult as to exceed one's ability. Given the latter, we examined whether theta- and alpha-power enhancements thought to reflect effortful listening show a quadratic trend across levels of listening difficulty from impossible to easy. Listeners (n = 14) performed an auditory delayed match-to-sample task with frequency-modulated tonal sweeps under impossible, difficult (at ∼70.7% correct threshold), and easy (well above threshold) conditions. Frontal midline theta-power and posterior alpha-power enhancements were observed during the retention interval, with greatest enhancement in the difficult condition. Independent component-based analyses of data suggest that theta-power enhancements stemmed from medial frontal sources at or near the anterior cingulate cortex, whereas alpha-power effects stemmed from occipital cortices. Results support the notion that theta- and alpha-power enhancements reflect effortful cognitive processes during listening, related to auditory working memory and the inhibition of task-irrelevant cortical processing regions, respectively. Theta- and alpha-power dynamics can be used to characterize the cognitive processes that make up effortful listening, including qualitatively different types of listening effort.

A roadmap for the study of conscious audition and its neural basis

How and which aspects of neural activity give rise to subjective perceptual experience-i.e. conscious perception-is a fundamental question of neuroscience. To date, the vast majority of work concerning this question has come from vision, raising the issue of generalizability of prominent resulting theories. However, recent work has begun to shed light on the neural processes subserving conscious perception in other modalities, particularly audition. Here, we outline a roadmap for the future study of conscious auditory perception and its neural basis, paying particular attention to how conscious perception emerges (and of which elements or groups of elements) in complex auditory scenes. We begin by discussing the functional role of the auditory system, particularly as it pertains to conscious perception. Next, we ask: what are the phenomena that need to be explained by a theory of conscious auditory perception? After surveying the available literature for candidate neural correlates, we end by considering the implications that such results have for a general theory of conscious perception as well as prominent outstanding questions and what approaches/techniques can best be used to address them.This article is part of the themed issue 'Auditory and visual scene analysis'.

An inhibitory pull-push circuit in frontal cortex

Push-pull is a canonical computation of excitatory cortical circuits. Here we identify a pull-push inhibitory circuit in frontal cortex that originates in vasoactive intestinal polypeptide (VIP) expressing interneurons. During arousal, VIP cells rapidly and directly inhibit pyramidal neurons; VIP cells also indirectly excite these pyramidal neurons via parallel disinhibition. Thus, arousal exerts a feed-back pull-push influence on excitatory neurons – an inversion of the canonical push-pull of feed-forward input.

The Spatial Release of Cognitive Load in Cocktail Party Is Determined by the Relative Levels of the Talkers

In a multi-talker situation, spatial separation between talkers reduces cognitive processing load: this is the "spatial release of cognitive load". The present study investigated the role played by the relative levels of the talkers on this spatial release of cognitive load. During the experiment, participants had to report the speech emitted by a target talker in the presence of a concurrent masker talker. The spatial separation (0° and 120° angular distance in azimuth) and the relative levels of the talkers (adverse, intermediate, and favorable target-to-masker ratio) were manipulated. The cognitive load was assessed with a prefrontal functional near-infrared spectroscopy. Data from 14 young normal-hearing listeners revealed that the target-to-masker ratio had a direct impact on the spatial release of cognitive load. Spatial separation significantly reduced the prefrontal activity only for the intermediate target-to-masker ratio and had no effect on prefrontal activity for the favorable and the adverse target-to-masker ratios. Therefore, the relative levels of the talkers might be a key point to determine the spatial release of cognitive load and more specifically the prefrontal activity induced by spatial cues in multi-talker situations.

Neural correlates of pupil dilation during human fear learning

BACKGROUND

Fear conditioning and extinction are prevailing experimental and etiological models for normal and pathological anxiety. Pupil dilations in response to conditioned stimuli are increasingly used as a robust psychophysiological readout of fear learning, but their neural correlates remain unknown. We aimed at identifying the neural correlates of pupil responses to threat and safety cues during a fear learning task.

METHODS

Thirty-four healthy subjects underwent a fear conditioning and extinction paradigm with simultaneous functional magnetic resonance imaging (fMRI) and pupillometry. After a stringent preprocessing and artifact rejection procedure, trial-wise pupil responses to threat and safety cues were entered as parametric modulations to the fMRI general linear models.

RESULTS

Trial-wise magnitude of pupil responses to both conditioned and safety stimuli correlated positively with activity in dorsal anterior cingulate cortex (dACC), thalamus, supramarginal gyrus and insula for the entire fear learning task, and with activity in the dACC during the fear conditioning phase in particular. Phasic pupil responses did not show habituation, but were negatively correlated with tonic baseline pupil diameter, which decreased during the task. Correcting phasic pupil responses for the tonic baseline pupil diameter revealed thalamic activity, which was also observed in an analysis employing a linear (declining) time modulation.

CONCLUSION

Pupil dilations during fear conditioning and extinction provide useful readouts to track fear learning on a trial-by-trial level, particularly with simultaneous fMRI. Whereas phasic pupil responses reflect activity in brain regions involved in fear learning and threat appraisal, most prominently in dACC, tonic changes in pupil diameter may reflect changes in general arousal.

Task-Related Vigilance During Word Recognition in Noise for Older Adults with Hearing Loss

BACKGROUND/STUDY CONTEXT

Vigilance refers to the ability to sustain and adapt attentional focus in response to changing task demands. For older adults with hearing loss, vigilant listening may be particularly effortful and variable across individuals. This study examined the extent to which neural responses to sudden, unexpected changes in task structure (e.g., from rest to word recognition epochs) were related to pupillometry measures of listening effort.

METHODS

Individual differences in the task-evoked pupil response during word recognition were used to predict functional magnetic resonance imaging (MRI) estimates of neural responses to salient transitions between quiet rest, noisy rest, and word recognition in unintelligible, fluctuating background noise. Participants included 29 older adults (M = 70.2 years old) with hearing loss (pure tone average across all frequencies = 36.1 dB HL [hearing level], SD = 6.7).

RESULTS

Individuals with a greater average pupil response exhibited a more vigilant pattern of responding on a standardized continuous performance test (response time variability across varying interstimulus intervals r(27) = .38, p = .04). Across participants there was widespread engagement of attention- and sensory-related cortices in response to transitions between blocks of rest and word recognition conditions. Individuals who exhibited larger task-evoked pupil dilation also showed even greater activity in the right primary auditory cortex in response to changes in task structure.

CONCLUSION

Pupillometric estimates of word recognition effort predicted variation in activity within cortical regions that were responsive to salient changes in the environment for older adults with hearing loss. The results of the current study suggest that vigilant attention is increased amongst older adults who exert greater listening effort.

Online Learners' Reading Ability Detection Based on Eye-Tracking Sensors

The detection of university online learners' reading ability is generally problematic and time-consuming. Thus the eye-tracking sensors have been employed in this study, to record temporal and spatial human eye movements. Learners' pupils, blinks, fixation, saccade, and regression are recognized as primary indicators for detecting reading abilities. A computational model is established according to the empirical eye-tracking data, and applying the multi-feature regularization machine learning mechanism based on a Low-rank Constraint. The model presents good generalization ability with an error of only 4.9% when randomly running 100 times. It has obvious advantages in saving time and improving precision, with only 20 min of testing required for prediction of an individual learner's reading ability.

Effect of response context and masker type on word recognition in school-age children and adults

In adults, masked speech recognition improves with the provision of a closed set of response alternatives. The present study evaluated whether school-age children (5-13 years) benefit to the same extent as adults from a forced-choice context, and whether this effect depends on masker type. Experiment 1 compared masked speech reception thresholds for disyllabic words in either an open-set or a four-alternative forced-choice (4AFC) task. Maskers were speech-shaped noise or two-talker speech. Experiment 2 compared masked speech reception thresholds for monosyllabic words in two 4AFC tasks, one in which the target and foils were phonetically similar and one in which they were dissimilar. Maskers were speech-shaped noise, amplitude-modulated noise, or two-talker speech. For both experiments, it was predicted that children would not benefit from the information provided by the 4AFC context to the same degree as adults, particularly when the masker was complex (two-talker) or when audible speech cues were temporally sparse (modulated-noise). Results indicate that young children do benefit from a 4AFC context to the same extent as adults in speech-shaped noise and amplitude-modulated noise, but the benefit of context increases with listener age for the two-talker speech masker.

Eye-Tracking Evidence that Happy Faces Impair Verbal Message Comprehension: The Case of Health Warnings in Direct-to-Consumer Pharmaceutical Television Commercials

Risk warning or disclosure information in advertising is only effective in correcting consumers' judgments if enough cognitive capacity is available to process that information. Hence, comprehension of verbal warnings in TV commercials may suffer if accompanied by positive visual elements. This research addresses this concern about cross-modality interference in the context of direct-to-consumer (DTC) pharmaceutical commercials in the United States by experimentally testing whether positive facial expressions reduce consumers' understanding of the mandated health warning. A content analysis of a sample of DTC commercials reveals that positive facial expressions are more prevalent during the verbal warning act of the commercials than during the other acts. An eye-tracking experiment conducted with specially produced DTC commercials, which vary the valence of characters' facial expressions during the health warning, provides evidence that happy faces reduce objective comprehension of the warning.

A solid frame for the window on cognition: Modeling event-related pupil responses

Pupil size is often used to infer central processes, including attention, memory, and emotion. Recent research has spotlighted its relation to behavioral variables from decision-making models and to neural variables such as locus coeruleus activity and cortical oscillations. As yet, a unified and principled approach for analyzing pupil responses is lacking. Here we seek to establish a formal, quantitative forward model for pupil responses by describing them with linear time-invariant systems. Based on empirical data from human participants, we show that a combination of two linear time-invariant systems can parsimoniously explain approximately all variance evoked by illuminance changes. Notably, the model makes a counterintuitive prediction that pupil constriction dominates the responses to darkness flashes, as in previous empirical reports. This prediction was quantitatively confirmed for responses to light and darkness flashes in an independent group of participants. Crucially, illuminance- and nonilluminance-related inputs to the pupillary system are presumed to share a common final pathway, composed of muscles and nerve terminals. Hence, we can harness our illuminance-based model to estimate the temporal evolution of this neural input for an auditory-oddball task, an emotional-words task, and a visual-detection task. Onset and peak latencies of the estimated neural inputs furnish plausible hypotheses for the complexity of the underlying neural circuit. To conclude, this mathematical description of pupil responses serves as a prerequisite to refining their relation to behavioral and brain indices of cognitive processes.

Effects of Aging and Adult-Onset Hearing Loss on Cortical Auditory Regions

Hearing loss is a common feature in human aging. It has been argued that dysfunctions in central processing are important contributing factors to hearing loss during older age. Aging also has well documented consequences for neural structure and function, but it is not clear how these effects interact with those that arise as a consequence of hearing loss. This paper reviews the effects of aging and adult-onset hearing loss in the structure and function of cortical auditory regions. The evidence reviewed suggests that aging and hearing loss result in atrophy of cortical auditory regions and stronger engagement of networks involved in the detection of salient events, adaptive control and re-allocation of attention. These cortical mechanisms are engaged during listening in effortful conditions in normal hearing individuals. Therefore, as a consequence of aging and hearing loss, all listening becomes effortful and cognitive load is constantly high, reducing the amount of available cognitive resources. This constant effortful listening and reduced cognitive spare capacity could be what accelerates cognitive decline in older adults with hearing loss.

The Timing and Effort of Lexical Access in Natural and Degraded Speech

Understanding speech is effortless in ideal situations, and although adverse conditions, such as caused by hearing impairment, often render it an effortful task, they do not necessarily suspend speech comprehension. A prime example of this is speech perception by cochlear implant users, whose hearing prostheses transmit speech as a significantly degraded signal. It is yet unknown how mechanisms of speech processing deal with such degraded signals, and whether they are affected by effortful processing of speech. This paper compares the automatic process of lexical competition between natural and degraded speech, and combines gaze fixations, which capture the course of lexical disambiguation, with pupillometry, which quantifies the mental effort involved in processing speech. Listeners' ocular responses were recorded during disambiguation of lexical embeddings with matching and mismatching durational cues. Durational cues were selected due to their substantial role in listeners' quick limitation of the number of lexical candidates for lexical access in natural speech. Results showed that lexical competition increased mental effort in processing natural stimuli in particular in presence of mismatching cues. Signal degradation reduced listeners' ability to quickly integrate durational cues in lexical selection, and delayed and prolonged lexical competition. The effort of processing degraded speech was increased overall, and because it had its sources at the pre-lexical level this effect can be attributed to listening to degraded speech rather than to lexical disambiguation. In sum, the course of lexical competition was largely comparable for natural and degraded speech, but showed crucial shifts in timing, and different sources of increased mental effort. We argue that well-timed progress of information from sensory to pre-lexical and lexical stages of processing, which is the result of perceptual adaptation during speech development, is the reason why in ideal situations speech is perceived as an undemanding task. Degradation of the signal or the receiver channel can quickly bring this well-adjusted timing out of balance and lead to increase in mental effort. Incomplete and effortful processing at the early pre-lexical stages has its consequences on lexical processing as it adds uncertainty to the forming and revising of lexical hypotheses.

The Impact of Auditory Spectral Resolution on Listening Effort Revealed by Pupil Dilation

OBJECTIVES

This study measured the impact of auditory spectral resolution on listening effort. Systematic degradation in spectral resolution was hypothesized to elicit corresponding systematic increases in pupil dilation, consistent with the notion of pupil dilation as a marker of cognitive load.

DESIGN

Spectral resolution of sentences was varied with two different vocoders: (1) a noise-channel vocoder with a variable number of spectral channels; and (2) a vocoder designed to simulate front-end processing of a cochlear implant, including peak-picking channel selection with variable synthesis filter slopes to simulate spread of neural excitation. Pupil dilation was measured after subject-specific luminance adjustment and trial-specific baseline measures. Mixed-effects growth curve analysis was used to model pupillary responses over time.

RESULTS

For both types of vocoder, pupil dilation grew with each successive degradation in spectral resolution. Within each condition, pupillary responses were not related to intelligibility scores, and the effect of spectral resolution on pupil dilation persisted even when only analyzing trials in which responses were 100% correct.

CONCLUSIONS

Intelligibility scores alone were not sufficient to quantify the effort required to understand speech with poor resolution. Degraded spectral resolution results in increased effort required to understand speech, even when intelligibility is at 100%. Pupillary responses were a sensitive and highly granular measurement to reveal changes in listening effort. Pupillary responses might potentially reveal the benefits of aural prostheses that are not captured by speech intelligibility performance alone as well as the disadvantages that are overcome by increased listening effort.

Autonomic Nervous System Responses During Perception of Masked Speech may Reflect Constructs other than Subjective Listening Effort

Typically, understanding speech seems effortless and automatic. However, a variety of factors may, independently or interactively, make listening more effortful. Physiological measures may help to distinguish between the application of different cognitive mechanisms whose operation is perceived as effortful. In the present study, physiological and behavioral measures associated with task demand were collected along with behavioral measures of performance while participants listened to and repeated sentences. The goal was to measure psychophysiological reactivity associated with three degraded listening conditions, each of which differed in terms of the source of the difficulty (distortion, energetic masking, and informational masking), and therefore were expected to engage different cognitive mechanisms. These conditions were chosen to be matched for overall performance (keywords correct), and were compared to listening to unmasked speech produced by a natural voice. The three degraded conditions were: (1) Unmasked speech produced by a computer speech synthesizer, (2) Speech produced by a natural voice and masked byspeech-shaped noise and (3) Speech produced by a natural voice and masked by two-talker babble. Masked conditions were both presented at a -8 dB signal to noise ratio (SNR), a level shown in previous research to result in comparable levels of performance for these stimuli and maskers. Performance was measured in terms of proportion of key words identified correctly, and task demand or effort was quantified subjectively by self-report. Measures of psychophysiological reactivity included electrodermal (skin conductance) response frequency and amplitude, blood pulse amplitude and pulse rate. Results suggest that the two masked conditions evoked stronger psychophysiological reactivity than did the two unmasked conditions even when behavioral measures of listening performance and listeners’ subjective perception of task demand were comparable across the three degraded conditions.

A physiological signature of sound meaning in dementia

The meaning of sensory objects is often behaviourally and biologically salient and decoding of semantic salience is potentially vulnerable in dementia. However, it remains unclear how sensory semantic processing is linked to physiological mechanisms for coding object salience and how that linkage is affected by neurodegenerative diseases. Here we addressed this issue using the paradigm of complex sounds. We used pupillometry to compare physiological responses to real versus synthetic nonverbal sounds in patients with canonical dementia syndromes (behavioural variant frontotemporal dementia – bvFTD, semantic dementia – SD; progressive nonfluent aphasia – PNFA; typical Alzheimer's disease – AD) relative to healthy older individuals. Nonverbal auditory semantic competence was assessed using a novel within-modality sound classification task and neuroanatomical associations of pupillary responses were assessed using voxel-based morphometry (VBM) of patients' brain MR images. After taking affective stimulus factors into account, patients with SD and AD showed significantly increased pupil responses to real versus synthetic sounds relative to healthy controls. The bvFTD, SD and AD groups had a nonverbal auditory semantic deficit relative to healthy controls and nonverbal auditory semantic performance was inversely correlated with the magnitude of the enhanced pupil response to real versus synthetic sounds across the patient cohort. A region of interest analysis demonstrated neuroanatomical associations of overall pupil reactivity and differential pupil reactivity to sound semantic content in superior colliculus and left anterior temporal cortex respectively. Our findings suggest that autonomic coding of auditory semantic ambiguity in the setting of a damaged semantic system may constitute a novel physiological signature of neurodegenerative diseases.

Does Signal Degradation Affect Top-Down Processing of Speech?

Speech perception is formed based on both the acoustic signal and listeners' knowledge of the world and semantic context. Access to semantic information can facilitate interpretation of degraded speech, such as speech in background noise or the speech signal transmitted via cochlear implants (CIs). This paper focuses on the latter, and investigates the time course of understanding words, and how sentential context reduces listeners' dependency on the acoustic signal for natural and degraded speech via an acoustic CI simulation.In an eye-tracking experiment we combined recordings of listeners' gaze fixations with pupillometry, to capture effects of semantic information on both the time course and effort of speech processing. Normal-hearing listeners were presented with sentences with or without a semantically constraining verb (e.g., crawl) preceding the target (baby), and their ocular responses were recorded to four pictures, including the target, a phonological (bay) competitor and a semantic (worm) and an unrelated distractor.The results show that in natural speech, listeners' gazes reflect their uptake of acoustic information, and integration of preceding semantic context. Degradation of the signal leads to a later disambiguation of phonologically similar words, and to a delay in integration of semantic information. Complementary to this, the pupil dilation data show that early semantic integration reduces the effort in disambiguating phonologically similar words. Processing degraded speech comes with increased effort due to the impoverished nature of the signal. Delayed integration of semantic information further constrains listeners' ability to compensate for inaudible signals.

Neural bases of accented speech perception

The recognition of unfamiliar regional and foreign accents represents a challenging task for the speech perception system (Floccia et al., 2006; Adank et al., 2009). Despite the frequency with which we encounter such accents, the neural mechanisms supporting successful perception of accented speech are poorly understood. Nonetheless, candidate neural substrates involved in processing speech in challenging listening conditions, including accented speech, are beginning to be identified. This review will outline neural bases associated with perception of accented speech in the light of current models of speech perception, and compare these data to brain areas associated with processing other speech distortions. We will subsequently evaluate competing models of speech processing with regards to neural processing of accented speech. See Cristia et al. (2012) for an in-depth overview of behavioral aspects of accent processing.

Evidence of Conjoint Activation of the Anterior Insular and Cingulate Cortices during Effortful Tasks

The ability to perform effortful tasks is a topic that has received considerable interest in the research of higher functions of the human brain. Neuroimaging studies show that the anterior insular and the anterior cingulate cortices are involved in a multitude of cognitive tasks that require mental effort. In this study, we investigated brain responses to effort using cognitive tasks with task-difficulty modulations and functional magnetic resonance imaging (fMRI). We hypothesized that effortful performance involves modulation of activation in the anterior insular and the anterior cingulate cortices, and that the modulation correlates with individual performance levels. Healthy participants performed tasks probing verbal working memory capacity using the reading span task, and visual perception speed using the inspection time task. In the fMRI analysis, we focused on identifying effort-related brain activation. The results showed that working memory and inspection time performances were directly related. The bilateral anterior insular and anterior cingulate cortices showed significantly increased activation during each task with common portions that were active across both tasks. We observed increased brain activation in the right anterior insula and the anterior cingulate cortex in participants with low working memory performance. In line with the reported results, we suggest that activation in the anterior insular and cingulate cortices is consistent with the neural efficiency hypothesis (Neubauer).