Getting the Cocktail Party Started: Masking Effects in Speech Perception

Attention-Driven Modulation of Auditory Cortex Activity during Selective Listening in a Multispeaker Setting

Real-world listening settings often consist of multiple concurrent sound streams. To limit perceptual interference during selective listening, the auditory system segregates and filters the relevant sensory input. Previous work provided evidence that the auditory cortex is critically involved in this process and selectively gates attended input toward subsequent processing stages. We studied at which level of auditory cortex processing this filtering of attended information occurs using functional magnetic resonance imaging (fMRI) and a naturalistic selective listening task. Forty-five human listeners (of either sex) attended to one of two continuous speech streams, presented either concurrently or in isolation. Functional data were analyzed using an inter-subject analysis to assess stimulus-specific components of ongoing auditory cortex activity. Our results suggest that stimulus-related activity in the primary auditory cortex and the adjacent planum temporale are hardly affected by attention, whereas brain responses at higher stages of the auditory cortex processing hierarchy become progressively more selective for the attended input. Consistent with these findings, a complementary analysis of stimulus-driven functional connectivity further demonstrated that information on the to-be-ignored speech stream is shared between the primary auditory cortex and the planum temporale but largely fails to reach higher processing stages. Our findings suggest that the neural processing of ignored speech cannot be effectively suppressed at the level of early cortical processing of acoustic features but is gradually attenuated once the competing speech streams are fully segregated.

Perceiving and misperceiving speech: lexical and sublexical processing in the superior temporal lobes

Listeners can use prior knowledge to predict the content of noisy speech signals, enhancing perception. However, this process can also elicit misperceptions. For the first time, we employed a prime-probe paradigm and transcranial magnetic stimulation to investigate causal roles for the left and right posterior superior temporal gyri (pSTG) in the perception and misperception of degraded speech. Listeners were presented with spectrotemporally degraded probe sentences preceded by a clear prime. To produce misperceptions, we created partially mismatched pseudo-sentence probes via homophonic nonword transformations (e.g. The little girl was excited to lose her first tooth-Tha fittle girmn wam expited du roos har derst cooth). Compared to a control site (vertex), inhibitory stimulation of the left pSTG selectively disrupted priming of real but not pseudo-sentences. Conversely, inhibitory stimulation of the right pSTG enhanced priming of misperceptions with pseudo-sentences, but did not influence perception of real sentences. These results indicate qualitatively different causal roles for the left and right pSTG in perceiving degraded speech, supporting bilateral models that propose engagement of the right pSTG in sublexical processing.

A systematic review on speech-in-noise perception in autism

Individuals with autism spectrum disorder (ASD) exhibit atypical speech-in-noise (SiN) perception, but the scope of these impairments has not been clearly defined. We conducted a systematic review of the behavioural research on SiN perception in ASD, using a comprehensive search strategy across databases (Embase, Pubmed, Web of Science, APA PsycArticles, LLBA, clinicaltrials.gov and PsyArXiv). We withheld 20 studies that generally revealed intact speech perception in stationary noise, while impairments in speech discrimination were found in temporally modulated noise, concurrent speech, and audiovisual speech perception. An association with auditory temporal processing deficits, exacerbated by suboptimal language skills, is shown. Speech-in-speech perception might be further impaired due to deficient top-down processing of speech. Further research is needed to address remaining challenges and gaps in our understanding of these impairments, including the developmental aspects of SiN processing in ASD, and the impact of gender and social attentional orienting on this ability. Our findings have important implications for improving communication in ASD, both in daily interactions and in clinical and educational settings.

An Inverse Relationship Between Gray Matter Volume and Speech-in-Noise Performance in Tinnitus Patients with Normal Hearing Sensitivity

Speech-in-noise (SiN) recognition difficulties are often reported in patients with tinnitus. Although brain structural changes such as reduced gray matter (GM) volume in auditory and cognitive processing regions have been reported in the tinnitus population, it remains unclear how such changes influence speech understanding, such as SiN performance. In this study, pure-tone audiometry and Quick Speech-in-Noise test were conducted on individuals with tinnitus and normal hearing and hearing-matched controls. T1-weighted structural MRI images were obtained from all participants. After preprocessing, GM volumes were compared between tinnitus and control groups using whole-brain and region-of-interest analyses. Further, regression analyses were performed to examine the correlation between regional GM volume and SiN scores in each group. The results showed decreased GM volume in the right inferior frontal gyrus in the tinnitus group relative to the control group. In the tinnitus group, SiN performance showed a negative correlation with GM volume in the left cerebellum (Crus I/II) and the left superior temporal gyrus; no significant correlation between SiN performance and regional GM volume was found in the control group. Even with clinically defined normal hearing and comparable SiN performance relative to controls, tinnitus appears to change the association between SiN recognition and regional GM volume. This change may reflect compensatory mechanisms utilized by individuals with tinnitus who maintain behavioral performance.

Listening to two speakers: Capacity and tradeoffs in neural speech tracking during Selective and Distributed Attention

Speech comprehension is severely compromised when several people talk at once, due to limited perceptual and cognitive resources. In such circumstances, top-down attention mechanisms can actively prioritize processing of task-relevant speech. However, behavioral and neural evidence suggest that this selection is not exclusive, and the system may have sufficient capacity to process additional speech input as well. Here we used a data-driven approach to contrast two opposing hypotheses regarding the system's capacity to co-represent competing speech: Can the brain represent two speakers equally or is the system fundamentally limited, resulting in tradeoffs between them? Neural activity was measured using magnetoencephalography (MEG) as human participants heard concurrent speech narratives and engaged in two tasks: Selective Attention, where only one speaker was task-relevant and Distributed Attention, where both speakers were equally relevant. Analysis of neural speech-tracking revealed that both tasks engaged a similar network of brain regions involved in auditory processing, attentional control and speech processing. Interestingly, during both Selective and Distributed Attention the neural representation of competing speech showed a bias towards one speaker. This is in line with proposed 'bottlenecks' for co-representation of concurrent speech and suggests that good performance on distributed attention tasks may be achieved by toggling attention between speakers over time.

Rostro-caudal networks for sound processing in the primate brain

Sound is processed in primate brains along anatomically and functionally distinct streams: this pattern can be seen in both human and non-human primates. We have previously proposed a general auditory processing framework in which these different perceptual profiles are associated with different computational characteristics. In this paper we consider how recent work supports our framework.

Cocktail party training induces increased speech intelligibility and decreased cortical activity in bilateral inferior frontal gyri. A functional near-infrared study

The human brain networks responsible for selectively listening to a voice amid other talkers remain to be clarified. The present study aimed to investigate relationships between cortical activity and performance in a speech-in-speech task, before (Experiment I) and after training-induced improvements (Experiment II). In Experiment I, 74 participants performed a speech-in-speech task while their cortical activity was measured using a functional near infrared spectroscopy (fNIRS) device. One target talker and one masker talker were simultaneously presented at three different target-to-masker ratios (TMRs): adverse, intermediate and favorable. Behavioral results show that performance may increase monotonically with TMR in some participants and failed to decrease, or even improved, in the adverse-TMR condition for others. On the neural level, an extensive brain network including the frontal (left prefrontal cortex, right dorsolateral prefrontal cortex and bilateral inferior frontal gyri) and temporal (bilateral auditory cortex) regions was more solicited by the intermediate condition than the two others. Additionally, bilateral frontal gyri and left auditory cortex activities were found to be positively correlated with behavioral performance in the adverse-TMR condition. In Experiment II, 27 participants, whose performance was the poorest in the adverse-TMR condition of Experiment I, were trained to improve performance in that condition. Results show significant performance improvements along with decreased activity in bilateral inferior frontal gyri, the right dorsolateral prefrontal cortex, the left inferior parietal cortex and the right auditory cortex in the adverse-TMR condition after training. Arguably, lower neural activity reflects higher efficiency in processing masker inhibition after speech-in-speech training. As speech-in-noise tasks also imply frontal and temporal regions, we suggest that regardless of the type of masking (speech or noise) the complexity of the task will prompt the implication of a similar brain network. Furthermore, the initial significant cognitive recruitment will be reduced following a training leading to an economy of cognitive resources.

Intelligibility of audiovisual sentences drives multivoxel response patterns in human superior temporal cortex

Regions of the human posterior superior temporal gyrus and sulcus (pSTG/S) respond to the visual mouth movements that constitute visual speech and the auditory vocalizations that constitute auditory speech, and neural responses in pSTG/S may underlie the perceptual benefit of visual speech for the comprehension of noisy auditory speech. We examined this possibility through the lens of multivoxel pattern responses in pSTG/S. BOLD fMRI data was collected from 22 participants presented with speech consisting of English sentences presented in five different formats: visual-only; auditory with and without added auditory noise; and audiovisual with and without auditory noise. Participants reported the intelligibility of each sentence with a button press and trials were sorted post-hoc into those that were more or less intelligible. Response patterns were measured in regions of the pSTG/S identified with an independent localizer. Noisy audiovisual sentences with very similar physical properties evoked very different response patterns depending on their intelligibility. When a noisy audiovisual sentence was reported as intelligible, the pattern was nearly identical to that elicited by clear audiovisual sentences. In contrast, an unintelligible noisy audiovisual sentence evoked a pattern like that of visual-only sentences. This effect was less pronounced for noisy auditory-only sentences, which evoked similar response patterns regardless of intelligibility. The successful integration of visual and auditory speech produces a characteristic neural signature in pSTG/S, highlighting the importance of this region in generating the perceptual benefit of visual speech.

Attention to Speech: Mapping Distributed and Selective Attention Systems

When faced with situations where many people talk at once, individuals can employ different listening strategies to deal with the cacophony of speech sounds and to achieve different goals. In this fMRI study, we investigated how the pattern of neural activity is affected by the type of attention applied to speech in a simulated "cocktail party." Specifically, we compared brain activation patterns when listeners "attended selectively" to only one speaker and ignored all others, versus when they "distributed their attention" and followed several concurrent speakers. Conjunction analysis revealed a highly overlapping network of regions activated for both types of attention, including auditory association cortex (bilateral STG/STS) and frontoparietal regions related to speech processing and attention (bilateral IFG/insula, right MFG, left IPS). Activity within nodes of this network, though, was modulated by the type of attention required as well as the number of competing speakers. Auditory and speech-processing regions exhibited higher activity during distributed attention, whereas frontoparietal regions were activated more strongly during selective attention. These results suggest a common "attention to speech" network, which provides the computational infrastructure to deal effectively with multi-speaker input, but with sufficient flexibility to implement different prioritization strategies and to adapt to different listener goals.

Attentional Modulation of Hierarchical Speech Representations in a Multitalker Environment

Humans are remarkably adept in listening to a desired speaker in a crowded environment, while filtering out nontarget speakers in the background. Attention is key to solving this difficult cocktail-party task, yet a detailed characterization of attentional effects on speech representations is lacking. It remains unclear across what levels of speech features and how much attentional modulation occurs in each brain area during the cocktail-party task. To address these questions, we recorded whole-brain blood-oxygen-level-dependent (BOLD) responses while subjects either passively listened to single-speaker stories, or selectively attended to a male or a female speaker in temporally overlaid stories in separate experiments. Spectral, articulatory, and semantic models of the natural stories were constructed. Intrinsic selectivity profiles were identified via voxelwise models fit to passive listening responses. Attentional modulations were then quantified based on model predictions for attended and unattended stories in the cocktail-party task. We find that attention causes broad modulations at multiple levels of speech representations while growing stronger toward later stages of processing, and that unattended speech is represented up to the semantic level in parabelt auditory cortex. These results provide insights on attentional mechanisms that underlie the ability to selectively listen to a desired speaker in noisy multispeaker environments.

The cortical organization of listening effort: New insight from functional near-infrared spectroscopy

Everyday challenges impact our ability to hear and comprehend spoken language with ease, such as accented speech (source factors), spectral degradation (transmission factors), complex or unfamiliar language use (message factors), and predictability (context factors). Auditory degradation and linguistic complexity in the brain and behavior have been well investigated, and several computational models have emerged. The work here provides a novel test of the hypotheses that listening effort is partially reliant on higher cognitive auditory attention and working memory mechanisms in the frontal lobe, and partially reliant on hierarchical linguistic computation in the brain's left hemisphere. We specifically hypothesize that these models are robust and can be applied in ecologically relevant and coarse-grain contexts that rigorously control for acoustic and linguistic listening challenges. Using functional near-infrared spectroscopy during an auditory plausibility judgment task, we show the hierarchical cortical organization for listening effort in the frontal and left temporal-parietal brain regions. In response to increasing levels of cognitive demand, we found (i) poorer comprehension, (ii) slower reaction times, (iii) increasing levels of perceived mental effort, (iv) increasing levels of brain activity in the prefrontal cortex, (v) hierarchical modulation of core language processing regions that reflect increasingly higher-order auditory-linguistic processing, and (vi) a correlation between participants' mental effort ratings and their performance on the task. Our results demonstrate that listening effort is partly reliant on higher cognitive auditory attention and working memory mechanisms in the frontal lobe and partly reliant on hierarchical linguistic computation in the brain's left hemisphere. Further, listening effort is driven by a voluntary, motivation-based attention system for which our results validate the use of a single-item post-task questionnaire for measuring perceived levels of mental effort and predicting listening performance. We anticipate our study to be a starting point for more sophisticated models of listening effort and even cognitive neuroplasticity in hearing aid and cochlear implant users.

School-age children benefit from voice gender cue differences for the perception of speech in competing speech

Differences in speakers' voice characteristics, such as mean fundamental frequency (F0) and vocal-tract length (VTL), that primarily define speakers' so-called perceived voice gender facilitate the perception of speech in competing speech. Perceiving speech in competing speech is particularly challenging for children, which may relate to their lower sensitivity to differences in voice characteristics than adults. This study investigated the development of the benefit from F0 and VTL differences in school-age children (4-12 years) for separating two competing speakers while tasked with comprehending one of them and also the relationship between this benefit and their corresponding voice discrimination thresholds. Children benefited from differences in F0, VTL, or both cues at all ages tested. This benefit proportionally remained the same across age, although overall accuracy continued to differ from that of adults. Additionally, children's benefit from F0 and VTL differences and their overall accuracy were not related to their discrimination thresholds. Hence, although children's voice discrimination thresholds and speech in competing speech perception abilities develop throughout the school-age years, children already show a benefit from voice gender cue differences early on. Factors other than children's discrimination thresholds seem to relate more closely to their developing speech in competing speech perception abilities.

Speech-evoked brain activity is more robust to competing speech when it is spoken by someone familiar

When speech is masked by competing sound, people are better at understanding what is said if the talker is familiar compared to unfamiliar. The benefit is robust, but how does processing of familiar voices facilitate intelligibility? We combined high-resolution fMRI with representational similarity analysis to quantify the difference in distributed activity between clear and masked speech. We demonstrate that brain representations of spoken sentences are less affected by a competing sentence when they are spoken by a friend or partner than by someone unfamiliar-effectively, showing a cortical signal-to-noise ratio (SNR) enhancement for familiar voices. This effect correlated with the familiar-voice intelligibility benefit. We functionally parcellated auditory cortex, and found that the most prominent familiar-voice advantage was manifest along the posterior superior and middle temporal gyri. Overall, our results demonstrate that experience-driven improvements in intelligibility are associated with enhanced multivariate pattern activity in posterior temporal cortex.

Predisposing and Precipitating Factors for Delirium in the Very Old (≥80 Years): A Prospective Cohort Study of 3,076 Patients

BACKGROUND

Predisposing and precipitating factors for delirium for the elderly, over the age of 65 years, are known, but not for the very old, over 80 years. As the society is getting older and evermore patients will reach >80 years, more evidence of the factors and their contribution to delirium is required in this patient group.

METHODS

In the course of 1 year, 3,076 patients above 80 years were screened prospectively for delirium based on a Delirium Observation Screening (DOS) scale, Intensive Care Delirium Screening Checklist (ICDSC), and a DSM (Diagnostic and Statistical Manual)-5 nursing instrument (ePA-AC) construct. Relevant predisposing and precipitating factors for delirium were assessed with a multiple regression analysis.

RESULTS

Of 3,076 patients above 80 years, 1,285 (41.8%) developed a delirium, which led to twice prolonged hospitalization (p < 0.001), requirement for subsequent assisted living (OR 2.2, CI: 1.73-2.8, p < 0.001), and increased mortality (OR 24.88, CI: 13.75-45.03, p < 0.001). Relevant predisposing factors were dementia (OR 15.6, CI: 10.17-23.91, p < 0.001), pressure sores (OR 4.61, CI: 2.74-7.76, p < 0.001), and epilepsy (OR 3.65, CI: 2.12-6.28, p < 0.0001). Relevant precipitating factors were acute renal failure (4.96, CI: 2.38-10.3, p < 0.001), intracranial hemorrhage (OR 8.7, CI: 4.27-17.7, p < 0.001), and pleural effusions (OR 3.25, CI: 1.77-17.8, p < 0.001).

CONCLUSION

Compared to the general delirium rate of approximately 20%, the prevalence of delirium doubled above the age of 80 years (41.8%) due to predisposing factors uncommon in younger patients.

Auditory Processing Disorder and Its Comorbidities: A Need for Consistency in Test Cutoff Scores

Purpose No gold standard criteria exist for diagnosing developmental auditory processing disorder (APD). This study aimed to identify APD criteria, which are consistent with that used for comorbidities, and how comorbidities predicted APD. Method A retrospective study of 167 participants (males = 105, females = 62; age: 6-16 years; nonverbal IQ > 80) with suspected APD is presented. Five SCAN-3 tests evaluated auditory processing (AP). Comorbidities included attention-deficit/hyperactivity disorder, language impairment, and impaired manual dexterity, which were identified using percentile ≤ 5 in the Swanson, Nolan and Pelham parental rating scale; Children's Communication Checklist-2; and Movement Assessment Battery for Children-2, respectively. Results Percentiles ≤ 9, ≤ 5, and < 2 in two or more AP tests had sensitivities (specificities) of 76% (70.6%), 59.3% (76.5%), and 26% (82.4%), respectively, in predicting comorbidities, which were present in 150 of the 167 participants. The criterion of "≤ 9 percentile in two or more AP tests" (Approach I) diagnosed APD in 119 participants, and criterion "≤ 5 percentile in two or more AP tests or ≤ 5 percentile in one AP plus one or more measures of comorbidities" (Approach II) diagnosed 123. The combination of approaches diagnosed 128 participants (76.6%) with APD, of which 114 were diagnosed by each approach (89%). Language impairment and impaired manual dexterity, but not attention-deficit/hyperactivity disorder, predicted APD. Conclusions "Percentile ≤ 9 in two or more AP tests" or "percentile ≤ 5 in one AP plus one or more measures of comorbidities" are evidence-based APD diagnostic criteria. Holistic and interprofessional practice evaluating comorbidities including motor skills is important for APD.

Breaking down the cocktail party: Attentional modulation of cerebral audiovisual speech processing

Recent studies utilizing electrophysiological speech envelope reconstruction have sparked renewed interest in the cocktail party effect by showing that auditory neurons entrain to selectively attended speech. Yet, the neural networks of attention to speech in naturalistic audiovisual settings with multiple sound sources remain poorly understood. We collected functional brain imaging data while participants viewed audiovisual video clips of lifelike dialogues with concurrent distracting speech in the background. Dialogues were presented in a full-factorial design, comprising task (listen to the dialogues vs. ignore them), audiovisual quality and semantic predictability. We used univariate analyses in combination with multivariate pattern analysis (MVPA) to study modulations of brain activity related to attentive processing of audiovisual speech. We found attentive speech processing to cause distinct spatiotemporal modulation profiles in distributed cortical areas including sensory and frontal-control networks. Semantic coherence modulated attention-related activation patterns in the earliest stages of auditory cortical processing, suggesting that the auditory cortex is involved in high-level speech processing. Our results corroborate views that emphasize the dynamic nature of attention, with task-specificity and context as cornerstones of the underlying neuro-cognitive mechanisms.

Neural Representation Enhanced for Speech and Reduced for Background Noise With a Hearing Aid Noise Reduction Scheme During a Selective Attention Task

Objectives

Selectively attending to a target talker while ignoring multiple interferers (competing talkers and background noise) is more difficult for hearing-impaired (HI) individuals compared to normal-hearing (NH) listeners. Such tasks also become more difficult as background noise levels increase. To overcome these difficulties, hearing aids (HAs) offer noise reduction (NR) schemes. The objective of this study was to investigate the effect of NR processing (inactive, where the NR feature was switched off, vs. active, where the NR feature was switched on) on the neural representation of speech envelopes across two different background noise levels [+3 dB signal-to-noise ratio (SNR) and +8 dB SNR] by using a stimulus reconstruction (SR) method.

Design

To explore how NR processing supports the listeners’ selective auditory attention, we recruited 22 HI participants fitted with HAs. To investigate the interplay between NR schemes, background noise, and neural representation of the speech envelopes, we used electroencephalography (EEG). The participants were instructed to listen to a target talker in front while ignoring a competing talker in front in the presence of multi-talker background babble noise.

Results

The results show that the neural representation of the attended speech envelope was enhanced by the active NR scheme for both background noise levels. The neural representation of the attended speech envelope at lower (+3 dB) SNR was shifted, approximately by 5 dB, toward the higher (+8 dB) SNR when the NR scheme was turned on. The neural representation of the ignored speech envelope was modulated by the NR scheme and was mostly enhanced in the conditions with more background noise. The neural representation of the background noise was modulated (i.e., reduced) by the NR scheme and was significantly reduced in the conditions with more background noise. The neural representation of the net sum of the ignored acoustic scene (ignored talker and background babble) was not modulated by the NR scheme but was significantly reduced in the conditions with a reduced level of background noise. Taken together, we showed that the active NR scheme enhanced the neural representation of both the attended and the ignored speakers and reduced the neural representation of background noise, while the net sum of the ignored acoustic scene was not enhanced.

Conclusion

Altogether our results support the hypothesis that the NR schemes in HAs serve to enhance the neural representation of speech and reduce the neural representation of background noise during a selective attention task. We contend that these results provide a neural index that could be useful for assessing the effects of HAs on auditory and cognitive processing in HI populations.

Modulation of Brain Activity by Selective Attention to Audiovisual Dialogues

In real-life noisy situations, we can selectively attend to conversations in the presence of irrelevant voices, but neurocognitive mechanisms in such natural listening situations remain largely unexplored. Previous research has shown distributed activity in the mid superior temporal gyrus (STG) and sulcus (STS) while listening to speech and human voices, in the posterior STS and fusiform gyrus when combining auditory, visual and linguistic information, as well as in left-hemisphere temporal and frontal cortical areas during comprehension. In the present functional magnetic resonance imaging (fMRI) study, we investigated how selective attention modulates neural responses to naturalistic audiovisual dialogues. Our healthy adult participants (N = 15) selectively attended to video-taped dialogues between a man and woman in the presence of irrelevant continuous speech in the background. We modulated the auditory quality of dialogues with noise vocoding and their visual quality by masking speech-related facial movements. Both increased auditory quality and increased visual quality were associated with bilateral activity enhancements in the STG/STS. In addition, decreased audiovisual stimulus quality elicited enhanced fronto-parietal activity, presumably reflecting increased attentional demands. Finally, attention to the dialogues, in relation to a control task where a fixation cross was attended and the dialogue ignored, yielded enhanced activity in the left planum polare, angular gyrus, the right temporal pole, as well as in the orbitofrontal/ventromedial prefrontal cortex and posterior cingulate gyrus. Our findings suggest that naturalistic conversations effectively engage participants and reveal brain networks related to social perception in addition to speech and semantic processing networks.

Understanding rostral-caudal auditory cortex contributions to auditory perception

There are functional and anatomical distinctions between the neural systems involved in the recognition of sounds in the environment and those involved in the sensorimotor guidance of sound production and the spatial processing of sound. Evidence for the separation of these processes has historically come from disparate literatures on the perception and production of speech, music and other sounds. More recent evidence indicates that there are computational distinctions between the rostral and caudal primate auditory cortex that may underlie functional differences in auditory processing. These functional differences may originate from differences in the response times and temporal profiles of neurons in the rostral and caudal auditory cortex, suggesting that computational accounts of primate auditory pathways should focus on the implications of these temporal response differences.

The Causal Role of Left and Right Superior Temporal Gyri in Speech Perception in Noise: A Transcranial Magnetic Stimulation Study

Successful perception of speech in everyday listening conditions requires effective listening strategies to overcome common acoustic distortions, such as background noise. Convergent evidence from neuroimaging and clinical studies identify activation within the temporal lobes as key to successful speech perception. However, current neurobiological models disagree on whether the left temporal lobe is sufficient for successful speech perception or whether bilateral processing is required. We addressed this issue using TMS to selectively disrupt processing in either the left or right superior temporal gyrus (STG) of healthy participants to test whether the left temporal lobe is sufficient or whether both left and right STG are essential. Participants repeated keywords from sentences presented in background noise in a speech reception threshold task while receiving online repetitive TMS separately to the left STG, right STG, or vertex or while receiving no TMS. Results show an equal drop in performance following application of TMS to either left or right STG during the task. A separate group of participants performed a visual discrimination threshold task to control for the confounding side effects of TMS. Results show no effect of TMS on the control task, supporting the notion that the results of Experiment 1 can be attributed to modulation of cortical functioning in STG rather than to side effects associated with online TMS. These results indicate that successful speech perception in everyday listening conditions requires both left and right STG and thus have ramifications for our understanding of the neural organization of spoken language processing.

Autonomic Nervous System Correlates of Speech Categorization Revealed Through Pupillometry

Human perception requires the many-to-one mapping between continuous sensory elements and discrete categorical representations. This grouping operation underlies the phenomenon of categorical perception (CP)-the experience of perceiving discrete categories rather than gradual variations in signal input. Speech perception requires CP because acoustic cues do not share constant relations with perceptual-phonetic representations. Beyond facilitating perception of unmasked speech, we reasoned CP might also aid the extraction of target speech percepts from interfering sound sources (i.e., noise) by generating additional perceptual constancy and reducing listening effort. Specifically, we investigated how noise interference impacts cognitive load and perceptual identification of unambiguous (i.e., categorical) vs. ambiguous stimuli. Listeners classified a speech vowel continuum (/u/-/a/) at various signal-to-noise ratios (SNRs [unmasked, 0 and -5 dB]). Continuous recordings of pupil dilation measured processing effort, with larger, later dilations reflecting increased listening demand. Critical comparisons were between time-locked changes in eye data in response to unambiguous (i.e., continuum endpoints) tokens vs. ambiguous tokens (i.e., continuum midpoint). Unmasked speech elicited faster responses and sharper psychometric functions, which steadily declined in noise. Noise increased pupil dilation across stimulus conditions, but not straightforwardly. Noise-masked speech modulated peak pupil size (i.e., [0 and -5 dB] > unmasked). In contrast, peak dilation latency varied with both token and SNR. Interestingly, categorical tokens elicited earlier pupil dilation relative to ambiguous tokens. Our pupillary data suggest CP reconstructs auditory percepts under challenging listening conditions through interactions between stimulus salience and listeners' internalized effort and/or arousal.

From speech and talkers to the social world: The neural processing of human spoken language

Human speech perception is a paradigm example of the complexity of human linguistic processing; however, it is also the dominant way of expressing vocal identity and is critically important for social interactions. Here, I review the ways that the speech, the talker, and the social nature of speech interact and how this may be computed in the human brain, using models and approaches from nonhuman primate studies. I explore the extent to which domain-general approaches may be able to account for some of these neural findings. Finally, I address the importance of extending these findings into a better understanding of the social use of speech in conversations.

An fMRI-study on single-sided deafness: Spectral-temporal properties and side of stimulation modulates hemispheric dominance

Objective

Our main aim was to investigate the blood oxygenation level dependent (BOLD) response to monaural and binaural speech- and non-speech stimuli as measured with fMRI in subjects with single-sided deafness and in normal hearing controls. We hypothesised that the response to monaural stimulation in both normal hearing subjects and persons with single-sided deafness would vary with the complexity and nature of the stimuli and the side of stimulation.

Design

Patients with left- and right single-sided deafness and controls with normal hearing receiving either binaural or monaural stimuli were tested using speech and non-speech auditory stimuli in an event-related fMRI experiment.

Study sample

Twenty-two patients with single-sided deafness after treatment for vestibular schwannoma and 50 normal hearing controls.

Results

Normal hearing persons receiving right side monaural stimuli activate bilateral temporal regions. Activation following left side monaural stimulation is more right lateralized. Persons with single-sided deafness respond similarly to controls to monaural stimulation. Persons with right side single-sided deafness show activation of frontal cortical regions not seen in persons with left side single-sided deafness following speech stimuli. This is possibly related to increased effort and more frequently reported problems with communication. Right side single-sided deafness is related to increased activation of areas usually related to processing of degraded input, including the thalamus.

Conclusion

Hemispheric dominance following monaural auditory stimulation is modulated by the spectral-temporal properties of the stimuli and by which ear is stimulated. Differences between patients with right- and left side deafness suggests that right side deafness is related to increased activation of areas involved in processing of degraded input.

•

Hemispheric dominance following monaural stimulation is modulated by stimuli properties and by which ear is stimulated

•

Results suggests that right side deafness is related to increased activation of eares involved in processing degraded input

•

Self-assessed communication ability and the BOLD-response to phonetic stimuli correlate in several brain regions.

EEG rhythms lateralization patterns in children with unilateral hearing loss are different from the patterns of normal hearing controls during speech-in-noise listening

Unilateral hearing loss constitutes a field of growing interest in the scientific community. In fact, this kind of patients represent a unique and physiological way to investigate how neuroplasticity overcame unilateral deafferentation by implementing particular strategies that produce apparently next- to- normal hearing behavioural performances. This explains why such patients have been underinvestigated for a long time. Thanks to the availability of techniques able to study the cerebral activity underlying the mentioned behavioural outcomes, the aim of the present research was to elucidate whether different electroencephalographic (EEG) patterns occurred in unilateral hearing loss (UHL) children in comparison to normal hearing (NH) controls during speech-in-noise listening. Given the intrinsic lateralized nature of such patients, due to the unilateral side of hearing impairment, the experimental question was to assess whether this would reflect a different EEG pattern while performing a word in noise recognition task varying the direction of the noise source. Results showed a correlation between the period of deafness and the cortical activity asymmetry toward the hearing ear side in the frontal, parietal and occipital areas in all the experimental conditions. Concerning alpha and beta activity in the frontal and central areas highlighted that in the NH group, the lateralization was always left-sided during the Quiet condition, while it was right-sided in noise conditions; this evidence was not, however, detected also in the UHL group. In addition, focusing on the theta and alpha activity in the frontal areas (Broca area) during noise conditions, while the activity was always left-lateralized in the NH group, it was ipsilateral to the direction of the background noise in the UHL group, and of a weaker extent than in NH controls. Furthermore, in noise conditions, only the UHL group showed a higher theta activity in the temporal areas ipsilateral to the side where the background noise was directed to. Finally, in the case of bilateral noise (background noise and word signal both coming from the same two sources), the theta and alpha activity in the frontal areas (Broca area) was left-lateralized in the case of the NH group and lateralized towards the side of the better hearing ear in the case of the UHL group. Taken together, this evidence supports the establishment of a particular EEG pattern occurrence in UHL children taking place in the frontal (Broca area), temporal and parietal lobes, probably physiologically established in order to deal with different sound and noise source directions.

A Tutorial on Auditory Attention Identification Methods

Auditory attention identification methods attempt to identify the sound source of a listener's interest by analyzing measurements of electrophysiological data. We present a tutorial on the numerous techniques that have been developed in recent decades, and we present an overview of current trends in multivariate correlation-based and model-based learning frameworks. The focus is on the use of linear relations between electrophysiological and audio data. The way in which these relations are computed differs. For example, canonical correlation analysis (CCA) finds a linear subset of electrophysiological data that best correlates to audio data and a similar subset of audio data that best correlates to electrophysiological data. Model-based (encoding and decoding) approaches focus on either of these two sets. We investigate the similarities and differences between these linear model philosophies. We focus on (1) correlation-based approaches (CCA), (2) encoding/decoding models based on dense estimation, and (3) (adaptive) encoding/decoding models based on sparse estimation. The specific focus is on sparsity-driven adaptive encoding models and comparing the methodology in state-of-the-art models found in the auditory literature. Furthermore, we outline the main signal processing pipeline for how to identify the attended sound source in a cocktail party environment from the raw electrophysiological data with all the necessary steps, complemented with the necessary MATLAB code and the relevant references for each step. Our main aim is to compare the methodology of the available methods, and provide numerical illustrations to some of them to get a feeling for their potential. A thorough performance comparison is outside the scope of this tutorial.

Listening Effort: How the Cognitive Consequences of Acoustic Challenge Are Reflected in Brain and Behavior

Everyday conversation frequently includes challenges to the clarity of the acoustic speech signal, including hearing impairment, background noise, and foreign accents. Although an obvious problem is the increased risk of making word identification errors, extracting meaning from a degraded acoustic signal is also cognitively demanding, which contributes to increased listening effort. The concepts of cognitive demand and listening effort are critical in understanding the challenges listeners face in comprehension, which are not fully predicted by audiometric measures. In this article, the authors review converging behavioral, pupillometric, and neuroimaging evidence that understanding acoustically degraded speech requires additional cognitive support and that this cognitive load can interfere with other operations such as language processing and memory for what has been heard. Behaviorally, acoustic challenge is associated with increased errors in speech understanding, poorer performance on concurrent secondary tasks, more difficulty processing linguistically complex sentences, and reduced memory for verbal material. Measures of pupil dilation support the challenge associated with processing a degraded acoustic signal, indirectly reflecting an increase in neural activity. Finally, functional brain imaging reveals that the neural resources required to understand degraded speech extend beyond traditional perisylvian language networks, most commonly including regions of prefrontal cortex, premotor cortex, and the cingulo-opercular network. Far from being exclusively an auditory problem, acoustic degradation presents listeners with a systems-level challenge that requires the allocation of executive cognitive resources. An important point is that a number of dissociable processes can be engaged to understand degraded speech, including verbal working memory and attention-based performance monitoring. The specific resources required likely differ as a function of the acoustic, linguistic, and cognitive demands of the task, as well as individual differences in listeners' abilities. A greater appreciation of cognitive contributions to processing degraded speech is critical in understanding individual differences in comprehension ability, variability in the efficacy of assistive devices, and guiding rehabilitation approaches to reducing listening effort and facilitating communication.

Perceptual grouping in the cocktail party: Contributions of voice-feature continuity

Cocktail parties pose a difficult yet solvable problem for the auditory system. Previous work has shown that the cocktail-party problem is considerably easier when all sounds in the target stream are spoken by the same talker (the voice-continuity benefit). The present study investigated the contributions of two of the most salient voice features-glottal-pulse rate (GPR) and vocal-tract length (VTL)-to the voice-continuity benefit. Twenty young, normal-hearing listeners participated in two experiments. On each trial, listeners heard concurrent sequences of spoken digits from three different spatial locations and reported the digits coming from a target location. Critically, across conditions, GPR and VTL either remained constant or varied across target digits. Additionally, across experiments, the target location either remained constant (Experiment 1) or varied (Experiment 2) within a trial. In Experiment 1, listeners benefited from continuity in either voice feature, but VTL continuity was more helpful than GPR continuity. In Experiment 2, spatial discontinuity greatly hindered listeners' abilities to exploit continuity in GPR and VTL. The present results suggest that selective attention benefits from continuity in target voice features and that VTL and GPR play different roles for perceptual grouping and stream segregation in the cocktail party.

Sensorimotor Representation of Speech Perception. Cross-Decoding of Place of Articulation Features during Selective Attention to Syllables in 7T fMRI

Sensorimotor integration, the translation between acoustic signals and motoric programs, may constitute a crucial mechanism for speech. During speech perception, the acoustic-motoric translations include the recruitment of cortical areas for the representation of speech articulatory features, such as place of articulation. Selective attention can shape the processing and performance of speech perception tasks. Whether and where sensorimotor integration takes place during attentive speech perception remains to be explored. Here, we investigate articulatory feature representations of spoken consonant-vowel (CV) syllables during two distinct tasks. Fourteen healthy humans attended to either the vowel or the consonant within a syllable in separate delayed-match-to-sample tasks. Single-trial fMRI blood oxygenation level-dependent (BOLD) responses from perception periods were analyzed using multivariate pattern classification and a searchlight approach to reveal neural activation patterns sensitive to the processing of place of articulation (i.e., bilabial/labiodental vs. alveolar). To isolate place of articulation representation from acoustic covariation, we applied a cross-decoding (generalization) procedure across distinct features of manner of articulation (i.e., stop, fricative, and nasal). We found evidence for the representation of place of articulation across tasks and in both tasks separately: for attention to vowels, generalization maps included bilateral clusters of superior and posterior temporal, insular, and frontal regions; for attention to consonants, generalization maps encompassed clusters in temporoparietal, insular, and frontal regions within the right hemisphere only. Our results specify the cortical representation of place of articulation features generalized across manner of articulation during attentive syllable perception, thus supporting sensorimotor integration during attentive speech perception and demonstrating the value of generalization.

Listening under difficult conditions: An activation likelihood estimation meta-analysis

The brain networks supporting speech identification and comprehension under difficult listening conditions are not well specified. The networks hypothesized to underlie effortful listening include regions responsible for executive control. We conducted meta-analyses of auditory neuroimaging studies to determine whether a common activation pattern of the frontal lobe supports effortful listening under different speech manipulations. Fifty-three functional neuroimaging studies investigating speech perception were divided into three independent Activation Likelihood Estimate analyses based on the type of speech manipulation paradigm used: Speech-in-noise (SIN, 16 studies, involving 224 participants); spectrally degraded speech using filtering techniques (15 studies involving 270 participants); and linguistic complexity (i.e., levels of syntactic, lexical and semantic intricacy/density, 22 studies, involving 348 participants). Meta-analysis of the SIN studies revealed higher effort was associated with activation in left inferior frontal gyrus (IFG), left inferior parietal lobule, and right insula. Studies using spectrally degraded speech demonstrated increased activation of the insula bilaterally and the left superior temporal gyrus (STG). Studies manipulating linguistic complexity showed activation in the left IFG, right middle frontal gyrus, left middle temporal gyrus and bilateral STG. Planned contrasts revealed left IFG activation in linguistic complexity studies, which differed from activation patterns observed in SIN or spectral degradation studies. Although there were no significant overlap in prefrontal activation across these three speech manipulation paradigms, SIN and spectral degradation showed overlapping regions in left and right insula. These findings provide evidence that there is regional specialization within the left IFG and differential executive networks underlie effortful listening.

Listening in Naturalistic Scenes: What Can Functional Near-Infrared Spectroscopy and Intersubject Correlation Analysis Tell Us About the Underlying Brain Activity?

Listening to speech in the noisy conditions of everyday life can be effortful, reflecting the increased cognitive workload involved in extracting meaning from a degraded acoustic signal. Studying the underlying neural processes has the potential to provide mechanistic insight into why listening is effortful under certain conditions. In a move toward studying listening effort under ecologically relevant conditions, we used the silent and flexible neuroimaging technique functional near-infrared spectroscopy (fNIRS) to examine brain activity during attentive listening to speech in naturalistic scenes. Thirty normally hearing participants listened to a series of narratives continuously varying in acoustic difficulty while undergoing fNIRS imaging. Participants then listened to another set of closely matched narratives and rated perceived effort and intelligibility for each scene. As expected, self-reported effort generally increased with worsening signal-to-noise ratio. After controlling for better-ear signal-to-noise ratio, perceived effort was greater in scenes that contained competing speech than in those that did not, potentially reflecting an additional cognitive cost of overcoming informational masking. We analyzed the fNIRS data using intersubject correlation, a data-driven approach suitable for analyzing data collected under naturalistic conditions. Significant intersubject correlation was seen in the bilateral auditory cortices and in a range of channels across the prefrontal cortex. The involvement of prefrontal regions is consistent with the notion that higher order cognitive processes are engaged during attentive listening to speech in complex real-world conditions. However, further research is needed to elucidate the relationship between perceived listening effort and activity in these extended cortical networks.

Intelligibility of degraded speech and the relationship between symptoms of inattention, hyperactivity/impulsivity and language impairment in children with suspected auditory processing disorder

OBJECTIVES

To compare the sensitivity and specificity of Auditory Figure Ground sub-tests of the SCAN-3 battery, using signal to noise ratio (SNR) of +8 dB (AFG+8) and 0 dB (AFG0), in identifying auditory processing disorder (APD). A secondary objective was to evaluate any difference in auditory processing (AP) between children with symptoms of inattention versus combined sub-types of Attention Deficit Hyperactivity Disorder (ADHD).

METHODS

Data from 201 children, aged 6 to 16 years (mean: 10 years 6 months, SD: 2 years 8 months), who were assessed for suspected APD were reviewed retrospectively. The outcomes of the SCAN-3 APD test battery, Swanson Nolan and Pelham-IV parental rating (SNAP-IV) and Children's Communication Checklist-2 (CCC-2) were analysed.

RESULTS

AFG0 had a sensitivity of 56.3% and specificity of 100% in identifying children performing poorly in at least two of six SCAN-3 sub-tests or one of the two questionnaires, in contrast to 42.1% and 80% respectively for AFG+8. Impaired AP was mostly associated with symptoms of ADHD and /or language impairment (LI). LI was present in 92.9% of children with ADHD symptoms. Children with symptoms of combined ADHD plus LI performed significantly poorly (p < 0.05) compared to inattention ADHD plus LI in Filtered Words (FW) sub-test, but not in the rest of the SCAN-3 sub-tests.

CONCLUSION

Speech in noise tests using SNR of 0 dB is better than +8 dB in assessing APD. The better FW performance of the inattention ADHD plus LI group can be speculated to be related to known difference in activity in a neural network between different sub-types of ADHD. The findings of the study and existing literature suggest that neural networks connecting the cerebral hemispheres, basal ganglia and cerebellum are involved in APD, ADHD and LI.

How Auditory Experience Differentially Influences the Function of Left and Right Superior Temporal Cortices

To investigate how hearing status, sign language experience, and task demands influence functional responses in the human superior temporal cortices (STC) we collected fMRI data from deaf and hearing participants (male and female), who either acquired sign language early or late in life. Our stimuli in all tasks were pictures of objects. We varied the linguistic and visuospatial processing demands in three different tasks that involved decisions about (1) the sublexical (phonological) structure of the British Sign Language (BSL) signs for the objects, (2) the semantic category of the objects, and (3) the physical features of the objects.

Neuroimaging data revealed that in participants who were deaf from birth, STC showed increased activation during visual processing tasks. Importantly, this differed across hemispheres. Right STC was consistently activated regardless of the task whereas left STC was sensitive to task demands. Significant activation was detected in the left STC only for the BSL phonological task. This task, we argue, placed greater demands on visuospatial processing than the other two tasks. In hearing signers, enhanced activation was absent in both left and right STC during all three tasks. Lateralization analyses demonstrated that the effect of deafness was more task-dependent in the left than the right STC whereas it was more task-independent in the right than the left STC. These findings indicate how the absence of auditory input from birth leads to dissociable and altered functions of left and right STC in deaf participants.

SIGNIFICANCE STATEMENT Those born deaf can offer unique insights into neuroplasticity, in particular in regions of superior temporal cortex (STC) that primarily respond to auditory input in hearing people. Here we demonstrate that in those deaf from birth the left and the right STC have altered and dissociable functions. The right STC was activated regardless of demands on visual processing. In contrast, the left STC was sensitive to the demands of visuospatial processing. Furthermore, hearing signers, with the same sign language experience as the deaf participants, did not activate the STCs. Our data advance current understanding of neural plasticity by determining the differential effects that hearing status and task demands can have on left and right STC function.

Schizophrenia affects speech-induced functional connectivity of the superior temporal gyrus under cocktail-party listening conditions

The superior temporal gyrus (STG) is involved in speech recognition against informational masking under cocktail-party-listening conditions. Compared to healthy listeners, people with schizophrenia perform worse in speech recognition under informational speech-on-speech masking conditions. It is not clear whether the schizophrenia-related vulnerability to informational masking is associated with certain changes in FC of the STG with some critical brain regions. Using sparse-sampling fMRI design, this study investigated the differences between people with schizophrenia and healthy controls in FC of the STG for target-speech listening against informational speech-on-speech masking, when a listening condition with either perceived spatial separation (PSS, with a spatial release of informational masking) or perceived spatial co-location (PSC, without the spatial release) between target speech and masking speech was introduced. The results showed that in healthy participants, but not participants with schizophrenia, the contrast of either the PSS or PSC condition against the masker-only condition induced an enhancement of functional connectivity (FC) of the STG with the left superior parietal lobule and the right precuneus. Compared to healthy participants, participants with schizophrenia showed declined FC of the STG with the bilateral precuneus, right SPL, and right supplementary motor area. Thus, FC of the STG with the parietal areas is normally involved in speech listening against informational masking under either the PSS or PSC conditions, and declined FC of the STG in people with schizophrenia with the parietal areas may be associated with the increased vulnerability to informational masking.

Brain substrates underlying auditory speech priming in healthy listeners and listeners with schizophrenia

BACKGROUND

Under 'cocktail party' listening conditions, healthy listeners and listeners with schizophrenia can use temporally pre-presented auditory speech-priming (ASP) stimuli to improve target-speech recognition, even though listeners with schizophrenia are more vulnerable to informational speech masking.

METHOD

Using functional magnetic resonance imaging, this study searched for both brain substrates underlying the unmasking effect of ASP in 16 healthy controls and 22 patients with schizophrenia, and brain substrates underlying schizophrenia-related speech-recognition deficits under speech-masking conditions.

RESULTS

In both controls and patients, introducing the ASP condition (against the auditory non-speech-priming condition) not only activated the left superior temporal gyrus (STG) and left posterior middle temporal gyrus (pMTG), but also enhanced functional connectivity of the left STG/pMTG with the left caudate. It also enhanced functional connectivity of the left STG/pMTG with the left pars triangularis of the inferior frontal gyrus (TriIFG) in controls and that with the left Rolandic operculum in patients. The strength of functional connectivity between the left STG and left TriIFG was correlated with target-speech recognition under the speech-masking condition in both controls and patients, but reduced in patients.

CONCLUSIONS

The left STG/pMTG and their ASP-related functional connectivity with both the left caudate and some frontal regions (the left TriIFG in healthy listeners and the left Rolandic operculum in listeners with schizophrenia) are involved in the unmasking effect of ASP, possibly through facilitating the following processes: masker-signal inhibition, target-speech encoding, and speech production. The schizophrenia-related reduction of functional connectivity between the left STG and left TriIFG augments the vulnerability of speech recognition to speech masking.

Cortical Alpha Oscillations Predict Speech Intelligibility

Understanding speech in noise (SiN) is a complex task involving sensory encoding and cognitive resources including working memory and attention. Previous work has shown that brain oscillations, particularly alpha rhythms (8–12 Hz) play important roles in sensory processes involving working memory and attention. However, no previous study has examined brain oscillations during performance of a continuous speech perception test. The aim of this study was to measure cortical alpha during attentive listening in a commonly used SiN task (digits-in-noise, DiN) to better understand the neural processes associated with “top-down” cognitive processing in adverse listening environments. We recruited 14 normal hearing (NH) young adults. DiN speech reception threshold (SRT) was measured in an initial behavioral experiment. EEG activity was then collected: (i) while performing the DiN near SRT; and (ii) while attending to a silent, close-caption video during presentation of identical digit stimuli that the participant was instructed to ignore. Three main results were obtained: (1) during attentive (“active”) listening to the DiN, a number of distinct neural oscillations were observed (mainly alpha with some beta; 15–30 Hz). No oscillations were observed during attention to the video (“passive” listening); (2) overall, alpha event-related synchronization (ERS) of central/parietal sources were observed during active listening when data were grand averaged across all participants. In some participants, a smaller magnitude alpha event-related desynchronization (ERD), originating in temporal regions, was observed; and (3) when individual EEG trials were sorted according to correct and incorrect digit identification, the temporal alpha ERD was consistently greater on correctly identified trials. No such consistency was observed with the central/parietal alpha ERS. These data demonstrate that changes in alpha activity are specific to listening conditions. To our knowledge, this is the first report that shows almost no brain oscillatory changes during a passive task compared to an active task in any sensory modality. Temporal alpha ERD was related to correct digit identification.

Cognitive Compensation of Speech Perception With Hearing Impairment, Cochlear Implants, and Aging

External degradations in incoming speech reduce understanding, and hearing impairment further compounds the problem. While cognitive mechanisms alleviate some of the difficulties, their effectiveness may change with age. In our research, reviewed here, we investigated cognitive compensation with hearing impairment, cochlear implants, and aging, via (a) phonemic restoration as a measure of top-down filling of missing speech, (b) listening effort and response times as a measure of increased cognitive processing, and (c) visual world paradigm and eye gazing as a measure of the use of context and its time course. Our results indicate that between speech degradations and their cognitive compensation, there is a fine balance that seems to vary greatly across individuals. Hearing impairment or inadequate hearing device settings may limit compensation benefits. Cochlear implants seem to allow the effective use of sentential context, but likely at the cost of delayed processing. Linguistic and lexical knowledge, which play an important role in compensation, may be successfully employed in advanced age, as some compensatory mechanisms seem to be preserved. These findings indicate that cognitive compensation in hearing impairment can be highly complicated—not always absent, but also not easily predicted by speech intelligibility tests only.