Auditory perceptual decision-making based on semantic categorization of environmental sounds

Longstanding Auditory Sensory and Semantic Differences in Preterm Born Children

More than 10% of births are preterm, and the long-term consequences on sensory and semantic processing of non-linguistic information remain poorly understood. 17 very preterm-born children (born at < 33 weeks gestational age) and 15 full-term controls were tested at 10 years old with an auditory object recognition task, while 64-channel auditory evoked potentials (AEPs) were recorded. Sounds consisted of living (animal and human vocalizations) and manmade objects (e.g. household objects, instruments, and tools). Despite similar recognition behavior, AEPs strikingly differed between full-term and preterm children. Starting at 50ms post-stimulus onset, AEPs from preterm children differed topographically from their full-term counterparts. Over the 108-224ms post-stimulus period, full-term children showed stronger AEPs in response to living objects, whereas preterm born children showed the reverse pattern; i.e. stronger AEPs in response to manmade objects. Differential brain activity between semantic categories could reliably classify children according to their preterm status. Moreover, this opposing pattern of differential responses to semantic categories of sounds was also observed in source estimations within a network of occipital, temporal and frontal regions. This study highlights how early life experience in terms of preterm birth shapes sensory and object processing later on in life.

Real-Time Audio Processing of Real-Life Soundscapes for EEG Analysis: ERPs Based on Natural Sound Onsets

With smartphone-based mobile electroencephalography (EEG), we can investigate sound perception beyond the lab. To understand sound perception in the real world, we need to relate naturally occurring sounds to EEG data. For this, EEG and audio information need to be synchronized precisely, only then it is possible to capture fast and transient evoked neural responses and relate them to individual sounds. We have developed Android applications (AFEx and Record-a) that allow for the concurrent acquisition of EEG data and audio features, i.e., sound onsets, average signal power (RMS), and power spectral density (PSD) on smartphone. In this paper, we evaluate these apps by computing event-related potentials (ERPs) evoked by everyday sounds. One participant listened to piano notes (played live by a pianist) and to a home-office soundscape. Timing tests showed a stable lag and a small jitter (< 3 ms) indicating a high temporal precision of the system. We calculated ERPs to sound onsets and observed the typical P1-N1-P2 complex of auditory processing. Furthermore, we show how to relate information on loudness (RMS) and spectra (PSD) to brain activity. In future studies, we can use this system to study sound processing in everyday life.

Neural Decoding Reveals Concurrent Phonemic and Subphonemic Representations of Speech Across Tasks

Robust and efficient speech perception relies on the interpretation of acoustically variable phoneme realizations, yet prior neuroimaging studies are inconclusive regarding the degree to which subphonemic detail is maintained over time as categorical representations arise. It is also unknown whether this depends on the demands of the listening task. We addressed these questions by using neural decoding to quantify the (dis)similarity of brain response patterns evoked during two different tasks. We recorded magnetoencephalography (MEG) as adult participants heard isolated, randomized tokens from a /ba/-/da/ speech continuum. In the passive task, their attention was diverted. In the active task, they categorized each token as ba or da. We found that linear classifiers successfully decoded ba vs. da perception from the MEG data. Data from the left hemisphere were sufficient to decode the percept early in the trial, while the right hemisphere was necessary but not sufficient for decoding at later time points. We also decoded stimulus representations and found that they were maintained longer in the active task than in the passive task; however, these representations did not pattern more like discrete phonemes when an active categorical response was required. Instead, in both tasks, early phonemic patterns gave way to a representation of stimulus ambiguity that coincided in time with reliable percept decoding. Our results suggest that the categorization process does not require the loss of subphonemic detail, and that the neural representation of isolated speech sounds includes concurrent phonemic and subphonemic information.

Neurocognitive dynamics of near-threshold voice signal detection and affective voice evaluation

Communication and voice signal detection in noisy environments are universal tasks for many species. The fundamental problem of detecting voice signals in noise (VIN) is underinvestigated especially in its temporal dynamic properties. We investigated VIN as a dynamic signal-to-noise ratio (SNR) problem to determine the neurocognitive dynamics of subthreshold evidence accrual and near-threshold voice signal detection. Experiment 1 showed that dynamic VIN, including a varying SNR and subthreshold sensory evidence accrual, is superior to similar conditions with nondynamic SNRs or with acoustically matched sounds. Furthermore, voice signals with affective meaning have a detection advantage during VIN. Experiment 2 demonstrated that VIN is driven by an effective neural integration in an auditory cortical-limbic network at and beyond the near-threshold detection point, which is preceded by activity in subcortical auditory nuclei. This demonstrates the superior recognition advantage of communication signals in dynamic noise contexts, especially when carrying socio-affective meaning.

Selective attention to sound features mediates cross-modal activation of visual cortices

Contemporary schemas of brain organization now include multisensory processes both in low-level cortices as well as at early stages of stimulus processing. Evidence has also accumulated showing that unisensory stimulus processing can result in cross-modal effects. For example, task-irrelevant and lateralised sounds can activate visual cortices; a phenomenon referred to as the auditory-evoked contralateral occipital positivity (ACOP). Some claim this is an example of automatic attentional capture in visual cortices. Other results, however, indicate that context may play a determinant role. Here, we investigated whether selective attention to spatial features of sounds is a determining factor in eliciting the ACOP. We recorded high-density auditory evoked potentials (AEPs) while participants selectively attended and discriminated sounds according to four possible stimulus attributes: location, pitch, speaker identity or syllable. Sound acoustics were held constant, and their location was always equiprobable (50% left, 50% right). The only manipulation was to which sound dimension participants attended. We analysed the AEP data from healthy participants within an electrical neuroimaging framework. The presence of sound-elicited activations of visual cortices depended on the to-be-discriminated, goal-based dimension. The ACOP was elicited only when participants were required to discriminate sound location, but not when they attended to any of the non-spatial features. These results provide a further indication that the ACOP is not automatic. Moreover, our findings showcase the interplay between task-relevance and spatial (un)predictability in determining the presence of the cross-modal activation of visual cortices.

Spatial attention affects the early processing of neutral versus fearful faces when they are task-irrelevant: a classifier study of the EEG C1 component

EEG studies suggest that the emotional content of visual stimuli is processed rapidly. In particular, the C1 component, which occurs up to 100 ms after stimulus onset and likely reflects activity in primary visual cortex V1, has been reported to be sensitive to emotional faces. However, difficulties replicating these results have been reported. We hypothesized that the nature of the task and attentional condition are key to reconcile the conflicting findings. We report three experiments of EEG activity during the C1 time range elicited by peripherally presented neutral and fearful faces under various attentional conditions: the faces were spatially attended or unattended and were either task-relevant or not. Using traditional event-related potential analysis, we found that the early activity changed depending on facial expression, attentional condition, and task. In addition, we trained classifiers to discriminate the different conditions from the EEG signals. Although the classifiers were not able to discriminate between facial expressions in any condition, they uncovered differences between spatially attended and unattended faces but solely when these were task-irrelevant. In addition, this effect was only present for neutral faces. Our study provides further indication that attention and task are key parameters when measuring early differences between emotional and neutral visual stimuli.

Auditory Categorization of Man-Made Sounds Versus Natural Sounds by Means of MEG Functional Brain Connectivity

Previous neuroimaging studies have shown that sounds can be discriminated due to living-related or man-made-related characteristics and involve different brain regions. However, these studies have mainly provided source space analyses, which offer simple maps of activated brain regions but do not explain how regions of a distributed system are functionally organized under a specific task. In the present study, we aimed to further examine the functional connectivity of the auditory processing pathway across different categories of non-speech sounds in healthy adults, by means of MEG. Our analyses demonstrated significant activation and interconnection differences between living and man-made object sounds, in the prefrontal areas, anterior-superior temporal gyrus (aSTG), posterior cingulate cortex (PCC), and supramarginal gyrus (SMG), occurring within 80–120 ms post-stimulus interval. Current findings replicated previous ones, in that other regions beyond the auditory cortex are involved during auditory processing. According to the functional connectivity analysis, differential brain networks across the categories exist, which proposes that sound category discrimination processing relies on distinct cortical networks, a notion that has been strongly argued in the literature also in relation to the visual system.

Categorization of everyday sounds by cochlear implanted children

Auditory categorization is an important process in the perception and understanding of everyday sounds. The use of cochlear implants (CIs) may affect auditory categorization and result in poor abilities. The current study was designed to compare how children with normal hearing (NH) and children with CIs categorize a set of everyday sounds. We tested 24 NH children and 24 children with CI on a free-sorting task of 18 everyday sounds corresponding to four a priori categories: nonlinguistic human vocalizations, environmental sounds, musical sounds, and animal vocalizations. Multiple correspondence analysis revealed considerable variation within both groups of child listeners, although the human vocalizations and musical sounds were similarly categorized. In contrast to NH children, children with CIs categorized some sounds according to their acoustic content rather than their associated semantic information. These results show that despite identification deficits, children with CIs are able to categorize environmental and vocal sounds in a similar way to NH children, and are able to use categorization as an adaptive process when dealing with everyday sounds.

Somatosensory and auditory deviance detection for outcome prediction during postanoxic coma

Objective

Prominent research in patients with disorders of consciousness investigated the electrophysiological correlates of auditory deviance detection as a marker of consciousness recovery. Here, we extend previous studies by investigating whether somatosensory deviance detection provides an added value for outcome prediction in postanoxic comatose patients.

Methods

Electroencephalography responses to frequent and rare stimuli were obtained from 66 patients on the first and second day after coma onset.

Results

Multivariate decoding analysis revealed an above chance‐level auditory discrimination in 25 patients on the first day and in 31 patients on the second day. Tactile discrimination was significant in 16 patients on the first day and in 23 patients on the second day. Single‐day sensory discrimination was unrelated to patients’ outcome in both modalities. However, improvement of auditory discrimination from first to the second day was predictive of good outcome with a positive predictive power (PPV) of 0.73 (CI = 0.52–0.88). Analyses considering the improvement of tactile, auditory and tactile, or either auditory or tactile discrimination showed no significant prediction of good outcome (PPVs = 0.58–0.68).

Interpretation

Our results show that in the acute phase of coma deviance detection is largely preserved for both auditory and tactile modalities. However, we found no evidence for an added value of somatosensory to auditory deviance detection function for coma‐outcome prediction.

Categorization of common sounds by cochlear implanted and normal hearing adults

Auditory categorization involves grouping of acoustic events along one or more shared perceptual dimensions which can relate to both semantic and physical attributes. This process involves both high level cognitive processes (categorization) and low-level perceptual encoding of the acoustic signal, both of which are affected by the use of a cochlear implant (CI) device. The goal of this study was twofold: I) compare the categorization strategies of CI users and normal hearing listeners (NHL) II) investigate if any characteristics of the raw acoustic signal could explain the results. 16 experienced CI users and 20 NHL were tested using a Free-Sorting Task of 16 common sounds divided into 3 predefined categories of environmental, musical and vocal sounds. Multiple Correspondence Analysis (MCA) and Hierarchical Clustering based on Principal Components (HCPC) show that CI users followed a similar categorization strategy to that of NHL and were able to discriminate between the three different types of sounds. However results for CI users were more varied and showed less inter-participant agreement. Acoustic analysis also highlighted the average pitch salience and average autocorrelation peak as being important for the perception and categorization of the sounds. The results therefore show that on a broad level of categorization CI users may not have as many difficulties as previously thought in discriminating certain kinds of sound; however the perception of individual sounds remains challenging.

Experience-based Auditory Predictions Modulate Brain Activity to Silence as do Real Sounds

Interactions between stimuli's acoustic features and experience-based internal models of the environment enable listeners to compensate for the disruptions in auditory streams that are regularly encountered in noisy environments. However, whether auditory gaps are filled in predictively or restored a posteriori remains unclear. The current lack of positive statistical evidence that internal models can actually shape brain activity as would real sounds precludes accepting predictive accounts of filling-in phenomenon. We investigated the neurophysiological effects of internal models by testing whether single-trial electrophysiological responses to omitted sounds in a rule-based sequence of tones with varying pitch could be decoded from the responses to real sounds and by analyzing the ERPs to the omissions with data-driven electrical neuroimaging methods. The decoding of the brain responses to different expected, but omitted, tones in both passive and active listening conditions was above chance based on the responses to the real sound in active listening conditions. Topographic ERP analyses and electrical source estimations revealed that, in the absence of any stimulation, experience-based internal models elicit an electrophysiological activity different from noise and that the temporal dynamics of this activity depend on attention. We further found that the expected change in pitch direction of omitted tones modulated the activity of left posterior temporal areas 140-200 msec after the onset of omissions. Collectively, our results indicate that, even in the absence of any stimulation, internal models modulate brain activity as do real sounds, indicating that auditory filling in can be accounted for by predictive activity.

Representation of Sound Objects within Early-Stage Auditory Areas: A Repetition Effect Study Using 7T fMRI

Environmental sounds are highly complex stimuli whose recognition depends on the interaction of top-down and bottom-up processes in the brain. Their semantic representations were shown to yield repetition suppression effects, i. e. a decrease in activity during exposure to a sound that is perceived as belonging to the same source as a preceding sound. Making use of the high spatial resolution of 7T fMRI we have investigated the representations of sound objects within early-stage auditory areas on the supratemporal plane. The primary auditory cortex was identified by means of tonotopic mapping and the non-primary areas by comparison with previous histological studies. Repeated presentations of different exemplars of the same sound source, as compared to the presentation of different sound sources, yielded significant repetition suppression effects within a subset of early-stage areas. This effect was found within the right hemisphere in primary areas A1 and R as well as two non-primary areas on the antero-medial part of the planum temporale, and within the left hemisphere in A1 and a non-primary area on the medial part of Heschl’s gyrus. Thus, several, but not all early-stage auditory areas encode the meaning of environmental sounds.

Neural detection of complex sound sequences in the absence of consciousness

The neural response to a violation of sequences of identical sounds is a typical example of the brain's sensitivity to auditory regularities. Previous literature interprets this effect as a pre-attentive and unconscious processing of sensory stimuli. By contrast, a violation to auditory global regularities, i.e. based on repeating groups of sounds, is typically detectable when subjects can consciously perceive them. Here, we challenge the notion that global detection implies consciousness by testing the neural response to global violations in a group of 24 patients with post-anoxic coma (three females, age range 45-87 years), treated with mild therapeutic hypothermia and sedation. By applying a decoding analysis to electroencephalographic responses to standard versus deviant sound sequences, we found above-chance decoding performance in 10 of 24 patients (Wilcoxon signed-rank test, P < 0.001), despite five of them being mildly hypothermic, sedated and unarousable. Furthermore, consistently with previous findings based on the mismatch negativity the progression of this decoding performance was informative of patients' chances of awakening (78% predictive of awakening). Our results show for the first time that detection of global regularities at neural level exists despite a deeply unconscious state.

Decoding auditory EEG responses in healthy and clinical populations: A comparative study

BACKGROUND

Analyses of brain responses to external stimuli are typically based on the means computed across conditions. However in many cognitive and clinical applications, taking into account their variability across trials has turned out to be statistically more sensitive than comparing their means.

NEW METHOD

In this study we present a novel implementation of a single-trial topographic analysis (STTA) for discriminating auditory evoked potentials at predefined time-windows. This analysis has been previously introduced for extracting spatio-temporal features at the level of the whole neural response. Adapting the STTA on specific time windows is an essential step for comparing its performance to other time-window based algorithms.

RESULTS

We analyzed responses to standard vs. deviant sounds and showed that the new implementation of the STTA gives above-chance decoding results in all subjects (in comparison to 7 out of 11 with the original method). In comatose patients, the improvement of the decoding performance was even more pronounced than in healthy controls and doubled the number of significant results.

COMPARISON WITH EXISTING METHOD(S)

We compared the results obtained with the new STTA to those based on a logistic regression in healthy controls and patients. We showed that the first of these two comparisons provided a better performance of the logistic regression; however only the new STTA provided significant results in comatose patients at group level.

CONCLUSIONS

Our results provide quantitative evidence that a systematic investigation of the accuracy of established methods in normal and clinical population is an essential step for optimizing decoding performance.

The speed and accuracy of perceptual decisions in a random-tone pitch task

Research in perceptual decision making is dominated by paradigms that tap the visual system, such as the random-dot motion (RDM) paradigm. In this study, we investigated whether the behavioral signature of perceptual decisions in the auditory domain is similar to those observed in the visual domain. We developed an auditory version of the RDM task, in which tones correspond to dots and pitch corresponds to motion (the random-tone pitch task, RTP). In this task, participants have to decide quickly whether the pitch of a “sound cloud” of tones is moving up or down. Stimulus strength and speed–accuracy trade-off were manipulated. To describe the relationship between stimulus strength and performance, we fitted the proportional-rate diffusion model to the data. The results showed a close coupling between stimulus strength and the speed and accuracy of perceptual decisions in both tasks. Additionally, we fitted the full drift diffusion model (DDM) to the data and showed that three of the four participants had similar speed–accuracy trade-offs in both tasks. However, for the RTP task, drift rates were larger and nondecision times slower, suggesting that some DDM parameters might be dependent on stimulus modality (drift rate and nondecision time), whereas others might not be (decision bound). The results illustrate that the RTP task is suitable for investigating the dynamics of auditory perceptual choices. Future studies using the task might help to investigate modality-specific effects on decision making at both the behavioral and neuronal levels.

Robust discrimination between EEG responses to categories of environmental sounds in early coma

Humans can recognize categories of environmental sounds, including vocalizations produced by humans and animals and the sounds of man-made objects. Most neuroimaging investigations of environmental sound discrimination have studied subjects while consciously perceiving and often explicitly recognizing the stimuli. Consequently, it remains unclear to what extent auditory object processing occurs independently of task demands and consciousness. Studies in animal models have shown that environmental sound discrimination at a neural level persists even in anesthetized preparations, whereas data from anesthetized humans has thus far provided null results. Here, we studied comatose patients as a model of environmental sound discrimination capacities during unconsciousness. We included 19 comatose patients treated with therapeutic hypothermia (TH) during the first 2 days of coma, while recording nineteen-channel electroencephalography (EEG). At the level of each individual patient, we applied a decoding algorithm to quantify the differential EEG responses to human vs. animal vocalizations as well as to sounds of living vocalizations vs. man-made objects. Discrimination between vocalization types was accurate in 11 patients and discrimination between sounds from living and man-made sources in 10 patients. At the group level, the results were significant only for the comparison between vocalization types. These results lay the groundwork for disentangling truly preferential activations in response to auditory categories, and the contribution of awareness to auditory category discrimination.

Progression of auditory discrimination based on neural decoding predicts awakening from coma

Auditory evoked potentials are informative of intact cortical functions of comatose patients. The integrity of auditory functions evaluated using mismatch negativity paradigms has been associated with their chances of survival. However, because auditory discrimination is assessed at various delays after coma onset, it is still unclear whether this impairment depends on the time of the recording. We hypothesized that impairment in auditory discrimination capabilities is indicative of coma progression, rather than of the comatose state itself and that rudimentary auditory discrimination remains intact during acute stages of coma. We studied 30 post-anoxic comatose patients resuscitated from cardiac arrest and five healthy, age-matched controls. Using a mismatch negativity paradigm, we performed two electroencephalography recordings with a standard 19-channel clinical montage: the first within 24 h after coma onset and under mild therapeutic hypothermia, and the second after 1 day and under normothermic conditions. We analysed electroencephalography responses based on a multivariate decoding algorithm that automatically quantifies neural discrimination at the single patient level. Results showed high average decoding accuracy in discriminating sounds both for control subjects and comatose patients. Importantly, accurate decoding was largely independent of patients' chance of survival. However, the progression of auditory discrimination between the first and second recordings was informative of a patient's chance of survival. A deterioration of auditory discrimination was observed in all non-survivors (equivalent to 100% positive predictive value for survivors). We show, for the first time, evidence of intact auditory processing even in comatose patients who do not survive and that progression of sound discrimination over time is informative of a patient's chance of survival. Tracking auditory discrimination in comatose patients could provide new insight to the chance of awakening in a quantitative and automatic fashion during early stages of coma.