Predictive Ensemble Decoding of Acoustical Features Explains Context-Dependent Receptive Fields

Abstract representations in temporal cortex support generative linguistic processing

Generativity, the ability to create and evaluate novel constructions, is a fundamental property of human language and cognition. The productivity of generative processes is determined by the scope of the representations they engage. Here we examine the neural representation of reduplication, a productive phonological process that can create novel forms through patterned syllable copying (e.g. ba-mih → ba-ba-mih, ba-mih-mih, or ba-mih-ba). Using MRI-constrained source estimates of combined MEG/EEG data collected during an auditory artificial grammar task, we identified localized cortical activity associated with syllable reduplication pattern contrasts in novel trisyllabic nonwords. Neural decoding analyses identified a set of predominantly right hemisphere temporal lobe regions whose activity reliably discriminated reduplication patterns evoked by untrained, novel stimuli. Effective connectivity analyses suggested that sensitivity to abstracted reduplication patterns was propagated between these temporal regions. These results suggest that localized temporal lobe activity patterns function as abstract representations that support linguistic generativity.

[Neural mechanism for modulation of auditory response of the striatum by locomotion]

OBJECTIVE

To explore whether the characteristic responses to sound stimulations of the auditory neurons in the striatum is regulated in different behavioral states.

METHODS

The auditory neurons in the striatum of awake C57BL/6J mice were selected for this study. We recorded the auditory response of the striatum to noises over a long period of time by building a synchronous in vivo electrophysiological and locomotion recording system and using glass microelectrode attachment recording. By analyzing the running speed of the mice, the behavioral states of the mice were divided into the quiet state and the active state, and the spontaneous activity and evoked responses of the auditory neurons in the striatum were analyzed in these two states.

RESULTS

Compared with those recorded in the quiet state, the spontaneous activity of the auditory neurons in the striatum of the mice increased significantly (37.06±12.02 vs 18.51±10.91, P < 0.001) while the auditory response of the neurons decreased significantly (noise intensity=60 dB, 3.45±2.99 vs 3.04±2.76, P < 0.001) in the active state.

CONCLUSION

Locomotion has a significant inhibitory effect on the auditory response of the striatum, which may importantly contribute to the decline of sound information recognition ability in the active state.

[Evolution of auditory response signal-to-noise ratio in ascending auditory pathways]

OBJECTIVE

To explore the characteristics of the evolution of auditory response signal-to-noise ratio at all levels of the ascending auditory pathway, its modulation by different brain states in different brain regions, and its potential value as an effective indicator for encoding sound characteristics.

METHODS

Eighty C57BL/6J awake mice were used for recording the best frequency auditory response of the neurons in the inferior colliculus (n=20), medial geniculate body (n=20), and primary auditory cortex using a glass microelectrode. The probability density of spontaneous and evoked firing of the neurons was calculated to establish a distribution model of spontaneous and evoked firing, and the evolution of the auditory response signal-to-noise ratio was statistically analyzed. The changes in spontaneous and evoked firing of the neurons and the auditory response signal-to-noise ratio in different brain regions were analyzed at rest and during running.

RESULTS

In different brain regions in the ascending auditory pathway, the spontaneous firing of the neurons all showed a Poisson distribution, and the evoked firing showed a lognormal distribution. The auditory response signal-to-noise ratio was significantly greater in the inferior colliculus than in the medial geniculate body and auditory cortex (P < 0.001). The auditory response signal-to-noise ratio in the 3 brain regions remained stable irrespective of the states of motion (P>0.05).

CONCLUSION

Auditory response signal-to-noise ratio may serve as an effective indicator of encoding sound characteristics.

Cortical voice processing is grounded in elementary sound analyses for vocalization relevant sound patterns

A subregion of the auditory cortex (AC) was proposed to selectively process voices. This selectivity of the temporal voice area (TVA) and its role in processing non-voice sounds however have remained elusive. For a better functional description of the TVA, we investigated its neural responses both to voice and non-voice sounds, and critically also to textural sound patterns (TSPs) that share basic features with natural sounds but that are perceptually very distant from voices. Listening to these TSPs, first, elicited activity in large subregions of the TVA, which was mainly driven by perpetual ratings of TSPs along a voice similarity scale. This similar TVA activity in response to TSPs might partially explain activation patterns typically observed during voice processing. Second, we reconstructed the TVA activity that is usually observed in voice processing with a linear combination of activation patterns from TSPs. An analysis of the reconstruction model weights demonstrated that the TVA similarly processes both natural voice and non-voice sounds as well as TSPs along their acoustic and perceptual features. The predominant factor in reconstructing the TVA pattern by TSPs were the perceptual voice similarity ratings. Third, a multi-voxel pattern analysis confirms that the TSPs contain sufficient sound information to explain TVA activity for voice processing. Altogether, rather than being restricted to higher-order voice processing only, the human "voice area" uses mechanisms to evaluate the perceptual and acoustic quality of non-voice sounds, and responds to the latter with a "voice-like" processing pattern when detecting some rudimentary perceptual similarity with voices.

Observation and assessment of acoustic contamination of electrophysiological brain signals during speech production and sound perception

OBJECTIVE

A current challenge of neurotechnologies is to develop speech brain-computer interfaces aiming at restoring communication in people unable to speak. To achieve a proof of concept of such system, neural activity of patients implanted for clinical reasons can be recorded while they speak. Using such simultaneously recorded audio and neural data, decoders can be built to predict speech features using features extracted from brain signals. A typical neural feature is the spectral power of field potentials in the high-gamma frequency band, which happens to overlap the frequency range of speech acoustic signals, especially the fundamental frequency of the voice. Here, we analyzed human electrocorticographic and intracortical recordings during speech production and perception as well as a rat microelectrocorticographic recording during sound perception. We observed that several datasets, recorded with different recording setups, contained spectrotemporal features highly correlated with those of the sound produced by or delivered to the participants, especially within the high-gamma band and above, strongly suggesting a contamination of electrophysiological recordings by the sound signal. This study investigated the presence of acoustic contamination and its possible source.

APPROACH

We developed analysis methods and a statistical criterion to objectively assess the presence or absence of contamination-specific correlations, which we used to screen several datasets from five centers worldwide.

MAIN RESULTS

Not all but several datasets, recorded in a variety of conditions, showed significant evidence of acoustic contamination. Three out of five centers were concerned by the phenomenon. In a recording showing high contamination, the use of high-gamma band features dramatically facilitated the performance of linear decoding of acoustic speech features, while such improvement was very limited for another recording showing no significant contamination. Further analysis and in vitro replication suggest that the contamination is caused by the mechanical action of the sound waves onto the cables and connectors along the recording chain, transforming sound vibrations into an undesired electrical noise affecting the biopotential measurements.

SIGNIFICANCE

Although this study does not per se question the presence of speech-relevant physiological information in the high-gamma range and above (multiunit activity), it alerts on the fact that acoustic contamination of neural signals should be proofed and eliminated before investigating the cortical dynamics of these processes. To this end, we make available a toolbox implementing the proposed statistical approach to quickly assess the extent of contamination in an electrophysiological recording (https://doi.org/10.5281/zenodo.3929296).

The Encoding of Speech Sounds in the Superior Temporal Gyrus

The human superior temporal gyrus (STG) is critical for extracting meaningful linguistic features from speech input. Local neural populations are tuned to acoustic-phonetic features of all consonants and vowels and to dynamic cues for intonational pitch. These populations are embedded throughout broader functional zones that are sensitive to amplitude-based temporal cues. Beyond speech features, STG representations are strongly modulated by learned knowledge and perceptual goals. Currently, a major challenge is to understand how these features are integrated across space and time in the brain during natural speech comprehension. We present a theory that temporally recurrent connections within STG generate context-dependent phonological representations, spanning longer temporal sequences relevant for coherent percepts of syllables, words, and phrases.

STRFs in primary auditory cortex emerge from masking-based statistics of natural sounds

We investigate how the neural processing in auditory cortex is shaped by the statistics of natural sounds. Hypothesising that auditory cortex (A1) represents the structural primitives out of which sounds are composed, we employ a statistical model to extract such components. The input to the model are cochleagrams which approximate the non-linear transformations a sound undergoes from the outer ear, through the cochlea to the auditory nerve. Cochleagram components do not superimpose linearly, but rather according to a rule which can be approximated using the max function. This is a consequence of the compression inherent in the cochleagram and the sparsity of natural sounds. Furthermore, cochleagrams do not have negative values. Cochleagrams are therefore not matched well by the assumptions of standard linear approaches such as sparse coding or ICA. We therefore consider a new encoding approach for natural sounds, which combines a model of early auditory processing with maximal causes analysis (MCA), a sparse coding model which captures both the non-linear combination rule and non-negativity of the data. An efficient truncated EM algorithm is used to fit the MCA model to cochleagram data. We characterize the generative fields (GFs) inferred by MCA with respect to in vivo neural responses in A1 by applying reverse correlation to estimate spectro-temporal receptive fields (STRFs) implied by the learned GFs. Despite the GFs being non-negative, the STRF estimates are found to contain both positive and negative subfields, where the negative subfields can be attributed to explaining away effects as captured by the applied inference method. A direct comparison with ferret A1 shows many similar forms, and the spectral and temporal modulation tuning of both ferret and model STRFs show similar ranges over the population. In summary, our model represents an alternative to linear approaches for biological auditory encoding while it captures salient data properties and links inhibitory subfields to explaining away effects.

The information carried by natural sounds enters the cortex of mammals in a specific format: the cochleagram. Instead of representing the original pressure waveforms, the inner ear represents how the energy in a sound is distributed across frequency bands and how the energy distribution evolves over time. The generation of cochleagrams is highly non-linear resulting in the dominance of one sound source per time-frequency bin under natural conditions (masking). Auditory cortex is believed to decompose cochleagrams into structural primitives, i.e., reappearing regular spectro-temporal subpatterns that make up cochleagram patterns (similar to edges in images). However, such a decomposition has so far only been modeled without considering masking and non-negativity. Here we apply a novel non-linear sparse coding model that can capture masking non-linearities and non-negativities. When trained on cochleagrams of natural sounds, the model gives rise to an encoding primarily based-on spectro-temporally localized components. If stimulated by a sound, the encoding units compete to explain its contents. The competition is a direct consequence of the statistical sound model, and it results in neural responses being best described by spectro-temporal receptive fields (STRFs) with positive and negative subfields. The emerging STRFs show a higher similarity to experimentally measured STRFs than a model without masking, which provides evidence for cortical encoding being consistent with the masking based sound statistics of cochleagrams. Furthermore, and more generally, our study suggests for the first time that negative subfields of STRFs may be direct evidence for explaining away effects resulting from performing inference in an underlying statistical model.

Recent advances in understanding the auditory cortex

Our ability to make sense of the auditory world results from neural processing that begins in the ear, goes through multiple subcortical areas, and continues in the cortex. The specific contribution of the auditory cortex to this chain of processing is far from understood. Although many of the properties of neurons in the auditory cortex resemble those of subcortical neurons, they show somewhat more complex selectivity for sound features, which is likely to be important for the analysis of natural sounds, such as speech, in real-life listening conditions. Furthermore, recent work has shown that auditory cortical processing is highly context-dependent, integrates auditory inputs with other sensory and motor signals, depends on experience, and is shaped by cognitive demands, such as attention. Thus, in addition to being the locus for more complex sound selectivity, the auditory cortex is increasingly understood to be an integral part of the network of brain regions responsible for prediction, auditory perceptual decision-making, and learning. In this review, we focus on three key areas that are contributing to this understanding: the sound features that are preferentially represented by cortical neurons, the spatial organization of those preferences, and the cognitive roles of the auditory cortex.

Go/No-Go task engagement enhances population representation of target stimuli in primary auditory cortex

Primary sensory cortices are classically considered to extract and represent stimulus features, while association and higher-order areas are thought to carry information about stimulus meaning. Here we show that this information can in fact be found in the neuronal population code of the primary auditory cortex (A1). A1 activity was recorded in awake ferrets while they either passively listened or actively discriminated stimuli in a range of Go/No-Go paradigms, with different sounds and reinforcements. Population-level dimensionality reduction techniques reveal that task engagement induces a shift in stimulus encoding from a sensory to a behaviorally driven representation that specifically enhances the target stimulus in all paradigms. This shift partly relies on task-engagement-induced changes in spontaneous activity. Altogether, we show that A1 population activity bears strong similarities to frontal cortex responses. These findings indicate that primary sensory cortices implement a crucial change in the structure of population activity to extract task-relevant information during behavior.

Sensory areas are thought to process stimulus information while higher-order processing occurs in association cortices. Here the authors report that during task engagement population activity in ferret primary auditory cortex shifts away from encoding stimulus features toward detection of the behaviourally relevant targets.

Phonemes: Lexical access and beyond

Phonemes play a central role in traditional theories as units of speech perception and access codes to lexical representations. Phonemes have two essential properties: they are 'segment-sized' (the size of a consonant or vowel) and abstract (a single phoneme may be have different acoustic realisations). Nevertheless, there is a long history of challenging the phoneme hypothesis, with some theorists arguing for differently sized phonological units (e.g. features or syllables) and others rejecting abstract codes in favour of representations that encode detailed acoustic properties of the stimulus. The phoneme hypothesis is the minority view today. We defend the phoneme hypothesis in two complementary ways. First, we show that rejection of phonemes is based on a flawed interpretation of empirical findings. For example, it is commonly argued that the failure to find acoustic invariances for phonemes rules out phonemes. However, the lack of invariance is only a problem on the assumption that speech perception is a bottom-up process. If learned sublexical codes are modified by top-down constraints (which they are), then this argument loses all force. Second, we provide strong positive evidence for phonemes on the basis of linguistic data. Almost all findings that are taken (incorrectly) as evidence against phonemes are based on psycholinguistic studies of single words. However, phonemes were first introduced in linguistics, and the best evidence for phonemes comes from linguistic analyses of complex word forms and sentences. In short, the rejection of phonemes is based on a false analysis and a too-narrow consideration of the relevant data.