Large-scale electrophysiology and deep learning reveal distorted neural signal dynamics after hearing loss

Listeners with hearing loss often struggle to understand speech in noise, even with a hearing aid. To better understand the auditory processing deficits that underlie this problem, we made large-scale brain recordings from gerbils, a common animal model for human hearing, while presenting a large database of speech and noise sounds. We first used manifold learning to identify the neural subspace in which speech is encoded and found that it is low-dimensional and that the dynamics within it are profoundly distorted by hearing loss. We then trained a deep neural network (DNN) to replicate the neural coding of speech with and without hearing loss and analyzed the underlying network dynamics. We found that hearing loss primarily impacts spectral processing, creating nonlinear distortions in cross-frequency interactions that result in a hypersensitivity to background noise that persists even after amplification with a hearing aid. Our results identify a new focus for efforts to design improved hearing aids and demonstrate the power of DNNs as a tool for the study of central brain structures.

Harnessing the Power of Artificial Intelligence in Otolaryngology and the Communication Sciences

Use of artificial intelligence (AI) is a burgeoning field in otolaryngology and the communication sciences. A virtual symposium on the topic was convened from Duke University on October 26, 2020, and was attended by more than 170 participants worldwide. This review presents summaries of all but one of the talks presented during the symposium; recordings of all the talks, along with the discussions for the talks, are available at https://www.youtube.com/watch?v=ktfewrXvEFg and https://www.youtube.com/watch?v=-gQ5qX2v3rg . Each of the summaries is about 2500 words in length and each summary includes two figures. This level of detail far exceeds the brief summaries presented in traditional reviews and thus provides a more-informed glimpse into the power and diversity of current AI applications in otolaryngology and the communication sciences and how to harness that power for future applications.

Why Do Hearing Aids Fail to Restore Normal Auditory Perception?

Hearing loss is a widespread condition that is linked to declines in quality of life and mental health. Hearing aids remain the treatment of choice, but, unfortunately, even state-of-the-art devices provide only limited benefit for the perception of speech in noisy environments. While traditionally viewed primarily as a loss of sensitivity, hearing loss is also known to cause complex distortions of sound-evoked neural activity that cannot be corrected by amplification alone. This Opinion article describes the effects of hearing loss on neural activity to illustrate the reasons why current hearing aids are insufficient and to motivate the use of new technologies to explore directions for improving the next generation of devices.

Midbrain adaptation may set the stage for the perception of musical beat

The ability to spontaneously feel a beat in music is a phenomenon widely believed to be unique to humans. Though beat perception involves the coordinated engagement of sensory, motor and cognitive processes in humans, the contribution of low-level auditory processing to the activation of these networks in a beat-specific manner is poorly understood. Here, we present evidence from a rodent model that midbrain preprocessing of sounds may already be shaping where the beat is ultimately felt. For the tested set of musical rhythms, on-beat sounds on average evoked higher firing rates than off-beat sounds, and this difference was a defining feature of the set of beat interpretations most commonly perceived by human listeners over others. Basic firing rate adaptation provided a sufficient explanation for these results. Our findings suggest that midbrain adaptation, by encoding the temporal context of sounds, creates points of neural emphasis that may influence the perceptual emergence of a beat.

Stimulus relevance modulates contrast adaptation in visual cortex

A general principle of sensory processing is that neurons adapt to sustained stimuli by reducing their response over time. Most of our knowledge on adaptation in single cells is based on experiments in anesthetized animals. How responses adapt in awake animals, when stimuli may be behaviorally relevant or not, remains unclear. Here we show that contrast adaptation in mouse primary visual cortex depends on the behavioral relevance of the stimulus. Cells that adapted to contrast under anesthesia maintained or even increased their activity in awake naïve mice. When engaged in a visually guided task, contrast adaptation re-occurred for stimuli that were irrelevant for solving the task. However, contrast adaptation was reversed when stimuli acquired behavioral relevance. Regulation of cortical adaptation by task demand may allow dynamic control of sensory-evoked signal flow in the neocortex.

DOI:http://dx.doi.org/10.7554/eLife.21589.001

Inhibitory control of correlated intrinsic variability in cortical networks

Cortical networks exhibit intrinsic dynamics that drive coordinated, large-scale fluctuations across neuronal populations and create noise correlations that impact sensory coding. To investigate the network-level mechanisms that underlie these dynamics, we developed novel computational techniques to fit a deterministic spiking network model directly to multi-neuron recordings from different rodent species, sensory modalities, and behavioral states. The model generated correlated variability without external noise and accurately reproduced the diverse activity patterns in our recordings. Analysis of the model parameters suggested that differences in noise correlations across recordings were due primarily to differences in the strength of feedback inhibition. Further analysis of our recordings confirmed that putative inhibitory neurons were indeed more active during desynchronized cortical states with weak noise correlations. Our results demonstrate that network models with intrinsically-generated variability can accurately reproduce the activity patterns observed in multi-neuron recordings and suggest that inhibition modulates the interactions between intrinsic dynamics and sensory inputs to control the strength of noise correlations.

DOI:http://dx.doi.org/10.7554/eLife.19695.001

Our brains contain billions of neurons, which are continually producing electrical signals to relay information around the brain. Yet most of our knowledge of how the brain works comes from studying the activity of one neuron at a time. Recently, studies of multiple neurons have shown that they tend to be active together in short bursts called “up” states, which are followed by periods in which they are less active called “down” states. When we are sleeping or under a general anesthetic, the neurons may be completely silent during down states, but when we are awake the difference in activity between the two states is usually less extreme. However, it is still not clear how the neurons generate these patterns of activity.

To address this question, Stringer et al. studied the activity of neurons in the brains of awake and anesthetized rats, mice and gerbils. The experiments recorded electrical activity from many neurons at the same time and found a wide range of different activity patterns. A computational model based on these data suggests that differences in the degree to which some neurons suppress the activity of other neurons may account for this variety. Increasing the strength of these inhibitory signals in the model decreased the fluctuations in electrical activity across entire areas of the brain. Further analysis of the experimental data supported the model’s predictions by showing that inhibitory neurons – which act to reduce electrical activity in other neurons – were more active when there were fewer fluctuations in activity across the brain.

The next step following on from this work would be to develop ways to build computer models that can mimic the activity of many more neurons at the same time. The models could then be used to interpret the electrical activity produced by many different kinds of neuron. This will enable researchers to test more sophisticated hypotheses about how the brain works.

DOI:http://dx.doi.org/10.7554/eLife.19695.002

Periodotopy in the gerbil inferior colliculus: local clustering rather than a gradient map

Periodicities in sound waveforms are widespread, and shape important perceptual attributes of sound including rhythm and pitch. Previous studies have indicated that, in the inferior colliculus (IC), a key processing stage in the auditory midbrain, neurons tuned to different periodicities might be arranged along a periodotopic axis which runs approximately orthogonal to the tonotopic axis. Here we map out the topography of frequency and periodicity tuning in the IC of gerbils in unprecedented detail, using pure tones and different periodic sounds, including click trains, sinusoidally amplitude modulated (SAM) noise and iterated rippled noise. We found that while the tonotopic map exhibited a clear and highly reproducible gradient across all animals, periodotopic maps varied greatly across different types of periodic sound and from animal to animal. Furthermore, periodotopic gradients typically explained only about 10% of the variance in modulation tuning between recording sites. However, there was a strong local clustering of periodicity tuning at a spatial scale of ca. 0.5 mm, which also differed from animal to animal.

Analysis and modelling of variability and covariability of population spike trains across multiple time scales

As multi-electrode and imaging technology begin to provide us with simultaneous recordings of large neuronal populations, new methods for modelling such data must also be developed. We present a model of responses to repeated trials of a sensory stimulus based on thresholded Gaussian processes that allows for analysis and modelling of variability and covariability of population spike trains across multiple time scales. The model framework can be used to specify the values of many different variability measures including spike timing precision across trials, coefficient of variation of the interspike interval distribution, and Fano factor of spike counts for individual neurons, as well as signal and noise correlations and correlations of spike counts across multiple neurons. Using both simulated data and data from different stages of the mammalian auditory pathway, we demonstrate the range of possible independent manipulations of different variability measures, and explore how this range depends on the sensory stimulus. The model provides a powerful framework for the study of experimental and surrogate data and for analyzing dependencies between different statistical properties of neuronal populations.

Incremental mutual information: a new method for characterizing the strength and dynamics of connections in neuronal circuits

Understanding the computations performed by neuronal circuits requires characterizing the strength and dynamics of the connections between individual neurons. This characterization is typically achieved by measuring the correlation in the activity of two neurons. We have developed a new measure for studying connectivity in neuronal circuits based on information theory, the incremental mutual information (IMI). By conditioning out the temporal dependencies in the responses of individual neurons before measuring the dependency between them, IMI improves on standard correlation-based measures in several important ways: 1) it has the potential to disambiguate statistical dependencies that reflect the connection between neurons from those caused by other sources (e.g. shared inputs or intrinsic cellular or network mechanisms) provided that the dependencies have appropriate timescales, 2) for the study of early sensory systems, it does not require responses to repeated trials of identical stimulation, and 3) it does not assume that the connection between neurons is linear. We describe the theory and implementation of IMI in detail and demonstrate its utility on experimental recordings from the primate visual system.

Modeling Population Spike Trains with Specified Time-Varying Spike Rates, Trial-to-Trial Variability, and Pairwise Signal and Noise Correlations

As multi-electrode and imaging technology begin to provide us with simultaneous recordings of large neuronal populations, new methods for modeling such data must also be developed. Here, we present a model for the type of data commonly recorded in early sensory pathways: responses to repeated trials of a sensory stimulus in which each neuron has it own time-varying spike rate (as described by its PSTH) and the dependencies between cells are characterized by both signal and noise correlations. This model is an extension of previous attempts to model population spike trains designed to control only the total correlation between cells. In our model, the response of each cell is represented as a binary vector given by the dichotomized sum of a deterministic “signal” that is repeated on each trial and a Gaussian random “noise” that is different on each trial. This model allows the simulation of population spike trains with PSTHs, trial-to-trial variability, and pairwise correlations that match those measured experimentally. Furthermore, the model also allows the noise correlations in the spike trains to be manipulated independently of the signal correlations and single-cell properties. To demonstrate the utility of the model, we use it to simulate and manipulate experimental responses from the mammalian auditory and visual systems. We also present a general form of the model in which both the signal and noise are Gaussian random processes, allowing the mean spike rate, trial-to-trial variability, and pairwise signal and noise correlations to be specified independently. Together, these methods for modeling spike trains comprise a potentially powerful set of tools for both theorists and experimentalists studying population responses in sensory systems.

Enhancement of ITD Coding Within the Initial Stages of the Auditory Pathway

Sensory systems use a variety of strategies to increase the signal-to-noise ratio in their inputs at the receptor level. However, important cues for sound localization are not present at the individual ears but are computed after inputs from the two ears converge within the brain, and we hypothesized that additional strategies to enhance the representation of these cues might be employed in the initial stages after binaural convergence. Specifically, we investigated the transformation that takes place between the first two stages of the gerbil auditory pathway that are sensitive to differences in the arrival time of a sound at the two ears (interaural time differences; ITDs): the medial superior olive (MSO), where ITD tuning originates, and the dorsal nucleus of the lateral lemniscus (DNLL), to which the MSO sends direct projections. We use a combined experimental and computational approach to demonstrate that the coding of ITDs is dramatically enhanced between these two stages, with the mutual information in the responses of single neurons increasing by a factor of 2. We also show that this enhancement is related to an increase in dynamic range for neurons with high preferred frequencies and a decrease in variability for neurons with low preferred frequencies. These results suggest that a major role of the initial stages of the ITD pathway may be to enhance the representation created at the site of coincidence detection and illustrate the potential of this pathway as a model system for the study of strategies for enhancing sensory representations in the mammalian brain.

Timing precision in population coding of natural scenes in the early visual system

The timing of spiking activity across neurons is a fundamental aspect of the neural population code. Individual neurons in the retina, thalamus, and cortex can have very precise and repeatable responses but exhibit degraded temporal precision in response to suboptimal stimuli. To investigate the functional implications for neural populations in natural conditions, we recorded in vivo the simultaneous responses, to movies of natural scenes, of multiple thalamic neurons likely converging to a common neuronal target in primary visual cortex. We show that the response of individual neurons is less precise at lower contrast, but that spike timing precision across neurons is relatively insensitive to global changes in visual contrast. Overall, spike timing precision within and across cells is on the order of 10 ms. Since closely timed spikes are more efficient in inducing a spike in downstream cortical neurons, and since fine temporal precision is necessary to represent the more slowly varying natural environment, we argue that preserving relative spike timing at a ∼10-ms resolution is a crucial property of the neural code entering cortex.

Neurons convey information about the world in the form of trains of action potentials (spikes). These trains are highly repeatable when the same stimulus is presented multiple times, and this temporal precision across repetitions can be as fine as a few milliseconds. It is usually assumed that this time scale also corresponds to the timing precision of several neighboring neurons firing in concert. However, the relative timing of spikes emitted by different neurons in a local population is not necessarily as fine as the temporal precision across repetitions within a single neuron. In the visual system of the brain, the level of contrast in the image entering the retina can affect single-neuron temporal precision, but the effects of contrast on the neural population code are unknown. Here we show that the temporal scale of the population code entering visual cortex is on the order of 10 ms and is largely insensitive to changes in visual contrast. Since closely timed spikes are more efficient in inducing a spike in downstream cortical neurons, and since fine temporal precision is necessary in representing the more slowly varying natural environment, preserving relative spike timing at a ∼10-ms resolution may be a crucial property of the neural code entering cortex.

Early neural representation of visual scenes occurs with a temporal precision on the order of 10 ms, which is precise enough to strongly drive downstream neurons in the visual pathway. Unlike individual neurons, the neural population code is largely insensitive to pronounced changes in visual contrast.

The representation of amplitude modulations in the mammalian auditory midbrain

Temporal modulations in stimulus amplitude are essential for recognizing and categorizing behaviorally relevant acoustic signals such as speech. Despite this behavioral importance, it remains unclear how amplitude modulations (AMs) are represented in the responses of neurons at higher levels of the auditory system. Studies using stimuli with sinusoidal amplitude modulations (SAMs) have shown that the responses of many neurons are strongly tuned to modulation frequency, leading to the hypothesis that AMs are represented by their periodicity in the auditory midbrain. However, AMs in general are defined not only by their modulation frequency, but also by a number of other parameters (duration, duty cycle, etc.), which covary with modulation frequency in SAM stimuli. Thus the relationship between modulation frequency and neural responses as characterized with SAM stimuli alone is ambiguous. In this study, we characterize the representation of AMs in the gerbil inferior colliculus by analyzing neural responses to a series of pulse trains in which duration and interpulse interval are systematically varied to quantify the importance of duration, interpulse interval, duty cycle, and modulation frequency independently. We find that, although modulation frequency is indeed an important parameter for some neurons, the responses of many neurons are also strongly influenced by other AM parameters, typically duration and duty cycle. These results suggest that AMs are represented in the auditory midbrain not only by their periodicity, but by a complex combination of several important parameters.

Dynamic spectrotemporal feature selectivity in the auditory midbrain

The transformation of auditory information from the cochlea to the cortex is a highly nonlinear process. Studies using tone stimuli have revealed that changes in even the most basic parameters of the auditory stimulus can alter neural response properties; for example, a change in stimulus intensity can cause a shift in a neuron's preferred frequency. However, it is not yet clear how such nonlinearities contribute to the processing of spectrotemporal features in complex sounds. Here, we use spectrotemporal receptive fields (STRFs) to characterize the effects of stimulus intensity on feature selectivity in the mammalian inferior colliculus (IC). At low intensities, we find that STRFs are relatively simple, typically consisting of a single excitatory region, indicating that the neural response is simply a reflection of the stimulus amplitude at the preferred frequency. In contrast, we find that STRFs at high intensities typically consist of a combination of an excitatory region and one or more inhibitory regions, often in a spectrotemporally inseparable arrangement, indicating selectivity for complex auditory features. We show that a linear-nonlinear model with the appropriate STRF can predict neural responses to stimuli with a fixed intensity, and we demonstrate that a simple extension of the model with an intensity-dependent STRF can predict responses to stimuli with varying intensity. These results illustrate the complexity of auditory feature selectivity in the IC, but also provide encouraging evidence that the prediction of nonlinear responses to complex stimuli is a tractable problem.

Efficient temporal processing of naturalistic sounds

In this study, we investigate the ability of the mammalian auditory pathway to adapt its strategy for temporal processing under natural stimulus conditions. We derive temporal receptive fields from the responses of neurons in the inferior colliculus to vocalization stimuli with and without additional ambient noise. We find that the onset of ambient noise evokes a change in receptive field dynamics that corresponds to a change from bandpass to lowpass temporal filtering. We show that these changes occur within a few hundred milliseconds of the onset of the noise and are evident across a range of overall stimulus intensities. Using a simple model, we illustrate how these changes in temporal processing exploit differences in the statistical properties of vocalizations and ambient noises to increase the information in the neural response in a manner consistent with the principles of efficient coding.

Adaptation to stimulus contrast and correlations during natural visual stimulation

In this study, we characterize the adaptation of neurons in the cat lateral geniculate nucleus to changes in stimulus contrast and correlations. By comparing responses to high- and low-contrast natural scene movie and white noise stimuli, we show that an increase in contrast or correlations results in receptive fields with faster temporal dynamics and stronger antagonistic surrounds, as well as decreases in gain and selectivity. We also observe contrast- and correlation-induced changes in the reliability and sparseness of neural responses. We find that reliability is determined primarily by processing in the receptive field (the effective contrast of the stimulus), while sparseness is determined by the interactions between several functional properties. These results reveal a number of adaptive phenomena and suggest that adaptation to stimulus contrast and correlations may play an important role in visual coding in a dynamic natural environment.

Temporal precision in the neural code and the timescales of natural vision

The timing of action potentials relative to sensory stimuli can be precise down to milliseconds in the visual system, even though the relevant timescales of natural vision are much slower. The existence of such precision contributes to a fundamental debate over the basis of the neural code and, specifically, what timescales are important for neural computation. Using recordings in the lateral geniculate nucleus, here we demonstrate that the relevant timescale of neuronal spike trains depends on the frequency content of the visual stimulus, and that 'relative', not absolute, precision is maintained both during spatially uniform white-noise visual stimuli and naturalistic movies. Using information-theoretic techniques, we demonstrate a clear role of relative precision, and show that the experimentally observed temporal structure in the neuronal response is necessary to represent accurately the more slowly changing visual world. By establishing a functional role of precision, we link visual neuron function on slow timescales to temporal structure in the response at faster timescales, and uncover a straightforward purpose of fine-timescale features of neuronal spike trains.

Dynamic encoding of natural luminance sequences by LGN bursts

In the lateral geniculate nucleus (LGN) of the thalamus, visual stimulation produces two distinct types of responses known as tonic and burst. Due to the dynamics of the T-type Ca ²⁺ channels involved in burst generation, the type of response evoked by a particular stimulus depends on the resting membrane potential, which is controlled by a network of modulatory connections from other brain areas. In this study, we use simulated responses to natural scene movies to describe how modulatory and stimulus-driven changes in LGN membrane potential interact to determine the luminance sequences that trigger burst responses. We find that at low resting potentials, when the T channels are de-inactivated and bursts are relatively frequent, an excitatory stimulus transient alone is sufficient to evoke a burst. However, to evoke a burst at high resting potentials, when the T channels are inactivated and bursts are relatively rare, prolonged inhibitory stimulation followed by an excitatory transient is required. We also observe evidence of these effects in vivo, where analysis of experimental recordings demonstrates that the luminance sequences that trigger bursts can vary dramatically with the overall burst percentage of the response. To characterize the functional consequences of the effects of resting potential on burst generation, we simulate LGN responses to different luminance sequences at a range of resting potentials with and without a mechanism for generating bursts. Using analysis based on signal detection theory, we show that bursts enhance detection of specific luminance sequences, ranging from the onset of excitatory sequences at low resting potentials to the offset of inhibitory sequences at high resting potentials. These results suggest a dynamic role for burst responses during visual processing that may change according to behavioral state.

This visual neuroscience paper simulates how resting potential and stimulus driven modulations in membrane potential interact to determine the response mode of LGN neurons to natural images.

Decoupling functional mechanisms of adaptive encoding

In a natural setting, adaptive mechanisms constantly modulate the encoding properties of sensory neurons in response to changes in the external environment. Recent experiments have revealed that adaptation affects both the spatiotemporal integration properties and baseline membrane potential of sensory neurons. However, the precise functional role of adaptation remains an open question, due in part to contradictory experimental results. Here, we develop a framework to characterize adaptive encoding, including a cascade model with a time-varying receptive field (reflecting spatiotemporal integration properties) and offset (reflecting baseline membrane potential), and a recursive technique for tracking changes in the model parameters during a single stimulus/response trial. Simulated and experimental responses from retinal neurons are used to track adaptive changes in receptive field structure and offset during nonstationary stimulation. Due to the nonlinear nature of spiking neurons, the parameters of the receptive field and offset must be estimated simultaneously, or changes in the offset (or even in the statistical distribution of the stimulus) can mask, confound, or create the illusion of adaptive changes in the receptive field. Our analysis suggests that these confounding effects may be at the root of the inconsistency in the literature and shows that seemingly conflicting experimental results can be reconciled within our framework.

Improved tracking of time-varying encoding properties of visual neurons by extended recursive least-squares

Traditional approaches to characterizing the transformation from stimulus to response in sensory systems assume both stationarity of the stimulus and time-invariance of the stimulus/response mapping. However, recent studies of sensory function under natural stimulus conditions have demonstrated important features of neural encoding that are in violation of these assumptions. Many sensory neurons respond to changes in the statistical distribution of the stimulus that are characteristic of the natural environment with corresponding changes in their encoding properties. In this paper, an extended recursive least-squares (ERLS) approach to adaptive estimation from stimulus/response observations is detailed. The ERLS approach improves the tracking ability of standard RLS approaches to adaptive estimation by removing a number of assumptions about the underlying system and the stimulus environment. The ERLS framework also incorporates an adaptive learning rate, so that prior knowledge of the relationship between the stimulus and the adaptive nature of the system under investigation can be used to improve tracking performance. Simulated and experimental neural responses are used to demonstrate the ability of the ERLS approach to track adaptation of encoding properties during a single stimulus/response trial. The ERLS framework lends tremendous flexibility to experimental design, facilitating the investigation of sensory function under naturalistic stimulus conditions.

Encoding of natural scene movies by tonic and burst spikes in the lateral geniculate nucleus

The role of the lateral geniculate nucleus (LGN) of the thalamus in visual encoding remains an open question. Here, we characterize the function of tonic and burst spikes in cat LGN X-cells in signaling features of natural stimuli. A significant increase in bursting was observed during natural stimulation (relative to white noise stimulation) and was linked to the strong correlation structure of the natural scene movies. Burst responses were triggered by specific stimulus events consisting of a prolonged inhibitory stimulus, followed by an excitatory stimulus, such as the movement of an object into the receptive field. LGN responses to natural scene movies were predicted using an integrate-and-fire (IF) framework and compared with experimentally observed responses. The standard IF model successfully predicted LGN responses to natural scene movies during tonic firing, indicating a linear relationship between stimulus and response. However, the IF model typically underpredicted the LGN response during periods of bursting, indicating a nonlinear amplification of the stimulus in the actual response. The addition of a burst mechanism to the IF model was necessary to accurately predict the entire LGN response. These results suggest that LGN bursts are an important part of the neural code, providing a nonlinear amplification of stimulus features that are typical of the natural environment.

Adaptive encoding in the visual pathway

In a natural setting, the mean luminance and contrast of the light within a visual neuron's receptive field are constantly changing as the eyes saccade across complex scenes. Adaptive mechanisms modulate filtering properties of the early visual pathway in response to these variations, allowing the system to maintain differential sensitivity to nonstationary stimuli. An adaptive variant of the reverse correlation technique is used to characterize these changes during single trials. Properties of the adaptive reverse correlation algorithm were investigated via simulation. Analysis of data collected from the mammalian visual system demonstrates the ability to continuously track adaptive changes in the encoding scheme. The adaptive estimation approach provides a framework for characterizing the role of adaptation in natural scene viewing.