How the brain uses time to represent and process visual information(1)

Multi-timescale neural dynamics for multisensory integration

Carrying out any everyday task, be it driving in traffic, conversing with friends or playing basketball, requires rapid selection, integration and segregation of stimuli from different sensory modalities. At present, even the most advanced artificial intelligence-based systems are unable to replicate the multisensory processes that the human brain routinely performs, but how neural circuits in the brain carry out these processes is still not well understood. In this Perspective, we discuss recent findings that shed fresh light on the oscillatory neural mechanisms that mediate multisensory integration (MI), including power modulations, phase resetting, phase-amplitude coupling and dynamic functional connectivity. We then consider studies that also suggest multi-timescale dynamics in intrinsic ongoing neural activity and during stimulus-driven bottom-up and cognitive top-down neural network processing in the context of MI. We propose a new concept of MI that emphasizes the critical role of neural dynamics at multiple timescales within and across brain networks, enabling the simultaneous integration, segregation, hierarchical structuring and selection of information in different time windows. To highlight predictions from our multi-timescale concept of MI, real-world scenarios in which multi-timescale processes may coordinate MI in a flexible and adaptive manner are considered.

Toward an interventional science of recovery after coma

Recovery of consciousness after coma remains one of the most challenging areas for accurate diagnosis and effective therapeutic engagement in the clinical neurosciences. Recovery depends on preservation of neuronal integrity and evolving changes in network function that re-establish environmental responsiveness. It typically occurs in defined steps: it begins with eye opening and unresponsiveness in a vegetative state, then limited recovery of responsiveness characterizes the minimally conscious state, and this is followed by recovery of reliable communication. This review considers several points for novel interventions, for example, in persons with cognitive motor dissociation in whom a hidden cognitive reserve is revealed. Circuit mechanisms underlying restoration of behavioral responsiveness and communication are discussed. An emerging theme is the possibility to rescue latent capacities in partially damaged human networks across time. These opportunities should be exploited for therapeutic engagement to achieve individualized solutions for restoration of communication and environmental interaction across varying levels of recovery.

Evidence for Multiscale Multiplexed Representation of Visual Features in EEG

Distinct neural processes such as sensory and memory processes are often encoded over distinct timescales of neural activations. Animal studies have shown that this multiscale coding strategy is also implemented for individual components of a single process, such as individual features of a multifeature stimulus in sensory coding. However, the generalizability of this encoding strategy to the human brain has remained unclear. We asked if individual features of visual stimuli were encoded over distinct timescales. We applied a multiscale time-resolved decoding method to electroencephalography (EEG) collected from human subjects presented with grating visual stimuli to estimate the timescale of individual stimulus features. We observed that the orientation and color of the stimuli were encoded in shorter timescales, whereas spatial frequency and the contrast of the same stimuli were encoded in longer timescales. The stimulus features appeared in temporally overlapping windows along the trial supporting a multiplexed coding strategy. These results provide evidence for a multiplexed, multiscale coding strategy in the human visual system.

Mesocircuit mechanisms in the diagnosis and treatment of disorders of consciousness

The 'mesocircuit hypothesis' proposes mechanisms underlying the recovery of consciousness following severe brain injuries. The model builds up from a single premise that multifocal brain injuries resulting in coma and subsequent disorders of consciousness produce widespread neuronal death and dysfunction. Considering the general properties of cortical, thalamic, and striatal neurons, a lawful and specific circuit-level mechanism is constructed based on these known anatomical and physiological specializations of neuronal subtypes. The mesocircuit model generates many testable predictions at the mesocircuit, local circuit, and cellular level across multiple cerebral structures to correlate diagnostic measurements and interpret therapeutic interventions. The anterior forebrain mesocircuit is integrally related to the frontal-parietal network, another network demonstrated to show strong correlation with levels of recovery in disorders of consciousness. A further extension known as the "ABCD" model has been used to examine interaction of these models in recovery of consciousness using electrophysiological data types. Many studies have examined predictions of the mesocircuit model; here we first present the model and review the accumulated evidence for several predictions of model across multiple stages of recovery function in human subjects. Recent studies linking the mesocircuit model, the ABCD model, and interactions with the frontoparietal network are reviewed. Finally, theoretical implications of the mesocircuit model at the neuronal level are considered to interpret recent studies of deep brain stimulation in the central lateral thalamus in patients recovering from coma and in new experimental models in the context of emerging understanding of neuronal and local circuit mechanisms underlying conscious brain states.

Computational methods to study information processing in neural circuits

The brain is an information processing machine and thus naturally lends itself to be studied using computational tools based on the principles of information theory. For this reason, computational methods based on or inspired by information theory have been a cornerstone of practical and conceptual progress in neuroscience. In this Review, we address how concepts and computational tools related to information theory are spurring the development of principled theories of information processing in neural circuits and the development of influential mathematical methods for the analyses of neural population recordings. We review how these computational approaches reveal mechanisms of essential functions performed by neural circuits. These functions include efficiently encoding sensory information and facilitating the transmission of information to downstream brain areas to inform and guide behavior. Finally, we discuss how further progress and insights can be achieved, in particular by studying how competing requirements of neural encoding and readout may be optimally traded off to optimize neural information processing.

When the Whole Is Less Than the Sum of Its Parts: Maximum Object Category Information and Behavioral Prediction in Multiscale Activation Patterns

Neural codes are reflected in complex neural activation patterns. Conventional electroencephalography (EEG) decoding analyses summarize activations by averaging/down-sampling signals within the analysis window. This diminishes informative fine-grained patterns. While previous studies have proposed distinct statistical features capable of capturing variability-dependent neural codes, it has been suggested that the brain could use a combination of encoding protocols not reflected in any one mathematical feature alone. To check, we combined 30 features using state-of-the-art supervised and unsupervised feature selection procedures (n = 17). Across three datasets, we compared decoding of visual object category between these 17 sets of combined features, and between combined and individual features. Object category could be robustly decoded using the combined features from all of the 17 algorithms. However, the combination of features, which were equalized in dimension to the individual features, were outperformed across most of the time points by the multiscale feature of Wavelet coefficients. Moreover, the Wavelet coefficients also explained the behavioral performance more accurately than the combined features. These results suggest that a single but multiscale encoding protocol may capture the EEG neural codes better than any combination of protocols. Our findings put new constraints on the models of neural information encoding in EEG.

PV+ Cells Enhance Temporal Population Codes but not Stimulus-Related Timing in Auditory Cortex

Spatio-temporal cortical activity patterns relative to both peripheral input and local network activity carry information about stimulus identity and context. GABAergic interneurons are reported to regulate spiking at millisecond precision in response to sensory stimulation and during gamma oscillations; their role in regulating spike timing during induced network bursts is unclear. We investigated this issue in murine auditory thalamo-cortical (TC) brain slices, in which TC afferents induced network bursts similar to previous reports in vivo. Spike timing relative to TC afferent stimulation during bursts was poor in pyramidal cells and SOM+ interneurons. It was more precise in PV+ interneurons, consistent with their reported contribution to spiking precision in pyramidal cells. Optogenetic suppression of PV+ cells unexpectedly improved afferent-locked spike timing in pyramidal cells. In contrast, our evidence suggests that PV+ cells do regulate the spatio-temporal spike pattern of pyramidal cells during network bursts, whose organization is suited to ensemble coding of stimulus information. Simulations showed that suppressing PV+ cells reduces the capacity of pyramidal cell networks to produce discriminable spike patterns. By dissociating temporal precision with respect to a stimulus versus internal cortical activity, we identified a novel role for GABAergic cells in regulating information processing in cortical networks.

Multiscale dynamics of interstimulus interval integration in visual cortex

Although the visual cortex receives information at multiple temporal patterns, much of the research in the field has focused only on intervals shorter than 1 second. Consequently, there is almost no information on what happens at longer temporal intervals. We have tried to address this question recording neuronal populations of the primary visual cortex during visual stimulation with repetitive grating stimuli and intervals ranging from 1 to 7 seconds. Our results showed that firing rate and response stability were dependent of interval duration. In addition, there were collective oscillations with different properties in response to changes in intervals duration. These results suggest that visual cortex could encode visual information at several time scales using oscillations at multiple frequencies.

Cracking the Neural Code for Sensory Perception by Combining Statistics, Intervention, and Behavior

The two basic processes underlying perceptual decisions-how neural responses encode stimuli, and how they inform behavioral choices-have mainly been studied separately. Thus, although many spatiotemporal features of neural population activity, or "neural codes," have been shown to carry sensory information, it is often unknown whether the brain uses these features for perception. To address this issue, we propose a new framework centered on redefining the neural code as the neural features that carry sensory information used by the animal to drive appropriate behavior; that is, the features that have an intersection between sensory and choice information. We show how this framework leads to a new statistical analysis of neural activity recorded during behavior that can identify such neural codes, and we discuss how to combine intersection-based analysis of neural recordings with intervention on neural activity to determine definitively whether specific neural activity features are involved in a task.

Using Matrix and Tensor Factorizations for the Single-Trial Analysis of Population Spike Trains

Advances in neuronal recording techniques are leading to ever larger numbers of simultaneously monitored neurons. This poses the important analytical challenge of how to capture compactly all sensory information that neural population codes carry in their spatial dimension (differences in stimulus tuning across neurons at different locations), in their temporal dimension (temporal neural response variations), or in their combination (temporally coordinated neural population firing). Here we investigate the utility of tensor factorizations of population spike trains along space and time. These factorizations decompose a dataset of single-trial population spike trains into spatial firing patterns (combinations of neurons firing together), temporal firing patterns (temporal activation of these groups of neurons) and trial-dependent activation coefficients (strength of recruitment of such neural patterns on each trial). We validated various factorization methods on simulated data and on populations of ganglion cells simultaneously recorded in the salamander retina. We found that single-trial tensor space-by-time decompositions provided low-dimensional data-robust representations of spike trains that capture efficiently both their spatial and temporal information about sensory stimuli. Tensor decompositions with orthogonality constraints were the most efficient in extracting sensory information, whereas non-negative tensor decompositions worked well even on non-independent and overlapping spike patterns, and retrieved informative firing patterns expressed by the same population in response to novel stimuli. Our method showed that populations of retinal ganglion cells carried information in their spike timing on the ten-milliseconds-scale about spatial details of natural images. This information could not be recovered from the spike counts of these cells. First-spike latencies carried the majority of information provided by the whole spike train about fine-scale image features, and supplied almost as much information about coarse natural image features as firing rates. Together, these results highlight the importance of spike timing, and particularly of first-spike latencies, in retinal coding.

Feature-Linking Model for Image Enhancement

Inspired by gamma-band oscillations and other neurobiological discoveries, neural networks research shifts the emphasis toward temporal coding, which uses explicit times at which spikes occur as an essential dimension in neural representations. We present a feature-linking model (FLM) that uses the timing of spikes to encode information. The first spiking time of FLM is applied to image enhancement, and the processing mechanisms are consistent with the human visual system. The enhancement algorithm achieves boosting the details while preserving the information of the input image. Experiments are conducted to demonstrate the effectiveness of the proposed method. Results show that the proposed method is effective.

Complementary contributions of spike timing and spike rate to perceptual decisions in rat S1 and S2 cortex

When a neuron responds to a sensory stimulus, two fundamental codes [1-6] may transmit the information specifying stimulus identity--spike rate (the total number of spikes in the sequence, normalized by time) and spike timing (the detailed millisecond-scale temporal structure of the response). To assess the functional significance of these codes, we recorded neuronal responses in primary (S1) and secondary (S2) somatosensory cortex of five rats as they used their whiskers to identify textured surfaces. From the spike trains evoked during whisker contact with the texture, we computed the information that rate and timing codes carried about texture identity and about the rat's choice. S1 and S2 spike timing carried more information about stimulus and about choice than spike rates; the conjunction of rate and timing carried more information than either code alone. Moreover, on trials when our spike-timing-decoding algorithm extracted faithful texture information, the rat was more likely to choose correctly; when our spike-timing-decoding algorithm extracted misleading texture information, the rat was more likely to err. For spike rate information, the relationship between faithfulness of the message and correct choice was significant but weaker. These results indicate that spike timing makes crucial contributions to tactile perception, complementing and surpassing those made by rate. The language by which somatosensory cortical neurons transmit information, and the readout mechanism used to produce behavior, appears to rely on multiplexed signals from spike rate and timing.

Extracting information in spike time patterns with wavelets and information theory

We present a new method to assess the information carried by temporal patterns in spike trains. The method first performs a wavelet decomposition of the spike trains, then uses Shannon information to select a subset of coefficients carrying information, and finally assesses timing information in terms of decoding performance: the ability to identify the presented stimuli from spike train patterns. We show that the method allows: 1) a robust assessment of the information carried by spike time patterns even when this is distributed across multiple time scales and time points; 2) an effective denoising of the raster plots that improves the estimate of stimulus tuning of spike trains; and 3) an assessment of the information carried by temporally coordinated spikes across neurons. Using simulated data, we demonstrate that the Wavelet-Information (WI) method performs better and is more robust to spike time-jitter, background noise, and sample size than well-established approaches, such as principal component analysis, direct estimates of information from digitized spike trains, or a metric-based method. Furthermore, when applied to real spike trains from monkey auditory cortex and from rat barrel cortex, the WI method allows extracting larger amounts of spike timing information. Importantly, the fact that the WI method incorporates multiple time scales makes it robust to the choice of partly arbitrary parameters such as temporal resolution, response window length, number of response features considered, and the number of available trials. These results highlight the potential of the proposed method for accurate and objective assessments of how spike timing encodes information.

Understanding Neural Population Coding: Information Theoretic Insights from the Auditory System

In recent years, our research in computational neuroscience has focused on understanding how populations of neurons encode naturalistic stimuli. In particular, we focused on how populations of neurons use the time domain to encode sensory information. In this focused review, we summarize this recent work from our laboratory. We focus in particular on the mathematical methods that we developed for the quantification of how information is encoded by populations of neurons and on how we used these methods to investigate the encoding of complex naturalistic sounds in auditory cortex. We review how these methods revealed a complementary role of low frequency oscillations and millisecond precise spike patterns in encoding complex sounds and in making these representations robust to imprecise knowledge about the timing of the external stimulus. Further, we discuss challenges in extending this work to understand how large populations of neurons encode sensory information. Overall, this previous work provides analytical tools and conceptual understanding necessary to study the principles of how neural populations reflect sensory inputs and achieve a stable representation despite many uncertainties in the environment.

Reading spike timing without a clock: intrinsic decoding of spike trains

The precise timing of action potentials of sensory neurons relative to the time of stimulus presentation carries substantial sensory information that is lost or degraded when these responses are summed over longer time windows. However, it is unclear whether and how downstream networks can access information in precise time-varying neural responses. Here, we review approaches to test the hypothesis that the activity of neural populations provides the temporal reference frames needed to decode temporal spike patterns. These approaches are based on comparing the single-trial stimulus discriminability obtained from neural codes defined with respect to network-intrinsic reference frames to the discriminability obtained from codes defined relative to the experimenter's computer clock. Application of this formalism to auditory, visual and somatosensory data shows that information carried by millisecond-scale spike times can be decoded robustly even with little or no independent external knowledge of stimulus time. In cortex, key components of such intrinsic temporal reference frames include dedicated neural populations that signal stimulus onset with reliable and precise latencies, and low-frequency oscillations that can serve as reference for partitioning extended neuronal responses into informative spike patterns.

Neural codes formed by small and temporally precise populations in auditory cortex

The encoding of sensory information by populations of cortical neurons forms the basis for perception but remains poorly understood. To understand the constraints of cortical population coding we analyzed neural responses to natural sounds recorded in auditory cortex of primates (Macaca mulatta). We estimated stimulus information while varying the composition and size of the considered population. Consistent with previous reports we found that when choosing subpopulations randomly from the recorded ensemble, the average population information increases steadily with population size. This scaling was explained by a model assuming that each neuron carried equal amounts of information, and that any overlap between the information carried by each neuron arises purely from random sampling within the stimulus space. However, when studying subpopulations selected to optimize information for each given population size, the scaling of information was strikingly different: a small fraction of temporally precise cells carried the vast majority of information. This scaling could be explained by an extended model, assuming that the amount of information carried by individual neurons was highly nonuniform, with few neurons carrying large amounts of information. Importantly, these optimal populations can be determined by a single biophysical marker-the neuron's encoding time scale-allowing their detection and readout within biologically realistic circuits. These results show that extrapolations of population information based on random ensembles may overestimate the population size required for stimulus encoding, and that sensory cortical circuits may process information using small but highly informative ensembles.

Different timescales for the neural coding of consonant and vowel sounds

Psychophysical, clinical, and imaging evidence suggests that consonant and vowel sounds have distinct neural representations. This study tests the hypothesis that consonant and vowel sounds are represented on different timescales within the same population of neurons by comparing behavioral discrimination with neural discrimination based on activity recorded in rat inferior colliculus and primary auditory cortex. Performance on 9 vowel discrimination tasks was highly correlated with neural discrimination based on spike count and was not correlated when spike timing was preserved. In contrast, performance on 11 consonant discrimination tasks was highly correlated with neural discrimination when spike timing was preserved and not when spike timing was eliminated. These results suggest that in the early stages of auditory processing, spike count encodes vowel sounds and spike timing encodes consonant sounds. These distinct coding strategies likely contribute to the robust nature of speech sound representations and may help explain some aspects of developmental and acquired speech processing disorders.

Computing complex visual features with retinal spike times

Neurons in sensory systems can represent information not only by their firing rate, but also by the precise timing of individual spikes. For example, certain retinal ganglion cells, first identified in the salamander, encode the spatial structure of a new image by their first-spike latencies. Here we explore how this temporal code can be used by downstream neural circuits for computing complex features of the image that are not available from the signals of individual ganglion cells. To this end, we feed the experimentally observed spike trains from a population of retinal ganglion cells to an integrate-and-fire model of post-synaptic integration. The synaptic weights of this integration are tuned according to the recently introduced tempotron learning rule. We find that this model neuron can perform complex visual detection tasks in a single synaptic stage that would require multiple stages for neurons operating instead on neural spike counts. Furthermore, the model computes rapidly, using only a single spike per afferent, and can signal its decision in turn by just a single spike. Extending these analyses to large ensembles of simulated retinal signals, we show that the model can detect the orientation of a visual pattern independent of its phase, an operation thought to be one of the primitives in early visual processing. We analyze how these computations work and compare the performance of this model to other schemes for reading out spike-timing information. These results demonstrate that the retina formats spatial information into temporal spike sequences in a way that favors computation in the time domain. Moreover, complex image analysis can be achieved already by a simple integrate-and-fire model neuron, emphasizing the power and plausibility of rapid neural computing with spike times.

A spike-timing-based integrated model for pattern recognition

During the past few decades, remarkable progress has been made in solving pattern recognition problems using networks of spiking neurons. However, the issue of pattern recognition involving computational process from sensory encoding to synaptic learning remains underexplored, as most existing models or algorithms target only part of the computational process. Furthermore, many learning algorithms proposed in the literature neglect or pay little attention to sensory information encoding, which makes them incompatible with neural-realistic sensory signals encoded from real-world stimuli. By treating sensory coding and learning as a systematic process, we attempt to build an integrated model based on spiking neural networks (SNNs), which performs sensory neural encoding and supervised learning with precisely timed sequences of spikes. With emerging evidence of precise spike-timing neural activities, the view that information is represented by explicit firing times of action potentials rather than mean firing rates has been receiving increasing attention. The external sensory stimulation is first converted into spatiotemporal patterns using a latency-phase encoding method and subsequently transmitted to the consecutive network for learning. Spiking neurons are trained to reproduce target signals encoded with precisely timed spikes. We show that when a supervised spike-timing-based learning is used, different spatiotemporal patterns are recognized by different spike patterns with a high time precision in milliseconds.

Analysis of slow (theta) oscillations as a potential temporal reference frame for information coding in sensory cortices

While sensory neurons carry behaviorally relevant information in responses that often extend over hundreds of milliseconds, the key units of neural information likely consist of much shorter and temporally precise spike patterns. The mechanisms and temporal reference frames by which sensory networks partition responses into these shorter units of information remain unknown. One hypothesis holds that slow oscillations provide a network-intrinsic reference to temporally partitioned spike trains without exploiting the millisecond-precise alignment of spikes to sensory stimuli. We tested this hypothesis on neural responses recorded in visual and auditory cortices of macaque monkeys in response to natural stimuli. Comparing different schemes for response partitioning revealed that theta band oscillations provide a temporal reference that permits extracting significantly more information than can be obtained from spike counts, and sometimes almost as much information as obtained by partitioning spike trains using precisely stimulus-locked time bins. We further tested the robustness of these partitioning schemes to temporal uncertainty in the decoding process and to noise in the sensory input. This revealed that partitioning using an oscillatory reference provides greater robustness than partitioning using precisely stimulus-locked time bins. Overall, these results provide a computational proof of concept for the hypothesis that slow rhythmic network activity may serve as internal reference frame for information coding in sensory cortices and they foster the notion that slow oscillations serve as key elements for the computations underlying perception.

Beyond traditional approaches to understanding the functional role of neuromodulators in sensory cortices

Over the last two decades, a vast literature has described the influence of neuromodulatory systems on the responses of sensory cortex neurons (review in Gu, 2002; Edeline, 2003; Weinberger, 2003; Metherate, 2004, 2011). At the single cell level, facilitation of evoked responses, increases in signal-to-noise ratio, and improved functional properties of sensory cortex neurons have been reported in the visual, auditory, and somatosensory modality. At the map level, massive cortical reorganizations have been described when repeated activation of a neuromodulatory system are associated with a particular sensory stimulus. In reviewing our knowledge concerning the way the noradrenergic and cholinergic system control sensory cortices, I will point out that the differences between the protocols used to reveal these effects most likely reflect different assumptions concerning the role of the neuromodulators. More importantly, a gap still exists between the descriptions of neuromodulatory effects and the concepts that are currently applied to decipher the neural code operating in sensory cortices. Key examples that bring this gap into focus are the concept of cell assemblies and the role played by the spike timing precision (i.e., by the temporal organization of spike trains at the millisecond time-scale) which are now recognized as essential in sensory physiology but are rarely considered in experiments describing the role of neuromodulators in sensory cortices. Thus, I will suggest that several lines of research, particularly in the field of computational neurosciences, should help us to go beyond traditional approaches and, ultimately, to understand how neuromodulators impact on the cortical mechanisms underlying our perceptual abilities.

Shaping the dynamics of a bidirectional neural interface

Progress in decoding neural signals has enabled the development of interfaces that translate cortical brain activities into commands for operating robotic arms and other devices. The electrical stimulation of sensory areas provides a means to create artificial sensory information about the state of a device. Taken together, neural activity recording and microstimulation techniques allow us to embed a portion of the central nervous system within a closed-loop system, whose behavior emerges from the combined dynamical properties of its neural and artificial components. In this study we asked if it is possible to concurrently regulate this bidirectional brain-machine interaction so as to shape a desired dynamical behavior of the combined system. To this end, we followed a well-known biological pathway. In vertebrates, the communications between brain and limb mechanics are mediated by the spinal cord, which combines brain instructions with sensory information and organizes coordinated patterns of muscle forces driving the limbs along dynamically stable trajectories. We report the creation and testing of the first neural interface that emulates this sensory-motor interaction. The interface organizes a bidirectional communication between sensory and motor areas of the brain of anaesthetized rats and an external dynamical object with programmable properties. The system includes (a) a motor interface decoding signals from a motor cortical area, and (b) a sensory interface encoding the state of the external object into electrical stimuli to a somatosensory area. The interactions between brain activities and the state of the external object generate a family of trajectories converging upon a selected equilibrium point from arbitrary starting locations. Thus, the bidirectional interface establishes the possibility to specify not only a particular movement trajectory but an entire family of motions, which includes the prescribed reactions to unexpected perturbations.

Active engagement improves primary auditory cortical neurons' ability to discriminate temporal modulation

The effect of attention on single neuron responses in the auditory system is unresolved. We found that when monkeys discriminated temporally amplitude modulated (AM) from unmodulated sounds, primary auditory cortical (A1) neurons better discriminated those sounds than when the monkeys were not discriminating them. This was observed for both average firing rate and vector strength (VS), a measure of how well neurons temporally follow the stimulus' temporal modulation. When data were separated by nonsynchronized and synchronized responses, the firing rate of nonsynchronized responses best distinguished AM- noise from unmodulated noise, followed by VS for synchronized responses, with firing rate for synchronized neurons providing the poorest AM discrimination. Firing rate-based AM discrimination for synchronized neurons, however, improved most with task engagement, showing that the least sensitive code in the passive condition improves the most with task engagement. Rate coding improved due to larger increases in absolute firing rate at higher modulation depths than for lower depths and unmodulated sounds. Relative to spontaneous activity (which increased with engagement), the response to unmodulated sounds decreased substantially. The temporal coding improvement--responses more precisely temporally following a stimulus when animals were required to attend to it--expands the framework of possible mechanisms of attention to include increasing temporal precision of stimulus following. These findings provide a crucial step to understanding the coding of temporal modulation and support a model in which rate and temporal coding work in parallel, permitting a multiplexed code for temporal modulation, and for a complementary representation of rate and temporal coding.

Neurons with stereotyped and rapid responses provide a reference frame for relative temporal coding in primate auditory cortex

The precise timing of spikes of cortical neurons relative to stimulus onset carries substantial sensory information. To access this information the sensory systems would need to maintain an internal temporal reference that reflects the precise stimulus timing. Whether and how sensory systems implement such reference frames to decode time-dependent responses, however, remains debated. Studying the encoding of naturalistic sounds in primate (Macaca mulatta) auditory cortex we here investigate potential intrinsic references for decoding temporally precise information. Within the population of recorded neurons, we found one subset responding with stereotyped fast latencies that varied little across trials or stimuli, while the remaining neurons had stimulus-modulated responses with longer and variable latencies. Computational analysis demonstrated that the neurons with stereotyped short latencies constitute an effective temporal reference for relative coding. Using the response onset of a simultaneously recorded stereotyped neuron allowed decoding most of the stimulus information carried by onset latencies and the full spike train of stimulus-modulated neurons. Computational modeling showed that few tens of such stereotyped reference neurons suffice to recover nearly all information that would be available when decoding the same responses relative to the actual stimulus onset. These findings reveal an explicit neural signature of an intrinsic reference for decoding temporal response patterns in the auditory cortex of alert animals. Furthermore, they highlight a role for apparently unselective neurons as an early saliency signal that provides a temporal reference for extracting stimulus information from other neurons.

A diversity of synaptic filters are created by temporal summation of excitation and inhibition

Temporal filtering is a fundamental operation of nervous systems. In peripheral sensory systems, the temporal pattern of spiking activity can encode various stimulus qualities, and temporal filtering allows postsynaptic neurons to detect behaviorally relevant stimulus features from these spike trains. Intrinsic excitability, short-term synaptic plasticity, and voltage-dependent dendritic conductances have all been identified as mechanisms that can establish temporal filtering behavior in single neurons. Here we show that synaptic integration of temporally summating excitation and inhibition can establish diverse temporal filters of presynaptic input. Mormyrid electric fish communicate by varying the intervals between electric organ discharges. The timing of each discharge is coded by peripheral receptors into precisely timed spikes. Within the midbrain posterior exterolateral nucleus, temporal filtering by individual neurons results in selective responses to a particular range of presynaptic interspike intervals. These neurons are diverse in their temporal filtering properties, reflecting the wide range of intervals that must be detected during natural communication behavior. By manipulating presynaptic spike timing with high temporal resolution, we demonstrate that tuning to behaviorally relevant patterns of presynaptic input is similar in vivo and in vitro. We reveal that GABAergic inhibition plays a critical role in establishing different temporal filtering properties. Further, our results demonstrate that temporal summation of excitation and inhibition establishes selective responses to high and low rates of synaptic input, respectively. Simple models of synaptic integration reveal that variation in these two competing influences provides a basic mechanism for generating diverse temporal filters of synaptic input.

Lateral information processing by spiking neurons: a theoretical model of the neural correlate of consciousness

Cognitive brain functions, for example, sensory perception, motor control and learning, are understood as computation by axonal-dendritic chemical synapses in networks of integrate-and-fire neurons. Cognitive brain functions may occur either consciously or nonconsciously (on "autopilot"). Conscious cognition is marked by gamma synchrony EEG, mediated largely by dendritic-dendritic gap junctions, sideways connections in input/integration layers. Gap-junction-connected neurons define a sub-network within a larger neural network. A theoretical model (the "conscious pilot") suggests that as gap junctions open and close, a gamma-synchronized subnetwork, or zone moves through the brain as an executive agent, converting nonconscious "auto-pilot" cognition to consciousness, and enhancing computation by coherent processing and collective integration. In this study we implemented sideways "gap junctions" in a single-layer artificial neural network to perform figure/ground separation. The set of neurons connected through gap junctions form a reconfigurable resistive grid or sub-network zone. In the model, outgoing spikes are temporally integrated and spatially averaged using the fixed resistive grid set up by neurons of similar function which are connected through gap-junctions. This spatial average, essentially a feedback signal from the neuron's output, determines whether particular gap junctions between neurons will open or close. Neurons connected through open gap junctions synchronize their output spikes. We have tested our gap-junction-defined sub-network in a one-layer neural network on artificial retinal inputs using real-world images. Our system is able to perform figure/ground separation where the laterally connected sub-network of neurons represents a perceived object. Even though we only show results for visual stimuli, our approach should generalize to other modalities. The system demonstrates a moving sub-network zone of synchrony, within which the contents of perception are represented and contained. This mobile zone can be viewed as a model of the neural correlate of consciousness in the brain.

The timing of vision - how neural processing links to different temporal dynamics

In this review, we describe our recent attempts to model the neural correlates of visual perception with biologically inspired networks of spiking neurons, emphasizing the dynamical aspects. Experimental evidence suggests distinct processing modes depending on the type of task the visual system is engaged in. A first mode, crucial for object recognition, deals with rapidly extracting the glimpse of a visual scene in the first 100 ms after its presentation. The promptness of this process points to mainly feedforward processing, which relies on latency coding, and may be shaped by spike timing-dependent plasticity (STDP). Our simulations confirm the plausibility and efficiency of such a scheme. A second mode can be engaged whenever one needs to perform finer perceptual discrimination through evidence accumulation on the order of 400 ms and above. Here, our simulations, together with theoretical considerations, show how predominantly local recurrent connections and long neural time-constants enable the integration and build-up of firing rates on this timescale. In particular, we review how a non-linear model with attractor states induced by strong recurrent connectivity provides straightforward explanations for several recent experimental observations. A third mode, involving additional top-down attentional signals, is relevant for more complex visual scene processing. In the model, as in the brain, these top-down attentional signals shape visual processing by biasing the competition between different pools of neurons. The winning pools may not only have a higher firing rate, but also more synchronous oscillatory activity. This fourth mode, oscillatory activity, leads to faster reaction times and enhanced information transfers in the model. This has indeed been observed experimentally. Moreover, oscillatory activity can format spike times and encode information in the spike phases with respect to the oscillatory cycle. This phenomenon is referred to as “phase-of-firing coding,” and experimental evidence for it is accumulating in the visual system. Simulations show that this code can again be efficiently decoded by STDP. Future work should focus on continuous natural vision, bio-inspired hardware vision systems, and novel experimental paradigms to further distinguish current modeling approaches.

Estimating Neuronal Information: Logarithmic Binning of Neuronal Inter-Spike Intervals

Neurons communicate via the relative timing of all-or-none biophysical signals called spikes. For statistical analysis, the time between spikes can be accumulated into inter-spike interval histograms. Information theoretic measures have been estimated from these histograms to assess how information varies across organisms, neural systems, and disease conditions. Because neurons are computational units that, to the extent they process time, work not by discrete clock ticks but by the exponential decays of numerous intrinsic variables, we propose that neuronal information measures scale more naturally with the logarithm of time. For the types of inter-spike interval distributions that best describe neuronal activity, the logarithm of time enables fewer bins to capture the salient features of the distributions. Thus, discretizing the logarithm of inter-spike intervals, as compared to the inter-spike intervals themselves, yields histograms that enable more accurate entropy and information estimates for fewer bins and less data. Additionally, as distribution parameters vary, the entropy and information calculated from the logarithm of the inter-spike intervals are substantially better behaved, e.g., entropy is independent of mean rate, and information is equally affected by rate gains and divisions. Thus, when compiling neuronal data for subsequent information analysis, the logarithm of the inter-spike intervals is preferred, over the untransformed inter-spike intervals, because it yields better information estimates and is likely more similar to the construction used by nature herself.

Millisecond encoding precision of auditory cortex neurons

Neurons in auditory cortex are central to our perception of sounds. However, the underlying neural codes, and the relevance of millisecond-precise spike timing in particular, remain debated. Here, we addressed this issue in the auditory cortex of alert nonhuman primates by quantifying the amount of information carried by precise spike timing about complex sounds presented for extended periods of time (random tone sequences and natural sounds). We investigated the dependence of stimulus information on the temporal precision at which spike times were registered and found that registering spikes at a precision coarser than a few milliseconds significantly reduced the encoded information. This dependence demonstrates that auditory cortex neurons can carry stimulus information at high temporal precision. In addition, we found that the main determinant of finely timed information was rapid modulation of the firing rate, whereas higher-order correlations between spike times contributed negligibly. Although the neural coding precision was high for random tone sequences and natural sounds, the information lost at a precision coarser than a few milliseconds was higher for the stimulus sequence that varied on a faster time scale (random tones), suggesting that the precision of cortical firing depends on the stimulus dynamics. Together, these results provide a neural substrate for recently reported behavioral relevance of precisely timed activity patterns with auditory cortex. In addition, they highlight the importance of millisecond-precise neural coding as general functional principle of auditory processing--from the periphery to cortex.

Information Carried by Population Spike Times in the Whisker Sensory Cortex can be Decoded Without Knowledge of Stimulus Time

Computational analyses have revealed that precisely timed spikes emitted by somatosensory cortical neuronal populations encode basic stimulus features in the rat's whisker sensory system. Efficient spike time based decoding schemes both for the spatial location of a stimulus and for the kinetic features of complex whisker movements have been defined. To date, these decoding schemes have been based upon spike times referenced to an external temporal frame – the time of the stimulus itself. Such schemes are limited by the requirement of precise knowledge of the stimulus time signal, and it is not clear whether stimulus times are known to rats making sensory judgments. Here, we first review studies of the information obtained from spike timing referenced to the stimulus time. Then we explore new methods for extracting spike train information independently of any external temporal reference frame. These proposed methods are based on the detection of stimulus-dependent differences in the firing time within a neuronal population. We apply them to a data set using single-whisker stimulation in anesthetized rats and find that stimulus site can be decoded based on the millisecond-range relative differences in spike times even without knowledge of stimulus time. If spike counts alone are measured over tens or hundreds of milliseconds rather than milliseconds, such decoders are much less effective. These results suggest that decoding schemes based on millisecond-precise spike times are likely to subserve robust and information-rich transmission of information in the somatosensory system.

Sensory neural codes using multiplexed temporal scales

Determining how neuronal activity represents sensory information is central for understanding perception. Recent work shows that neural responses at different timescales can encode different stimulus attributes, resulting in a temporal multiplexing of sensory information. Multiplexing increases the encoding capacity of neural responses, enables disambiguation of stimuli that cannot be discriminated at a single response timescale, and makes sensory representations stable to the presence of variability in the sensory world. Thus, as we discuss here, temporal multiplexing could be a key strategy used by the brain to form an information-rich and stable representation of the environment.

A copulas approach to neuronal networks models

Simultaneous recordings from groups of neurons request to improve models. To switch from single unit description to multivariate models describing the coding activity of two or more neurons, we propose to use the copula notion. This mathematical object catches the coupling properties and allows a mathematical description of the dependencies between two or more random variables. Its use is here illustrated by means of toy examples and further applications are discussed.

Implications on cerebellar function from information coding

One function of the cerebellar cortex is to process information. There are at least two types of information. Temporal information is encoded in the timing pattern of action and synaptic potentials, whereas structural information is encoded in the spatial pattern of the cerebellar synaptic circuitry. Intuitively, analysis of highly complex information in the time domain would require a cerebellar cortex with structural complexity to match. Information theory offers a way to estimate quantitatively both types of information and thereby helps to test hypotheses or advance theories of cerebellar neurobiology. These estimates suggest: (i) That the mossy-fiber-granule-cell system carries far more (temporal) information than the climbing fiber system, (ii) that Purkinje cells extract only a fraction of the (temporal) information from their afferents, and (iii) that the cerebellar cortex has a large (spatial) information coding capacity. Concerning information, one can argue that the cerebellar cortex analyzes temporal information in its afferents as a search engine, in search of coincidental mossy fiber events based on timing cues provided by climbing fiber events. Results of successive searches are continuously being converted into structural information encoded in the spatial distribution pattern of granule-cell-Purkinje-cell synapses along granule cell axons, thereby providing an adaptive and indeed self-correcting dimension to the structural information code. The search engine operation involves cellular mechanisms acting on temporal events and is part of an associative learning process. The conversion and generation of structural information involves neuroplasticity mechanisms acting at the synaptic level, with electrophysiological as well as structural consequences, and may be part of the short- and long-term memory process. These and other attributes qualify the cerebellar cortex as a dynamic information processing center, contributing to memory and learning while linking motor output with sensory events.

Contribution of spike timing to the information transmitted by HVC neurons

In many species, neurons with highly selective stimulus-response properties characterize higher order sensory areas and/or sensory motor areas of the CNS. In the songbird nuclei, the responses of HVC (used as a proper name) neurons during playback of the bird's own song (BOS) are probably one of the most striking examples of selectivity for natural stimuli. We examined here to what extent spike-timing carries information about natural and time-reversed versions of the BOS. From a heterogenous population of 107 HVC neurons recorded in long-day or short-day conditions, a standard indicator of stimulus preference based on spike-count (the d' index) indicates that a limited proportion of cells can be classified as selective for the BOS (20% with a |d'| > 1). In contrast, quantifying the information conveyed by spike trains with the metric-space of J.D. Victor & K.P Purpura [(1996) J. Neurophysiol., 76, 1310-1326] indicates that 62% of the cells display significant amounts of transmitted information, among which 77% are 'temporal cells'. 'Temporal cells' correspond to cells transmitting significant amounts of information when spike-timing is considered, whereas no information, or lower amounts of transmitted information, is obtained when only spike-count is considered. Computing a correlation index between spike trains [S. Schreiber et al. (2003) Neurocomputing, 52-54,925-931] revealed that spike-timing reliability is higher for the forward than for the reverse BOS, whatever the day length and the cell type are. Cells classified as selective in terms of spike-counts (d' index) had greater amounts of transmitted information, but cells classified as non-selective (d' < 0.5) can also transmit significant amounts of information. Thus, information theory methods demonstrate that a much larger proportion of neurons than expected based on spike-count only participate in the discrimination between stimuli.

Spatio-temporal information analysis of event-related BOLD responses

A new approach for analysis of event-related fMRI (BOLD) signals is proposed. The technique is based on measures from information theory and is used both for spatial localization of task-related activity, as well as for extracting temporal information regarding the task-dependent propagation of activation across different brain regions. This approach enables whole brain visualization of voxels (areas) most involved in coding of a specific task condition, the time at which they are most informative about the condition, as well as their average amplitude at that preferred time. The approach does not require prior assumptions about the shape of the hemodynamic response function (HRF) nor about linear relations between BOLD response and presented stimuli (or task conditions). We show that relative delays between different brain regions can also be computed without prior knowledge of the experimental design, suggesting a general method that could be applied for analysis of differential time delays that occur during natural, uncontrolled conditions. Here we analyze BOLD signals recorded during performance of a motor learning task. We show that, during motor learning, the BOLD response of unimodal motor cortical areas precedes the response in higher-order multimodal association areas, including posterior parietal cortex. Brain areas found to be associated with reduced activity during motor learning, predominantly in prefrontal brain regions, are informative about the task typically at significantly later times.

The selectivity of canary HVC neurons for the Bird's Own Song: rate coding, temporal coding, or both?

The neuronal selectivity observed in the avian song system for the Bird's Own Song progressively emerged as an extraordinary fruitful model to investigate the neural code underlying the recognition of complex stimuli and the occurrence of learned behaviors. In adult zebra finch, neurons from the HVC (used as a proper name) show very selective auditory responses, firing more to presentation of the Bird's Own Song (BOS) than to reverse BOS or other conspecific songs. However, as adult zebra finches always produce the same stereotyped song, the presence of such highly selective neurons in birds with larger repertoire still remains an open question. Data presented here show that neurons selective for the BOS can be found in adult canary, a seasonal breeding bird which display a large repertoire. More precisely, we found that a large proportion of neurons (29/36) exhibits higher responses to presentation of the forward than to the reverse BOS, and that 22% of the cells were identified as selective on the basis of the d' value. For a cell that was extensively studied, we evaluated to what extent temporal stimulus-related structure predicts the acoustic stimulus using linear or non-linear artificial neural networks (ANN). These analyses indicated that the temporal structure contained in spike trains characterizes more accurately the stimulus than the firing rate. The limitations of applying ANN analyses to electrophysiological data are discussed and potential solutions to increase the confidence in these analysis are proposed.

Maximum Likelihood Estimation of a Stochastic Integrate-and-Fire Neural Encoding Model

We examine a cascade encoding model for neural response in which a linear filtering stage is followed by a noisy, leaky, integrate-and-fire spike generation mechanism. This model provides a biophysically more realistic alternative to models based on Poisson (memoryless) spike generation, and can effectively reproduce a variety of spiking behaviors seen in vivo. We describe the maximum likelihood estimator for the model parameters, given only extracellular spike train responses (not intracellular voltage data). Specifically, we prove that the log-likelihood function is concave and thus has an essentially unique global maximum that can be found using gradient ascent techniques. We develop an efficient algorithm for computing the maximum likelihood solution, demonstrate the effectiveness of the resulting estimator with numerical simulations, and discuss a method of testing the model's validity using time-rescaling and density evolution techniques.

Estimation of Entropy and Mutual Information

We present some new results on the nonparametric estimation of entropy and mutual information. First, we use an exact local expansion of the entropy function to prove almost sure consistency and central limit theorems for three of the most commonly used discretized information estimators. The setup is related to Grenander's method of sieves and places no assumptions on the underlying probability measure generating the data. Second, we prove a converse to these consistency theorems, demonstrating that a misapplication of the most common estimation techniques leads to an arbitrarily poor estimate of the true information, even given unlimited data. This “inconsistency” theorem leads to an analytical approximation of the bias, valid in surprisingly small sample regimes and more accurate than the usual [Formula: see text] formula of Miller and Madow over a large region of parameter space. The two most practical implications of these results are negative: (1) information estimates in a certain data regime are likely contaminated by bias, even if “bias-corrected” estimators are used, and (2) confidence intervals calculated by standard techniques drastically underestimate the error of the most common estimation methods.

Finally, we note a very useful connection between the bias of entropy estimators and a certain polynomial approximation problem. By casting bias calculation problems in this approximation theory framework, we obtain the best possible generalization of known asymptotic bias results. More interesting, this framework leads to an estimator with some nice properties: the estimator comes equipped with rigorous bounds on the maximum error over all possible underlying probability distributions, and this maximum error turns out to be surprisingly small. We demonstrate the application of this new estimator on both real and simulated data.

Cooperation between area 17 neuron pairs enhances fine discrimination of orientation

We examined 66 complex cells in area 17 of cats that were paralyzed and anesthetized with propofol and N2O. We studied changes in ensemble responses for small (<10 degrees ) and large (>10 degrees ) differences in orientation. Examination of temporal resolution and discharge history revealed advantages in discrimination from both dependent (e.g., synchronization) and independent (e.g., bursting) interspike interval properties. For 27 pairs of neurons, we found that the average cooperation (the advantage gained from the joint activity) was 57.6% for fine discrimination of orientation but <5% for gross discrimination. Dependency (probabilistic quantification of the interaction between the cells) was measured between 29 pairs of neurons while varying orientation. On average, the dependency tuning for orientation was 35.5% narrower than the average firing rate tuning. The changes in dependency around the peak orientation (at which the firing rate remains relatively constant) lead to substantial cooperation that can improve discrimination in this region. The narrow tuning of dependency and the cooperation provide evidence to support a population-encoding scheme that is based on biologically plausible mechanisms and that could account for hyperacuities.

Surfing a spike wave down the ventral stream

Numerous theories of neural processing, often motivated by experimental observations, have explored the computational properties of neural codes based on the absolute or relative timing of spikes in spike trains. Spiking neuron models and theories however, as well as their experimental counterparts, have generally been limited to the simulation or observation of isolated neurons, isolated spike trains, or reduced neural populations. Such theories would therefore seem inappropriate to capture the properties of a neural code relying on temporal spike patterns distributed across large neuronal populations. Here we report a range of computer simulations and theoretical considerations that were designed to explore the possibilities of one such code and its relevance for visual processing. In a unified framework where the relation between stimulus saliency and spike relative timing plays the central role, we describe how the ventral stream of the visual system could process natural input scenes and extract meaningful information, both rapidly and reliably. The first wave of spikes generated in the retina in response to a visual stimulation carries information explicitly in its spatio-temporal structure: the most salient information is represented by the first spikes over the population. This spike wave, propagating through a hierarchy of visual areas, is regenerated at each processing stage, where its temporal structure can be modified by (i). the selectivity of the cortical neurons, (ii). lateral interactions and (iii). top-down attentional influences from higher order cortical areas. The resulting model could account for the remarkable efficiency and rapidity of processing observed in the primate visual system.