Tuning dissimilarity explains short distance decline of spontaneous spike correlation in macaque V1

Dependence of Contextual Modulation in Macaque V1 on Interlaminar Signal Flow

In visual cortex, neural correlates of subjective perception can be generated by modulation of activity from beyond the classical receptive field (CRF). In macaque V1, activity generated by nonclassical receptive field (nCRF) stimulation involves different intracortical circuitry than activity generated by CRF stimulation, suggesting that interactions between neurons across V1 layers differ under CRF and nCRF stimulus conditions. We measured border ownership modulation within large populations of V1 neurons. We found that neurons in single columns preferred the same side of objects located outside of the CRF. In addition, we found that interactions between pairs of neurons situated across feedback/horizontal and input layers differed between CRF and nCRF stimulation. Furthermore, the magnitude of border ownership modulation was predicted by greater information flow from feedback/horizontal to input layers. These results demonstrate that the flow of signals between layers covaries with the degree to which neurons integrate information from beyond the CRF.

DualSort: online spike sorting with a running neural network

Objective.Spike sorting, i.e. the detection and separation of measured action potentials from different extracellularly recorded neurons, remains one of the bottlenecks in deciphering the brain. In recent years, the application of neural networks (NNs) for spike sorting has garnered significant attention. Most methods focus on specific sub-problems within the conventional spike sorting pipeline, such as spike detection or feature extraction, and attempt to solve them with complex network architectures. This paper presents DualSort, a simple NN that gets combined with downstream post-processing for real-time spike sorting. It shows high efficiency, low complexity, and requires a comparatively small amount of human interaction.Approach.Synthetic and experimentally obtained extracellular single-channel recordings were utilized to train and evaluate the proposed NN. For training, spike waveforms were labeled with respect to their associated neuron and position in the signal, allowing the detection and categorization of spikes in unison. DualSort classifies a single spike multiple times in succession, as it runs over the signal in a step-by-step manner and uses a post-processing algorithm that transmits the network output into spike trains. Main results.With the used datasets, DualSort was able to detect and distinguish different spike waveforms and separate them from background activity. The post-processing algorithm significantly strengthened the overall performance of the model, making the system more robust as a whole. Although DualSort is an end-to-end solution that efficiently transforms filtered signals into spike trains, it competes with contemporary state-of-the-art technologies that exclusively target single sub-problems in the conventional spike sorting pipeline.Significance.This work demonstrates that even under high noise levels, complex NNs are not necessary by any means to achieve high performance in spike detection and sorting. The utilization of data augmentation on a limited quantity of spikes could substantially decrease hand-labeling compared to other studies. Furthermore, the proposed framework can be utilized without human interaction when combined with an unsupervised technique that provides pseudo labels for DualSort. Due to the low complexity of our network, it works efficiently and enables real-time processing on basic hardware. The proposed approach is not limited to spike sorting, as it may also be used to process different signals, such as electroencephalogram (EEG), which needs to be investigated in future research.

Canonical circuit computations for computer vision

Advanced computer vision mechanisms have been inspired by neuroscientific findings. However, with the focus on improving benchmark achievements, technical solutions have been shaped by application and engineering constraints. This includes the training of neural networks which led to the development of feature detectors optimally suited to the application domain. However, the limitations of such approaches motivate the need to identify computational principles, or motifs, in biological vision that can enable further foundational advances in machine vision. We propose to utilize structural and functional principles of neural systems that have been largely overlooked. They potentially provide new inspirations for computer vision mechanisms and models. Recurrent feedforward, lateral, and feedback interactions characterize general principles underlying processing in mammals. We derive a formal specification of core computational motifs that utilize these principles. These are combined to define model mechanisms for visual shape and motion processing. We demonstrate how such a framework can be adopted to run on neuromorphic brain-inspired hardware platforms and can be extended to automatically adapt to environment statistics. We argue that the identified principles and their formalization inspires sophisticated computational mechanisms with improved explanatory scope. These and other elaborated, biologically inspired models can be employed to design computer vision solutions for different tasks and they can be used to advance neural network architectures of learning.

Application of deep reinforcement learning for spike sorting under multi-class imbalance

Spike sorting is the basis for analyzing spike firing patterns encoded in high-dimensional information spaces. With the fact that high-density microelectrode arrays record multiple neurons simultaneously, the data collected often suffers from two problems: a few overlapping spikes and different neuronal firing rates, which both belong to the multi-class imbalance problem. Since deep reinforcement learning (DRL) assign targeted attention to categories through reward functions, we propose ImbSorter to implement spike sorting under multi-class imbalance. We describe spike sorting as a Markov sequence decision and construct a dynamic reward function (DRF) to improve the sensitivity of the agent to minor classes based on the inter-class imbalance ratios. The agent is eventually guided by the optimal strategy to classify spikes. We consider the Wave_Clus dataset, which contains overlapping spikes and diverse noise levels, and the macaque dataset, which has a multi-scale imbalance. ImbSorter is compared with classical DRL architectures, traditional machine learning algorithms, and advanced overlapping spike sorting techniques on these two above datasets. ImbSorter obtained improved results on the Macro_F1. The results show ImbSorter has a promising ability to resist overlapping and noise interference. It has high stability and promising performance in processing spikes with different degrees of skewed distribution.

A deep learning network based on CNN and sliding window LSTM for spike sorting

Spike sorting plays an essential role to obtain electrophysiological activity of single neuron in the fields of neural signal decoding. With the development of electrode array, large numbers of spikes are recorded simultaneously, which rises the need for accurate automatic and generalization algorithms. Hence, this paper proposes a spike sorting model with convolutional neural network (CNN) and a spike classification model with combination of CNN and Long-Short Term Memory (LSTM). The recall rate of our detector could reach 94.40% in low noise level dataset. Although the recall declined with the increasing noise level, our model still presented higher feasibility and better robustness than other models. In addition, the results of our classification model presented an accuracy of greater than 99% in simulated data and an average accuracy of about 95% in experimental data, suggesting our classifier outperforms the current "WMsorting" and other deep learning models. Moreover, the performance of our whole algorithm was evaluated through simulated data and the results shows that the accuracy of spike sorting reached about 97%. It is noteworthy to say that, this proposed algorithm could be used to achieve accurate and robust automated spike detection and spike classification.

Functional interactions among neurons within single columns of macaque V1

Recent developments in high-density neurophysiological tools now make it possible to record from hundreds of single neurons within local, highly interconnected neural networks. Among the many advantages of such recordings is that they dramatically increase the quantity of identifiable, functional interactions between neurons thereby providing an unprecedented view of local circuits. Using high-density, Neuropixels recordings from single neocortical columns of primary visual cortex in nonhuman primates, we identified 1000s of functionally interacting neuronal pairs using established crosscorrelation approaches. Our results reveal clear and systematic variations in the synchrony and strength of functional interactions within single cortical columns. Despite neurons residing within the same column, both measures of interactions depended heavily on the vertical distance separating neuronal pairs, as well as on the similarity of stimulus tuning. In addition, we leveraged the statistical power afforded by the large numbers of functionally interacting pairs to categorize interactions between neurons based on their crosscorrelation functions. These analyses identified distinct, putative classes of functional interactions within the full population. These classes of functional interactions were corroborated by their unique distributions across defined laminar compartments and were consistent with known properties of V1 cortical circuitry, such as the lead-lag relationship between simple and complex cells. Our results provide a clear proof-of-principle for the use of high-density neurophysiological recordings to assess circuit-level interactions within local neuronal networks.

Classification of overlapping spikes using convolutional neural networks and long short term memory

Spike sorting is one of the key techniques to understand brain activity. In this paper, we propose a novel deep learning approach based on convolutional neural networks (CNN) and long short term memory (LSTM) to implement overlapping spike sorting. The results of the simulated data demonstrated that the clustering accuracy was greater than 99.9% and 99.0% for non-overlapping spikes and overlapping spikes, respectively. Moreover, the proposed method performed better than our previous deep learning approach named "1D-CNN". In addition, the experimental data recorded from the primary visual cortex of a macaque monkey were used to evaluate the proposed method in a practical application. It was shown that the method could successfully isolate most overlapping spikes of different neurons (ranging from two to five). In summary, the CNN + LSTM method proposed in this paper is of great advantage for overlapping spike sorting with high accuracy. It lays the foundation for application in more challenging works, such as distinguishing the simultaneous recordings of multichannel neuronal activities.

Fast Simulation of a Multi-Area Spiking Network Model of Macaque Cortex on an MPI-GPU Cluster

Spiking neural network models are increasingly establishing themselves as an effective tool for simulating the dynamics of neuronal populations and for understanding the relationship between these dynamics and brain function. Furthermore, the continuous development of parallel computing technologies and the growing availability of computational resources are leading to an era of large-scale simulations capable of describing regions of the brain of ever larger dimensions at increasing detail. Recently, the possibility to use MPI-based parallel codes on GPU-equipped clusters to run such complex simulations has emerged, opening up novel paths to further speed-ups. NEST GPU is a GPU library written in CUDA-C/C++ for large-scale simulations of spiking neural networks, which was recently extended with a novel algorithm for remote spike communication through MPI on a GPU cluster. In this work we evaluate its performance on the simulation of a multi-area model of macaque vision-related cortex, made up of about 4 million neurons and 24 billion synapses and representing 32 mm2 surface area of the macaque cortex. The outcome of the simulations is compared against that obtained using the well-known CPU-based spiking neural network simulator NEST on a high-performance computing cluster. The results show not only an optimal match with the NEST statistical measures of the neural activity in terms of three informative distributions, but also remarkable achievements in terms of simulation time per second of biological activity. Indeed, NEST GPU was able to simulate a second of biological time of the full-scale macaque cortex model in its metastable state 3.1× faster than NEST using 32 compute nodes equipped with an NVIDIA V100 GPU each. Using the same configuration, the ground state of the full-scale macaque cortex model was simulated 2.4× faster than NEST.

Abrupt darkening under high dynamic range (HDR) luminance invokes facilitation for high-contrast targets and grouping by luminance similarity

When scanning across a scene, luminance can vary by up to 100,000-to-1 (high dynamic range, HDR), requiring multiple normalizing mechanisms spanning from the retina to the cortex to support visual acuity and recognition. Vision models based on standard dynamic range (SDR) luminance contrast ratios below 100-to-1 have limited ability to generalize to real-world scenes with HDR luminance. To characterize how orientation and luminance are linked in brain mechanisms for luminance normalization, we measured orientation discrimination of Gabor targets under HDR luminance dynamics. We report a novel phenomenon, that abrupt 10- to 100-fold darkening engages contextual facilitation, distorting the apparent orientation of a high-contrast central target. Surprisingly, facilitation was influenced by grouping by luminance similarity, as well as by the degree of luminance variability in the surround. These results challenge vision models based solely on activity normalization and raise new questions that will lead to models that perform better in real-world scenes.

An Accurate and Robust Method for Spike Sorting Based on Convolutional Neural Networks

In the fields of neuroscience and biomedical signal processing, spike sorting is a crucial step to extract the information of single neurons from extracellular recordings. In this paper, we propose a novel deep learning approach based on one-dimensional convolutional neural networks (1D-CNNs) to implement accurate and robust spike sorting. The results of the simulated data demonstrated that the clustering accuracy in most datasets was greater than 99%, despite the multiple levels of noise and various degrees of overlapped spikes. Moreover, the proposed method performed significantly better than the state-of-the-art method named “WMsorting” and a deep-learning-based multilayer perceptron (MLP) model. In addition, the experimental data recorded from the primary visual cortex of a macaque monkey were used to evaluate the proposed method in a practical application. It was shown that the method could successfully isolate most spikes of different neurons (ranging from two to five) by training the 1D-CNN model with a small number of manually labeled spikes. Considering the above, the deep learning method proposed in this paper is of great advantage for spike sorting with high accuracy and strong robustness. It lays the foundation for application in more challenging works, such as distinguishing overlapped spikes and the simultaneous sorting of multichannel recordings.

A 100,000-to-1 high dynamic range (HDR) luminance display for investigating visual perception under real-world luminance dynamics

BACKGROUND

Real-world illumination challenges both autonomous sensing and displays, because scene luminance can vary by up to 10⁹-to-1, whereas vision models have limited ability to generalize beyond 100-to-1 luminance contrast. Brain mechanisms automatically normalize the visual input based on feature context, but they remain poorly understood because of the limitations of commercially available displays.

NEW METHOD

Here, we describe procedures for setup, calibration, and precision check of an HDR display system, based on a JVC DLA-RS600U reference projector, with over 100,000-to-1 luminance dynamic range (636-0.006055 cd/m²), pseudo 11 bit grayscale precision, and 3 ms temporal precision in the MATLAB/Psychtoolbox software environment. The setup is synchronized with electroencephalography (EEG) and infrared eye-tracking measurements.

RESULTS

We show display metrics including light scatter versus average display luminance (ADL), spatial uniformity, and spatial uniformity at high spatial frequency. We also show a luminance normalization phenomenon, contextual facilitation of a high contrast target, whose discovery required HDR display.

COMPARISON WITH EXISTING METHODS

This system provides 100-fold greater dynamic range than standard 1000-to-1 contrast displays and increases the number of gray levels from 256 or 1024 (8 or 10 bits) to 2048 (pseudo 11 bits), enabling the study of mesopic-to-photopic vision, at the expense of spatial non-uniformities.

CONCLUSIONS

This HDR research capability opens new questions of how visual perception is resilient to real-world luminance dynamics and will lead to improved visual modeling of dense urban and forest environments and of mixed indoor-outdoor environments such as cockpits and augmented reality. Our display metrics code can be found at https://github.com/USArmyResearchLab/ARL-Display-Metrics-and-Average-Display-Luminance.

Stimulus-Tuned Structure of Correlated fMRI Activity in Human Visual Cortex

Processing units are interconnected in the visual system, where a sensory organ and downstream cortical regions communicate through hierarchical connections, and local sites within the regions communicate through horizontal connections. In such networks, neural activities at local sites are likely to influence one another in complex ways and thus are intricately correlated. Recognizing the functional importance of correlated activity in sensory representation, spontaneous activities have been studied via diverse local or global measures in various time scales. Here, measuring functional magnetic resonance imaging (fMRI) signals in human early visual cortex, we explored systematic patterns that govern the correlated activities arising spontaneously. Specifically, guided by previously identified biases in anatomical connection patterns, we characterized all possible pairs of gray matter sites in 3 relational factors: "retinotopic distance," "cortical distance," and "stimulus tuning similarity." By evaluating and comparing the unique contributions of these factors to the correlated activity, we found that tuning similarity factors overrode distance factors in accounting for the structure of correlated fMRI activity both within and between V1, V2, and V3, irrespective of the presence or degree of visual stimulation. Our findings indicate that the early human visual cortex is intrinsically organized as a network tuned to the stimulus features.

A multi-scale layer-resolved spiking network model of resting-state dynamics in macaque visual cortical areas

Cortical activity has distinct features across scales, from the spiking statistics of individual cells to global resting-state networks. We here describe the first full-density multi-area spiking network model of cortex, using macaque visual cortex as a test system. The model represents each area by a microcircuit with area-specific architecture and features layer- and population-resolved connectivity between areas. Simulations reveal a structured asynchronous irregular ground state. In a metastable regime, the network reproduces spiking statistics from electrophysiological recordings and cortico-cortical interaction patterns in fMRI functional connectivity under resting-state conditions. Stable inter-area propagation is supported by cortico-cortical synapses that are moderately strong onto excitatory neurons and stronger onto inhibitory neurons. Causal interactions depend on both cortical structure and the dynamical state of populations. Activity propagates mainly in the feedback direction, similar to experimental results associated with visual imagery and sleep. The model unifies local and large-scale accounts of cortex, and clarifies how the detailed connectivity of cortex shapes its dynamics on multiple scales. Based on our simulations, we hypothesize that in the spontaneous condition the brain operates in a metastable regime where cortico-cortical projections target excitatory and inhibitory populations in a balanced manner that produces substantial inter-area interactions while maintaining global stability.

The mammalian cortex fulfills its complex tasks by operating on multiple temporal and spatial scales from single cells to entire areas comprising millions of cells. These multi-scale dynamics are supported by specific network structures at all levels of organization. Since models of cortex hitherto tend to concentrate on a single scale, little is known about how cortical structure shapes the multi-scale dynamics of the network. We here present dynamical simulations of a multi-area network model at neuronal and synaptic resolution with population-specific connectivity based on extensive experimental data which accounts for a wide range of dynamical phenomena. Our model elucidates relationships between local and global scales in cortex and provides a platform for future studies of cortical function.

Criteria on Balance, Stability, and Excitability in Cortical Networks for Constraining Computational Models

During ongoing and Up state activity, cortical circuits manifest a set of dynamical features that are conserved across these states. The present work systematizes these phenomena by three notions: excitability, the ability to sustain activity without external input; balance, precise coordination of excitatory and inhibitory neuronal inputs; and stability, maintenance of activity at a steady level. Slice preparations exhibiting Up states demonstrate that balanced activity can be maintained by small local circuits. While computational models of cortical circuits have included different combinations of excitability, balance, and stability, they have done so without a systematic quantitative comparison with experimental data. Our study provides quantitative criteria for this purpose, by analyzing in-vitro and in-vivo neuronal activity and characterizing the dynamics on the neuronal and population levels. The criteria are defined with a tolerance that allows for differences between experiments, yet are sufficient to capture commonalities between persistently depolarized cortical network states and to help validate computational models of cortex. As test cases for the derived set of criteria, we analyze three widely used models of cortical circuits and find that each model possesses some of the experimentally observed features, but none satisfies all criteria simultaneously, showing that the criteria are able to identify weak spots in computational models. The criteria described here form a starting point for the systematic validation of cortical neuronal network models, which will help improve the reliability of future models, and render them better building blocks for larger models of the brain.

The Primary Visual Cortex Is Differentially Modulated by Stimulus-Driven and Top-Down Attention

Selective attention can be focused either volitionally, by top-down signals derived from task demands, or automatically, by bottom-up signals from salient stimuli. Because the brain mechanisms that underlie these two attention processes are poorly understood, we recorded local field potentials (LFPs) from primary visual cortical areas of cats as they performed stimulus-driven and anticipatory discrimination tasks. Consistent with our previous observations, in both tasks, we found enhanced beta activity, which we have postulated may serve as an attention carrier. We characterized the functional organization of task-related beta activity by (i) cortical responses (EPs) evoked by electrical stimulation of the optic chiasm and (ii) intracortical LFP correlations. During the anticipatory task, peripheral stimulation that was preceded by high-amplitude beta oscillations evoked large-amplitude EPs compared with EPs that followed low-amplitude beta. In contrast, during the stimulus-driven task, cortical EPs preceded by high-amplitude beta oscillations were, on average, smaller than those preceded by low-amplitude beta. Analysis of the correlations between the different recording sites revealed that beta activation maps were heterogeneous during the bottom-up task and homogeneous for the top-down task. We conclude that bottom-up attention activates cortical visual areas in a mosaic-like pattern, whereas top-down attentional modulation results in spatially homogeneous excitation.

Network Anisotropy Trumps Noise for Efficient Object Coding in Macaque Inferior Temporal Cortex

How neuronal ensembles compute information is actively studied in early visual cortex. Much less is known about how local ensembles function in inferior temporal (IT) cortex, the last stage of the ventral visual pathway that supports visual recognition. Previous reports suggested that nearby neurons carry information mostly independently, supporting efficient processing (Barlow, 1961). However, others postulate that noise covariation effects may depend on network anisotropy/homogeneity and on how the covariation relates to representation. Do slow trial-by-trial noise covariations increase or decrease IT's object coding capability, how does encoding capability relate to correlational structure (i.e., the spatial pattern of signal and noise redundancy/homogeneity across neurons), and does knowledge of correlational structure matter for decoding? We recorded simultaneously from ∼80 spiking neurons in ∼1 mm(3) of macaque IT under light neurolept anesthesia. Noise correlations were stronger for neurons with correlated tuning, and noise covariations reduced object encoding capability, including generalization across object pose and illumination. Knowledge of noise covariations did not lead to better decoding performance. However, knowledge of anisotropy/homogeneity improved encoding and decoding efficiency by reducing the number of neurons needed to reach a given performance level. Such correlated neurons were found mostly in supragranular and infragranular layers, supporting theories that link recurrent circuitry to manifold representation. These results suggest that redundancy benefits manifold learning of complex high-dimensional information and that subsets of neurons may be more immune to noise covariation than others.

SIGNIFICANCE STATEMENT

How noise affects neuronal population coding is poorly understood. By sampling densely from local populations supporting visual object recognition, we show that recurrent circuitry supports useful representations and that subsets of neurons may be more immune to noise covariation than others.

Sloppiness in spontaneously active neuronal networks

Various plasticity mechanisms, including experience-dependent, spontaneous, as well as homeostatic ones, continuously remodel neural circuits. Yet, despite fluctuations in the properties of single neurons and synapses, the behavior and function of neuronal assemblies are generally found to be very stable over time. This raises the important question of how plasticity is coordinated across the network. To address this, we investigated the stability of network activity in cultured rat hippocampal neurons recorded with high-density multielectrode arrays over several days. We used parametric models to characterize multineuron activity patterns and analyzed their sensitivity to changes. We found that the models exhibited sloppiness, a property where the model behavior is insensitive to changes in many parameter combinations, but very sensitive to a few. The activity of neurons with sloppy parameters showed faster and larger fluctuations than the activity of a small subset of neurons associated with sensitive parameters. Furthermore, parameter sensitivity was highly correlated with firing rates. Finally, we tested our observations from cell cultures on an in vivo recording from monkey visual cortex and we confirm that spontaneous cortical activity also shows hallmarks of sloppy behavior and firing rate dependence. Our findings suggest that a small subnetwork of highly active and stable neurons supports group stability, and that this endows neuronal networks with the flexibility to continuously remodel without compromising stability and function.

Correlated activity supports efficient cortical processing

Visual recognition is a computational challenge that is thought to occur via efficient coding. An important concept is sparseness, a measure of coding efficiency. The prevailing view is that sparseness supports efficiency by minimizing redundancy and correlations in spiking populations. Yet, we recently reported that "choristers", neurons that behave more similarly (have correlated stimulus preferences and spontaneous coincident spiking), carry more generalizable object information than uncorrelated neurons ("soloists") in macaque inferior temporal (IT) cortex. The rarity of choristers (as low as 6% of IT neurons) indicates that they were likely missed in previous studies. Here, we report that correlation strength is distinct from sparseness (choristers are not simply broadly tuned neurons), that choristers are located in non-granular output layers, and that correlated activity predicts human visual search efficiency. These counterintuitive results suggest that a redundant correlational structure supports efficient processing and behavior.

Tuning and spontaneous spike time synchrony share a common structure in macaque inferior temporal cortex

Investigating the relationship between tuning and spike timing is necessary to understand how neuronal populations in anterior visual cortex process complex stimuli. Are tuning and spontaneous spike time synchrony linked by a common spatial structure (do some cells covary more strongly, even in the absence of visual stimulation?), and what is the object coding capability of this structure? Here, we recorded from spiking populations in macaque inferior temporal (IT) cortex under neurolept anesthesia. We report that, although most nearby IT neurons are weakly correlated, neurons with more similar tuning are also more synchronized during spontaneous activity. This link between tuning and synchrony was not simply due to cell separation distance. Instead, it expands on previous reports that neurons along an IT penetration are tuned to similar but slightly different features. This constraint on possible population firing rate patterns was consistent across stimulus sets, including animate vs. inanimate object categories. A classifier trained on this structure was able to generalize category "read-out" to untrained objects using only a few dimensions (a few patterns of site weightings per electrode array). We suggest that tuning and spike synchrony are linked by a common spatial structure that is highly efficient for object representation.