Ignoring correlated activity causes a failure of retinal population codes

Biophysical neural adaptation mechanisms enable artificial neural networks to capture dynamic retinal computation

Adaptation is a universal aspect of neural systems that changes circuit computations to match prevailing inputs. These changes facilitate efficient encoding of sensory inputs while avoiding saturation. Conventional artificial neural networks (ANNs) have limited adaptive capabilities, hindering their ability to reliably predict neural output under dynamic input conditions. Can embedding neural adaptive mechanisms in ANNs improve their performance? To answer this question, we develop a new deep learning model of the retina that incorporates the biophysics of photoreceptor adaptation at the front-end of conventional convolutional neural networks (CNNs). These conventional CNNs build on 'Deep Retina,' a previously developed model of retinal ganglion cell (RGC) activity. CNNs that include this new photoreceptor layer outperform conventional CNN models at predicting male and female primate and rat RGC responses to naturalistic stimuli that include dynamic local intensity changes and large changes in the ambient illumination. These improved predictions result directly from adaptation within the phototransduction cascade. This research underscores the potential of embedding models of neural adaptation in ANNs and using them to determine how neural circuits manage the complexities of encoding natural inputs that are dynamic and span a large range of light levels.

Advancements in brain-machine interfaces for application in the metaverse

In recent years, with the shift of focus in metaverse research toward content exchange and social interaction, breaking through the current bottleneck of audio-visual media interaction has become an urgent issue. The use of brain-machine interfaces for sensory simulation is one of the proposed solutions. Currently, brain-machine interfaces have demonstrated irreplaceable potential as physiological signal acquisition tools in various fields within the metaverse. This study explores three application scenarios: generative art in the metaverse, serious gaming for healthcare in metaverse medicine, and brain-machine interface applications for facial expression synthesis in the virtual society of the metaverse. It investigates existing commercial products and patents (such as MindWave Mobile, GVS, and Galea), draws analogies with the development processes of network security and neurosecurity, bioethics and neuroethics, and discusses the challenges and potential issues that may arise when brain-machine interfaces mature and are widely applied. Furthermore, it looks ahead to the diverse possibilities of deep and varied applications of brain-machine interfaces in the metaverse in the future.

Calibrating Bayesian Decoders of Neural Spiking Activity

Accurately decoding external variables from observations of neural activity is a major challenge in systems neuroscience. Bayesian decoders, which provide probabilistic estimates, are some of the most widely used. Here we show how, in many common settings, the probabilistic predictions made by traditional Bayesian decoders are overconfident. That is, the estimates for the decoded stimulus or movement variables are more certain than they should be. We then show how Bayesian decoding with latent variables, taking account of low-dimensional shared variability in the observations, can improve calibration, although additional correction for overconfidence is still needed. Using data from males, we examine (1) decoding the direction of grating stimuli from spike recordings in the primary visual cortex in monkeys, (2) decoding movement direction from recordings in the primary motor cortex in monkeys, (3) decoding natural images from multiregion recordings in mice, and (4) decoding position from hippocampal recordings in rats. For each setting, we characterize the overconfidence, and we describe a possible method to correct miscalibration post hoc. Properly calibrated Bayesian decoders may alter theoretical results on probabilistic population coding and lead to brain-machine interfaces that more accurately reflect confidence levels when identifying external variables.

Small-correlation expansion to quantify information in noisy sensory systems

Neural networks encode information through their collective spiking activity in response to external stimuli. This population response is noisy and strongly correlated, with a complex interplay between correlations induced by the stimulus, and correlations caused by shared noise. Understanding how these correlations affect information transmission has so far been limited to pairs or small groups of neurons, because the curse of dimensionality impedes the evaluation of mutual information in larger populations. Here, we develop a small-correlation expansion to compute the stimulus information carried by a large population of neurons, yielding interpretable analytical expressions in terms of the neurons' firing rates and pairwise correlations. We validate the approximation on synthetic data and demonstrate its applicability to electrophysiological recordings in the vertebrate retina, allowing us to quantify the effects of noise correlations between neurons and of memory in single neurons.

Dynamic structure of motor cortical neuron coactivity carries behaviorally relevant information

Skillful, voluntary movements are underpinned by computations performed by networks of interconnected neurons in the primary motor cortex (M1). Computations are reflected by patterns of coactivity between neurons. Using pairwise spike time statistics, coactivity can be summarized as a functional network (FN). Here, we show that the structure of FNs constructed from an instructed-delay reach task in nonhuman primates is behaviorally specific: Low-dimensional embedding and graph alignment scores show that FNs constructed from closer target reach directions are also closer in network space. Using short intervals across a trial, we constructed temporal FNs and found that temporal FNs traverse a low-dimensional subspace in a reach-specific trajectory. Alignment scores show that FNs become separable and correspondingly decodable shortly after the Instruction cue. Finally, we observe that reciprocal connections in FNs transiently decrease following the Instruction cue, consistent with the hypothesis that information external to the recorded population temporarily alters the structure of the network at this moment.

Large-scale interrogation of retinal cell functions by 1-photon light-sheet microscopy

Visual processing in the retina depends on the collective activity of large ensembles of neurons organized in different layers. Current techniques for measuring activity of layer-specific neural ensembles rely on expensive pulsed infrared lasers to drive 2-photon activation of calcium-dependent fluorescent reporters. We present a 1-photon light-sheet imaging system that can measure the activity in hundreds of neurons in the ex vivo retina over a large field of view while presenting visual stimuli. This allows for a reliable functional classification of different retinal cell types. We also demonstrate that the system has sufficient resolution to image calcium entry at individual synaptic release sites across the axon terminals of dozens of simultaneously imaged bipolar cells. The simple design, large field of view, and fast image acquisition make this a powerful system for high-throughput and high-resolution measurements of retinal processing at a fraction of the cost of alternative approaches.

Differences in the spatial fidelity of evoked and spontaneous signals in the degenerating retina

Vision restoration strategies aim to reestablish vision by replacing the function of lost photoreceptors with optoelectronic hardware or through gene therapy. One complication to these approaches is that retinal circuitry undergoes remodeling after photoreceptor loss. Circuit remodeling following perturbation is ubiquitous in the nervous system and understanding these changes is crucial for treating neurodegeneration. Spontaneous oscillations that arise during retinal degeneration have been well-studied, however, other changes in the spatiotemporal processing of evoked and spontaneous activity have received less attention. Here we use subretinal electrical stimulation to measure the spatial and temporal spread of both spontaneous and evoked activity during retinal degeneration. We found that electrical stimulation synchronizes spontaneous oscillatory activity, over space and through time, thus leading to increased correlations in ganglion cell activity. Intriguingly, we found that spatial selectivity was maintained in rd10 retina for evoked responses, with spatial receptive fields comparable to wt retina. These findings indicate that different biophysical mechanisms are involved in mediating feed forward excitation, and the lateral spread of spontaneous activity in the rd10 retina, lending support toward the possibility of high-resolution vision restoration.

Retinal Processing: Insights from Mathematical Modelling

The retina is the entrance of the visual system. Although based on common biophysical principles, the dynamics of retinal neurons are quite different from their cortical counterparts, raising interesting problems for modellers. In this paper, I address some mathematically stated questions in this spirit, discussing, in particular: (1) How could lateral amacrine cell connectivity shape the spatio-temporal spike response of retinal ganglion cells? (2) How could spatio-temporal stimuli correlations and retinal network dynamics shape the spike train correlations at the output of the retina? These questions are addressed, first, introducing a mathematically tractable model of the layered retina, integrating amacrine cells' lateral connectivity and piecewise linear rectification, allowing for computing the retinal ganglion cells receptive field together with the voltage and spike correlations of retinal ganglion cells resulting from the amacrine cells networks. Then, I review some recent results showing how the concept of spatio-temporal Gibbs distributions and linear response theory can be used to characterize the collective spike response to a spatio-temporal stimulus of a set of retinal ganglion cells, coupled via effective interactions corresponding to the amacrine cells network. On these bases, I briefly discuss several potential consequences of these results at the cortical level.

Modelling the neural code in large populations of correlated neurons

Neurons respond selectively to stimuli, and thereby define a code that associates stimuli with population response patterns. Certain correlations within population responses (noise correlations) significantly impact the information content of the code, especially in large populations. Understanding the neural code thus necessitates response models that quantify the coding properties of modelled populations, while fitting large-scale neural recordings and capturing noise correlations. In this paper, we propose a class of response model based on mixture models and exponential families. We show how to fit our models with expectation-maximization, and that they capture diverse variability and covariability in recordings of macaque primary visual cortex. We also show how they facilitate accurate Bayesian decoding, provide a closed-form expression for the Fisher information, and are compatible with theories of probabilistic population coding. Our framework could allow researchers to quantitatively validate the predictions of neural coding theories against both large-scale neural recordings and cognitive performance.

Nonlinear Decoding of Natural Images From Large-Scale Primate Retinal Ganglion Recordings

Decoding sensory stimuli from neural activity can provide insight into how the nervous system might interpret the physical environment, and facilitates the development of brain-machine interfaces. Nevertheless, the neural decoding problem remains a significant open challenge. Here, we present an efficient nonlinear decoding approach for inferring natural scene stimuli from the spiking activities of retinal ganglion cells (RGCs). Our approach uses neural networks to improve on existing decoders in both accuracy and scalability. Trained and validated on real retinal spike data from more than 1000 simultaneously recorded macaque RGC units, the decoder demonstrates the necessity of nonlinear computations for accurate decoding of the fine structures of visual stimuli. Specifically, high-pass spatial features of natural images can only be decoded using nonlinear techniques, while low-pass features can be extracted equally well by linear and nonlinear methods. Together, these results advance the state of the art in decoding natural stimuli from large populations of neurons.

Variations in photoreceptor throughput to mouse visual cortex and the unique effects on tuning

Visual input to primary visual cortex (V1) depends on highly adaptive filtering in the retina. In turn, isolation of V1 computations requires experimental control of retinal adaptation to infer its spatio-temporal-chromatic output. Here, we measure the balance of input to mouse V1, in the anesthetized setup, from the three main photoreceptor opsins-M-opsin, S-opsin, and rhodopsin-as a function of two stimulus dimensions. The first dimension is the level of light adaptation within the mesopic range, which governs the balance of rod and cone inputs to cortex. The second stimulus dimension is retinotopic position, which governs the balance of S- and M-cone opsin input due to the opsin expression gradient in the retina. The fitted model predicts opsin input under arbitrary lighting environments, which provides a much-needed handle on in-vivo studies of the mouse visual system. We use it here to reveal that V1 is rod-mediated in common laboratory settings yet cone-mediated in natural daylight. Next, we compare functional properties of V1 under rod and cone-mediated inputs. The results show that cone-mediated V1 responds to 2.5-fold higher temporal frequencies than rod-mediated V1. Furthermore, cone-mediated V1 has smaller receptive fields, yet similar spatial frequency tuning. V1 responses in rod-deficient (Gnat1-/-) mice confirm that the effects are due to differences in photoreceptor opsin contribution.

Reconstruction of natural images from responses of primate retinal ganglion cells

The visual message conveyed by a retinal ganglion cell (RGC) is often summarized by its spatial receptive field, but in principle also depends on the responses of other RGCs and natural image statistics. This possibility was explored by linear reconstruction of natural images from responses of the four numerically-dominant macaque RGC types. Reconstructions were highly consistent across retinas. The optimal reconstruction filter for each RGC - its visual message - reflected natural image statistics, and resembled the receptive field only when nearby, same-type cells were included. ON and OFF cells conveyed largely independent, complementary representations, and parasol and midget cells conveyed distinct features. Correlated activity and nonlinearities had statistically significant but minor effects on reconstruction. Simulated reconstructions, using linear-nonlinear cascade models of RGC light responses that incorporated measured spatial properties and nonlinearities, produced similar results. Spatiotemporal reconstructions exhibited similar spatial properties, suggesting that the results are relevant for natural vision.