Innate visual learning through spontaneous activity patterns

Unsupervised learning on spontaneous retinal activity leads to efficient neural representation geometry

Prior to the onset of vision, neurons in the developing mammalian retina spontaneously fire in correlated activity patterns known as retinal waves. Experimental evidence suggests that retinal waves strongly influence the emergence of sensory representations before visual experience. We aim to model this early stage of functional development by using movies of neurally active developing retinas as pre-training data for neural networks. Specifically, we pre-train a ResNet-18 with an unsupervised contrastive learning objective (SimCLR) on both simulated and experimentally-obtained movies of retinal waves, then evaluate its performance on image classification tasks. We find that pre-training on retinal waves significantly improves performance on tasks that test object invariance to spatial translation, while slightly improving performance on more complex tasks like image classification. Notably, these performance boosts are realized on held-out natural images even though the pre-training procedure does not include any natural image data. We then propose a geometrical explanation for the increase in network performance, namely that the spatiotemporal characteristics of retinal waves facilitate the formation of separable feature representations. In particular, we demonstrate that networks pre-trained on retinal waves are more effective at separating image manifolds than randomly initialized networks, especially for manifolds defined by sets of spatial translations. These findings indicate that the broad spatiotemporal properties of retinal waves prepare networks for higher order feature extraction.

Not so spontaneous: Multi-dimensional representations of behaviors and context in sensory areas

Sensory areas are spontaneously active in the absence of sensory stimuli. This spontaneous activity has long been studied; however, its functional role remains largely unknown. Recent advances in technology, allowing large-scale neural recordings in the awake and behaving animal, have transformed our understanding of spontaneous activity. Studies using these recordings have discovered high-dimensional spontaneous activity patterns, correlation between spontaneous activity and behavior, and dissimilarity between spontaneous and sensory-driven activity patterns. These findings are supported by evidence from developing animals, where a transition toward these characteristics is observed as the circuit matures, as well as by evidence from mature animals across species. These newly revealed characteristics call for the formulation of a new role for spontaneous activity in neural sensory computation.

Principles of GABAergic signaling in developing cortical network dynamics

GABAergic signaling provides inhibitory stabilization and spatiotemporally coordinates the firing of recurrently connected excitatory neurons in mature cortical circuits. Inhibition thus enables self-generated neuronal activity patterns that underlie various aspects of sensation and cognition. In this review, we aim to provide a conceptual framework describing how and when GABA-releasing interneurons acquire their network functions during development. Focusing on the developing visual neocortex and hippocampus in mice and rats in vivo, we hypothesize that at the onset of patterned activity, glutamatergic neurons are stable by themselves and inhibitory stabilization is not yet functional. We review important milestones in the development of GABAergic signaling and illustrate how the cell-type-specific strengthening of synaptic inhibition toward eye opening shapes cortical network dynamics and allows the developing cortex to progressively disengage from extra-cortical synaptic drive. We translate this framework to human cortical development and discuss clinical implications for the treatment of neonatal seizures.

Neocortex saves energy by reducing coding precision during food scarcity

Information processing is energetically expensive. In the mammalian brain, it is unclear how information coding and energy use are regulated during food scarcity. Using whole-cell recordings and two-photon imaging in layer 2/3 mouse visual cortex, we found that food restriction reduced AMPA receptor conductance, reducing synaptic ATP use by 29%. Neuronal excitability was nonetheless preserved by a compensatory increase in input resistance and a depolarized resting potential. Consequently, neurons spiked at similar rates as controls but spent less ATP on underlying excitatory currents. This energy-saving strategy had a cost because it amplified the variability of visually-evoked subthreshold responses, leading to a 32% broadening of orientation tuning and impaired fine visual discrimination. This reduction in coding precision was associated with reduced levels of the fat mass-regulated hormone leptin and was restored by exogenous leptin supplementation. Our findings reveal that metabolic state dynamically regulates the energy spent on coding precision in neocortex.

Network instability dynamics drive a transient bursting period in the developing hippocampus in vivo

Spontaneous correlated activity is a universal hallmark of immature neural circuits. However, the cellular dynamics and intrinsic mechanisms underlying network burstiness in the intact developing brain are largely unknown. Here, we use two-photon Ca²⁺ imaging to comprehensively map the developmental trajectories of spontaneous network activity in the hippocampal area CA1 of mice in vivo. We unexpectedly find that network burstiness peaks after the developmental emergence of effective synaptic inhibition in the second postnatal week. We demonstrate that the enhanced network burstiness reflects an increased functional coupling of individual neurons to local population activity. However, pairwise neuronal correlations are low, and network bursts (NBs) recruit CA1 pyramidal cells in a virtually random manner. Using a dynamic systems modeling approach, we reconcile these experimental findings and identify network bi-stability as a potential regime underlying network burstiness at this age. Our analyses reveal an important role of synaptic input characteristics and network instability dynamics for NB generation. Collectively, our data suggest a mechanism, whereby developing CA1 performs extensive input-discrimination learning prior to the onset of environmental exploration.

On the Role of LGN/V1 Spontaneous Activity as an Innate Learning Pattern for Visual Development

Correlated, spontaneous neural activity is known to play a necessary role in visual development, but the higher-order statistical structure of these coherent, amorphous patterns has only begun to emerge in the past decade. Several computational studies have demonstrated how this endogenous activity can be used to train a developing visual system. Models that generate spontaneous activity analogous to retinal waves have shown that these waves can serve as stimuli for efficient coding models of V1. This general strategy in development has one clear advantage: The same learning algorithm can be used both before and after eye-opening. This same insight can be applied to understanding LGN/V1 spontaneous activity. Although lateral geniculate nucleus (LGN) activity has been less discussed in the literature than retinal waves, here we argue that the waves found in the LGN have a number of properties that fill the role of a training pattern. We make the case that the role of “innate learning” with spontaneous activity is not only possible, but likely in later stages of visual development, and worth pursuing further using an efficient coding paradigm.

Unsupervised learning in images and audio to produce neural receptive fields: a primer and accessible notebook

Sensory processing relies on efficient computation driven by a combination of low-level unsupervised, statistical structural learning, and high-level task-dependent learning. In the earliest stages of sensory processing, sparse and independent coding strategies are capable of modeling neural processing using the same coding strategy with only a change in the input (e.g., grayscale images, color images, and audio). We present a consolidated review of Independent Component Analysis (ICA) as an efficient neural coding scheme with the ability to model early visual and auditory neural processing. We created a self-contained, accessible Jupyter notebook using Python to demonstrate the efficient coding principle for different modalities following a consistent five-step strategy. For each modality, derived receptive field models from natural and non-natural inputs are contrasted, demonstrating how neural codes are not produced when the inputs sufficiently deviate from those animals were evolved to process. Additionally, the demonstration shows that ICA produces more neurally-appropriate receptive field models than those based on common compression strategies, such as Principal Component Analysis. The five-step strategy not only produces neural-like models but also promotes reuse of code to emphasize the input-agnostic nature where each modality can be modeled with only a change in inputs. This notebook can be used to readily observe the links between unsupervised machine learning strategies and early sensory neuroscience, improving our understanding of flexible data-driven neural development in nature and future applications.

The development of active binocular vision under normal and alternate rearing conditions

The development of binocular vision is an active learning process comprising the development of disparity tuned neurons in visual cortex and the establishment of precise vergence control of the eyes. We present a computational model for the learning and self-calibration of active binocular vision based on the Active Efficient Coding framework, an extension of classic efficient coding ideas to active perception. Under normal rearing conditions with naturalistic input, the model develops disparity tuned neurons and precise vergence control, allowing it to correctly interpret random dot stereograms. Under altered rearing conditions modeled after neurophysiological experiments, the model qualitatively reproduces key experimental findings on changes in binocularity and disparity tuning. Furthermore, the model makes testable predictions regarding how altered rearing conditions impede the learning of precise vergence control. Finally, the model predicts a surprising new effect that impaired vergence control affects the statistics of orientation tuning in visual cortical neurons.

Somatostatin interneurons restrict cell recruitment to retinally driven spontaneous activity in the developing cortex

During early development, before the eyes open, synaptic refinement of sensory networks depends on activity generated by developing neurons themselves. In the mouse visual system, retinal cells spontaneously depolarize and recruit downstream neurons to bursts of activity, where the number of recruited cells determines the resolution of synaptic retinotopic refinement. Here we show that during the second post-natal week in mouse visual cortex, somatostatin (SST)-expressing interneurons control the recruitment of cells to retinally driven spontaneous activity. Suppressing SST interneurons increases cell participation and allows events to spread farther along the cortex. During the same developmental period, a second type of high-participation, retina-independent event occurs. During these events, cells receive such large excitatory charge that inhibition is overwhelmed and large parts of the cortex participate in each burst. These results reveal a role of SST interneurons in restricting retinally driven activity in the visual cortex, which may contribute to the refinement of retinotopy.

Unsupervised experience with temporal continuity of the visual environment is causally involved in the development of V1 complex cells

Unsupervised adaptation to the spatiotemporal statistics of visual experience is a key computational principle that has long been assumed to govern postnatal development of visual cortical tuning, including orientation selectivity of simple cells and position tolerance of complex cells in primary visual cortex (V1). Yet, causal empirical evidence supporting this hypothesis is scant. Here, we show that degrading the temporal continuity of visual experience during early postnatal life leads to a sizable reduction of the number of complex cells and to an impairment of their functional properties while fully sparing the development of simple cells. This causally implicates adaptation to the temporal structure of the visual input in the development of transformation tolerance but not of shape tuning, thus tightly constraining computational models of unsupervised cortical learning.

Theoretical Models of Neural Development

Constructing a functioning nervous system requires the precise orchestration of a vast array of mechanical, molecular, and neural-activity-dependent cues. Theoretical models can play a vital role in helping to frame quantitative issues, reveal mathematical commonalities between apparently diverse systems, identify what is and what is not possible in principle, and test the abilities of specific mechanisms to explain the data. This review focuses on the progress that has been made over the last decade in our theoretical understanding of neural development.

Spontaneous Activity in the Zebrafish Tectum Reorganizes over Development and Is Influenced by Visual Experience

Spontaneous patterns of activity in the developing visual system may play an important role in shaping the brain for function. During the period 4-9 dpf (days post-fertilization), larval zebrafish learn to hunt prey, a behavior that is critically dependent on the optic tectum. However, how spontaneous activity develops in the tectum over this period and the effect of visual experience are unknown. Here we performed two-photon calcium imaging of GCaMP6s zebrafish larvae at all days from 4 to 9 dpf. Using recently developed graph theoretic techniques, we found significant changes in both single-cell and population activity characteristics over development. In particular, we identified days 5-6 as a critical moment in the reorganization of the underlying functional network. Altering visual experience early in development altered the statistics of tectal activity, and dark rearing also caused a long-lasting deficit in the ability to capture prey. Thus, tectal development is shaped by both intrinsic factors and visual experience.

Simple cubic random-site percolation thresholds for neighborhoods containing fourth-nearest neighbors

In this paper, random-site percolation thresholds for a simple cubic (SC) lattice with site neighborhoods containing next-next-next-nearest neighbors (4NN) are evaluated with Monte Carlo simulations. A recently proposed algorithm with low sampling for percolation thresholds estimation (Bastas et al., arXiv:1411.5834) is implemented for the studies of the top-bottom wrapping probability. The obtained percolation thresholds are p(C)(4NN)=0.31160(12),p(C)(4NN+NN)=0.15040(12),p(C)(4NN+2NN)=0.15950(12),p(C)(4NN+3NN)=0.20490(12),p(C)(4NN+2NN+NN)=0.11440(12),p(C)(4NN+3NN+NN)=0.11920(12),p(C)(4NN+3NN+2NN)=0.11330(12), and p(C)(4NN+3NN+2NN+NN)=0.10000(12), where 3NN, 2NN, and NN stand for next-next-nearest neighbors, next-nearest neighbors, and nearest neighbors, respectively. As an SC lattice with 4NN neighbors may be mapped onto two independent interpenetrated SC lattices but with a lattice constant that is twice as large, the percolation threshold p(C)(4NN) is exactly equal to p(C)(NN). The simplified method of Bastas et al. allows for uncertainty of the percolation threshold value p(C) to be reached, similar to that obtained with the classical method but ten times faster.

Slow feature analysis on retinal waves leads to V1 complex cells

The developing visual system of many mammalian species is partially structured and organized even before the onset of vision. Spontaneous neural activity, which spreads in waves across the retina, has been suggested to play a major role in these prenatal structuring processes. Recently, it has been shown that when employing an efficient coding strategy, such as sparse coding, these retinal activity patterns lead to basis functions that resemble optimal stimuli of simple cells in primary visual cortex (V1). Here we present the results of applying a coding strategy that optimizes for temporal slowness, namely Slow Feature Analysis (SFA), to a biologically plausible model of retinal waves. Previously, SFA has been successfully applied to model parts of the visual system, most notably in reproducing a rich set of complex-cell features by training SFA with quasi-natural image sequences. In the present work, we obtain SFA units that share a number of properties with cortical complex-cells by training on simulated retinal waves. The emergence of two distinct properties of the SFA units (phase invariance and orientation tuning) is thoroughly investigated via control experiments and mathematical analysis of the input-output functions found by SFA. The results support the idea that retinal waves share relevant temporal and spatial properties with natural visual input. Hence, retinal waves seem suitable training stimuli to learn invariances and thereby shape the developing early visual system such that it is best prepared for coding input from the natural world.

Sparse coding can predict primary visual cortex receptive field changes induced by abnormal visual input

Receptive fields acquired through unsupervised learning of sparse representations of natural scenes have similar properties to primary visual cortex (V1) simple cell receptive fields. However, what drives in vivo development of receptive fields remains controversial. The strongest evidence for the importance of sensory experience in visual development comes from receptive field changes in animals reared with abnormal visual input. However, most sparse coding accounts have considered only normal visual input and the development of monocular receptive fields. Here, we applied three sparse coding models to binocular receptive field development across six abnormal rearing conditions. In every condition, the changes in receptive field properties previously observed experimentally were matched to a similar and highly faithful degree by all the models, suggesting that early sensory development can indeed be understood in terms of an impetus towards sparsity. As previously predicted in the literature, we found that asymmetries in inter-ocular correlation across orientations lead to orientation-specific binocular receptive fields. Finally we used our models to design a novel stimulus that, if present during rearing, is predicted by the sparsity principle to lead robustly to radically abnormal receptive fields.

The responses of neurons in the primary visual cortex (V1), a region of the brain involved in encoding visual input, are modified by the visual experience of the animal during development. For example, most neurons in animals reared viewing stripes of a particular orientation only respond to the orientation that the animal experienced. The responses of V1 cells in normal animals are similar to responses that simple optimisation algorithms can learn when trained on images. However, whether the similarity between these algorithms and V1 responses is merely coincidental has been unclear. Here, we used the results of a number of experiments where animals were reared with modified visual experience to test the explanatory power of three related optimisation algorithms. We did this by filtering the images for the algorithms in ways that mimicked the visual experience of the animals. This allowed us to show that the changes in V1 responses in experiment were consistent with the algorithms. This is evidence that the precepts of the algorithms, notably sparsity, can be used to understand the development of V1 responses. Further, we used our model to propose a novel rearing condition which we expect to have a dramatic effect on development.

On the dynamics of cortical development: synchrony and synaptic self-organization

We describe a model for cortical development that resolves long-standing difficulties of earlier models. It is proposed that, during embryonic development, synchronous firing of neurons and their competition for limited metabolic resources leads to selection of an array of neurons with ultra-small-world characteristics. Consequently, in the visual cortex, macrocolumns linked by superficial patchy connections emerge in anatomically realistic patterns, with an ante-natal arrangement which projects signals from the surrounding cortex onto each macrocolumn in a form analogous to the projection of a Euclidean plane onto a Möbius strip. This configuration reproduces typical cortical response maps, and simulations of signal flow explain cortical responses to moving lines as functions of stimulus velocity, length, and orientation. With the introduction of direct visual inputs, under the operation of Hebbian learning, development of mature selective response “tuning” to stimuli of given orientation, spatial frequency, and temporal frequency would then take place, overwriting the earlier ante-natal configuration. The model is provisionally extended to hierarchical interactions of the visual cortex with higher centers, and a general principle for cortical processing of spatio-temporal images is sketched.

Sparse coding on the spot: spontaneous retinal waves suffice for orientation selectivity

Ohshiro, Hussain, and Weliky (2011) recently showed that ferrets reared with exposure to flickering spot stimuli, in the absence of oriented visual experience, develop oriented receptive fields. They interpreted this as refutation of efficient coding models, which require oriented input in order to develop oriented receptive fields. Here we show that these data are compatible with the efficient coding hypothesis if the influence of spontaneous retinal waves is considered. We demonstrate that independent component analysis learns predominantly oriented receptive fields when trained on a mixture of spot stimuli and spontaneous retinal waves. Further, we show that the efficient coding hypothesis provides a compelling explanation for the contrast between the lack of receptive field changes seen in animals reared with spot stimuli and the significant cortical reorganisation observed in stripe-reared animals.

A theory of slow feature analysis for transformation-based input signals with an application to complex cells

We develop a group-theoretical analysis of slow feature analysis for the case where the input data are generated by applying a set of continuous transformations to static templates. As an application of the theory, we analytically derive nonlinear visual receptive fields and show that their optimal stimuli, as well as the orientation and frequency tuning, are in good agreement with previous simulations of complex cells in primary visual cortex (Berkes and Wiskott, 2005). The theory suggests that side and end stopping can be interpreted as a weak breaking of translation invariance. Direction selectivity is also discussed.

How Infants Learn About the Visual World

The visual world of adults consists of objects at various distances, partly occluding one another, substantial and stable across space and time. The visual world of young infants, in contrast, is often fragmented and unstable, consisting not of coherent objects but rather surfaces that move in unpredictable ways. Evidence from computational modeling and from experiments with human infants highlights three kinds of learning that contribute to infants' knowledge of the visual world: learning via association, learning via active assembly, and learning via visual-manual exploration. Infants acquire knowledge by observing objects move in and out of sight, forming associations of these different views. In addition, the infant's own self-produced behavior-oculomotor patterns and manual experience, in particular-are important means by which infants discover and construct their visual world.

Effects of early pattern deprivation on visual development

Studies of children treated for dense cataract shed light on the extent to which pattern stimulation drives normal visual development and whether there are sensitive periods during which an abnormal visual environment is especially detrimental. Here, we summarize the findings to date into five general principles: (1) At least for low-level vision, aspects of vision that develop the earliest are the least likely to be adversely affected by abnormal visual input whereas those that develop later are affected more severely. (2) Early visual input is necessary to preserve the neural infrastructure for later visual learning, even for visual capabilities that will not appear until later in development. (3) The development of both the dorsal and ventral streams depends on normal visual input. (4) After monocular deprivation has been treated by surgical removal of the cataractous lens, the interactions between the aphakic and phakic eyes are competitive for low-level vision but are complementary for high-level vision. (5) There are multiple sensitive periods during which experience can influence visual development.The studies described here have important implications for understanding normal development. They indicate that patterned visual input immediately after birth plays a vital role in the construction and preservation of the neural architecture that will later mediate sensitivity to both basic and higher level aspects of vision. The period during which patterned visual input is necessary for normal visual development varies widely across different aspects of vision and can range from only a few months after birth to more than the first 10 years of life. The results point to new research questions on why early visual deprivation can cause later deficits, what limits adult plasticity, and whether effective rehabilitation in other areas can provide new clues for the treatment of amblyopia.

Sparsification of neuronal activity in the visual cortex at eye-opening

Eye-opening represents a turning point in the function of the visual cortex. Before eye-opening, the visual cortex is largely devoid of sensory inputs and neuronal activities are generated intrinsically. After eye-opening, the cortex starts to integrate visual information. Here we used in vivo two-photon calcium imaging to explore the developmental changes of the mouse visual cortex by analyzing the ongoing spontaneous activity. We found that before eye-opening, the activity of layer 2/3 neurons consists predominantly of slow wave oscillations. These waves were first detected at postnatal day 8 (P8). Their initial very low frequency (0.01 Hz) gradually increased during development to approximately 0.5 Hz in adults. Before eye-opening, a large fraction of neurons (>75%) was active during each wave. One day after eye-opening, this dense mode of recruitment changed to a sparse mode with only 36% of active neurons per wave. This was followed by a progressive decrease during the following weeks, reaching 12% of active neurons per wave in adults. The possible role of visual experience for this process of sparsification was investigated by analyzing dark-reared mice. We found that sparsification also occurred in these mice, but that the switch from a dense to a sparse activity pattern was delayed by 3-4 days as compared with normally-reared mice. These results reveal a modulatory contribution of visual experience during the first days after eye-opening, but an overall dominating role of intrinsic factors. We propose that the transformation in network activity from dense to sparse is a prerequisite for the changed cortical function at eye-opening.