How close are we to understanding v1?

Uncovering 2-D toroidal representations in grid cell ensemble activity during 1-D behavior

Minimal experiments, such as head-fixed wheel-running and sleep, offer experimental advantages but restrict the amount of observable behavior, making it difficult to classify functional cell types. Arguably, the grid cell, and its striking periodicity, would not have been discovered without the perspective provided by free behavior in an open environment. Here, we show that by shifting the focus from single neurons to populations, we change the minimal experimental complexity required. We identify grid cell modules and show that the activity covers a similar, stable toroidal state space during wheel running as in open field foraging. Trajectories on grid cell tori correspond to single trial runs in virtual reality and path integration in the dark, and the alignment of the representation rapidly shifts with changes in experimental conditions. Thus, we provide a methodology to discover and study complex internal representations in even the simplest of experiments.

More than expected: extracellular waveforms and functional responses in monkey LGN

Unlike the exhaustive determination of cell types in the retina, key populations in the lateral geniculate nucleus of the thalamus (LGN) may have been missed. Here, we have begun to characterize the full range of extracellular neuronal responses in the LGN of awake monkeys using multi-electrodes during the presentation of colored noise visual stimuli to identify any previously overlooked signals. Extracellular spike waveforms of single units were classified into seven distinct classes, revealing previously unrecognized diversity: four negative-dominant classes that were narrow or broad, one triphasic class, and two positive-dominant classes. Based on their mapped receptive field (RF), these units were further categorized into either magnocellular (M), parvocellular (P), koniocellular (K), or non-RF (N). We found correlations between spike shape and mapped RF and response characteristics, with negative and narrow spiking waveform units predominantly associated with P and N RFs, and positive waveforms mostly linked to M RFs. Responses from positive waveforms exhibited shorter latencies, larger RF sizes, and were associated with larger eccentricities in the visual field than the other waveform classes. Additionally, N cells, those without an estimated RF, were consistently responsive to the visually presented mapping stimulus at a lower and more sustained rate than units with an RF. These findings suggest that the LGN cell population may be more diverse than previously believed.

Deep neural networks are not a single hypothesis but a language for expressing computational hypotheses

An ideal vision model accounts for behavior and neurophysiology in both naturalistic conditions and designed lab experiments. Unlike psychological theories, artificial neural networks (ANNs) actually perform visual tasks and generate testable predictions for arbitrary inputs. These advantages enable ANNs to engage the entire spectrum of the evidence. Failures of particular models drive progress in a vibrant ANN research program of human vision.

Normalization by orientation-tuned surround in human V1-V3

An influential account of neuronal responses in primary visual cortex is the normalized energy model. This model is often implemented as a multi-stage computation. The first stage is linear filtering. The second stage is the extraction of contrast energy, whereby a complex cell computes the squared and summed outputs of a pair of the linear filters in quadrature phase. The third stage is normalization, in which a local population of complex cells mutually inhibit one another. Because the population includes cells tuned to a range of orientations and spatial frequencies, the result is that the responses are effectively normalized by the local stimulus contrast. Here, using evidence from human functional MRI, we show that the classical model fails to account for the relative responses to two classes of stimuli: straight, parallel, band-passed contours (gratings), and curved, band-passed contours (snakes). The snakes elicit fMRI responses that are about twice as large as the gratings, yet a traditional divisive normalization model predicts responses that are about the same. Motivated by these observations and others from the literature, we implement a divisive normalization model in which cells matched in orientation tuning ("tuned normalization") preferentially inhibit each other. We first show that this model accounts for differential responses to these two classes of stimuli. We then show that the model successfully generalizes to other band-pass textures, both in V1 and in extrastriate cortex (V2 and V3). We conclude that even in primary visual cortex, complex features of images such as the degree of heterogeneity, can have large effects on neural responses.

Enhancing performance of electron holography with mathematical and machine learning-based denoising techniques

Electron holography is a useful tool for analyzing functional properties, such as electromagnetic fields and strains of materials and devices. The performance of electron holography is limited by the 'shot noise' inherent in electron micrographs (holograms), which are composed of a finite number of electrons. A promising approach for addressing this issue is to use mathematical and machine learning-based image-processing techniques for hologram denoising. With the advancement of information science, denoising methods have become capable of extracting signals that are completely buried in noise, and they are being applied to electron microscopy, including electron holography. However, these advanced denoising methods are complex and have many parameters to be tuned; therefore, it is necessary to understand their principles in depth and use them carefully. Herein, we present an overview of the principles and usage of sparse coding, the wavelet hidden Markov model and tensor decomposition, which have been applied to electron holography. We also present evaluation results for the denoising performance of these methods obtained through their application to simulated and experimentally recorded holograms. Our analysis, review and comparison of the methods clarify the impact of denoising on electron holography research.

Compact deep neural network models of visual cortex

A powerful approach to understanding the computations carried out in visual cortex is to develop models that predict neural responses to arbitrary images. Deep neural network (DNN) models have worked remarkably well at predicting neural responses [1, 2, 3], yet their underlying computations remain buried in millions of parameters. Have we simply replaced one complicated system in vivo with another in silico ? Here, we train a data-driven deep ensemble model that predicts macaque V4 responses ∼50% more accurately than currently-used task-driven DNN models. We then compress this deep ensemble to identify compact models that have 5,000x fewer parameters yet equivalent accuracy as the deep ensemble. We verified that the stimulus preferences of the compact models matched those of the real V4 neurons by measuring V4 responses to both 'maximizing' and adversarial images generated using compact models. We then analyzed the inner workings of the compact models and discovered a common circuit motif: Compact models share a similar set of filters in early stages of processing but then specialize by heavily consolidating this shared representation with a precise readout. This suggests that a V4 neuron's stimulus preference is determined entirely by its consolidation step. To demonstrate this, we investigated the compression step of a dot-detecting compact model and found a set of simple computations that may be carried out by dot-selective V4 neurons. Overall, our work demonstrates that the DNN models currently used in computational neuroscience are needlessly large; our approach provides a new way forward for obtaining explainable, high-accuracy models of visual cortical neurons.

Simple and complex cells revisited: toward a selectivity-invariance model of object recognition

This paper presents a theoretical perspective on modeling ventral stream processing by revisiting the computational abstraction of simple and complex cells. In parallel to David Marr's vision theory, we organize the new perspective into three levels. At the computational level, we abstract simple and complex cells into space partitioning and composition in a topological space based on the redundancy exploitation hypothesis of Horace Barlow. At the algorithmic level, we present a hierarchical extension of sparse coding by exploiting the manifold constraint in high-dimensional space (i.e., the blessing of dimensionality). The resulting over-parameterized models for object recognition differ from existing hierarchical models by disentangling the objectives of selectivity and invariance computation. It is possible to interpret our hierarchical construction as a computational implementation of cortically local subspace untangling for object recognition and face representation, which are closely related to exemplar-based and axis-based coding in the medial temporal lobe. At the implementation level, we briefly discuss two possible implementations based on asymmetric sparse autoencoders and divergent spiking neural networks.

State-of-the-art image and video quality assessment with a metric based on an intrinsically non-linear neural summation model

The development of automatic methods for image and video quality assessment that correlate well with the perception of human observers is a very challenging open problem in vision science, with numerous practical applications in disciplines such as image processing and computer vision, as well as in the media industry. In the past two decades, the goal of image quality research has been to improve upon classical metrics by developing models that emulate some aspects of the visual system, and while the progress has been considerable, state-of-the-art quality assessment methods still share a number of shortcomings, like their performance dropping considerably when they are tested on a database that is quite different from the one used to train them, or their significant limitations in predicting observer scores for high framerate videos. In this work we propose a novel objective method for image and video quality assessment that is based on the recently introduced Intrinsically Non-linear Receptive Field (INRF) formulation, a neural summation model that has been shown to be better at predicting neural activity and visual perception phenomena than the classical linear receptive field. Here we start by optimizing, on a classic image quality database, the four parameters of a very simple INRF-based metric, and proceed to test this metric on three other databases, showing that its performance equals or surpasses that of the state-of-the-art methods, some of them having millions of parameters. Next, we extend to the temporal domain this INRF image quality metric, and test it on several popular video quality datasets; again, the results of our proposed INRF-based video quality metric are shown to be very competitive.

Sharing neurophysiology data from the Allen Brain Observatory

Nullius in verba ('trust no one'), chosen as the motto of the Royal Society in 1660, implies that independently verifiable observations-rather than authoritative claims-are a defining feature of empirical science. As the complexity of modern scientific instrumentation has made exact replications prohibitive, sharing data is now essential for ensuring the trustworthiness of one's findings. While embraced in spirit by many, in practice open data sharing remains the exception in contemporary systems neuroscience. Here, we take stock of the Allen Brain Observatory, an effort to share data and metadata associated with surveys of neuronal activity in the visual system of laboratory mice. Data from these surveys have been used to produce new discoveries, to validate computational algorithms, and as a benchmark for comparison with other data, resulting in over 100 publications and preprints to date. We distill some of the lessons learned about open surveys and data reuse, including remaining barriers to data sharing and what might be done to address these.

Waves traveling over a map of visual space can ignite short-term predictions of sensory input

Recent analyses have found waves of neural activity traveling across entire visual cortical areas in awake animals. These traveling waves modulate the excitability of local networks and perceptual sensitivity. The general computational role of these spatiotemporal patterns in the visual system, however, remains unclear. Here, we hypothesize that traveling waves endow the visual system with the capacity to predict complex and naturalistic inputs. We present a network model whose connections can be rapidly and efficiently trained to predict individual natural movies. After training, a few input frames from a movie trigger complex wave patterns that drive accurate predictions many frames into the future solely from the network's connections. When the recurrent connections that drive waves are randomly shuffled, both traveling waves and the ability to predict are eliminated. These results suggest traveling waves may play an essential computational role in the visual system by embedding continuous spatiotemporal structures over spatial maps.

Efficient coding of natural scenes improves neural system identification

Neural system identification aims at learning the response function of neurons to arbitrary stimuli using experimentally recorded data, but typically does not leverage normative principles such as efficient coding of natural environments. Visual systems, however, have evolved to efficiently process input from the natural environment. Here, we present a normative network regularization for system identification models by incorporating, as a regularizer, the efficient coding hypothesis, which states that neural response properties of sensory representations are strongly shaped by the need to preserve most of the stimulus information with limited resources. Using this approach, we explored if a system identification model can be improved by sharing its convolutional filters with those of an autoencoder which aims to efficiently encode natural stimuli. To this end, we built a hybrid model to predict the responses of retinal neurons to noise stimuli. This approach did not only yield a higher performance than the "stand-alone" system identification model, it also produced more biologically plausible filters, meaning that they more closely resembled neural representation in early visual systems. We found these results applied to retinal responses to different artificial stimuli and across model architectures. Moreover, our normatively regularized model performed particularly well in predicting responses of direction-of-motion sensitive retinal neurons. The benefit of natural scene statistics became marginal, however, for predicting the responses to natural movies. In summary, our results indicate that efficiently encoding environmental inputs can improve system identification models, at least for noise stimuli, and point to the benefit of probing the visual system with naturalistic stimuli.

Perceptual bias contextualized in visually ambiguous stimuli

The visual appearance of an object is a function of stimulus properties as well as perceptual biases imposed by the observer. The context-specific trade-off between both can be measured accurately in a perceptual judgment task, involving grouping by proximity in ambiguous dot lattices. Such grouping depends lawfully on a stimulus parameter of the dot lattices known as their aspect ratio (AR), whose effect is modulated by a perceptual bias representing the preference for a cardinal orientation. In two experiments, we investigated how preceding context can lead to bias modulation, either in a top-down fashion via visual working memory (VWM) or bottom-up via sensory priming. In Experiment 1, we embedded the perceptual judgment task in a change detection paradigm and studied how the factors of VWM load (complexity of the memory array) and content (congruency in orientation to the ensuing dot lattice) affect the prominence of perceptual bias. A robust vertical orientation bias was observed, which was increased by VWM load and modulated by congruent VWM content. In Experiment 2, dot lattices were preceded by oriented primes. Here, primes regardless of orientation elicited a vertical orientation bias in dot lattices compared to a neutral baseline. Taken together, the two experiments demonstrate that top-down context (VWM load and content) effectively controls orientation bias modulation, while bottom-up context (i.e., priming) merely acts as an undifferentiated trigger to perceptual bias. These findings characterize the temporal context sensitivity of Gestalt perception, shed light on the processes responsible for different perceptual outcomes of ambiguous stimuli, and identify some of the mechanisms controlling perceptual bias.

Assessing the ecological validity of numerosity-selective neuronal populations with real-world natural scenes

Animals and humans are able to quickly and effortlessly estimate the number of items in a set: their numerosity. Numerosity perception is thought to be critical to behavior, from feeding to escaping predators to human mathematical cognition. Virtually, all scientific studies on numerosity mechanisms use well controlled but artificial stimuli to isolate the numerosity dimension from other physical quantities. Here, we probed the ecological validity of these artificial stimuli and evaluate whether an important component in numerosity processing, the numerosity-selective neural populations, also respond to numerosity of items in real-world natural scenes. Using 7T MRI and natural images from a wide range of categories, we provide evidence that the numerosity-tuned neuronal populations show numerosity-selective responses when viewing images from a real-world natural scene. Our findings strengthen the role of numerosity-selective neurons in numerosity perception and provide an important link to their function in numerosity perception in real-world settings.

Visual evoked feedforward-feedback traveling waves organize neural activity across the cortical hierarchy in mice

Sensory processing is distributed among many brain regions that interact via feedforward and feedback signaling. Neuronal oscillations have been shown to mediate intercortical feedforward and feedback interactions. Yet, the macroscopic structure of the multitude of such oscillations remains unclear. Here, we show that simple visual stimuli reliably evoke two traveling waves with spatial wavelengths that cover much of the cerebral hemisphere in awake mice. 30-50 Hz feedforward waves arise in primary visual cortex (V1) and propagate rostrally, while 3-6 Hz feedback waves originate in the association cortex and flow caudally. The phase of the feedback wave modulates the amplitude of the feedforward wave and synchronizes firing between V1 and parietal cortex. Altogether, these results provide direct experimental evidence that visual evoked traveling waves percolate through the cerebral cortex and coordinate neuronal activity across broadly distributed networks mediating visual processing.

Contrast discrimination in images of natural scenes

Contrast discrimination determines the threshold contrast required to distinguish between two suprathreshold visual stimuli. It is typically measured using sine-wave gratings. We first present a modification to Barten's semi-mechanistic contrast discrimination model to account for spatial frequency effects and demonstrate how the model can successfully predict visual thresholds obtained from published classical contrast discrimination studies. Contrast discrimination functions are then measured from images of natural scenes, using a psychophysical paradigm based on that employed in our previous study of contrast detection sensitivity. The proposed discrimination model modification is shown to successfully predict discrimination thresholds for structurally very different types of natural image stimuli. A comparison of results shows that, for normal contrast levels in natural scene viewing, contextual contrast detection and discrimination are approximately the same and almost independent of spatial frequency within the range of 1-20 c/deg. At higher frequencies, both sensitivities decrease in magnitude due to optical limitations of the eye. The results are discussed in relation to current image quality models.

Natural behavior is the language of the brain

The breadth and complexity of natural behaviors inspires awe. Understanding how our perceptions, actions, and internal thoughts arise from evolved circuits in the brain has motivated neuroscientists for generations. Researchers have traditionally approached this question by focusing on stereotyped behaviors, either natural or trained, in a limited number of model species. This approach has allowed for the isolation and systematic study of specific brain operations, which has greatly advanced our understanding of the circuits involved. At the same time, the emphasis on experimental reductionism has left most aspects of the natural behaviors that have shaped the evolution of the brain largely unexplored. However, emerging technologies and analytical tools make it possible to comprehensively link natural behaviors to neural activity across a broad range of ethological contexts and timescales, heralding new modes of neuroscience focused on natural behaviors. Here we describe a three-part roadmap that aims to leverage the wealth of behaviors in their naturally occurring distributions, linking their variance with that of underlying neural processes to understand how the brain is able to successfully navigate the everyday challenges of animals' social and ecological landscapes. To achieve this aim, experimenters must harness one challenge faced by all neurobiological systems, namely variability, in order to gain new insights into the language of the brain.

Cortical layer-specific differences in stimulus selectivity revealed with high-field fMRI and single-vessel resolution optical imaging of the primary visual cortex

The mammalian neocortex exhibits a stereotypical laminar organization, with feedforward inputs arriving primarily into layer 4, local computations shaping response selectivity in layers 2/3, and outputs to other brain areas emanating via layers 2/3, 5 and 6. It cannot be assumed a priori that these signatures of laminar differences in neuronal circuitry are reflected in hemodynamic signals that form the basis of functional magnetic resonance imaging (fMRI). Indeed, optical imaging of single-vessel functional responses has highlighted the potential limits of using vascular signals as surrogates for mapping the selectivity of neural responses. Therefore, before fMRI can be employed as an effective tool for studying critical aspects of laminar processing, validation with single-vessel resolution is needed. The primary visual cortex (V1) in cats, with its precise neuronal functional micro-architecture, offers an ideal model system to examine laminar differences in stimulus selectivity across imaging modalities. Here we used cerebral blood volume weighted (wCBV) fMRI to examine if layer-specific orientation-selective responses could be detected in cat V1. We found orientation preference maps organized tangential to the cortical surface that typically extended across depth in a columnar fashion. We then examined arterial dilation and blood velocity responses to identical visual stimuli by using two- and three- photon optical imaging at single-vessel resolution-which provides a measure of the hemodynamic signals with the highest spatial resolution. Both fMRI and optical imaging revealed a consistent laminar response pattern in which orientation selectivity in cortical layer 4 was significantly lower compared to layer 2/3. This systematic change in selectivity across cortical layers has a clear underpinning in neural circuitry, particularly when comparing layer 4 to other cortical layers.

Decoding Pixel-Level Image Features from Two-Photon Calcium Signals of Macaque Visual Cortex

Images of visual scenes comprise essential features important for visual cognition of the brain. The complexity of visual features lies at different levels, from simple artificial patterns to natural images with different scenes. It has been a focus of using stimulus images to predict neural responses. However, it remains unclear how to extract features from neuronal responses. Here we address this question by leveraging two-photon calcium neural data recorded from the visual cortex of awake macaque monkeys. With stimuli including various categories of artificial patterns and diverse scenes of natural images, we employed a deep neural network decoder inspired by image segmentation technique. Consistent with the notation of sparse coding for natural images, a few neurons with stronger responses dominated the decoding performance, whereas decoding of ar tificial patterns needs a large number of neurons. When natural images using the model pretrained on artificial patterns are decoded, salient features of natural scenes can be extracted, as well as the conventional category information. Altogether, our results give a new perspective on studying neural encoding principles using reverse-engineering decoding strategies.

Neural oscillations promoting perceptual stability and perceptual memory during bistable perception

Ambiguous images elicit bistable perception, wherein periods of momentary perceptual stability are interrupted by sudden perceptual switches. When intermittently presented, ambiguous images trigger a perceptual memory trace in the intervening blank periods. Understanding the neural bases of perceptual stability and perceptual memory during bistable perception may hold clues for explaining the apparent stability of visual experience in the natural world, where ambiguous and fleeting images are prevalent. Motivated by recent work showing the involvement of the right inferior frontal gyrus (rIFG) in bistable perception, we conducted a transcranial direct-current stimulation (tDCS) study with a double-blind, within-subject cross-over design to test a potential causal role of rIFG in these processes. Subjects viewed ambiguous images presented continuously or intermittently while under EEG recording. We did not find any significant tDCS effect on perceptual behavior. However, the fluctuations of oscillatory power in the alpha and beta bands predicted perceptual stability, with higher power corresponding to longer percept durations. In addition, higher alpha and beta power predicted enhanced perceptual memory during intermittent viewing. These results reveal a unified neurophysiological mechanism sustaining perceptual stability and perceptual memory when the visual system is faced with ambiguous input.

Inhibition, but not excitation, recovers from partial cone loss with greater spatiotemporal integration, synapse density, and frequency

Neural circuits function in the face of changing inputs, either caused by normal variation in stimuli or by cell death. To maintain their ability to perform essential computations with partial inputs, neural circuits make modifications. Here, we study the retinal circuit’s responses to changes in light stimuli or in photoreceptor inputs by inducing partial cone death in the mature mouse retina. Can the retina withstand or recover from input loss? We find that the excitatory pathways exhibit functional loss commensurate with cone death and with some aspects predicted by partial light stimulation. However, inhibitory pathways recover functionally from lost input by increasing spatiotemporal integration in a way that is not recapitulated by partially stimulating the control retina. Anatomically, inhibitory synapses are upregulated on secondary bipolar cells and output ganglion cells. These findings demonstrate the greater capacity for inhibition, compared with excitation, to modify spatiotemporal processing with fewer cone inputs.

Lee et al. find partial cone loss triggers inhibition, but not excitation, to increase spatiotemporal integration, recover contrast gain, and increase synaptic release onto retinal ganglion cells. Natural images filtered by cone-loss receptive fields perceptually match those of controls. Thus, inhibition compensates for fewer cones to potentially preserve perception.

Spatial structure, phase, and the contrast of natural images

The sensitivity of the human visual system is thought to be shaped by environmental statistics. A major endeavor in vision science, therefore, is to uncover the image statistics that predict perceptual and cognitive function. When searching for targets in natural images, for example, it has recently been proposed that target detection is inversely related to the spatial similarity of the target to its local background. We tested this hypothesis by measuring observers' sensitivity to targets that were blended with natural image backgrounds. Targets were designed to have a spatial structure that was either similar or dissimilar to the background. Contrary to masking from similarity, we found that observers were most sensitive to targets that were most similar to their backgrounds. We hypothesized that a coincidence of phase alignment between target and background results in a local contrast signal that facilitates detection when target-background similarity is high. We confirmed this prediction in a second experiment. Indeed, we show that, by solely manipulating the phase of a target relative to its background, the target can be rendered easily visible or undetectable. Our study thus reveals that, in addition to its structural similarity, the phase of the target relative to the background must be considered when predicting detection sensitivity in natural images.

The limbic memory circuit and the neural basis of contextual memory

The hippocampus, retrosplenial cortex and anterior thalamus are key components of a neural circuit known to be involved in a variety of memory functions, including spatial, contextual and episodic memory. In this review, we focus on the role of this circuit in contextual memory processes. The background environment, or context, is a powerful cue for memory retrieval, and neural representations of the context provide a mechanism for efficiently retrieving relevant memories while avoiding interference from memories that belong to other contexts. Data from experimental lesions and neural manipulation techniques indicate that each of these regions is critical for contextual memory. Neurophysiological evidence from the hippocampus and retrosplenial cortex suggest that contextual information is represented within this circuit by population-level neural firing patterns that reliably differentiate each context a subject encounters. These findings indicate that encoding contextual information to support context-dependent memory retrieval is a key function of this circuit.

Long-term priors influence visual perception through recruitment of long-range feedback

Perception results from the interplay of sensory input and prior knowledge. Despite behavioral evidence that long-term priors powerfully shape perception, the neural mechanisms underlying these interactions remain poorly understood. We obtained direct cortical recordings in neurosurgical patients as they viewed ambiguous images that elicit constant perceptual switching. We observe top-down influences from the temporal to occipital cortex, during the preferred percept that is congruent with the long-term prior. By contrast, stronger feedforward drive is observed during the non-preferred percept, consistent with a prediction error signal. A computational model based on hierarchical predictive coding and attractor networks reproduces all key experimental findings. These results suggest a pattern of large-scale information flow change underlying long-term priors' influence on perception and provide constraints on theories about long-term priors' influence on perception.

Visual exposure enhances stimulus encoding and persistence in primary cortex

Experience shapes sensory responses, already at the earliest stages of cortical processing. We provide evidence that, in the primary visual cortex of anesthetized cats, brief repetitive exposure to a set of simple, abstract stimuli expands the range and decreases the variability of neuronal responses that persist after stimulus offset. These refinements increase the stimulus-specific clustering of neuronal population responses and result in a more efficient encoding of both stimulus identity and stimulus structure, thus potentially benefiting simple readouts in higher cortical areas. Similar results can be achieved via local plasticity mechanisms in recurrent networks, through self-organized refinements of internal dynamics that do not require changes in firing amplitudes.

The brain adapts to the sensory environment. For example, simple sensory exposure can modify the response properties of early sensory neurons. How these changes affect the overall encoding and maintenance of stimulus information across neuronal populations remains unclear. We perform parallel recordings in the primary visual cortex of anesthetized cats and find that brief, repetitive exposure to structured visual stimuli enhances stimulus encoding by decreasing the selectivity and increasing the range of the neuronal responses that persist after stimulus presentation. Low-dimensional projection methods and simple classifiers demonstrate that visual exposure increases the segregation of persistent neuronal population responses into stimulus-specific clusters. These observed refinements preserve the representational details required for stimulus reconstruction and are detectable in postexposure spontaneous activity. Assuming response facilitation and recurrent network interactions as the core mechanisms underlying stimulus persistence, we show that the exposure-driven segregation of stimulus responses can arise through strictly local plasticity mechanisms, also in the absence of firing rate changes. Our findings provide evidence for the existence of an automatic, unguided optimization process that enhances the encoding power of neuronal populations in early visual cortex, thus potentially benefiting simple readouts at higher stages of visual processing.

Stable representation of a naturalistic movie emerges from episodic activity with gain variability

Visual cortical responses are known to be highly variable across trials within an experimental session. However, the long-term stability of visual cortical responses is poorly understood. Here using chronic imaging of V1 in mice we show that neural responses to repeated natural movie clips are unstable across weeks. Individual neuronal responses consist of sparse episodic activity which are stable in time but unstable in gain across weeks. Further, we find that the individual episode, instead of neuron, serves as the basic unit of the week-to-week fluctuation. To investigate how population activity encodes the stimulus, we extract a stable one-dimensional representation of the time in the natural movie, using an unsupervised method. Most week-to-week fluctuation is perpendicular to the stimulus encoding direction, thus leaving the stimulus representation largely unaffected. We propose that precise episodic activity with coordinated gain changes are keys to maintain a stable stimulus representation in V1.

The unbiased estimation of the fraction of variance explained by a model

The correlation coefficient squared, r², is commonly used to validate quantitative models on neural data, yet it is biased by trial-to-trial variability: as trial-to-trial variability increases, measured correlation to a model’s predictions decreases. As a result, models that perfectly explain neural tuning can appear to perform poorly. Many solutions to this problem have been proposed, but no consensus has been reached on which is the least biased estimator. Some currently used methods substantially overestimate model fit, and the utility of even the best performing methods is limited by the lack of confidence intervals and asymptotic analysis. We provide a new estimator, r^ER2, that outperforms all prior estimators in our testing, and we provide confidence intervals and asymptotic guarantees. We apply our estimator to a variety of neural data to validate its utility. We find that neural noise is often so great that confidence intervals of the estimator cover the entire possible range of values ([0, 1]), preventing meaningful evaluation of the quality of a model’s predictions. This leads us to propose the use of the signal-to-noise ratio (SNR) as a quality metric for making quantitative comparisons across neural recordings. Analyzing a variety of neural data sets, we find that up to ∼ 40% of some state-of-the-art neural recordings do not pass even a liberal SNR criterion. Moving toward more reliable estimates of correlation, and quantitatively comparing quality across recording modalities and data sets, will be critical to accelerating progress in modeling biological phenomena.

Quantifying the similarity between a model and noisy data is fundamental to advancing the scientific understanding of biological phenomena, and it is particularly relevant to modeling neuronal responses. A ubiquitous metric of similarity is the correlation coefficient, but this metric depends on a variety of factors that are irrelevant to the similarity between a model and data. While neuroscientists have recognized this problem and proposed corrected methods, no consensus has been reached as to which are effective. Prior methods have wide variation in their accuracy, and even the most successful methods lack confidence intervals, leaving uncertainty about the reliability of any particular estimate. We address these issues by developing a new estimator along with an associated confidence interval that outperforms all prior methods. We also demonstrate how a signal-to-noise ratio can be used to usefully threshold and compare noisy experimental data across studies and recording paradigms.

Illusory Essences: A Bias Holding Back Theorizing in Psychological Science

The reliance in psychology on verbal definitions means that psychological research is unusually moored to how humans think and communicate about categories. Psychological concepts (e.g., intelligence, attention) are easily assumed to represent objective, definable categories with an underlying essence. Like the “vital forces” previously thought to animate life, these assumed essences can create an illusion of understanding. By synthesizing a wide range of research lines from cognitive, clinical, and biological psychology and neuroscience, we describe a pervasive tendency across psychological science to assume that essences explain phenomena. Labeling a complex phenomenon can appear as theoretical progress before there is sufficient evidence that the described category has a definable essence or known boundary conditions. Category labels can further undermine progress by masking contingent and contextual relationships and obscuring the need to specify mechanisms. Finally, we highlight examples of promising methods that circumvent the lure of essences and suggest four concrete strategies for identifying and avoiding essentialist intuitions in theory development.

Does task-irrelevant music affect gaze allocation during real-world scene viewing?

Gaze control manifests from a dynamic integration of visual and auditory information, with sound providing important cues for how a viewer should behave. Some past research suggests that music, even if entirely irrelevant to the current task demands, may also sway the timing and frequency of fixations. The current work sought to further assess this idea as well as investigate whether task-irrelevant music could also impact how gaze is spatially allocated. In preparation for a later memory test, participants studied pictures of urban scenes in silence or while simultaneously listening to one of two types of music. Eye tracking was recorded, and nine gaze behaviors were measured to characterize the temporal and spatial aspects of gaze control. Findings showed that while these gaze behaviors changed over the course of viewing, music had no impact. Participants in the music conditions, however, did show better memory performance than those who studied in silence. These findings are discussed within theories of multimodal gaze control.

Sparse thalamocortical convergence

How many thalamic neurons converge onto a cortical cell? This is an important question, because the organization of thalamocortical projections can influence the cortical architecture.^1,2 Here, we estimate the degree of thalamocortical convergence in primary visual cortex by taking advantage of the cortical expansion-neurons within a restricted volume in primary visual cortex have overlapping receptive fields driven by a smaller set of inputs from the lateral geniculate nucleus.^3-5 Under these conditions, the measurements of cortical receptive fields in a population can be used to infer the receptive fields of their geniculate inputs and the weights of their projections using non-negative matrix factorization.⁶ The analysis reveals sparse connectivity,⁷ where a handful (~2-6) of thalamic inputs account for 90% of the total synaptic weight to a cortical neuron. Together with previous findings,⁸ these results paint a picture consistent with the idea that convergence of a few inputs partly determine the retinotopy and tuning properties of cortical cells.^8-13.

How inhibitory neurons increase information transmission under threshold modulation

Modulation of neuronal thresholds is ubiquitous in the brain. Phenomena such as figure-ground segmentation, motion detection, stimulus anticipation, and shifts in attention all involve changes in a neuron’s threshold based on signals from larger scales than its primary inputs. However, this modulation reduces the accuracy with which neurons can represent their primary inputs, creating a mystery as to why threshold modulation is so widespread in the brain. We find that modulation is less detrimental than other forms of neuronal variability and that its negative effects can be nearly completely eliminated if modulation is applied selectively to sparsely responding neurons in a circuit by inhibitory neurons. We verify these predictions in the retina where we find that inhibitory amacrine cells selectively deliver modulation signals to sparsely responding ganglion cell types. Our findings elucidate the central role that inhibitory neurons play in maximizing information transmission under modulation.

Modulation of neuronal thresholds is ubiquitous in the brain but reduces the accuracy of neural signaling. Hsu et al. show that the negative impact of threshold modulation can be almost completely eliminated when modulation is not delivered uniformly to all neurons but only to a subset and via inhibitory neurons.

Anomalous Behavior Detection Framework Using HTM-Based Semantic Folding Technique

Upon the working principles of the human neocortex, the Hierarchical Temporal Memory model has been developed which is a proposed theoretical framework for sequence learning. Both categorical and numerical types of data are handled by HTM. Semantic Folding Theory (SFT) is based on HTM to represent a data stream for processing in the form of sparse distributed representation (SDR). For natural language perception and production, SFT delivers a solid structural background for semantic evidence description to the fundamentals of the semantic foundation during the phase of language learning. Anomalies are the patterns from data streams that do not follow the expected behavior. Any stream of data patterns could have a number of anomaly types. In a data stream, a single pattern or combination of closely related patterns that diverges and deviates from standard, normal, or expected is called a static (spatial) anomaly. A temporal anomaly is a set of unexpected changes between patterns. When a change first appears, this is recorded as an anomaly. If this change looks a number of times, then it is set to a “new normal” and terminated as an anomaly. An HTM system detects the anomaly, and due to continuous learning nature, it quickly learns when they become the new normal. A robust anomalous behavior detection framework using HTM-based SFT for improving decision-making (SDR-ABDF/P2) is a proposed framework or model in this research. The researcher claims that the proposed model would be able to learn the order of several variables continuously in temporal sequences by using an unsupervised learning rule.

The Role of Edges in Line Drawing Perception

It has often been conjectured that the effectiveness of line drawings can be explained by the similarity of edge images to line drawings. This article presents several problems with explaining line drawing perception in terms of edges, and how the recently proposed Realism Hypothesis resolves these problems. There is nonetheless existing evidence that edges are often the best features for predicting where people draw lines; this article describes how the Realism Hypothesis can explain this evidence.

The Linguistic Analysis of Scene Semantics: LASS

In this paper, we define a new method for analyzing object-scene contextual relationships using computational linguistics: Linguistic Analysis of Scene Semantics, or LASS. LASS uses linguistic semantic similarity relationships between scene object and context labels embedded in a vector-space language model: Facebook Research's fastText. Importantly, the use of fastText permits semantic similarity score calculation between any set of strings and thus elements of any set of image data for which labels are available. Scene semantic similarity scores are then embedded in object segmentation mask locations in the image, creating a semantic similarity map. LASS can also be fully automated by generating context and object labels, as well as object segmentation masks, using deep learning. We compare semantic similarity maps between human- and neural network-generated annotations on a corpus of images taken from the LabelMe database. Semantic similarity maps produced by the fully automated LASS have a number of desirable properties, while maintaining a high degree of spatial and semantic similarity to them. Finally, we use LASS to evaluate the distribution of semantically consistent scene elements in space. Both show relatively uniform distributions of semantic relatedness to scene context, suggesting that contextually appropriate objects are likely to be found in all image regions. Taken together, these results suggest that LASS is accurate, automatic, flexible, and useful in a number of research contexts such as scene grammar and novelty detection.

Survey of spiking in the mouse visual system reveals functional hierarchy

The anatomy of the mammalian visual system, from the retina to the neocortex, is organized hierarchically¹. However, direct observation of cellular-level functional interactions across this hierarchy is lacking due to the challenge of simultaneously recording activity across numerous regions. Here we describe a large, open dataset-part of the Allen Brain Observatory²-that surveys spiking from tens of thousands of units in six cortical and two thalamic regions in the brains of mice responding to a battery of visual stimuli. Using cross-correlation analysis, we reveal that the organization of inter-area functional connectivity during visual stimulation mirrors the anatomical hierarchy from the Allen Mouse Brain Connectivity Atlas³. We find that four classical hierarchical measures-response latency, receptive-field size, phase-locking to drifting gratings and response decay timescale-are all correlated with the hierarchy. Moreover, recordings obtained during a visual task reveal that the correlation between neural activity and behavioural choice also increases along the hierarchy. Our study provides a foundation for understanding coding and signal propagation across hierarchically organized cortical and thalamic visual areas.

A Gradient of Sharpening Effects by Perceptual Prior across the Human Cortical Hierarchy

Prior knowledge profoundly influences perceptual processing. Previous studies have revealed consistent suppression of predicted stimulus information in sensory areas, but how prior knowledge modulates processing higher up in the cortical hierarchy remains poorly understood. In addition, the mechanism leading to suppression of predicted sensory information remains unclear, and studies thus far have revealed a mixed pattern of results in support of either the "sharpening" or "dampening" model. Here, using 7T fMRI in humans (both sexes), we observed that prior knowledge acquired from fast, one-shot perceptual learning sharpens neural representation throughout the ventral visual stream, generating suppressed sensory responses. In contrast, the frontoparietal and default mode networks exhibit similar sharpening of content-specific neural representation, but in the context of unchanged and enhanced activity magnitudes, respectively: a pattern we refer to as "selective enhancement." Together, these results reveal a heretofore unknown macroscopic gradient of prior knowledge's sharpening effect on neural representations across the cortical hierarchy.SIGNIFICANCE STATEMENT A fundamental question in neuroscience is how prior knowledge shapes perceptual processing. Perception is constantly informed by internal priors in the brain acquired from past experiences, but the neural mechanisms underlying this process are poorly understood. To date, research on this question has focused on early visual regions, reporting a consistent downregulation when predicted stimuli are encountered. Here, using a dramatic one-shot perceptual learning paradigm, we observed that prior knowledge results in sharper neural representations across the cortical hierarchy of the human brain through a gradient of mechanisms. In visual regions, neural responses tuned away from internal predictions are suppressed. In frontoparietal regions, neural activity consistent with priors is selectively enhanced. These results deepen our understanding of how prior knowledge informs perception.

Keep it real: rethinking the primacy of experimental control in cognitive neuroscience

Naturalistic experimental paradigms in neuroimaging arose from a pressure to test the validity of models we derive from highly-controlled experiments in real-world contexts. In many cases, however, such efforts led to the realization that models developed under particular experimental manipulations failed to capture much variance outside the context of that manipulation. The critique of non-naturalistic experiments is not a recent development; it echoes a persistent and subversive thread in the history of modern psychology. The brain has evolved to guide behavior in a multidimensional world with many interacting variables. The assumption that artificially decoupling and manipulating these variables will lead to a satisfactory understanding of the brain may be untenable. We develop an argument for the primacy of naturalistic paradigms, and point to recent developments in machine learning as an example of the transformative power of relinquishing control. Naturalistic paradigms should not be deployed as an afterthought if we hope to build models of brain and behavior that extend beyond the laboratory into the real world.

Evidence for the intrinsically nonlinear nature of receptive fields in vision

The responses of visual neurons, as well as visual perception phenomena in general, are highly nonlinear functions of the visual input, while most vision models are grounded on the notion of a linear receptive field (RF). The linear RF has a number of inherent problems: it changes with the input, it presupposes a set of basis functions for the visual system, and it conflicts with recent studies on dendritic computations. Here we propose to model the RF in a nonlinear manner, introducing the intrinsically nonlinear receptive field (INRF). Apart from being more physiologically plausible and embodying the efficient representation principle, the INRF has a key property of wide-ranging implications: for several vision science phenomena where a linear RF must vary with the input in order to predict responses, the INRF can remain constant under different stimuli. We also prove that Artificial Neural Networks with INRF modules instead of linear filters have a remarkably improved performance and better emulate basic human perception. Our results suggest a change of paradigm for vision science as well as for artificial intelligence.

Color illusions also deceive CNNs for low-level vision tasks: Analysis and implications

The study of visual illusions has proven to be a very useful approach in vision science. In this work we start by showing that, while convolutional neural networks (CNNs) trained for low-level visual tasks in natural images may be deceived by brightness and color illusions, some network illusions can be inconsistent with the perception of humans. Next, we analyze where these similarities and differences may come from. On one hand, the proposed linear eigenanalysis explains the overall similarities: in simple CNNs trained for tasks like denoising or deblurring, the linear version of the network has center-surround receptive fields, and global transfer functions are very similar to the human achromatic and chromatic contrast sensitivity functions in human-like opponent color spaces. These similarities are consistent with the long-standing hypothesis that considers low-level visual illusions as a by-product of the optimization to natural environments. Specifically, here human-like features emerge from error minimization. On the other hand, the observed differences must be due to the behavior of the human visual system not explained by the linear approximation. However, our study also shows that more 'flexible' network architectures, with more layers and a higher degree of nonlinearity, may actually have a worse capability of reproducing visual illusions. This implies, in line with other works in the vision science literature, a word of caution on using CNNs to study human vision: on top of the intrinsic limitations of the L + NL formulation of artificial networks to model vision, the nonlinear behavior of flexible architectures may easily be markedly different from that of the visual system.

Intracellular ATP levels in mouse cortical excitatory neurons varies with sleep-wake states

Whilst the brain is assumed to exert homeostatic functions to keep the cellular energy status constant under physiological conditions, this has not been experimentally proven. Here, we conducted in vivo optical recordings of intracellular concentration of adenosine 5'-triphosphate (ATP), the major cellular energy metabolite, using a genetically encoded sensor in the mouse brain. We demonstrate that intracellular ATP levels in cortical excitatory neurons fluctuate in a cortex-wide manner depending on the sleep-wake states, correlating with arousal. Interestingly, ATP levels profoundly decreased during rapid eye movement sleep, suggesting a negative energy balance in neurons despite a simultaneous increase in cerebral hemodynamics for energy supply. The reduction in intracellular ATP was also observed in response to local electrical stimulation for neuronal activation, whereas the hemodynamics were simultaneously enhanced. These observations indicate that cerebral energy metabolism may not always meet neuronal energy demands, consequently resulting in physiological fluctuations of intracellular ATP levels in neurons.

Where the eyes wander: The relationship between mind wandering and fixation allocation to visually salient and semantically informative static scene content

Vision is crucial for many everyday activities, but the mind is not always focused on what the eyes see. Mind wandering occurs frequently and is associated with attenuated visual and cognitive processing of external information. Corresponding changes in gaze behavior—namely, fewer, longer, and more dispersed fixations—suggest a shift in how the visual system samples external information. Using three computational models of visual salience and two innovative approaches for measuring semantic informativeness, the current work assessed whether these changes reflect how the visual system prioritizes visually salient and semantically informative scene content, two major determinants in most theoretical frameworks and computational models of gaze control. Findings showed that, in a static scene viewing task, fixations were allocated to scene content that was more visually salient 10 seconds prior to probe-caught, self-reported mind wandering compared to self-reported attentive viewing. The relationship between mind wandering and semantic content was more equivocal, with weaker evidence that fixations are more likely to fall on locally informative scene regions. This indicates that the visual system is still able to discriminate visually salient and semantically informative scene content during mind wandering and may fixate on such information more frequently than during attentive viewing. Theoretical implications are discussed in light of these findings.

Response nonlinearities in networks of spiking neurons

Combining information from multiple sources is a fundamental operation performed by networks of neurons in the brain, whose general principles are still largely unknown. Experimental evidence suggests that combination of inputs in cortex relies on nonlinear summation. Such nonlinearities are thought to be fundamental to perform complex computations. However, these non-linearities are inconsistent with the balanced-state model, one of the most popular models of cortical dynamics, which predicts networks have a linear response. This linearity is obtained in the limit of very large recurrent coupling strength. We investigate the stationary response of networks of spiking neurons as a function of coupling strength. We show that, while a linear transfer function emerges at strong coupling, nonlinearities are prominent at finite coupling, both at response onset and close to saturation. We derive a general framework to classify nonlinear responses in these networks and discuss which of them can be captured by rate models. This framework could help to understand the diversity of non-linearities observed in cortical networks.

A Comparative Analysis of Visual Encoding Models Based on Classification and Segmentation Task-Driven CNNs

Nowadays, visual encoding models use convolution neural networks (CNNs) with outstanding performance in computer vision to simulate the process of human information processing. However, the prediction performances of encoding models will have differences based on different networks driven by different tasks. Here, the impact of network tasks on encoding models is studied. Using functional magnetic resonance imaging (fMRI) data, the features of natural visual stimulation are extracted using a segmentation network (FCN32s) and a classification network (VGG16) with different visual tasks but similar network structure. Then, using three sets of features, i.e., segmentation, classification, and fused features, the regularized orthogonal matching pursuit (ROMP) method is used to establish the linear mapping from features to voxel responses. The analysis results indicate that encoding models based on networks performing different tasks can effectively but differently predict stimulus-induced responses measured by fMRI. The prediction accuracy of the encoding model based on VGG is found to be significantly better than that of the model based on FCN in most voxels but similar to that of fused features. The comparative analysis demonstrates that the CNN performing the classification task is more similar to human visual processing than that performing the segmentation task.

Movement-Related Signals in Sensory Areas: Roles in Natural Behavior

Recent studies have demonstrated prominent and widespread movement-related signals in the brain of head-fixed mice, even in primary sensory areas. However, it is still unknown what role these signals play in sensory processing. Why are these sensory areas 'contaminated' by movement signals? During natural behavior, animals actively acquire sensory information as they move through the environment and use this information to guide ongoing actions. In this context, movement-related signals could allow sensory systems to predict self-induced sensory changes and extract additional information about the environment. In this review we summarize recent findings on the presence of movement-related signals in sensory areas and discuss how their study, in the context of natural freely moving behaviors, could advance models of sensory processing.

Inhibition stabilization is a widespread property of cortical networks

Many cortical network models use recurrent coupling strong enough to require inhibition for stabilization. Yet it has been experimentally unclear whether inhibition-stabilized network (ISN) models describe cortical function well across areas and states. Here, we test several ISN predictions, including the counterintuitive (paradoxical) suppression of inhibitory firing in response to optogenetic inhibitory stimulation. We find clear evidence for ISN operation in mouse visual, somatosensory, and motor cortex. Simple two-population ISN models describe the data well and let us quantify coupling strength. Although some models predict a non-ISN to ISN transition with increasingly strong sensory stimuli, we find ISN effects without sensory stimulation and even during light anesthesia. Additionally, average paradoxical effects result only with transgenic, not viral, opsin expression in parvalbumin (PV)-positive neurons; theory and expression data show this is consistent with ISN operation. Taken together, these results show strong coupling and inhibition stabilization are common features of the cortex.

Simulation-Trained Sparse Coding for High-Precision Phase Imaging in Low-Dose Electron Holography

We broaden the applicability of sparse coding, a machine learning method, to low-dose electron holography by using simulated holograms for learning and validation processes. The holograms, with shot noise, are prepared to generate a model, or a dictionary, that includes basic features representing interference fringes. The dictionary is applied to sparse representations of other simulated holograms with various signal-to-noise ratios (SNRs). Results demonstrate that this approach successfully removes noise for holograms with an extremely small SNR of 0.10, and that the denoised holograms provide the accurate phase distribution. Furthermore, this study demonstrates that the dictionary learned from the simulated holograms can be applied to denoising of experimental holograms of a p-n junction specimen recorded with different exposure times. The results indicate that the simulation-trained sparse coding is suitable for use over a wide range of imaging conditions, in particular for observing electron beam-sensitive materials.

Mechanisms of Feature Selectivity and Invariance in Primary Visual Cortex

Visual object identification requires both selectivity for specific visual features that are important to the object's identity and invariance to feature manipulations. For example, a hand can be shifted in position, rotated, or contracted but still be recognized as a hand. How are the competing requirements of selectivity and invariance built into the early stages of visual processing? Typically, cells in the primary visual cortex are classified as either simple or complex. They both show selectivity for edge-orientation but complex cells develop invariance to edge position within the receptive field (spatial phase). Using a data-driven model that extracts the spatial structures and nonlinearities associated with neuronal computation, we quantitatively describe the balance between selectivity and invariance in complex cells. Phase invariance is frequently partial, while invariance to orientation and spatial frequency are more extensive than expected. The invariance arises due to two independent factors: (1) the structure and number of filters and (2) the form of nonlinearities that act upon the filter outputs. Both vary more than previously considered, so primary visual cortex forms an elaborate set of generic feature sensitivities, providing the foundation for more sophisticated object processing.

Network Analysis of Murine Cortical Dynamics Implicates Untuned Neurons in Visual Stimulus Coding

Unbiased and dense sampling of large populations of layer 2/3 pyramidal neurons in mouse primary visual cortex (V1) reveals two functional sub-populations: neurons tuned and untuned to drifting gratings. Whether functional interactions between these two groups contribute to the representation of visual stimuli is unclear. To examine these interactions, we summarize the population partial pairwise correlation structure as a directed and weighted graph. We find that tuned and untuned neurons have distinct topological properties, with untuned neurons occupying central positions in functional networks (FNs). Implementation of a decoder that utilizes the topology of these FNs yields accurate decoding of visual stimuli. We further show that decoding performance degrades comparably following manipulations of either tuned or untuned neurons. Our results demonstrate that untuned neurons are an integral component of V1 FNs and suggest that network interactions contain information about the stimulus that is accessible to downstream elements.