Motion detection based on recurrent network dynamics

High-Level Visual Encoding Model Framework with Hierarchical Ventral Stream-Optimized Neural Networks

Visual encoding models based on deep neural networks (DNN) show good performance in predicting brain activity in low-level visual areas. However, due to the amount of neural data limitation, DNN-based visual encoding models are difficult to fit for high-level visual areas, resulting in insufficient encoding performance. The ventral stream suggests that higher visual areas receive information from lower visual areas, which is not fully reflected in the current encoding models. In the present study, we propose a novel visual encoding model framework which uses the hierarchy of representations in the ventral stream to improve the model’s performance in high-level visual areas. Under the framework, we propose two categories of hierarchical encoding models from the voxel and the feature perspectives to realize the hierarchical representations. From the voxel perspective, we first constructed an encoding model for the low-level visual area (V1 or V2) and extracted the voxel space predicted by the model. Then we use the extracted voxel space of the low-level visual area to predict the voxel space of the high-level visual area (V4 or LO) via constructing a voxel-to-voxel model. From the feature perspective, the feature space of the first model is extracted to predict the voxel space of the high-level visual area. The experimental results show that two categories of hierarchical encoding models effectively improve the encoding performance in V4 and LO. In addition, the proportion of the best-encoded voxels for different models in V4 and LO show that our proposed models have obvious advantages in prediction accuracy. We find that the hierarchy of representations in the ventral stream has a positive effect on improving the performance of the existing model in high-level visual areas.

End-to-end neural system identification with neural information flow

Neural information flow (NIF) provides a novel approach for system identification in neuroscience. It models the neural computations in multiple brain regions and can be trained end-to-end via stochastic gradient descent from noninvasive data. NIF models represent neural information processing via a network of coupled tensors, each encoding the representation of the sensory input contained in a brain region. The elements of these tensors can be interpreted as cortical columns whose activity encodes the presence of a specific feature in a spatiotemporal location. Each tensor is coupled to the measured data specific to a brain region via low-rank observation models that can be decomposed into the spatial, temporal and feature receptive fields of a localized neuronal population. Both these observation models and the convolutional weights defining the information processing within regions are learned end-to-end by predicting the neural signal during sensory stimulation. We trained a NIF model on the activity of early visual areas using a large-scale fMRI dataset recorded in a single participant. We show that we can recover plausible visual representations and population receptive fields that are consistent with empirical findings.

Neural mechanism of visual information degradation from retina to V1 area

The information processing mechanism of the visual nervous system is an unresolved scientific problem that has long puzzled neuroscientists. The amount of visual information is significantly degraded when it reaches the V1 after entering the retina; nevertheless, this does not affect our visual perception of the outside world. Currently, the mechanisms of visual information degradation from retina to V1 are still unclear. For this purpose, the current study used the experimental data summarized by Marcus E. Raichle to investigate the neural mechanisms underlying the degradation of the large amount of data from topological mapping from retina to V1, drawing on the photoreceptor model first. The obtained results showed that the image edge features of visual information were extracted by the convolution algorithm with respect to the function of synaptic plasticity when visual signals were hierarchically processed from low-level to high-level. The visual processing was characterized by the visual information degradation, and this compensatory mechanism embodied the principles of energy minimization and transmission efficiency maximization of brain activity, which matched the experimental data summarized by Marcus E. Raichle. Our results further the understanding of the information processing mechanism of the visual nervous system.

Short-Term Attractive Tilt Aftereffects Predicted by a Recurrent Network Model of Primary Visual Cortex

Adaptation is a multi-faceted phenomenon that is of interest in terms of both its function and its potential to reveal underlying neural processing. Many behavioral studies have shown that after exposure to an oriented adapter the perceived orientation of a subsequent test is repulsed away from the orientation of the adapter. This is the well-known Tilt Aftereffect (TAE). Recently, we showed that the dynamics of recurrently connected networks may contribute substantially to the neural changes induced by adaptation, especially on short time scales. Here we extended the network model and made the novel behavioral prediction that the TAE should be attractive, not repulsive, on a time scale of a few 100 ms. Our experiments, using a novel adaptation protocol that specifically targeted adaptation on a short time scale, confirmed this prediction. These results support our hypothesis that recurrent network dynamics may contribute to short-term adaptation. More broadly, they show that understanding the neural processing of visual inputs that change on the time scale of a typical fixation requires a detailed analysis of not only the intrinsic properties of neurons, but also the slow and complex dynamics that emerge from their recurrent connectivity. We argue that this is but one example of how even simple recurrent networks can underlie surprisingly complex information processing, and are involved in rudimentary forms of memory, spatio-temporal integration, and signal amplification.

Computational Foundations of Natural Intelligence

New developments in AI and neuroscience are revitalizing the quest to understanding natural intelligence, offering insight about how to equip machines with human-like capabilities. This paper reviews some of the computational principles relevant for understanding natural intelligence and, ultimately, achieving strong AI. After reviewing basic principles, a variety of computational modeling approaches is discussed. Subsequently, I concentrate on the use of artificial neural networks as a framework for modeling cognitive processes. This paper ends by outlining some of the challenges that remain to fulfill the promise of machines that show human-like intelligence.

Adaptation without Plasticity

Sensory adaptation is a phenomenon in which neurons are affected not only by their immediate input but also by the sequence of preceding inputs. In visual cortex, for example, neurons shift their preferred orientation after exposure to an oriented stimulus. This adaptation is traditionally attributed to plasticity. We show that a recurrent network generates tuning curve shifts observed in cat and macaque visual cortex, even when all synaptic weights and intrinsic properties in the model are fixed. This demonstrates that, in a recurrent network, adaptation on timescales of hundreds of milliseconds does not require plasticity. Given the ubiquity of recurrent connections, this phenomenon likely contributes to responses observed across cortex and shows that plasticity cannot be inferred solely from changes in tuning on these timescales. More broadly, our findings show that recurrent connections can endow a network with a powerful mechanism to store and integrate recent contextual information.

Recurrent network dynamics reconciles visual motion segmentation and integration

In sensory systems, a range of computational rules are presumed to be implemented by neuronal subpopulations with different tuning functions. For instance, in primate cortical area MT, different classes of direction-selective cells have been identified and related either to motion integration, segmentation or transparency. Still, how such different tuning properties are constructed is unclear. The dominant theoretical viewpoint based on a linear-nonlinear feed-forward cascade does not account for their complex temporal dynamics and their versatility when facing different input statistics. Here, we demonstrate that a recurrent network model of visual motion processing can reconcile these different properties. Using a ring network, we show how excitatory and inhibitory interactions can implement different computational rules such as vector averaging, winner-take-all or superposition. The model also captures ordered temporal transitions between these behaviors. In particular, depending on the inhibition regime the network can switch from motion integration to segmentation, thus being able to compute either a single pattern motion or to superpose multiple inputs as in motion transparency. We thus demonstrate that recurrent architectures can adaptively give rise to different cortical computational regimes depending upon the input statistics, from sensory flow integration to segmentation.

Recurrent Network Dynamics; a Link between Form and Motion

To discriminate visual features such as corners and contours, the brain must be sensitive to spatial correlations between multiple points in an image. Consistent with this, macaque V2 neurons respond selectively to patterns with well-defined multipoint correlations. Here, we show that a standard feedforward model (a cascade of linear–non-linear filters) does not capture this multipoint selectivity. As an alternative, we developed an artificial neural network model with two hierarchical stages of processing and locally recurrent connectivity. This model faithfully reproduced neurons’ selectivity for multipoint correlations. By probing the model, we gained novel insights into early form processing. First, the diverse selectivity for multipoint correlations and complex response dynamics of the hidden units in the model were surprisingly similar to those observed in V1 and V2. This suggests that both transient and sustained response dynamics may be a vital part of form computations. Second, the model self-organized units with speed and direction selectivity that was correlated with selectivity for multipoint correlations. In other words, the model units that detected multipoint spatial correlations also detected space-time correlations. This leads to the novel hypothesis that higher-order spatial correlations could be computed by the rapid, sequential assessment and comparison of multiple low-order correlations within the receptive field. This computation links spatial and temporal processing and leads to the testable prediction that the analysis of complex form and motion are closely intertwined in early visual cortex.

Evidence and Counterevidence in Motion Perception

Sensory neurons gather evidence in favor of the specific stimuli to which they are tuned, but they could improve their sensitivity by also taking counterevidence into account. The Bours-Lankheet model for motion detection uses counterevidence that relies on a specific combination of the ON and OFF channels in the early visual system. Specifically, the model detects pairs of flashes that occur separated in space and time. If the flashes have the same contrast polarity, they are interpreted as evidence in favor of the corresponding motion. But if they have opposite contrasts, they are interpreted as evidence against it. This mechanism provides an explanation for reverse-phi (the perceived reversal of an apparent motion stimulus due to periodic contrast-inversions) that is a conceptual departure from the standard explanations of the effect. Here, we investigate this counterevidence mechanism by measuring directional tuning curves of neurons in the primary visual and middle temporal cortex areas of awake, behaving macaques using constant-contrast and inverting-contrast moving dot stimuli. Our electrophysiological data support the Bours-Lankheet model and suggest that the counterevidence computation occurs at an early stage of neural processing not captured by the standard models.

Testing the assumptions underlying fMRI adaptation using intracortical recordings in area MT

We investigated how neural activity in the middle temporal area of the macaque monkey changes after 3 sec of exposure to a visual stimulus and used this to gain insight into the assumptions underlying the fMRI adaptation method (fMRIa). We studied both changes in tuning curves following weak and strong motion stimuli (adaptation) and the differences between a first and second exposure to the same stimulus (repetition suppression). Typically, tuning curves had smaller amplitudes and narrower tuning widths after strong adaptation; this was true for single neurons, multi-unit activity (MUA), the evoked local field potential (LFP), as well as gamma band activity. Repetition typically led to reduced responses. This reduction was correlated with direction selectivity and not explained by neural fatigue. Our data, however, warn against a simplistic view of the consequences of adaptation. First, a considerable fraction of neurons and sites showed response enhancements after adaptation, especially when probed with a stimulus that moved opposite to the direction of the adapting stimulus. Second, adaptation was stimulus selective only on a time scale of ∼100 msec. Third, aggregate measures of neural activity (MUA, LFPs) had substantially different adaptation effects. Fourth, there were qualitative differences between our findings in MT and earlier findings in IT cortex. We conclude that selective adaptation effects in fMRIa are relatively easy to miss even when they exist (for instance by presenting stimuli for too long, or because neurons that enhance after adaptation cancel out the effect of neurons that suppress). Moreover, we argue that adaptation should be understood in the context of the computations that a neural circuit perform. Using fMRIa as a tool to uncover neural selectivity requires a better understanding of this circuitry and its consequences for adaptation.

Visual processing of informative multipoint correlations arises primarily in V2

Using the visual system as a model, we recently showed that the efficient coding principle accounted for the allocation of computational resources in central sensory processing: when sampling an image is the main limitation, resources are devoted to compute the statistical features that are the most variable, and therefore the most informative (eLife 2014;3:e03722. DOI: 10.7554/eLife.03722 Hermundstad et al., 2014). Building on these results, we use single-unit recordings in the macaque monkey to determine where these computations—sensitivity to specific multipoint correlations—occur. We find that these computations take place in visual area V2, primarily in its supragranular layers. The demonstration that V2 neurons are sensitive to the multipoint correlations that are informative about natural images provides a common computational underpinning for diverse but well-recognized aspects of neural processing in V2, including its sensitivity to corners, junctions, illusory contours, figure/ground, and ‘naturalness.’

DOI:http://dx.doi.org/10.7554/eLife.06604.001