A visual motion sensor based on the properties of V1 and MT neurons

Binocular Integration of Perceptually Suppressed Visual Information in Amblyopia

Purpose

The purpose of this study was to assess whether motion information from suppressed amblyopic eyes can influence visual perception.

Methods

Participants with normal vision (n = 20) and with amblyopia (n = 20; 11 anisometropic and 9 strabismic/mixed) viewed dichoptic, orthogonal drifting gratings through a mirror stereoscope. Participants continuously reported form and motion percepts as gratings rivaled for 60 seconds. Responses were binned into categories ranging from binocular integration to complete suppression. Periods when the grating presented to the nondominant/amblyopic eye was suppressed were analyzed further to determine the extent of binocular integration of motion.

Results

Individuals with amblyopia experienced longer periods of non-preferred eye suppression than controls. When the non-preferred eye grating was suppressed, binocular integration of motion occurred 48.1 ± 6.2% and 31.2 ± 5.8% of the time in control and amblyopic participants, respectively. Periods of motion integration from the suppressed eye were significantly non-zero for both groups.

Conclusions

Visual information seen only by a suppressed amblyopic eye can be binocularly integrated and influence the overall visual percept. These findings reveal that visual information subjected to interocular suppression can still contribute to binocular vision and suggest the use of appropriate optical correction for the amblyopic eye to improve image quality for binocular combination.

Compound Stimuli Reveal the Structure of Visual Motion Selectivity in Macaque MT Neurons

Motion selectivity in primary visual cortex (V1) is approximately separable in orientation, spatial frequency, and temporal frequency (“frequency-separable”). Models for area MT neurons posit that their selectivity arises by combining direction-selective V1 afferents whose tuning is organized around a tilted plane in the frequency domain, specifying a particular direction and speed (“velocity-separable”). This construction explains “pattern direction-selective” MT neurons, which are velocity-selective but relatively invariant to spatial structure, including spatial frequency, texture and shape. We designed a set of experiments to distinguish frequency-separable and velocity-separable models and executed them with single-unit recordings in macaque V1 and MT. Surprisingly, when tested with single drifting gratings, most MT neurons’ responses are fit equally well by models with either form of separability. However, responses to plaids (sums of two moving gratings) tend to be better described as velocity-separable, especially for pattern neurons. We conclude that direction selectivity in MT is primarily computed by summing V1 afferents, but pattern-invariant velocity tuning for complex stimuli may arise from local, recurrent interactions.

Pattern Motion Processing by MT Neurons

Based on stimulation with plaid patterns, neurons in the Middle Temporal (MT) area of primate visual cortex are divided into two types: pattern and component cells. The prevailing theory suggests that pattern selectivity results from the summation of the outputs of component cells as part of a hierarchical visual pathway. We present a computational model of the visual pathway from primary visual cortex (V1) to MT that suggests an alternate model where the progression from component to pattern selectivity is not required. Using standard orientation-selective V1 cells, end-stopped V1 cells, and V1 cells with extra-classical receptive fields (RFs) as inputs to MT, the model shows that the degree of pattern or component selectivity in MT could arise from the relative strengths of the three V1 input types. Dominance of end-stopped V1 neurons in the model leads to pattern selectivity in MT, while dominance of V1 cells with extra-classical RFs result in component selectivity. This model may assist in designing experiments to further understand motion processing mechanisms in primate MT.

Spectral Signatures of Feedforward and Recurrent Circuitry in Monkey Area MT

Recordings of local field potential (LFP) in the visual cortex can show rhythmic activity at gamma frequencies (30-100 Hz). While the gamma rhythms in the primary visual cortex have been well studied, the structural and functional characteristics of gamma rhythms in extrastriate visual cortex are less clear. Here, we studied the spatial distribution and functional specificity of gamma rhythms in extrastriate middle temporal (MT) area of visual cortex in marmoset monkeys. We found that moving gratings induced narrowband gamma rhythms across cortical layers that were coherent across much of area MT. Moving dot fields instead induced a broadband increase in LFP in middle and upper layers, with weaker narrowband gamma rhythms in deeper layers. The stimulus dependence of LFP response in middle and upper layers of area MT appears to reflect the presence (gratings) or absence (dot fields and other textures) of strongly oriented contours. Our results suggest that gamma rhythms in these layers are propagated from earlier visual cortex, while those in the deeper layers may emerge in area MT.

A Motion-from-Form Mechanism Contributes to Extracting Pattern Motion from Plaids

Since the discovery of neurons selective for pattern motion direction in primate middle temporal area MT (Albright, 1984; Movshon et al., 1985), the neural computation of this signal has been the subject of intense study. The bulk of this work has explored responses to plaids obtained by summing two drifting sinusoidal gratings. Unfortunately, with these stimuli, many different mechanisms are similarly effective at extracting pattern motion. We devised a new set of stimuli, obtained by summing two random line stimuli with different orientations. This allowed several novel manipulations, including generating plaids that do not contain rigid 2D motion. Importantly, these stimuli do not engage most of the previously proposed mechanisms. We then recorded the ocular following responses that such stimuli induce in human subjects. We found that pattern motion is computed even with stimuli that do not cohere perceptually, including those without rigid motion, and even when the two gratings are presented separately to the two eyes. Moderate temporal and/or spatial separation of the gratings impairs the computation. We show that, of the models proposed so far, only those based on the intersection-of-constraints rule, embedding a motion-from-form mechanism (in which orientation signals are used in the computation of motion direction signals), can account for our results. At least for the eye movements reported here, a motion-from-form mechanism is thus involved in one of the most basic functions of the visual motion system: extracting motion direction from complex scenes.

SIGNIFICANCE STATEMENT

Anatomical considerations led to the proposal that visual function is organized in separate processing streams: one (ventral) devoted to form and one (dorsal) devoted to motion. Several experimental results have challenged this view, arguing in favor of a more integrated view of visual processing. Here we add to this body of work, supporting a role for form information even in a function--extracting pattern motion direction from complex scenes--for which decisive evidence for the involvement of form signals has been lacking.

A Completely Blind Video Integrity Oracle

Considerable progress has been made toward developing still picture perceptual quality analyzers that do not require any reference picture and that are not trained on human opinion scores of distorted images. However, there do not yet exist any such completely blind video quality assessment (VQA) models. Here, we attempt to bridge this gap by developing a new VQA model called the video intrinsic integrity and distortion evaluation oracle (VIIDEO). The new model does not require the use of any additional information other than the video being quality evaluated. VIIDEO embodies models of intrinsic statistical regularities that are observed in natural vidoes, which are used to quantify disturbances introduced due to distortions. An algorithm derived from the VIIDEO model is thereby able to predict the quality of distorted videos without any external knowledge about the pristine source, anticipated distortions, or human judgments of video quality. Even with such a paucity of information, we are able to show that the VIIDEO algorithm performs much better than the legacy full reference quality measure MSE on the LIVE VQA database and delivers performance comparable with a leading human judgment trained blind VQA model. We believe that the VIIDEO algorithm is a significant step toward making real-time monitoring of completely blind video quality possible.

Neural Mechanisms of Cortical Motion Computation Based on a Neuromorphic Sensory System

The visual cortex analyzes motion information along hierarchically arranged visual areas that interact through bidirectional interconnections. This work suggests a bio-inspired visual model focusing on the interactions of the cortical areas in which a new mechanism of feedforward and feedback processing are introduced. The model uses a neuromorphic vision sensor (silicon retina) that simulates the spike-generation functionality of the biological retina. Our model takes into account two main model visual areas, namely V1 and MT, with different feature selectivities. The initial motion is estimated in model area V1 using spatiotemporal filters to locally detect the direction of motion. Here, we adapt the filtering scheme originally suggested by Adelson and Bergen to make it consistent with the spike representation of the DVS. The responses of area V1 are weighted and pooled by area MT cells which are selective to different velocities, i.e. direction and speed. Such feature selectivity is here derived from compositions of activities in the spatio-temporal domain and integrating over larger space-time regions (receptive fields). In order to account for the bidirectional coupling of cortical areas we match properties of the feature selectivity in both areas for feedback processing. For such linkage we integrate the responses over different speeds along a particular preferred direction. Normalization of activities is carried out over the spatial as well as the feature domains to balance the activities of individual neurons in model areas V1 and MT. Our model was tested using different stimuli that moved in different directions. The results reveal that the error margin between the estimated motion and synthetic ground truth is decreased in area MT comparing with the initial estimation of area V1. In addition, the modulated V1 cell activations shows an enhancement of the initial motion estimation that is steered by feedback signals from MT cells.

Redundancy reduction explains the expansion of visual direction space around the cardinal axes

Motion direction discrimination in humans is worse for oblique directions than for the cardinal directions (the oblique effect). For some unknown reason, the human visual system makes systematic errors in the estimation of particular motion directions; a direction displacement near a cardinal axis appears larger than it really is whereas the same displacement near an oblique axis appears to be smaller. Although the perceptual effects are robust and are clearly measurable in smooth pursuit eye movements, all attempts to identify the neural underpinnings for the oblique effect have failed. Here we show that a model of image velocity estimation based on the known properties of neurons in primary visual cortex (V1) and the middle temporal (MT) visual area of the primate brain produces the oblique effect. We also provide an explanation for the unusual asymmetric patterns of inhibition that have been found surrounding MT neurons. These patterns are consistent with a mechanism within the visual system that prevents redundant velocity signals from being passed onto the next motion-integration stage, (dorsal Medial superior temporal, MSTd). We show that model redundancy-reduction mechanisms within the MT-MSTd pathway produce the oblique effect.

Adaptation disrupts motion integration in the primate dorsal stream

Sensory systems adjust continuously to the environment. The effects of recent sensory experience-or adaptation-are typically assayed by recording in a relevant subcortical or cortical network. However, adaptation effects cannot be localized to a single, local network. Adjustments in one circuit or area will alter the input provided to others, with unclear consequences for computations implemented in downstream circuits. Here, we show that prolonged adaptation with drifting gratings, which alters responses in the early visual system, impedes the ability of area MT neurons to integrate motion signals in plaid stimuli. Perceptual experiments reveal a corresponding loss of plaid coherence. A simple computational model shows how the altered representation of motion signals in early cortex can derail integration in MT. Our results suggest that the effects of adaptation cascade through the visual system, derailing the downstream representation of distinct stimulus attributes.

Blind prediction of natural video quality

We propose a blind (no reference or NR) video quality evaluation model that is nondistortion specific. The approach relies on a spatio-temporal model of video scenes in the discrete cosine transform domain, and on a model that characterizes the type of motion occurring in the scenes, to predict video quality. We use the models to define video statistics and perceptual features that are the basis of a video quality assessment (VQA) algorithm that does not require the presence of a pristine video to compare against in order to predict a perceptual quality score. The contributions of this paper are threefold. 1) We propose a spatio-temporal natural scene statistics (NSS) model for videos. 2) We propose a motion model that quantifies motion coherency in video scenes. 3) We show that the proposed NSS and motion coherency models are appropriate for quality assessment of videos, and we utilize them to design a blind VQA algorithm that correlates highly with human judgments of quality. The proposed algorithm, called video BLIINDS, is tested on the LIVE VQA database and on the EPFL-PoliMi video database and shown to perform close to the level of top performing reduced and full reference VQA algorithms.

Speed and direction response profiles of neurons in macaque MT and MST show modest constraint line tuning

Several models of heading detection during smooth pursuit rely on the assumption of local constraint line tuning to exist in large scale motion detection templates. A motion detector that exhibits pure constraint line tuning responds maximally to any 2D-velocity in the set of vectors that can be decomposed into the central, or classic, preferred velocity (the shortest vector that still yields the maximum response) and any vector orthogonal to that. To test this assumption, we measured the firing rates of isolated middle temporal (MT) and medial superior temporal (MST) neurons to random dot stimuli moving in a range of directions and speeds. We found that as a function of 2D velocity, the pooled responses were best fit with a 2D Gaussian profile with a factor of elongation, orthogonal to the central preferred velocity, of roughly 1.5 for MST and 1.7 for MT. This means that MT and MST cells are more sharply tuned for speed than they are for direction; and that they indeed show some level of constraint line tuning. However, we argue that the observed elongation is insufficient to achieve behavioral heading discrimination accuracy on the order of 1-2 degrees as reported before.

The role of V1 surround suppression in MT motion integration

Neurons in the primate extrastriate cortex are highly selective for complex stimulus features such as faces, objects, and motion patterns. One explanation for this selectivity is that neurons in these areas carry out sophisticated computations on the outputs of lower-level areas such as primary visual cortex (V1), where neuronal selectivity is often modeled in terms of linear spatiotemporal filters. However, it has long been known that such simple V1 models are incomplete because they fail to capture important nonlinearities that can substantially alter neuronal selectivity for specific stimulus features. Thus a key step in understanding the function of higher cortical areas is the development of realistic models of their V1 inputs. We have addressed this issue by constructing a computational model of the V1 neurons that provide the strongest input to extrastriate cortical middle temporal (MT) area. We find that a modest elaboration to the standard model of V1 direction selectivity generates model neurons with strong end-stopping, a property that is also found in the V1 layers that provide input to MT. With this computational feature in place, the seemingly complex properties of MT neurons can be simulated by assuming that they perform a simple nonlinear summation of their inputs. The resulting model, which has a very small number of free parameters, can simulate many of the diverse properties of MT neurons. In particular, we simulate the invariance of MT tuning curves to the orientation and length of tilted bar stimuli, as well as the accompanying temporal dynamics. We also show how this property relates to the continuum from component to pattern selectivity observed when MT neurons are tested with plaids. Finally, we confirm several key predictions of the model by recording from MT neurons in the alert macaque monkey. Overall our results demonstrate that many of the seemingly complex computations carried out by high-level cortical neurons can in principle be understood by examining the properties of their inputs.

Motion tuned spatio-temporal quality assessment of natural videos

There has recently been a great deal of interest in the development of algorithms that objectively measure the integrity of video signals. Since video signals are being delivered to human end users in an increasingly wide array of applications and products, it is important that automatic methods of video quality assessment (VQA) be available that can assist in controlling the quality of video being delivered to this critical audience. Naturally, the quality of motion representation in videos plays an important role in the perception of video quality, yet existing VQA algorithms make little direct use of motion information, thus limiting their effectiveness. We seek to ameliorate this by developing a general, spatio-spectrally localized multiscale framework for evaluating dynamic video fidelity that integrates both spatial and temporal (and spatio-temporal) aspects of distortion assessment. Video quality is evaluated not only in space and time, but also in space-time, by evaluating motion quality along computed motion trajectories. Using this framework, we develop a full reference VQA algorithm for which we coin the term the MOtion-based Video Integrity Evaluation index, or MOVIE index. It is found that the MOVIE index delivers VQA scores that correlate quite closely with human subjective judgment, using the Video Quality Expert Group (VQEG) FRTV Phase 1 database as a test bed. Indeed, the MOVIE index is found to be quite competitive with, and even outperform, algorithms developed and submitted to the VQEG FRTV Phase 1 study, as well as more recent VQA algorithms tested on this database.

A double dissociation between striate and extrastriate visual cortex for pattern motion perception revealed using rTMS

The neural mechanisms underlying the integration and segregation of motion signals are often studied using plaid stimuli. These stimuli consist of two spatially coincident dynamic gratings of differing orientations, which are either perceived to move in two unique directions or are integrated by the visual system to elicit the percept of a checkerboard moving in a single direction. Computations pertaining to the motion of the individual component gratings are thought to take place in striate cortex (V1) whereas motion integration is thought to involve neurons in dorsal stream extrastriate visual areas, particularly V5/MT. By combining a psychophysical task that employed plaid stimuli with 1 Hz offline repetitive transcranial magnetic stimulation (rTMS), we demonstrated a double dissociation between striate and extrastriate visual cortex in terms of their contributions to motion integration. rTMS over striate cortex increased coherent motion percepts whereas rTMS over extrastriate cortex had the opposite effect. These effects were robust directly after the stimulation administration and gradually returned to baseline within 15 minutes. This double dissociation is consistent with previous patient data and the recent hypothesis that both coherent and transparent motion percepts are supported by the visual system simultaneously and compete for perceptual dominance.

The aperture problem in contoured stimuli

A moving object elicits responses from V1 neurons tuned to a broad range of locations, directions, and spatiotemporal frequencies. Global pooling of such signals can overcome their intrinsic ambiguity in relation to the object's direction/speed (the "aperture problem"); here we examine the role of low-spatial frequencies (SF) and second-order statistics in this process. Subjects made a 2AFC fine direction-discrimination judgement of 'naturally' contoured stimuli viewed rigidly translating behind a series of small circular apertures. This configuration allowed us to manipulate the scene by randomly switching which portion of the stimulus was presented behind each aperture or by occluding certain spatial frequency bands. We report that global motion integration is (a) largely insensitive to the second-order statistics of such stimuli and (b) is rigidly broadband even in the presence of a disrupted low SF component.

Aging affects the neural representation of speed in Macaque area MT

Human perception of speed declines with age. Much of the decline is probably mediated by changes in the middle temporal (MT) area, an extrastriate area whose neural activity is linked to the perception of speed. In the present study, we used random-dot patterns to study the effects of aging on speed-tuning curves in cortical area MT of macaque visual cortex. Our results provide evidence for a significant degradation of speed selectivity in MT. Cells in old animals preferred lower speeds than did those in young animals. Response modulation and discriminative capacity for speed in old monkeys were also significantly weaker than those in young ones. Concurrently, MT cells in old monkeys showed increased baseline responses, peak responses and response variability, and these changes were accompanied by decreased signal-to-noise ratios. We also found that speed discrimination thresholds in old animals were higher than in young ones. The foregoing neural changes may mediate the declines in visual motion perception that occur during senescence.

The effect of overall stimulus velocity on motion parallax

This study examined the effectiveness with which motion parallax information can be utilized by rhesus monkeys for depth perception. A visual display comprised of random-dots that mimicked a rigid, three-dimensional object rocking back and forth was used. Differential depth was produced by presenting sub-regions of the dots moving at different velocities from the rest of dots in the display. The tasks for the monkeys were to detect or discriminate a target region that was protruding the furthest from the background plane. To understand the role of stimulus movement, we examined the accuracy and the rapidity of the saccadic responses as a function of rocking velocity of the entire three-dimensional object. The results showed that performance accuracy improved and reaction times decreased with increasing rocking velocities. The monkeys can process the motion parallax information with remarkable rapidity such that the average reaction time ranged between 212 and 246 milliseconds. The data collected suggest that the successive activation of just two sets of cones is sufficient to perform the task.

Misperceptions of speed for chromatic and luminance grating stimuli

Errors in the perception of speed of moving visual stimuli can occur when presented stimuli are of unequal contrast and when they appear alongside additional modifier stimuli that move at different speeds. We have examined these misperceptions for chromatic and luminance grating stimuli in order to assess to what extent these different kinds of motion cue might be utilised in the analysis of speed of moving objects. We show that the dependence on contrast of speed matching for luminance and chromatic stimuli is similar over a range of stimulus speeds greater than 4 deg/s. Differences between the contrast dependencies of speed perception for chromatic and luminance stimuli are only evident at slow speeds (< 4 deg/s) and low contrasts. The presence of modifier stimuli can directly influence the perceived speed at both high and low velocities and contrasts. This influence was found to be independent of the modifiers' chromaticity and was greatest when the modifiers were adjacent to and presented simultaneously with the test and reference stimuli. However, the modifiers were still able to induce measurable changes in perceived speed for increased separations over space and time. Taken together these results indicate that whilst differences do exist in the contrast dependencies of speed perception for chromatic and luminance stimuli, they are evident only for a narrow range of stimulus parameters (i.e. low speed and low contrast). There appears to be ample scope for interactions between chromatic and luminance contrast in speed perception where there is the capacity to pool this information over a relatively broad spatio-temporal extent.

A single mechanism can explain the speed tuning properties of MT and V1 complex neurons

A recent study by Priebe et al., (2006) has shown that a small proportion (27%) of primate directionally selective, complex V1 neurons are tuned for the speed of image motion. In this study, I show that the weighted intersection mechanism (WIM) model, which was previously proposed to explain speed tuning in middle temporal neurons, can also explain the tuning found in complex V1 neurons. With the addition of a contrast gain mechanism, this model is able to replicate the effects of contrast on V1 speed tuning, a phenomenon that was recently discovered by Priebe et al., (2006). The WIM model simulations also indicate that V1 neuron spatiotemporal frequency response maps may be asymmetrical in shape and hence poorly characterized by the symmetrical two-dimensional Gaussian fitting function used by Priebe et al., (2006) to classify their cells. Therefore, the actual proportion of speed tuning among directional complex V1 cells may be higher than the 27% estimate suggested by these authors.

Measurement of rate of expansion in the perception of radial motion

Optic flow generated by rigid surface patches can be decomposed into a small number of elementary motion types. In these experiments, we show that the human visual system can evaluate expansion, one of these motion types, metrically. Moreover, we show that the discrimination of rates of expansion are spatially local. Because the estimation of the focus of expansion is somewhat imprecise, this locality sometimes produces predictable errors in the estimation of rate of expansion. One can make predictions like this with a model adapted from one previously developed for angular-velocity discrimination.