Neural correlates of structure-from-motion perception in macaque V1 and MT

Gestalt neurons and emergent properties in visual perception: A novel concept for the transformation from local to global processing

Gestalten in visual perception are defined by emergent properties of the whole, which cannot be predicted from the sum of its parts; rather, they arise by virtue of inherent principles, the Laws of Seeing. This review attempts to assign neurophysiological correlates to select emergent properties in motion and contour perception and proposes parallels to the processing of local versus global attributes by classical versus contextual receptive fields. The aim is to identify Gestalt neurons in the visual system to account for the Laws of Seeing in causal terms and to explain "Why do things look as they do" (Koffka, 1935, p. 76).

Identifying cortical areas that underlie the transformation from 2D retinal to 3D head-centric motion signals

Accurate motion perception requires that the visual system integrate the 2D retinal motion signals received by the two eyes into a single representation of 3D motion. However, most experimental paradigms present the same stimulus to the two eyes, signaling motion limited to a 2D fronto-parallel plane. Such paradigms are unable to dissociate the representation of 3D head-centric motion signals (i.e., 3D object motion relative to the observer) from the associated 2D retinal motion signals. Here, we used stereoscopic displays to present separate motion signals to the two eyes and examined their representation in visual cortex using fMRI. Specifically, we presented random-dot motion stimuli that specified various 3D head-centric motion directions. We also presented control stimuli, which matched the motion energy of the retinal signals, but were inconsistent with any 3D motion direction. We decoded motion direction from BOLD activity using a probabilistic decoding algorithm. We found that 3D motion direction signals can be reliably decoded in three major clusters in the human visual system. Critically, in early visual cortex (V1-V3), we found no significant difference in decoding performance between stimuli specifying 3D motion directions and the control stimuli, suggesting that these areas represent the 2D retinal motion signals, rather than 3D head-centric motion itself. In voxels in and surrounding hMT and IPS0 however, decoding performance was consistently superior for stimuli that specified 3D motion directions compared to control stimuli. Our results reveal the parts of the visual processing hierarchy that are critical for the transformation of retinal into 3D head-centric motion signals and suggest a role for IPS0 in their representation, in addition to its sensitivity to 3D object structure and static depth.

Attractive serial dependence overcomes repulsive neuronal adaptation

Sensory responses and behavior are strongly shaped by stimulus history. For example, perceptual reports are sometimes biased toward previously viewed stimuli (serial dependence). While behavioral studies have pointed to both perceptual and postperceptual origins of this phenomenon, neural data that could elucidate where these biases emerge is limited. We recorded functional magnetic resonance imaging (fMRI) responses while human participants (male and female) performed a delayed orientation discrimination task. While behavioral reports were attracted to the previous stimulus, response patterns in visual cortex were repelled. We reconciled these opposing neural and behavioral biases using a model where both sensory encoding and readout are shaped by stimulus history. First, neural adaptation reduces redundancy at encoding and leads to the repulsive biases that we observed in visual cortex. Second, our modeling work suggest that serial dependence is induced by readout mechanisms that account for adaptation in visual cortex. According to this account, the visual system can simultaneously improve efficiency via adaptation while still optimizing behavior based on the temporal structure of natural stimuli.

The coding principals of early sensory regions are in constant flux due to adaptation, but how does the brain interpret these labile signals from early sensory areas? This study finds that a visual illusion known as serial dependence can be explained by a model where readout of these early areas also changes dynamically; the model is supported by neuroimaging and behavioral data.

A neural correlate of perceptual segmentation in macaque middle temporal cortical area

High-resolution vision requires fine retinal sampling followed by integration to recover object properties. Importantly, accuracy is lost if local samples from different objects are intermixed. Thus, segmentation, grouping of image regions for separate processing, is crucial for perception. Previous work has used bi-stable plaid patterns, which can be perceived as either a single or multiple moving surfaces, to study this process. Here, we report a relationship between activity in a mid-level site in the primate visual pathways and segmentation judgments. Specifically, we find that direction selective middle temporal neurons are sensitive to texturing cues used to bias the perception of bi-stable plaids and exhibit a significant trial-by-trial correlation with subjective perception of a constant stimulus. This correlation is greater in units that signal global motion in patterns with multiple local orientations. Thus, we conclude the middle temporal area contains a signal for segmenting complex scenes into constituent objects and surfaces.

Bi-Stable Perception: Self-Coordinating Brain Regions to Make-Up the Mind

Bi-stable perception is a strong instance of cognitive self-organization, providing a research model for how ‘the brain makes up its mind.’ The complexity of perceptual bistability prevents a simple attribution of functions to areas, because many cognitive processes, recruiting multiple brain regions, are simultaneously involved. The functional magnetic resonance imaging (fMRI) evidence suggests the activation of a large network of distant brain areas. Concurrently, electroencephalographic and magnetoencephalographic (MEEG) literature shows sub second oscillatory activity and phase synchrony on several frequency bands. Strongly represented are beta and gamma bands, often associated with neural/cognitive integration processes. The spatial extension and short duration of brain activities suggests the need for a fast, large-scale neural coordination mechanism. To address the range of temporo-spatial scales involved, we systematize the current knowledge from mathematical models, cognitive sciences and neuroscience at large, from single-cell- to system-level research, including evidence from human and non-human primates. Surprisingly, despite evidence spanning through different organization levels, models, and experimental approaches, the scarcity of integrative studies is evident. In a final section of the review we dwell on the reasons behind such scarcity and on the need of integration in order to achieve a real understanding of the complexities underlying bi-stable perception processes.

Perceptual rivalry across animal species

This review in memoriam of Jack Pettigrew provides an overview of past and current research into the phenomenon of multistable perception across multiple animal species. Multistable perception is characterized by two or more perceptual interpretations spontaneously alternating, or rivaling, when animals are exposed to stimuli with inherent sensory ambiguity. There is a wide array of ambiguous stimuli across sensory modalities, ranging from the configural changes observed in simple line drawings, such as the famous Necker cube, to the alternating perception of entire visual scenes that can be instigated by interocular conflict. The latter phenomenon, called binocular rivalry, in particular caught the attention of the late Jack Pettigrew, who combined his interest in the neuronal basis of perception with a unique comparative biological approach that considered ambiguous sensation as a fundamental problem of sensory systems that has shaped the brain throughout evolution. Here, we examine the research findings on visual perceptual alternation and suppression in a wide variety of species including insects, fish, reptiles, and primates. We highlight several interesting commonalities across species and behavioral indicators of perceptual alternation. In addition, we show how the comparative approach provides new avenues for understanding how the brain suppresses opposing sensory signals and generates alternations in perceptual dominance.

Decreased amplitude and reliability of odor-evoked responses in two mouse models of autism

Sensory processing deficits are increasingly recognized as core symptoms of autism spectrum disorders (ASDs). However the molecular and circuit mechanisms that lead to sensory deficits are unknown. We show that two molecularly disparate mouse models of autism display similar deficits in sensory-evoked responses in the mouse olfactory system. We find that both Cntnap2- and Shank3-deficient mice of both sexes exhibit reduced response amplitude and trial-to-trial reliability during repeated odor presentation. Mechanistically, we show that both mouse models have weaker and fewer synapses between olfactory sensory nerve (OSN) terminals and olfactory bulb tufted cells and weaker synapses between OSN terminals and inhibitory periglomerular cells. Consequently, deficits in sensory processing provide an excellent candidate phenotype for analysis in ASDs.NEW & NOTEWORTHY The genetics of autism spectrum disorder (ASD) are complex. How the many risk genes generate the similar sets of symptoms that define the disorder is unknown. In particular, little is understood about the functional consequences of these genetic alterations. Sensory processing deficits are important aspects of the ASD diagnosis and may be due to unreliable neural circuits. We show that two mouse models of autism, Cntnap2- and Shank3-deficient mice, display reduced odor-evoked response amplitudes and reliability. These data suggest that altered sensory-evoked responses may constitute a circuit phenotype in ASDs.

Task-related hemodynamic responses are modulated by reward and task engagement

Hemodynamic recordings from visual cortex contain powerful endogenous task-related responses that may reflect task-related arousal, or "task engagement" distinct from attention. We tested this hypothesis with hemodynamic measurements (intrinsic-signal optical imaging) from monkey primary visual cortex (V1) while the animals' engagement in a periodic fixation task over several hours was varied through reward size and as animals took breaks. With higher rewards, animals appeared more task-engaged; task-related responses were more temporally precise at the task period (approximately 10-20 seconds) and modestly stronger. The 2-5 minute blocks of high-reward trials led to ramp-like decreases in mean local blood volume; these reversed with ramp-like increases during low reward. The blood volume increased even more sharply when the animal shut his eyes and disengaged completely from the task (5-10 minutes). We propose a mechanism that controls vascular tone, likely along with local neural responses in a manner that reflects task engagement over the full range of timescales tested.

Magnetoencephalography adaptation reveals depth-cue-invariant object representations in the visual cortex

Independent of edges and 2-D shape that can be highly informative of object identity, depth cues alone can also give rise to vivid and effective object percepts. The processing of different depth cues engages segregated cortical areas, and an efficient object representation would be one that is invariant to depth cues. Here, we investigated depth-cue invariance of object representations by measuring the category-specific response to faces-the M170 response measured with magnetoencephalography. The M170 response is strongest to faces and is sensitive to adaptation, such that repeated presentation of a face diminishes subsequent M170 responses. We used this feature of the M170 and measured the degree to which the adaptation effect is affected by variations in depth cue and 3-D object shape. Subjects viewed a rapid presentation of two stimuli-an adaptor and a test stimulus. The adaptor was either a face, a chair, or a face-like oval surface, and rendered with a single depth cue (shading, structure from motion, or texture). The test stimulus was always a shaded face of a random identity, thus completely controlling for low-level influences on the M170 response to the test stimulus. In the left fusiform face area, we found strong M170 adaptation when the adaptor was a face regardless of its depth cue. This adaptation was marginal in the right fusiform and negligible in the occipital regions. Our results support the presence of depth-cue-invariant representations in the human visual system, alongside size, position, and viewpoint invariance.

Dynamic Visual Cues for Differentiating Mirror and Glass

Mirror materials (perfect specular surfaces such as polished metal) and glass materials (transparent and refraction media) are quite commonly encountered in everyday life. The human visual system can discriminate these complex distorted images formed by reflection or transmission of the surrounding environment even though they do not intrinsically possess surface colour. In this study, we determined the cues that aid mirror and glass discrimination. From video analysis, we found that glass objects have more opposite motion components relative to the direction of object rotation. Then, we hypothesised a model developed using motion transparency because motion information is not only present on the front side, but also on the rear side of the object surface in the glass material object. In materials judging experiments, we found that human performance with rotating video stimuli is higher than that with static stimuli (simple images). Subsequently, we compared the developed model derived from motion coherency to human rating performance for transparency and specular reflection. The model sufficiently identified the different materials using dynamic information. These results suggest that the visual system relies on dynamic cues that indicate the difference between mirror and glass.

Multi-stability with ambiguous visual stimuli in Drosophila orientation behavior

It is widely accepted for humans and higher animals that vision is an active process in which the organism interprets the stimulus. To find out whether this also holds for lower animals, we designed an ambiguous motion stimulus, which serves as something like a multi-stable perception paradigm in Drosophila behavior. Confronted with a uniform panoramic texture in a closed-loop situation in stationary flight, the flies adjust their yaw torque to stabilize their virtual self-rotation. To make the visual input ambiguous, we added a second texture. Both textures got a rotatory bias to move into opposite directions at a constant relative angular velocity. The results indicate that the fly now had three possible frames of reference for self-rotation: either of the two motion components as well as the integrated motion vector of the two. In this ambiguous stimulus situation, the flies generated a continuous sequence of behaviors, each one adjusted to one or another of the three references.

Vision is considered an active process in humans and higher animals in which the stimulus is interpreted by the subject and can be perceived in different ways if it is ambiguous. We aimed to find out whether this also holds for lower animals, such as the fruit fly Drosophila melanogaster. To provide ambiguity, we exposed flies to transparent motion stimuli in a flight simulator and found their behavior to be multi-stable. These results show that the visual system of the fly can separate the individual components of a transparent motion stimulus, and that this kind of stimulus is ambiguous to the fly. The extent to which the fly shows component selectivity in its behavior depends on several properties of the stimulus, like pattern contrast and element density. The alternations between the different behaviors exhibit a stochasticity reminiscent of the temporal dynamics in human multi-stable perception.

Contrast sensitivity, V1 neural activity, and natural vision

Contrast sensitivity is fundamental to natural visual processing and an important tool for characterizing both visual function and clinical disorders. We simultaneously measured contrast sensitivity and neural contrast response functions and compared measurements in common laboratory conditions with naturalistic conditions. In typical experiments, a subject holds fixation and a stimulus is flashed on, whereas in natural vision, saccades bring stimuli into view. Motivated by our previous V1 findings, we tested the hypothesis that perceptual contrast sensitivity is lower in natural vision and that this effect is associated with corresponding changes in V1 activity. We found that contrast sensitivity and V1 activity are correlated and that the relationship is similar in laboratory and naturalistic paradigms. However, in the more natural situation, contrast sensitivity is reduced up to 25% compared with that in a standard fixation paradigm, particularly at lower spatial frequencies, and this effect correlates with significant reductions in V1 responses. Our data suggest that these reductions in natural vision result from fast adaptation on one fixation that lowers the response on a subsequent fixation. This is the first demonstration of rapid, natural-image adaptation that carries across saccades, a process that appears to constantly influence visual sensitivity in natural vision.

NEW & NOTEWORTHY

Visual sensitivity and activity in brain area V1 were studied in a paradigm that included saccadic eye movements and natural visual input. V1 responses and contrast sensitivity were significantly reduced compared with results in common laboratory paradigms. The parallel neural and perceptual effects of eye movements and stimulus complexity appear to be due to a form of rapid adaptation that carries across saccades.

How to Keep a Reversible Figure from Reversing: Teasing Out Top — Down and Bottom — Up Processes

The nature of processes underlying our perception of reversible figures was examined through two experiments investigating the effects of prior exposure conditions on an observer's report of figural reversal. In experiment 1, observers were adapted over several minutes to an unambiguous version of a rotating Necker cube prior to the presentation of the standard ambiguous figure. Results indicated that adaptation produced an immediate bias to perceive the ambiguous figure in the opposite configuration (ie reverse bias) and to reduce reports of reversal over the test period. The introduction of a brief delay between the adaptation and test periods revealed that this bias is a highly transient effect and is only clearly evident when the adaptation and test figures are matched in size. In experiment 2, observers were primed with an unambiguous figure for a few seconds prior to the presentation of the standard ambiguous figure. In this case, the obtained bias strongly favored the observer's reporting the ambiguous figure to be in the same configuration as the adapting figure (ie positive bias); and neither introducing a delay period nor changing figure size had any effect. We conclude that these experiments reveal the distinct roles of transient, retinally localized neural processes as well as more stable, global processes under specifiable conditions.

Dorsal and ventral stream contributions to form-from-motion perception in a patient with form-from motion deficit: a case report

The main model of visual processing in primates proposes an anatomo-functional distinction between the dorsal stream, specialized in spatio-temporal information, and the ventral stream, processing essentially form information. However, these two pathways also communicate to share much visual information. These dorso-ventral interactions have been studied using form-from-motion (FfM) stimuli, revealing that FfM perception first activates dorsal regions (e.g., MT+/V5), followed by successive activations of ventral regions (e.g., LOC). However, relatively little is known about the implications of focal brain damage of visual areas on these dorso-ventral interactions. In the present case report, we investigated the dynamics of dorsal and ventral activations related to FfM perception (using topographical ERP analysis and electrical source imaging) in a patient suffering from a deficit in FfM perception due to right extrastriate brain damage in the ventral stream. Despite the patient's FfM impairment, both successful (observed for the highest level of FfM signal) and absent/failed FfM perception evoked the same temporal sequence of three processing states observed previously in healthy subjects. During the first period, brain source localization revealed cortical activations along the dorsal stream, currently associated with preserved elementary motion processing. During the latter two periods, the patterns of activity differed from normal subjects: activations were observed in the ventral stream (as reported for normal subjects), but also in the dorsal pathway, with the strongest and most sustained activity localized in the parieto-occipital regions. On the other hand, absent/failed FfM perception was characterized by weaker brain activity, restricted to the more lateral regions. This study shows that in the present case report, successful FfM perception, while following the same temporal sequence of processing steps as in normal subjects, evoked different patterns of brain activity. By revealing a brain circuit involving the most rostral part of the dorsal pathway, this study provides further support for neuro-imaging studies and brain lesion investigations that have suggested the existence of different brain circuits associated with different profiles of interaction between the dorsal and the ventral streams.

The neural representation of objects formed through the spatiotemporal integration of visual transients

Oftentimes, objects are only partially and transiently visible as parts of them become occluded during observer or object motion. The visual system can integrate such object fragments across space and time into perceptual wholes or spatiotemporal objects. This integrative and dynamic process may involve both ventral and dorsal visual processing pathways, along which shape and spatial representations are thought to arise. We measured fMRI BOLD response to spatiotemporal objects and used multi-voxel pattern analysis (MVPA) to decode shape information across 20 topographic regions of visual cortex. Object identity could be decoded throughout visual cortex, including intermediate (V3A, V3B, hV4, LO1-2,) and dorsal (TO1-2, and IPS0-1) visual areas. Shape-specific information, therefore, may not be limited to early and ventral visual areas, particularly when it is dynamic and must be integrated. Contrary to the classic view that the representation of objects is the purview of the ventral stream, intermediate and dorsal areas may play a distinct and critical role in the construction of object representations across space and time.

Feedforward and feedback sources of choice probability in neural population responses

How the processing of signals carried by sensory neurons supports perceptual decisions is a long-standing question in neuroscience. The ability to record neuronal activity in awake animals while they perform psychophysical tasks near threshold has been a key advance in studying these questions. Trial-to-trial correlations between the activity of sensory neurons and the decisions reported by animals ('choice probabilities'), even when measured across repeated presentations of an identical stimulus provide insights into this problem. But understanding the sources of such co-variability between sensory neurons and behavior has proven more difficult than it initially appeared. Below, we discuss our current understanding of what gives rise to these correlations.

A computational relationship between thalamic sensory neural responses and contrast perception

Uncovering the relationship between sensory neural responses and perceptual decisions remains a fundamental problem in neuroscience. Decades of experimental and modeling work in the sensory cortex have demonstrated that a perceptual decision pool is usually composed of tens to hundreds of neurons, the responses of which are significantly correlated not only with each other, but also with the behavioral choices of an animal. Few studies, however, have measured neural activity in the sensory thalamus of awake, behaving animals. Therefore, it remains unclear how many thalamic neurons are recruited and how the information from these neurons is pooled at subsequent cortical stages to form a perceptual decision. In a previous study we measured neural activity in the macaque lateral geniculate nucleus (LGN) during a two alternative forced choice (2AFC) contrast detection task, and found that single LGN neurons were significantly correlated with the monkeys’ behavioral choices, despite their relatively poor contrast sensitivity and a lack of overall interneuronal correlations. We have now computationally tested a number of specific hypotheses relating these measured LGN neural responses to the contrast detection behavior of the animals. We modeled the perceptual decisions with different numbers of neurons and using a variety of pooling/readout strategies, and found that the most successful model consisted of about 50–200 LGN neurons, with individual neurons weighted differentially according to their signal-to-noise ratios (quantified as d-primes). These results supported the hypothesis that in contrast detection the perceptual decision pool consists of multiple thalamic neurons, and that the response fluctuations in these neurons can influence contrast perception, with the more sensitive thalamic neurons likely to exert a greater influence.

The functional asymmetry of ON and OFF channels in the perception of contrast

To fully understand the relationship between perception and single neural responses, one should take into consideration the early stages of sensory processing. Few studies, however, have directly examined the neural underpinning of visual perception in the lateral geniculate nucleus (LGN), only one synapse away from the retina. In this study we recorded from LGN parvocellular (P) ON-center and OFF-center neurons while monkeys either passively viewed or actively detected a full range of contrasts. We found that OFF neurons were more sensitive in detecting negative contrasts than ON neurons were in detecting positive contrasts. Also, OFF neurons had higher spontaneous activities, higher peak response amplitudes, and were more sustained than ON neurons in their contrast responses. Puzzlingly, OFF neurons failed to show any significant correlations with the monkeys' perceptual choices, despite their greater contrast sensitivities. If, however, choice probabilities were calculated from interspike intervals instead of spike counts (thus taking into account the higher firing rates of OFF neurons), OFF neurons but not ON neurons were significantly correlated with behavioral choices. Taken together, these results demonstrate in awake, behaving animals that: 1) the ON and OFF pathways do not simply mirror each other in their functionality but instead carry qualitatively different types of information, and 2) the responses of ON and OFF neurons can be correlated with perceptual choices even in the absence of physical stimuli and interneuronal correlations.

Perceptual decision related activity in the lateral geniculate nucleus

Fundamental to neuroscience is the understanding of how the language of neurons relates to behavior. In the lateral geniculate nucleus (LGN), cells show distinct properties such as selectivity for particular wavelengths, increments or decrements in contrast, or preference for fine detail versus rapid motion. No studies, however, have measured how LGN cells respond when an animal is challenged to make a perceptual decision using information within the receptive fields of those LGN cells. In this study we measured neural activity in the macaque LGN during a two-alternative, forced-choice (2AFC) contrast detection task or during a passive fixation task and found that a small proportion (13.5%) of single LGN parvocellular (P) and magnocellular (M) neurons matched the psychophysical performance of the monkey. The majority of LGN neurons measured in both tasks were not as sensitive as the monkey. The covariation between neural response and behavior (quantified as choice probability) was significantly above chance during active detection, even when there was no external stimulus. Interneuronal correlations and task-related gain modulations were negligible under the same condition. A bottom-up pooling model that used sensory neural responses to compute perceptual choices in the absence of interneuronal correlations could fully explain these results at the level of the LGN, supporting the hypothesis that the perceptual decision pool consists of multiple sensory neurons and that response fluctuations in these neurons can influence perception.

Unconscious local motion alters global image speed

Accurate motion perception of self and object speed is crucial for successful interaction in the world. The context in which we make such speed judgments has a profound effect on their accuracy. Misperceptions of motion speed caused by the context can have drastic consequences in real world situations, but they also reveal much about the underlying mechanisms of motion perception. Here we show that motion signals suppressed from awareness can warp simultaneous conscious speed perception. In Experiment 1, we measured global speed discrimination thresholds using an annulus of 8 local Gabor elements. We show that physically removing local elements from the array attenuated global speed discrimination. However, removing awareness of the local elements only had a small effect on speed discrimination. That is, unconscious local motion elements contributed to global conscious speed perception. In Experiment 2 we measured the global speed of the moving Gabor patterns, when half the elements moved at different speeds. We show that global speed averaging occurred regardless of whether local elements were removed from awareness, such that the speed of invisible elements continued to be averaged together with the visible elements to determine the global speed. These data suggest that contextual motion signals outside of awareness can both boost and affect our experience of motion speed, and suggest that such pooling of motion signals occurs before the conscious extraction of the surround motion speed.

Feature integration and object representations along the dorsal stream visual hierarchy

The visual system is split into two processing streams: a ventral stream that receives color and form information and a dorsal stream that receives motion information. Each stream processes that information hierarchically, with each stage building upon the previous. In the ventral stream this leads to the formation of object representations that ultimately allow for object recognition regardless of changes in the surrounding environment. In the dorsal stream, this hierarchical processing has classically been thought to lead to the computation of complex motion in three dimensions. However, there is evidence to suggest that there is integration of both dorsal and ventral stream information into motion computation processes, giving rise to intermediate object representations, which facilitate object selection and decision making mechanisms in the dorsal stream. First we review the hierarchical processing of motion along the dorsal stream and the building up of object representations along the ventral stream. Then we discuss recent work on the integration of ventral and dorsal stream features that lead to intermediate object representations in the dorsal stream. Finally we propose a framework describing how and at what stage different features are integrated into dorsal visual stream object representations. Determining the integration of features along the dorsal stream is necessary to understand not only how the dorsal stream builds up an object representation but also which computations are performed on object representations instead of local features.

Subjective visual perception: from local processing to emergent phenomena of brain activity

The combination of electrophysiological recordings with ambiguous visual stimulation made possible the detection of neurons that represent the content of subjective visual perception and perceptual suppression in multiple cortical and subcortical brain regions. These neuronal populations, commonly referred to as the neural correlates of consciousness, are more likely to be found in the temporal and prefrontal cortices as well as the pulvinar, indicating that the content of perceptual awareness is represented with higher fidelity in higher-order association areas of the cortical and thalamic hierarchy, reflecting the outcome of competitive interactions between conflicting sensory information resolved in earlier stages. However, despite the significant insights into conscious perception gained through monitoring the activities of single neurons and small, local populations, the immense functional complexity of the brain arising from correlations in the activity of its constituent parts suggests that local, microscopic activity could only partially reveal the mechanisms involved in perceptual awareness. Rather, the dynamics of functional connectivity patterns on a mesoscopic and macroscopic level could be critical for conscious perception. Understanding these emergent spatio-temporal patterns could be informative not only for the stability of subjective perception but also for spontaneous perceptual transitions suggested to depend either on the dynamics of antagonistic ensembles or on global intrinsic activity fluctuations that may act upon explicit neural representations of sensory stimuli and induce perceptual reorganization. Here, we review the most recent results from local activity recordings and discuss the potential role of effective, correlated interactions during perceptual awareness.

The role of human ventral visual cortex in motion perception

Visual motion perception is fundamental to many aspects of visual perception. Visual motion perception has long been associated with the dorsal (parietal) pathway and the involvement of the ventral 'form' (temporal) visual pathway has not been considered critical for normal motion perception. Here, we evaluated this view by examining whether circumscribed damage to ventral visual cortex impaired motion perception. The perception of motion in basic, non-form tasks (motion coherence and motion detection) and complex structure-from-motion, for a wide range of motion speeds, all centrally displayed, was assessed in five patients with a circumscribed lesion to either the right or left ventral visual pathway. Patients with a right, but not with a left, ventral visual lesion displayed widespread impairments in central motion perception even for non-form motion, for both slow and for fast speeds, and this held true independent of the integrity of areas MT/V5, V3A or parietal regions. In contrast with the traditional view in which only the dorsal visual stream is critical for motion perception, these novel findings implicate a more distributed circuit in which the integrity of the right ventral visual pathway is also necessary even for the perception of non-form motion.

Inferring decoding strategies from choice probabilities in the presence of correlated variability

The activity of cortical neurons in sensory areas covaries with perceptual decisions, a relationship that is often quantified by choice probabilities. Although choice probabilities have been measured extensively, their interpretation has remained fraught with difficulty. We derive the mathematical relationship between choice probabilities, read-out weights and correlated variability in the standard neural decision-making model. Our solution allowed us to prove and generalize earlier observations on the basis of numerical simulations and to derive new predictions. Notably, our results indicate how the read-out weight profile, or decoding strategy, can be inferred from experimentally measurable quantities. Furthermore, we developed a test to decide whether the decoding weights of individual neurons are optimal for the task, even without knowing the underlying correlations. We confirmed the practicality of our approach using simulated data from a realistic population model. Thus, our findings provide a theoretical foundation for a growing body of experimental results on choice probabilities and correlations.

Visual motion and the perception of surface material

Many critical perceptual judgments, from telling whether fruit is ripe to determining whether the ground is slippery, involve estimating the material properties of surfaces. Very little is known about how the brain recognizes materials, even though the problem is likely as important for survival as navigating or recognizing objects. Though previous research has focused nearly exclusively on the properties of static images, recent evidence suggests that motion may affect the appearance of surface material. However, what kind of information motion conveys and how this information may be used by the brain is still unknown. Here, we identify three motion cues that the brain could rely on to distinguish between matte and shiny surfaces. We show that these motion measurements can override static cues, leading to dramatic changes in perceived material depending on the image motion characteristics. A classifier algorithm based on these cues correctly predicts both successes and some striking failures of human material perception. Together these results reveal a previously unknown use for optic flow in the perception of surface material properties.

Disambiguation of Necker cube rotation by monocular and binocular depth cues: relative effectiveness for establishing long-term bias

The apparent direction of rotation of perceptually bistable wire-frame (Necker) cubes can be conditioned to depend on retinal location by interleaving their presentation with cubes that are disambiguated by depth cues (Haijiang, Saunders, Stone, & Backus, 2006; Harrison & Backus, 2010a). The long-term nature of the learned bias is demonstrated by resistance to counter-conditioning on a consecutive day. In previous work, either binocular disparity and occlusion, or a combination of monocular depth cues that included occlusion, internal occlusion, haze, and depth-from-shading, were used to control the rotation direction of disambiguated cubes. Here, we test the relative effectiveness of these two sets of depth cues in establishing the retinal location bias. Both cue sets were highly effective in establishing a perceptual bias on Day 1 as measured by the perceived rotation direction of ambiguous cubes. The effect of counter-conditioning on Day 2, on perceptual outcome for ambiguous cubes, was independent of whether the cue set was the same or different as Day 1. This invariance suggests that a common neural population instantiates the bias for rotation direction, regardless of the cue set used. However, in a further experiment where only disambiguated cubes were presented on Day 1, perceptual outcome of ambiguous cubes during Day 2 counter-conditioning showed that the monocular-only cue set was in fact more effective than disparity-plus-occlusion for causing long-term learning of the bias. These results can be reconciled if the conditioning effect of Day 1 ambiguous trials in the first experiment is taken into account (Harrison & Backus, 2010b). We suggest that monocular disambiguation leads to stronger bias either because it more strongly activates a single neural population that is necessary for perceiving rotation, or because ambiguous stimuli engage cortical areas that are also engaged by monocularly disambiguated stimuli but not by disparity-disambiguated stimuli.

Representation of speech categories in the primate auditory cortex

A "ventral" auditory pathway in nonhuman primates that originates in the core auditory cortex and ends in the prefrontal cortex is thought to be involved in components of nonspatial auditory processing. Previous work from our laboratory has indicated that neurons in the prefrontal cortex reflect monkeys' decisions during categorical judgments. Here, we tested the role of the superior temporal gyrus (STG), a region of the secondary auditory cortex that is part of this ventral pathway, during similar categorical judgments. While monkeys participated in a match-to-category task and reported whether two consecutive auditory stimuli belonged to the same category or to different categories, we recorded spiking activity from STG neurons. The auditory stimuli were morphs of two human-speech sounds (bad and dad). We found that STG neurons represented auditory categories. However, unlike activity in the prefrontal cortex, STG activity was not modulated by the monkeys' behavioral reports (choices). This finding is consistent with the anterolateral STG's role as a part of functional circuit involved in the coding, representation, and perception of the nonspatial features of an auditory stimulus.

Discerning nonrigid 3D shapes from motion cues

Many organisms and objects deform nonrigidly when moving, requiring perceivers to separate shape changes from object motions. Surprisingly, the abilities of observers to correctly infer nonrigid volumetric shapes from motion cues have not been measured, and structure from motion models predominantly use variants of rigidity assumptions. We show that observers are equally sensitive at discriminating cross-sections of flexing and rigid cylinders based on motion cues, when the cylinders are rotated simultaneously around the vertical and depth axes. A computational model based on motion perspective (i.e., assuming perceived depth is inversely proportional to local velocity) predicted the psychometric curves better than shape from motion factorization models using shape or trajectory basis functions. Asymmetric percepts of symmetric cylinders, arising because of asymmetric velocity profiles, provided additional evidence for the dominant role of relative velocity in shape perception. Finally, we show that inexperienced observers are generally incapable of using motion cues to detect inflation/deflation of rigid and flexing cylinders, but this handicap can be overcome with practice for both nonrigid and rigid shapes. The empirical and computational results of this study argue against the use of rigidity assumptions in extracting 3D shape from motion and for the primacy of motion deformations computed from motion shears.

Correlations between the activity of sensory neurons and behavior: how much do they tell us about a neuron's causality?

How the activity of sensory neurons elicits perceptions and guides behavior is central to our understanding of the brain and is a subject of intense investigation in neuroscience. Correlations between the activity of sensory neurons and behavior have been widely observed and are sometimes used to infer how neurons are used to guide a certain behavior. This view is challenged firstly by theoretical considerations that these correlations rely on the existence of correlated noise and its structure, and secondly by recent empirical observations suggesting that such correlated noise is not a fixed network property but that it depends on various sources, and varies with a subject's mental state.

The integration of disparity, shading and motion parallax cues for depth perception in humans and monkeys

A visual stimulus display was created that enabled us to examine how effectively the three depth cues of disparity, motion parallax and shading can be integrated in humans and monkeys. The display was designed to allow us to present these three depth cues separately and in various combinations. Depth was processed most effectively and most rapidly when all three cues were presented together indicating that these separate cues are integrated at yet unknown sites in the brain. Testing in humans and monkeys yielded similar results suggesting that monkeys are a good animal model for the study of the underlying neural mechanisms of depth perception.

A dissociation of attention and awareness in phase-sensitive but not phase-insensitive visual channels

The elements most vivid in our conscious awareness are the ones to which we direct our attention. Scientific study confirms the impression of a close bond between selective attention and visual awareness, yet the nature of this association remains elusive. Using visual afterimages as an index, we investigate neural processing of stimuli as they enter awareness and as they become the object of attention. We find evidence of response enhancement accompanying both attention and awareness, both in the phase-sensitive neural channels characteristic of early processing stages and in the phase-insensitive channels typical of higher cortical areas. The effects of attention and awareness on phase-insensitive responses are positively correlated, but in the same experiments, we observe no correlation between the effects on phase-sensitive responses. This indicates independent signatures of attention and awareness in early visual areas yet a convergence of their effects at more advanced processing stages.

Visual pathways serving motion detection in the mammalian brain

Motion perception is the process through which one gathers information on the dynamic visual world, in terms of the speed and movement direction of its elements. Motion sensation takes place from the retinal light sensitive elements, through the visual thalamus, the primary and higher visual cortices. In the present review we aim to focus on the extrageniculo-extrastriate cortical and subcortical visual structures of the feline and macaque brain and discuss their functional role in visual motion perception. Special attention is paid to the ascending tectofugal system that may serve for detection of the visual environment during self-motion.

A backward progression of attentional effects in the ventral stream

The visual processing of behaviorally relevant stimuli is enhanced through top-down attentional feedback. One possibility is that feedback targets early visual areas first and the attentional enhancement builds up at progressively later stages of the visual hierarchy. An alternative possibility is that the feedback targets the higher-order areas first and the attentional effects are communicated "backward" to early visual areas. Here, we compared the magnitude and latency of attentional enhancement of firing rates in V1, V2, and V4 in the same animals performing the same task. We found a reverse order of attentional effects, such that attentional enhancement was larger and earlier in V4 and smaller and later in V1, with intermediate results in V2. These results suggest that attentional mechanisms operate via feedback from higher-order areas to lower-order ones.

Prefrontal neurons predict choices during an auditory same-different task

The detection of stimuli is critical for an animal's survival [1]. However, it is not adaptive for an animal to respond automatically to every stimulus that is present in the environment [2-5]. Given that the prefrontal cortex (PFC) plays a key role in executive function [6-8], we hypothesized that PFC activity should be involved in context-dependent responses to uncommon stimuli. As a test of this hypothesis, monkeys participated in a same-different task, a variant of an oddball task [2]. During this task, a monkey heard multiple presentations of a "reference" stimulus that were followed by a "test" stimulus and reported whether these stimuli were the same or different. While they participated in this task, we recorded from neurons in the ventrolateral prefrontal cortex (vPFC; a cortical area involved in aspects of nonspatial auditory processing [9, 10]). We found that vPFC activity was correlated with the monkeys' choices. This finding demonstrates a direct link between single neurons and behavioral choices in the PFC on a nonspatial auditory task.

Higher order visual processing in macaque extrastriate cortex

The extrastriate cortex of primates encompasses a substantial portion of the cerebral cortex and is devoted to the higher order processing of visual signals and their dispatch to other parts of the brain. A first step towards the understanding of the function of this cortical tissue is a description of the selectivities of the various neuronal populations for higher order aspects of the image. These selectivities present in the various extrastriate areas support many diverse representations of the scene before the subject. The list of the known selectivities includes that for pattern direction and speed gradients in middle temporal/V5 area; for heading in medial superior temporal visual area, dorsal part; for orientation of nonluminance contours in V2 and V4; for curved boundary fragments in V4 and shape parts in infero-temporal area (IT); and for curvature and orientation in depth from disparity in IT and CIP. The most common putative mechanism for generating such emergent selectivity is the pattern of excitatory and inhibitory linear inputs from the afferent area combined with nonlinear mechanisms in the afferent and receiving area.

Modulation of V4 shifts from dependent to independent on feature during target selection

We analyzed neuronal activities of visual area V4 of monkeys performing a delayed visual search task. To examine the temporal profile of factors influencing the neuronal activities, we conducted multiple regression analyses at 5 ms steps. During the period from 110 to 155 ms after the stimulus onset, there were neurons whose activity was suppressed when a target was presented near but beyond the neuron's receptive field (RF) compared to that when a target was within the RF. We referred this suppressive effect as an early period modulation. During the period from 155 to 280 ms after the stimulus onset, V4 activities were suppressed when a target was presented in any location outside of the neuron's RF. We referred this suppressive effect as a late period modulation. The magnitudes of the effect were equivalent across target locations when a target was beyond its RF. At the population level, while the modulation in the early period was correlated with stimulus selectivity, the modulation in the late period did not show such a correlation. These results suggest that V4 neurons have at least two distinct phases of modulations and those modulations contribute to select a target in the visual search task.

Dynamics of response to perceptual pop-out stimuli in macaque V1

Contextual modulation due to feature contrast between the receptive field and surrounding region has been reported for numerous stimuli in primary visual cortex. One type of this modulation, iso-orientation surround suppression, has been studied extensively. The degree to which surround suppression is related to other forms of contextual modulation remains unknown. We used shape-from-shading stimuli in a field of distractors to test the latency and magnitude of contextual modulation to a stimulus that cannot be distinguished with an orientation-selective mechanism. This stimulus configuration readily elicits perceptual pop-out in human observers and induces a long-latency contextual modulation response in neurons in macaque early visual cortex. We found that animals trained to detect the location of a pop-out stimulus were better at finding a sphere that appeared to be lit from below in the presence of distractors that were lit from above. Furthermore, neuronal responses were stronger and had shorter latency in the condition where behavioral performance was best. This asymmetry is compatible with earlier psychophysical findings in human observers. In the population of V1 neurons, the latency of the contextual modulation response is 145 ms on average (ranging from 70 to 230 ms). This is much longer than the latency for iso-orientation surround suppression, indicating that the underlying circuitry is distinct. Our results support the idea that a feature-specific feedback signal generates the pop-out responses we observe and suggest that V1 neurons actively participate in the computation of perceptual salience.

Binocular depth perception and the cerebral cortex

Our ability to coordinate the use of our left and right eyes and to make use of subtle differences between the images received by each eye allows us to perceive stereoscopic depth, which is important for the visual perception of three-dimensional space. Binocular neurons in the visual cortex combine signals from the left and right eyes. Probing the roles of binocular neurons in different perceptual tasks has advanced our understanding of the stages within the visual cortex that lead to binocular depth perception.

Seeing the forest without losing sight of the trees: details are processed despite reversal insensitivity in structure from motion

Presenting a subject with two random-dot flow fields produces a persuasive 3D percept of a rotating cylinder whose perceived direction of rotation is bi-stable and reverses spontaneously. When real reversals are introduced subjects rarely perceive them. It is possible that although these reversals are consciously ignored, they are registered and available to the visual system for potential later use. We tested this prediction by comparing the number of spontaneously perceived reversals with and without real reversals. We found that although the actual reversals were usually not perceived, they nonetheless increased the number of spontaneous flips. The visual system apparently prefers to retain global structure rather than perceive a motion change of every element in the stimulus. However, it does not completely ignore the changes and incorporates them into other emerging interpretations. These findings argue against the fatigue hypothesis of spontaneous reversals which would predict a decrease in the number of reversals and can be interpreted as supporting the information-processing view by noting that due to the real reversals, the visual system continues to accrue data that argues against the current representation and facilitates switching to the rival representation.

Context-dependent perceptual modulation of single neurons in primate visual cortex

Some neurons in the visual cortex alter their spiking rate according to the perceptual interpretation of an observed stimulus, rather than its physical structure alone. Experiments in monkeys have suggested that, although the proportion of neurons showing this effect differs greatly between cortical areas, this proportion remains similar across different stimuli. These findings have raised the intriguing questions of whether the same neurons always participate in the disambiguation of sensory patterns and whether such neurons might represent a special class of cortical cells that relay perceptual signals to higher cortical areas. Here we explore this question by measuring activity in the middle temporal cortex of monkeys and asking to what degree the percept-related responses of individual neurons depend upon the specific sensory input. In contrast to our expectations, we found that even small differences in the stimuli led to significant changes in the signaling of the perceptual state by single neurons. We conclude that nearly all feature-responsive neurons in this area, rather than a select subset, can contribute to the resolution of sensory conflict, and that the role of individual cells in signaling the perceptual outcome is tightly linked to the fine details of the stimuli involved.

Active Maintenance of Associative Mnemonic Signal in Monkey Inferior Temporal Cortex

We investigated the contribution of the inferior temporal (IT) cortical neurons to the active maintenance of internal representations. The activity of single neurons in the IT cortex was recorded while the monkeys performed a sequential-type associative memory task in which distractor stimuli interrupted the delay epoch between the cue and target (paired-associate) stimuli. For each neuron, information about each stimulus conveyed by the delay activity was estimated as a coefficient of multiple regression analysis. We found that target information derived from long-term memory (LTM) persisted despite the distractors. By contrast, cue information derived from the visual system was attenuated and frequently replaced by distractor information. These results suggest that LTM-derived information required for upcoming behavior is actively maintained in the IT neurons, whereas visually derived information tends to be updated irrespective of behavioral relevance.

EEG detection of early Alzheimer's disease using psychophysical tasks

In this study, we hypothesized that a quantitative EEG (qEEG) method for measuring EEG variability combined with specific psychophysical tasks could improve the classification accuracy of subjects with normal aging vs. mild cognitive impairment (MCI) or mild dementia due to Alzheimer's Disease and Related Disorders (ADRD). The cross-sectional sample consisted of 48 subjects (32 normal aging and 16 ADRD: n = 3 mild dementia, n = 13 MCI FAST stage 3). During EEG recording, subjects performed two visual, delayed recognition memory tasks as well as a task that tested their ability to perceive structure-from-motion (SFM). These EEG data were used to compute qEEG measures of the (normalized) variance of posterior cortical activity during the first 150 milliseconds (ms) after stimulus onset and the variance of anterior cortical activity during the second 150 ms epoch. The ratio, anterior/posterior cerebral qEEG value, was then computed for each subject, and the optimal cutoff value identified to discriminate normal from impaired subjects. An optimal qEEG cutoff value for the delayed recognition memory tasks correctly discriminated 30 of the 32 normal aging subjects (94% specificity) and 14 of 16 MCI-to-mild ADRD subjects (88% sensitivity). On the other hand, the application of this qEEG measure to EEG data recorded while subjects performed a SFM task did not distinguish between ADRD and normal aging any better than chance. In conclusion, this qEEG measure is specific to the psychophysical task being performed by the subject. When it was combined with delayed recognition memory tasks, it yielded results that are comparable to the accuracies reported by PET scan studies of normal aging vs. AD with mild cognitive impairment. These results warrant further evaluation.

Quantitative prediction of perceptual decisions during near-threshold fear detection

A fundamental goal of cognitive neuroscience is to explain how mental decisions originate from basic neural mechanisms. The goal of the present study was to investigate the neural correlates of perceptual decisions in the context of emotional perception. To probe this question, we investigated how fluctuations in functional MRI (fMRI) signals were correlated with behavioral choice during a near-threshold fear detection task. fMRI signals predicted behavioral choice independently of stimulus properties and task accuracy in a network of brain regions linked to emotional processing: posterior cingulate cortex, medial prefrontal cortex, right inferior frontal gyrus, and left insula. We quantified the link between fMRI signals and behavioral choice in a whole-brain analysis by determining choice probabilities by means of signal-detection theory methods. Our results demonstrate that voxel-wise fMRI signals can reliably predict behavioral choice in a quantitative fashion (choice probabilities ranged from 0.63 to 0.78) at levels comparable to neuronal data. We suggest that the conscious decision that a fearful face has been seen is represented across a network of interconnected brain regions that prepare the organism to appropriately handle emotionally challenging stimuli and that regulate the associated emotional response.

Thalamic relays and cortical functioning

Studies on the visual thalamic relays, the lateral geniculate nucleus and pulvinar, provide three key properties that have dramatically changed the view that the thalamus serves as a simple relay to get information from subcortical sites to cortex. First, the retinal input, although a small minority (7%) in terms of numbers of synapses onto geniculate relay cells, dominates receptive field properties of these relay cells and strongly drives them; 93% of input thus is nonretinal and modulates the relay in dynamic and important ways related to behavioral state, including attention. We call the retinal input the driver input and the nonretinal, modulator input, and their unique morphological and functional differences allow us to recognize driver and modulator input to many other thalamic relays. Second, much of the modulation is related to control of a voltage-gated, low threshold Ca(2+) conductance that determines response properties of relay cells -burst or tonic - and this, among other things, affects the salience of information relayed. Third, the lateral geniculate nucleus and pulvinar (a massive but generally mysterious and ignored thalamic relay), are examples of two different types of relay: the LGN is a first order relay, transmitting information from a subcortical driver source (retina), while the pulvinar is mostly a higher order relay, transmitting information from a driver source emanating from layer 5 of one cortical area to another area. Higher order relays seem especially important to general corticocortical communication, and this view challenges the conventional dogma that such communication is based on direct corticocortical connections. In this sense, any new information reaching a cortical area, whether from a subcortical source or another cortical area, benefits from a thalamic relay. Other examples of first and higher order relays also exist, and generally higher order relays represent the majority of thalamus. A final property of interest emphasized in chapter 17 by Guillery (2005) is that most or all driver inputs to thalamus, whether from a subcortical source or from layer 5 of cortex, are axons that branch, with the extrathalamic branch innervating a motor or premotor region in the brainstem, or in some cases, spinal cord. This suggests that actual information relayed by thalamus to cortex is actually a copy of motor instructions (Guillery, 2005). Overall, these features of thalamic relays indicate that the thalamus not only provides a behaviorally relevant, dynamic control over the nature of information relayed, it also plays a key role in basic corticocortical communication.