A common neuronal code for perceptual processes in visual cortex? Comparing choice and attentional correlates in V5/MT

Interaction between spatial perception and temporal perception enables preservation of cause-effect relationship: Visual psychophysics and neuronal dynamics

INTRODUCTION

Visual perception of moving objects is integral to our day-to-day life, integrating visual spatial and temporal perception. Most research studies have focused on finding the brain regions activated during motion perception. However, an empirically validated general mathematical model is required to understand the modulation of the motion perception. Here, we develop a mathematical formulation of the modulation of the perception of a moving object due to a change in speed, under the formulation of the invariance of causality.

METHODS

We formulated the perception of a moving object as the coordinate transformation from a retinotopic space onto perceptual space and derived a quantitative relationship between spatiotemporal coordinates. To validate our model, we undertook the analysis of two experiments: (i) the perceived length of the moving arc, and (ii) the perceived time while observing moving stimuli. We performed a magnetic resonance imaging (MRI) tractography investigation of subjects to demarcate the anatomical correlation of the modulation of the perception of moving objects.

RESULTS

Our theoretical model shows that the interaction between visual-spatial and temporal perception, during the perception of moving object is described by coupled linear equations; and experimental observations validate our model. We observed that cerebral area V5 may be an anatomical correlate for this interaction. The physiological basis of interaction is shown by a Lotka-Volterra system delineating interplay between acetylcholine and dopamine neurotransmitters, whose concentrations vary periodically with the orthogonal phase shift between them, occurring at the axodendritic synapse of complex cells at area V5.

CONCLUSION

Under the invariance of causality in the representation of events in retinotopic space and perceptual space, the speed modulates the perception of a moving object. This modulation may be due to variations of the tuning properties of complex cells at area V5 due to the dynamic interaction between acetylcholine and dopamine. Our analysis is the first significant study, to our knowledge, that establishes a mathematical linkage between motion perception and causality invariance.

Decision-related feedback in visual cortex lacks spatial selectivity

Feedback in the brain is thought to convey contextual information that underlies our flexibility to perform different tasks. Empirical and computational work on the visual system suggests this is achieved by targeting task-relevant neuronal subpopulations. We combine two tasks, each resulting in selective modulation by feedback, to test whether the feedback reflected the combination of both selectivities. We used visual feature-discrimination specified at one of two possible locations and uncoupled the decision formation from motor plans to report it, while recording in macaque mid-level visual areas. Here we show that although the behavior is spatially selective, using only task-relevant information, modulation by decision-related feedback is spatially unselective. Population responses reveal similar stimulus-choice alignments irrespective of stimulus relevance. The results suggest a common mechanism across tasks, independent of the spatial selectivity these tasks demand. This may reflect biological constraints and facilitate generalization across tasks. Our findings also support a previously hypothesized link between feature-based attention and decision-related activity.

Feedback modulates visual neurons, thought to help achieve flexible task performance. Here, the authors show decision-related feedback is not only relayed to task-relevant neurons, suggesting a broader mechanism and supporting a previously hypothesized link to feature-based attention.

Axonal Projections from Middle Temporal Area to the Pulvinar in the Common Marmoset

The pulvinar, the largest thalamic nucleus in the primate brain, has connections with a variety of cortical areas and is involved in many aspects of higher brain functions. Among cortico-pulvino-cortical systems, the connection between the middle temporal area (MT) and the pulvinar has been thought to contribute significantly to complex motion recognition. Recently, the common marmoset (Callithrix jacchus), has become a valuable model for a variety of neuroscience studies, including visual neuroscience and translational research of neurological and psychiatric disorders. However, information on projections from MT to the pulvinar in the marmoset brain is scant. We addressed this deficiency by injecting sensitive anterograde viral tracers into MT to examine the distribution of labeled terminations in the pulvinar. The injection sites were placed retinotopically according to visual field coordinates mapped by optical intrinsic imaging. All injections produced anterograde terminal labeling, which was densest in the medial nucleus of the inferior pulvinar (PIm), sparser in the central nucleus of the inferior pulvinar, and weakest in the lateral pulvinar. Within each subnucleus, terminations formed separate retinotopic fields. Most labeled terminals were small but these comingled with a few large terminals, distributed mainly in the dorsomedial part of the PIm. Our results further delineate the organization of projections from MT to the pulvinar in the marmoset as forming parallel complex networks, which may differentially contribute to motion processing. It is interesting that the densest projections from MT target the PIm, the subnucleus recently reported to preferentially receive direct retinal projections.

Coding Perceptual Decisions: From Single Units to Emergent Signaling Properties in Cortical Circuits

Spiking activity in single neurons of the primate visual cortex has been tightly linked to perceptual decisions. Any mechanism that reads out these perceptual signals to support behavior must respect the underlying neuroanatomy that shapes the functional properties of sensory neurons. Spatial distribution and timing of inputs to the next processing levels are critical, as conjoint activity of precursor neurons increases the spiking rate of downstream neurons and ultimately drives behavior. I set out how correlated activity might coalesce into a micropool of task-sensitive neurons signaling a particular percept to determine perceptual decision signals locally and for flexible interarea transmission depending on the task context. As data from more and more neurons and their complex interactions are analyzed, the space of computational mechanisms must be constrained based on what is plausible within neurobiological limits. This review outlines experiments to test the new perspectives offered by these extended methods.

Interneuronal correlations at longer time scales predict decision signals for bistable structure-from-motion perception

Perceptual decisions are thought to depend on the activation of task-relevant neurons, whose activity is often correlated in time. Here, we examined how the temporal structure of shared variability in neuronal firing relates to perceptual choices. We recorded stimulus-selective neurons from visual area V5/MT while two monkeys (Macaca mulatta) made perceptual decisions about the rotation direction of structure-from-motion cylinders. Interneuronal correlations for a perceptually ambiguous cylinder stimulus were significantly higher than those for unambiguous cylinders or for random 2D motion during passive viewing. Much of the difference arose from correlations at relatively long timescales (hundreds of milliseconds). Choice-related neural activity (quantified as choice probability; CP) for ambiguous cylinders was positively correlated with interneuronal correlations and was specifically associated with their long timescale component. Furthermore, the slope of the long timescale - but not the instantaneous - component of the correlation predicted higher CPs towards the end of the trial i.e. close to the decision. Our results suggest that the perceptual stability of structure-from-motion cylinders may be controlled by enhanced interneuronal correlations on longer timescales. We propose this as a potential signature of top-down influences onto V5/MT processing that shape and stabilize the appearance of 3D-motion percepts.

Developmental trajectory of social influence integration into perceptual decisions in children

The opinions of others have a profound influence on decision making in adults. The impact of social influence appears to change during childhood, but the underlying mechanisms and their development remain unclear. We tested 125 neurotypical children between the ages of 6 and 14 years on a perceptual decision task about 3D-motion figures under informational social influence. In these children, a systematic bias in favor of the response of another person emerged at around 12 years of age, regardless of whether the other person was an age-matched peer or an adult. Drift diffusion modeling indicated that this social influence effect in neurotypical children was due to changes in the integration of sensory information, rather than solely a change in decision behavior. When we tested a smaller cohort of 30 age- and IQ-matched autistic children on the same task, we found some early decision bias to social influence, but no evidence for the development of systematic integration of social influence into sensory processing for any age group. Our results suggest that by the early teens, typical neurodevelopment allows social influence to systematically bias perceptual processes in a visual task previously linked to the dorsal visual stream. That the same bias did not appear to emerge in autistic adolescents in this study may explain some of their difficulties in social interactions.

Defining the V5/MT neuronal pool for perceptual decisions in a visual stereo-motion task

In the primate visual cortex, neurons signal differences in the appearance of objects with high precision. However, not all activated neurons contribute directly to perception. We defined the perceptual pool in extrastriate visual area V5/MT for a stereo-motion task, based on trial-by-trial co-variation between perceptual decisions and neuronal firing (choice probability (CP)). Macaque monkeys were trained to discriminate the direction of rotation of a cylinder, using the binocular depth between the moving dots that form its front and rear surfaces. We manipulated the activity of single neurons trial-to-trial by introducing task-irrelevant stimulus changes: dot motion in cylinders was aligned with neuronal preference on only half the trials, so that neurons were strongly activated with high firing rates on some trials and considerably less activated on others. We show that single neurons maintain high neurometric sensitivity for binocular depth in the face of substantial changes in firing rate. CP was correlated with neurometric sensitivity, not level of activation. In contrast, for individual neurons, the correlation between perceptual choice and neuronal activity may be fundamentally different when responding to different stimulus versions. Therefore, neuronal pools supporting sensory discrimination must be structured flexibly and independently for each stimulus configuration to be discriminated.

This article is part of the themed issue ‘Vision in our three-dimensional world'.

Decision-Related Activity in Macaque V2 for Fine Disparity Discrimination Is Not Compatible with Optimal Linear Readout

Fine judgments of stereoscopic depth rely mainly on relative judgments of depth (relative binocular disparity) between objects, rather than judgments of the distance to where the eyes are fixating (absolute disparity). In macaques, visual area V2 is the earliest site in the visual processing hierarchy for which neurons selective for relative disparity have been observed (Thomas et al., 2002). Here, we found that, in macaques trained to perform a fine disparity discrimination task, disparity-selective neurons in V2 were highly selective for the task, and their activity correlated with the animals' perceptual decisions (unexplained by the stimulus). This may partially explain similar correlations reported in downstream areas. Although compatible with a perceptual role of these neurons for the task, the interpretation of such decision-related activity is complicated by the effects of interneuronal "noise" correlations between sensory neurons. Recent work has developed simple predictions to differentiate decoding schemes (Pitkow et al., 2015) without needing measures of noise correlations, and found that data from early sensory areas were compatible with optimal linear readout of populations with information-limiting correlations. In contrast, our data here deviated significantly from these predictions. We additionally tested this prediction for previously reported results of decision-related activity in V2 for a related task, coarse disparity discrimination (Nienborg and Cumming, 2006), thought to rely on absolute disparity. Although these data followed the predicted pattern, they violated the prediction quantitatively. This suggests that optimal linear decoding of sensory signals is not generally a good predictor of behavior in simple perceptual tasks.

SIGNIFICANCE STATEMENT

Activity in sensory neurons that correlates with an animal's decision is widely believed to provide insights into how the brain uses information from sensory neurons. Recent theoretical work developed simple predictions to differentiate decoding schemes, and found support for optimal linear readout of early sensory populations with information-limiting correlations. Here, we observed decision-related activity for neurons in visual area V2 of macaques performing fine disparity discrimination, as yet the earliest site for this task. These findings, and previously reported results from V2 in a different task, deviated from the predictions for optimal linear readout of a population with information-limiting correlations. Our results suggest that optimal linear decoding of early sensory information is not a general decoding strategy used by the brain.

Functional interactions between the macaque dorsal and ventral visual pathways during three-dimensional object vision

The division of labor between the dorsal and the ventral visual stream in the primate brain has inspired numerous studies on the visual system in humans and in nonhuman primates. However, how and under which circumstances the two visual streams interact is still poorly understood. Here we review evidence from anatomy, modelling, electrophysiology, electrical microstimulation (EM), reversible inactivation and functional imaging in the macaque monkey aimed at clarifying at which levels in the hierarchy of visual areas the two streams interact, and what type of information might be exchanged between the two streams during three-dimensional (3D) object viewing. Neurons in both streams encode 3D structure from binocular disparity, synchronized activity between parietal and inferotemporal areas is present during 3D structure categorization, and clusters of 3D structure-selective neurons in parietal cortex are anatomically connected to ventral stream areas. In addition, caudal intraparietal cortex exerts a causal influence on 3D-structure related activations in more anterior parietal cortex and in inferotemporal cortex. Thus, both anatomical and functional evidence indicates that the dorsal and the ventral visual stream interact during 3D object viewing.

Sensory integration dynamics in a hierarchical network explains choice probabilities in cortical area MT

Neuronal variability in sensory cortex predicts perceptual decisions. This relationship, termed choice probability (CP), can arise from sensory variability biasing behaviour and from top-down signals reflecting behaviour. To investigate the interaction of these mechanisms during the decision-making process, we use a hierarchical network model composed of reciprocally connected sensory and integration circuits. Consistent with monkey behaviour in a fixed-duration motion discrimination task, the model integrates sensory evidence transiently, giving rise to a decaying bottom-up CP component. However, the dynamics of the hierarchical loop recruits a concurrently rising top-down component, resulting in sustained CP. We compute the CP time-course of neurons in the medial temporal area (MT) and find an early transient component and a separate late contribution reflecting decision build-up. The stability of individual CPs and the dynamics of noise correlations further support this decomposition. Our model provides a unified understanding of the circuit dynamics linking neural and behavioural variability.

Differences between primary auditory cortex and auditory belt related to encoding and choice for AM sounds

We recorded from middle-lateral (ML) and primary (A1) auditory cortex while macaques discriminated amplitude-modulated (AM) noise from unmodulated noise. Compared with A1, ML had a higher proportion of neurons that encoded increasing AM depth by decreasing their firing rates ("decreasing" neurons), particularly with responses that were not synchronized to the modulation. Choice probability (CP) analysis revealed that A1 and ML activity were different during the first half of the test stimulus. In A1, significant CP began before the test stimulus, remained relatively constant (or increased slightly) during the stimulus, and increased greatly within 200 ms of lever release. Neurons in ML behaved similarly, except that significant CP disappeared during the first half of the stimulus and reappeared during the second half and prerelease periods. CP differences between A1 and ML depend on neural response type. In ML (but not A1), when activity was lower during the first half of the stimulus in nonsynchronized, decreasing neurons, the monkey was more likely to report AM. Neurons that both increased firing rate with increasing modulation depth ("increasing" neurons) and synchronized their responses to AM had similar choice-related activity dynamics in ML and A1. These results suggest that, when ascending the auditory system, there is a transformation in coding AM from primarily synchronized increasing responses in A1 to nonsynchronized and dual (increasing/decreasing) coding in ML. This sensory transformation is accompanied by changes in the timing of activity related to choice, suggesting functional differences between A1 and ML related to attention and/or behavior.

Inattention blindness to motion in middle temporal area

Subjects naturally form and use expectations to solve familiar tasks, but the accuracy of these expectations and the neuronal mechanisms by which these expectations enhance behavior are unclear. We trained animals (Macaca mulatta) in a challenging perceptual task in which the likelihood of a very brief pulse of motion was consistently modulated over time and space. Pulse likelihood had dramatic effects on behavior: unexpected pulses were nearly invisible to the animals. To examine the neuronal basis of such inattention blindness, we recorded from single neurons in the middle temporal (MT) area, an area related to motion perception. Fluctuations in how reliably MT neurons both signaled stimulus events and predicted behavioral choices were highly correlated with changes in performance over the course of individual trials. A simple neuronal pooling model reveals that the dramatic behavioral effects of attention in this task can be completely explained by changes in the reliability of a small number of MT neurons.

Human decision making based on variations in internal noise: an EEG study

Perceptual decision making is prone to errors, especially near threshold. Physiological, behavioural and modeling studies suggest this is due to the intrinsic or ‘internal’ noise in neural systems, which derives from a mixture of bottom-up and top-down sources. We show here that internal noise can form the basis of perceptual decision making when the external signal lacks the required information for the decision. We recorded electroencephalographic (EEG) activity in listeners attempting to discriminate between identical tones. Since the acoustic signal was constant, bottom-up and top-down influences were under experimental control. We found that early cortical responses to the identical stimuli varied in global field power and topography according to the perceptual decision made, and activity preceding stimulus presentation could predict both later activity and behavioural decision. Our results suggest that activity variations induced by internal noise of both sensory and cognitive origin are sufficient to drive discrimination judgments.

Choice-related activity and correlated noise in subcortical vestibular neurons

Functional links between neuronal activity and perception are studied by examining trial-by-trial correlations (choice probabilities) between neural responses and perceptual decisions. We addressed fundamental issues regarding the nature and origin of choice probabilities by recording from subcortical (brainstem and cerebellar) neurons in rhesus monkeys during a vestibular heading discrimination task. Subcortical neurons showed robust choice probabilities that exceeded those seen in cortex (area MSTd) under identical conditions. The greater choice probabilities of subcortical neurons could be predicted by a stronger dependence of correlated noise on tuning similarity, as revealed by population decoding. Significant choice probabilities were observed almost exclusively for neurons that responded selectively to translation, whereas neurons that represent net gravito-inertial acceleration did not show choice probabilities. These findings suggest that the emergence of choice probabilities in the vestibular system depends on a critical signal transformation that occurs in subcortical pathways to distinguish translation from orientation relative to gravity.

Neural correlates of prior expectations of motion in the lateral intraparietal and middle temporal areas

Successful decision making involves combining observations of the external world with prior knowledge. Recent studies suggest that neural activity in macaque lateral intraparietal area (LIP) provides a useful window into this process. This study examines how rapidly changing prior knowledge about an upcoming sensory stimulus influences the computations that convert sensory signals into plans for action. Two monkeys performed a cued direction discrimination task, in which an arrow cue presented at the start of each trial communicated the prior probability of the direction of stimulus motion. We hypothesized that the cue would either shift the initial level of LIP activity before sensory evidence arrived, or it would scale sensory responses according to the prior probability of each stimulus, manifesting as a change in slope of LIP firing rates. Neural recordings demonstrated a clear shift in the activity level of LIP neurons following the arrow cue, which persisted into the presentation of the motion stimulus. No significant change in slope of responses was observed, suggesting that sensory gain was not strongly modulated. To confirm the latter observation, middle temporal area (MT) neurons were recorded during a version of the cued direction discrimination task, and we found no change in MT responses resulting from the presentation of the directional cue. These results suggest that information about an immediately upcoming stimulus does not scale the sensory response, but rather changes the amount of evidence that must be accumulated to reach a decision in areas that are involved in planning action.

Effect of feature-selective attention on neuronal responses in macaque area MT

Attention influences visual processing in striate and extrastriate cortex, which has been extensively studied for spatial-, object-, and feature-based attention. Most studies exploring neural signatures of feature-based attention have trained animals to attend to an object identified by a certain feature and ignore objects/displays identified by a different feature. Little is known about the effects of feature-selective attention, where subjects attend to one stimulus feature domain (e.g., color) of an object while features from different domains (e.g., direction of motion) of the same object are ignored. To study this type of feature-selective attention in area MT in the middle temporal sulcus, we trained macaque monkeys to either attend to and report the direction of motion of a moving sine wave grating (a feature for which MT neurons display strong selectivity) or attend to and report its color (a feature for which MT neurons have very limited selectivity). We hypothesized that neurons would upregulate their firing rate during attend-direction conditions compared with attend-color conditions. We found that feature-selective attention significantly affected 22% of MT neurons. Contrary to our hypothesis, these neurons did not necessarily increase firing rate when animals attended to direction of motion but fell into one of two classes. In one class, attention to color increased the gain of stimulus-induced responses compared with attend-direction conditions. The other class displayed the opposite effects. Feature-selective activity modulations occurred earlier in neurons modulated by attention to color compared with neurons modulated by attention to motion direction. Thus feature-selective attention influences neuronal processing in macaque area MT but often exhibited a mismatch between the preferred stimulus dimension (direction of motion) and the preferred attention dimension (attention to color).

The functional link between area MT neural fluctuations and detection of a brief motion stimulus

Fluctuations of neural firing rates in visual cortex are known to be correlated with variations in perceptual performance. It is important to know whether these fluctuations are functionally linked to perception in a causal manner or instead reflect non-causal processes that arise after the perceptual decision is made. We recorded from middle temporal (MT) neurons from monkey subjects while they detected the random occurrence of a brief 50 ms motion pulse that occurred in either of two (or simultaneously in both) random dot patches located in the same hemisphere. The receptive field parameters of the motion pulse were matched to that preferred by each MT neuron under study. This task contained uncertainty in both space and time because, on any given trial, the subjects did not know which patch would contain the motion pulse or when the motion pulse would occur. Covariations between MT activity and behavior began just before the motion pulse onset and peaked at the maximum neural response. These neural-behavioral covariations were strongest when only one patch contained the motion pulse and were still weakly present when a patch did not contain a motion pulse. A feedforward temporal integration model with two independent detector channels captured both the detection performance and evolution of the neural-behavior covariations over time and stimulus condition. The results suggest that, when detecting a brief visual stimulus, there is a causal relationship between fluctuations in neural activity and variations in behavior across trials.

Correlations between the activity of sensory neurons and behavior: how much do they tell us about a neuron's causality?

How the activity of sensory neurons elicits perceptions and guides behavior is central to our understanding of the brain and is a subject of intense investigation in neuroscience. Correlations between the activity of sensory neurons and behavior have been widely observed and are sometimes used to infer how neurons are used to guide a certain behavior. This view is challenged firstly by theoretical considerations that these correlations rely on the existence of correlated noise and its structure, and secondly by recent empirical observations suggesting that such correlated noise is not a fixed network property but that it depends on various sources, and varies with a subject's mental state.

The Neural Mechanisms of the Effects of Transcranial Magnetic Stimulation on Perception

Transcranial magnetic stimulation (TMS) is a technique used to study perceptual, motor, and cognitive functions in the human brain. Its effects have been likened to a “virtual brain lesion,” but a direct test of this assumption is lacking. To verify this hypothesis, we measured psychophysically the interaction between the neural activity induced by a visual motion-direction discrimination task and that induced by TMS. The visual stimulus featured two elements: a visual signal (dots that moved coherently in one direction) and visual noise (dots that moved randomly in many directions). Three hypotheses were tested to explain the impairment in performance as a result of TMS: 1) a decrease in signal strength; 2) an induction of randomly distributed neural noise with an accompanying decrement in system sensitivity; and 3) a suppression of relevant information processing and addition of neural noise. We provide evidence in favor of the second hypothesis by showing that TMS basically acts by adding neural noise to the perceptual process.

Multisensory integration: psychophysics, neurophysiology, and computation

Fundamental observations and principles derived from traditional physiological studies of multisensory integration have been difficult to reconcile with computational and psychophysical studies that share the foundation of probabilistic (Bayesian) inference. We review recent work on multisensory integration, focusing on experiments that bridge single-cell electrophysiology, psychophysics, and computational principles. These studies show that multisensory (visual-vestibular) neurons can account for near-optimal cue integration during the perception of self-motion. Unlike the nonlinear (superadditive) interactions emphasized in some previous studies, visual-vestibular neurons accomplish near-optimal cue integration through subadditive linear summation of their inputs, consistent with recent computational theories. Important issues remain to be resolved, including the observation that variations in cue reliability appear to change the weights that neurons apply to their different sensory inputs.

Combining spatial and feature-based attention within the receptive field of MT neurons

This study investigates the effects of feature-based attention on responses of direction-selective neurons in the middle temporal area (MT) of macaque visual cortex to attended stimuli inside the receptive field. Redirecting attention between the preferred and null direction of transparent random dot motion patterns caused a mean modulation of responses of approximately 32%, about half of what was observed when the two directions of motion in the receptive field were spatially separated allowing feature-based and spatial attention to work in concert. This is consistent with models of visual attention that interpret the attentional modulation of a neuron as the combination of all attentional influences, treating stimulus location simply as another feature.

Decision-related activity in sensory neurons reflects more than a neuron's causal effect

During perceptual decisions, the activity of sensory neurons correlates with a subject's percept, even when the physical stimulus is identical. The origin of this correlation is unknown. Current theory proposes a causal effect of noise in sensory neurons on perceptual decisions, but the correlation could result from different brain states associated with the perceptual choice (a top-down explanation). These two schemes have very different implications for the role of sensory neurons in forming decisions. Here we use white-noise analysis to measure tuning functions of V2 neurons associated with choice and simultaneously measure how the variation in the stimulus affects the subjects' (two macaques) perceptual decisions. In causal models, stronger effects of the stimulus upon decisions, mediated by sensory neurons, are associated with stronger choice-related activity. However, we find that over the time course of the trial these measures change in different directions-at odds with causal models. An analysis of the effect of reward size also supports this conclusion. Finally, we find that choice is associated with changes in neuronal gain that are incompatible with causal models. All three results are readily explained if choice is associated with changes in neuronal gain caused by top-down phenomena that closely resemble attention. We conclude that top-down processes contribute to choice-related activity. Thus, even forming simple sensory decisions involves complex interactions between cognitive processes and sensory neurons.

Temporal precision of neuronal information in a rapid perceptual judgment

In many situations, such as pedestrians crossing a busy street or prey evading predators, rapid decisions based on limited perceptual information are critical for survival. The brevity of these perceptual judgments constrains how neuronal signals are integrated or pooled over time because the underlying sequence of processes, from sensation to perceptual evaluation to motor planning and execution, all occur within several hundred milliseconds. Because most previous physiological studies of these processes have relied on tasks requiring considerably longer temporal integration, the neuronal basis of such rapid decisions remains largely unexplored. In this study, we examine the temporal precision of neuronal activity associated with a rapid perceptual judgment. We find that the activity of individual neurons over tens of milliseconds can reliably convey information about sensory events and was well correlated with the animals' judgments. There was a strong correlation between sensory reliability and the correlation with behavioral choice, suggesting that rapid decisions were preferentially based on the most reliable sensory signals. We also find that a simple model in which the responses of a small number of individual neurons (<5) are summed can completely explain behavioral performance. These results suggest that neuronal circuits are sufficiently precise to allow for cognitive decisions to be based on small numbers of action potentials from highly reliable neurons.

Neural correlates of multisensory cue integration in macaque MSTd

Human observers combine multiple sensory cues synergistically to achieve greater perceptual sensitivity, but little is known about the underlying neuronal mechanisms. We recorded the activity of neurons in the dorsal medial superior temporal (MSTd) area during a task in which trained monkeys combined visual and vestibular cues near-optimally to discriminate heading. During bimodal stimulation, MSTd neurons combined visual and vestibular inputs linearly with subadditive weights. Neurons with congruent heading preferences for visual and vestibular stimuli showed improvements in sensitivity that parallel behavioral effects. In contrast, neurons with opposite preferences showed diminished sensitivity under cue combination. Responses of congruent cells were more strongly correlated with monkeys' perceptual decisions than were responses of opposite cells, suggesting that the monkey monitored the activity of congruent cells to a greater extent during cue integration. These findings show that perceptual cue integration occurs in nonhuman primates and identify a population of neurons that may form its neural basis.

Neuronal firing rate, inter-neuron correlation and synchrony in area MT are correlated with directional choices during stimulus and reward expectation

Sensation, memories, and predictions contribute to choices in everyday life, and their relative impact should change with task constraints. To investigate how the impact from sensory cortex on decision making varies with task constraints we trained macaque monkeys in a direction discrimination task where they could maximize reward by waiting for sensory visual information early in a trial, while focusing on memory and reward prediction as a trial progressed. The task constraints caused animals to indicate decisions in complete absence of visual motion stimuli (stimulus independent decisions), as 25% of the trials were 'no stimulus' trials. On 'no stimulus' trials reward delivery depended on the current decision in relation to the decision history. Stimulus independent decisions occurred during an epoch when a stimulus could in principle have been presented, or afterwards when stimuli could not occur anymore. Stimulus independent decisions were significantly different during these two periods. Reward exploitation was more efficient late in the trial, but it was not associated with systematic activity changes in directionally selective neurons in area MT. Conversely, systematic changes of neuronal activity and firing rate correlation in directionally selective middle temporal area (MT) neurons were restricted to a short time period before early decisions. Changing task constraints in the course of a single trial thus determines how neurons in sensory areas contribute to decision making.

Bounded integration in parietal cortex underlies decisions even when viewing duration is dictated by the environment

Decisions about sensory stimuli are often based on an accumulation of evidence in time. When subjects control stimulus duration, the decision terminates when the accumulated evidence reaches a criterion level. Under many natural circumstances and in many laboratory settings, the environment, rather than the subject, controls the stimulus duration. In these settings, it is generally assumed that subjects commit to a choice at the end of the stimulus stream. Indeed, failure to benefit from the full stream of information is interpreted as a sign of imperfect accumulation or memory leak. Contrary to these assumptions, we show that monkeys performing a direction discrimination task commit to a choice when the accumulated evidence reaches a threshold level (or bound), sometimes long before the end of stimulus. This bounded accumulation of evidence is reflected in the activity of neurons in the lateral intraparietal cortex. Thus, the readout of visual cortex embraces a termination rule to limit processing even when potentially useful information is available.

Psychophysically measured task strategy for disparity discrimination is reflected in V2 neurons

In perceptual tasks, subjects attempt to rely on their most informative cues. Such strategic choices should be reflected in the types of sensory neurons that are used. We investigated this in a binocular-disparity discrimination task. Using psychophysical reverse-correlation, also known as image classification, we identified the perceptual strategies of two macaques (Macaca mulatta). Correlations between reported disparity signs and disparity noise samples for each trial yielded detection 'filters'. Filter amplitude was greater at near disparities than at far disparities, indicating that the subjects relied more on near disparities. Recordings from both macaques' disparity-selective V2 neurons showed a correlation between neuronal responses and perceptual judgment in near-preferring, but not far-preferring, units, mirroring the psychophysically measured strategy. After one monkey learned to weight near and far disparities equally, activity in its far-preferring neurons correlated with choice. Thus, the pattern of correlations between neuronal activity and perceptual reports indicates how subjects use their neuronal signals.

Activity of inferior temporal cortical neurons predicts recognition choice behavior and recognition time during visual search

Although the selectivity for complex stimuli exhibited by neurons in inferior temporal cortex is often taken as evidence of their role in visual perception, few studies have directly tested this hypothesis. Here, we sought to create a relatively natural task with few behavioral constraints to test whether activity in inferior temporal cortex neurons predicts whether or not a monkey will recognize and respond to a complex visual object. Monkeys were trained to freely view an array of images and report the presence of one of many possible target images previously associated with a hand response. On certain trials, the identity of the target was swapped during the monkeys' targeting saccade. Furthermore, the response association of the preswap target and the postswap target differed (e.g., right-to-left target swap). Neural activity in cells selective for the preswap target was significantly higher when the monkeys' response matched the hand association of the preswap target. Furthermore, the monkeys' response time was predicted by the magnitude of the presaccadic firing rate on nonswap trials. Our results provide additional support for the role of inferior temporal cortex in object recognition during natural behavior.

Context-dependent perceptual modulation of single neurons in primate visual cortex

Some neurons in the visual cortex alter their spiking rate according to the perceptual interpretation of an observed stimulus, rather than its physical structure alone. Experiments in monkeys have suggested that, although the proportion of neurons showing this effect differs greatly between cortical areas, this proportion remains similar across different stimuli. These findings have raised the intriguing questions of whether the same neurons always participate in the disambiguation of sensory patterns and whether such neurons might represent a special class of cortical cells that relay perceptual signals to higher cortical areas. Here we explore this question by measuring activity in the middle temporal cortex of monkeys and asking to what degree the percept-related responses of individual neurons depend upon the specific sensory input. In contrast to our expectations, we found that even small differences in the stimuli led to significant changes in the signaling of the perceptual state by single neurons. We conclude that nearly all feature-responsive neurons in this area, rather than a select subset, can contribute to the resolution of sensory conflict, and that the role of individual cells in signaling the perceptual outcome is tightly linked to the fine details of the stimuli involved.

Macaque V2 neurons, but not V1 neurons, show choice-related activity

In the macaque extrastriate cortex, robust correlations between perceptual choice and neuronal response have been demonstrated, frequently quantified as choice probabilities (CPs). Such correlations are modest in early visual cortex, suggesting that CPs may depend on the position of a neuron in the hierarchy of visual processing. However, previous studies have not compared neurons with similar precision in equivalent tasks. We investigated the role of cortical hierarchy on CP using a task for which significant CPs have been described previously for middle temporal area (MT). We measured CPs in disparity-selective neurons from both V1 and V2. The stimulus was a dynamic random dot stereogram, presented with a near or a far disparity, masked by varying numbers of binocularly uncorrelated dots. Two macaque monkeys reported whether they perceived a circular patch in front or behind a surrounding annulus in a forced choice task. For V2 (n = 69), CP was on average 0.56, the first demonstration of systematic CPs in a visual area as early as V2. In V1 (n = 74), average CP was at chance level (0.51). The pattern was similar in a subgroup of neurons selected such that the statistical precision in the task was on average identical to that reported for MT (mean CP, 0.51 for V1, n = 33; 0.58 for V2, n = 54). This difference between V1 and V2 could not be explained by eye movements, stimulus size relative to the receptive field, or differences in disparity tuning. Rather, it seems to reflect a functional difference (at least in disparity processing) between striate and extrastriate cortex.

Local field potential in cortical area MT: stimulus tuning and behavioral correlations

Low-frequency electrical signals like those that compose the local field potential (LFP) can be detected at substantial distances from their point of origin within the brain. It is thus unclear how useful the LFP might be for assessing local function, for example, on the spatial scale of cortical columns. We addressed this problem by comparing speed and direction tuning of LFPs obtained from middle temporal area MT with the tuning of multiunit (MU) activity recorded simultaneously. We found that the LFP can be well tuned for speed and direction and is highly correlated with that of MU activity, particularly for frequencies at and above the gamma band. LFP tuning is substantially poorer for lower frequencies, although tuning for direction extends to lower frequencies than does tuning for speed. Our data suggest that LFP signals at and above the gamma band reflect neural processing on the spatial scale of cortical columns, within a few hundred micrometers of the electrode tip. Consistent with this notion, we also found that frequencies at and above the gamma band measured during a speed discrimination task exhibit an effect known as "choice probability," which reveals a particularly close relationship between neural activity and behavioral choices. In the LFP, this signature of the perceptual choice comprises a shift in relative power from low-frequency bands (alpha and beta) to the gamma band. It remains to be determined how LFP choice probability, which is a temporal signature, is related to conventional choice probability effects observed in spike rates.

Comparing Perceptual Signals of Single V5/MT Neurons in Two Binocular Depth Tasks

Neurons in the extrastriate visual area V5/MT show perceptually relevant signals in binocular depth tasks, which can be measured as a choice probability (CP) for the neuron. The presence of a CP in a particular paradigm may be an indicator that the neuron is generally part of the substrate for the perception of binocular depth. We compared the responses of those single neurons that show CPs in one stereoscopic depth task with their responses in another stereo task. Each neuron was tested for the presence of 1) CPs during a task in which macaques responded to the sign of binocular depth in a structure-from-motion stimulus, to judge its direction of three-dimensional rotation and 2) a consistent response to the stereo disparity of binocularly anti-correlated stimuli. Previous work, confirmed here, shows that changing the disparity of these binocularly anti-correlated stimuli often fails to yield a coherent change in the depth percept. For each test alone, there are V5/MT neurons that carry signals that are congruent with the perceptual effects. However, on comparing tests, there is no fixed pool of neurons that can account for the binocular depth percept. Excitation of neurons with a measurable CP does not necessarily lead to a change in perception. The cortical circuitry must be able to make dynamic changes in the pools of neurons that underlie perceptual judgments according to the demands of the task.

Neuronal mechanisms for the perception of ambiguous stimuli

Ambiguous figures that may take on the appearance of two or more distinct forms have fascinated philosophers and psychologists for generations. Recently, several laboratories have studied the neuronal basis of perceptual appearance at the level of single neurons in the cerebral cortex. Experiments that integrate neuronal recording with analyses based on sensory detection theory reveal a remarkable degree of specificity in these neuronal responses. The new challenges are to understand how cognitive processes, such as attention and memory, interact with perception to generate these neuronal signals.