Dynamic properties of orientation discrimination assessed by using classification images

A computational account of transsaccadic attentional allocation based on visual gain fields

Coordination of goal-directed behavior depends on the brain's ability to recover the locations of relevant objects in the world. In humans, the visual system encodes the spatial organization of sensory inputs, but neurons in early visual areas map objects according to their retinal positions, rather than where they are in the world. How the brain computes world-referenced spatial information across eye movements has been widely researched and debated. Here, we tested whether shifts of covert attention are sufficiently precise in space and time to track an object's real-world location across eye movements. We found that observers' attentional selectivity is remarkably precise and is barely perturbed by the execution of saccades. Inspired by recent neurophysiological discoveries, we developed an observer model that rapidly estimates the real-world locations of objects and allocates attention within this reference frame. The model recapitulates the human data and provides a parsimonious explanation for previously reported phenomena in which observers allocate attention to task-irrelevant locations across eye movements. Our findings reveal that visual attention operates in real-world coordinates, which can be computed rapidly at the earliest stages of cortical processing.

The influence of natural image statistics on upright orientation judgements

Humans have well-documented priors for many features present in nature that guide visual perception. Despite being putatively grounded in the statistical regularities of the environment, scene priors are frequently violated due to the inherent variability of visual features from one scene to the next. However, these repeated violations do not appreciably challenge visuo-cognitive function, necessitating the broad use of priors in conjunction with context-specific information. We investigated the trade-off between participants' internal expectations formed from both longer-term priors and those formed from immediate contextual information using a perceptual inference task and naturalistic stimuli. Notably, our task required participants to make perceptual inferences about naturalistic images using their own internal criteria, rather than making comparative judgements. Nonetheless, we show that observers' performance is well approximated by a model that makes inferences using a prior for low-level image statistics, aggregated over many images. We further show that the dependence on this prior is rapidly re-weighted against contextual information, even when misleading. Our results therefore provide insight into how apparent high-level interpretations of scene appearances follow from the most basic of perceptual processes, which are grounded in the statistics of natural images.

Emergence of radial orientation selectivity: Effect of cell density changes and eccentricity in a layered network

We establish a simple mechanism by which radially oriented simple cells can emerge in the primary visual cortex. In 1986, R. Linsker. proposed a means by which radially symmetric, spatial opponent cells can evolve, driven entirely by noise, from structure in the initial synaptic connectivity distribution. We provide an analytical derivation of Linsker's results, and further show that radial eigenfunctions can be expressed as a weighted sum of degenerate Cartesian eigenfunctions, and vice-versa. These results are extended to allow for radially dependent cell density, from which we show that, despite a circularly symmetric synaptic connectivity distribution, radially biased orientation selectivity emerges in the third layer when cell density in the first layer, or equivalently, synaptic radius, changes with eccentricity; i.e., distance to the center of the lamina. This provides a potential mechanism for the emergence of radial orientation in the primary visual cortex before eye opening and the onset of structured visual input after birth.

Spatial attention in perceptual decision making as revealed by response-locked classification image analysis

In many situations, humans serially sample information from many locations in an image to make an appropriate decision about a visual target. Spatial attention and eye movements play a crucial role in this serial vision process. To investigate the effect of spatial attention in such dynamic decision making, we applied a classification image (CI) analysis locked to the observer's reaction time (RT). We asked human observers to detect as rapidly as possible a target whose contrast gradually increased on the left or right side of dynamic noise, with the presentation of a spatial cue. The analysis revealed a spatiotemporally biphasic profile of the CI which peaked at ~ 350 ms before the observer's response. We found that a valid cue presented at the target location shortened the RT and increased the overall amplitude of the CI, especially when the cue appeared 500-1250 ms before the observer's response. The results were quantitatively accounted for by a simple perceptual decision mechanism that accumulates the outputs of the spatiotemporal contrast detector, whose gain is increased by sustained attention to the cued location.

Genetic algorithms reveal profound individual differences in emotion recognition

Emotional communication relies on a mutual understanding, between expresser and viewer, of facial configurations that broadcast specific emotions. However, we do not know whether people share a common understanding of how emotional states map onto facial expressions. This is because expressions exist in a high-dimensional space too large to explore in conventional experimental paradigms. Here, we address this by adapting genetic algorithms and combining them with photorealistic three-dimensional avatars to efficiently explore the high-dimensional expression space. A total of 336 people used these tools to generate facial expressions that represent happiness, fear, sadness, and anger. We found substantial variability in the expressions generated via our procedure, suggesting that different people associate different facial expressions to the same emotional state. We then examined whether variability in the facial expressions created could account for differences in performance on standard emotion recognition tasks by asking people to categorize different test expressions. We found that emotion categorization performance was explained by the extent to which test expressions matched the expressions generated by each individual. Our findings reveal the breadth of variability in people's representations of facial emotions, even among typical adult populations. This has profound implications for the interpretation of responses to emotional stimuli, which may reflect individual differences in the emotional category people attribute to a particular facial expression, rather than differences in the brain mechanisms that produce emotional responses.

Dynamics and Mechanisms of Contrast-Dependent Modulation of Spatial-Frequency Tuning in the Early Visual Cortex

The spatial-frequency (SF) tuning of neurons in the early visual cortex is adjusted for stimulus contrast. As the contrast increases, SF tuning is modulated so that the transmission of fine features is facilitated. A variety of mechanisms are involved in shaping SF tunings, but those responsible for the contrast-dependent modulations are unclear. To address this, we measured the time course of SF tunings of area 17 neurons in male cats under different contrasts with a reverse correlation. After response onset, the optimal SF continuously shifted to a higher SF over time, with a larger shift for higher contrast. At high contrast, whereas neurons with a large shift of optimal SF exhibited a large bandwidth decrease, those with a negligible shift increased the bandwidth over time. Between these two extremes, the degree of SF shift and bandwidth change continuously varied. At low contrast, bandwidth generally decreased over time. These dynamic effects enhanced the processing of high-frequency range under a high-contrast condition and allowed time-average SF tuning curves to show contrast-dependent modulation, like that of steady-state SF tuning curves reported previously. Combinations of two mechanisms, one that decreases bandwidth and shifts optimal SF, and another that increases bandwidth without shifting optimal SF, would explain the full range of SF tuning dynamics. Our results indicate that one of the essential roles of tuning dynamics of area 17 neurons, which have been observed for various visual features, is to adjust tunings depending on contrast.SIGNIFICANCE STATEMENT The spatial scales of features transmitted by cortical neurons are adjusted depending on stimulus contrast. However, the underlying mechanism is not fully understood. We measured the time course of spatial frequency tunings of cat area 17 neurons under different contrast conditions and observed a variety of dynamic effects that contributed to spatial-scale adjustment, allowing neurons to adjust their spatial frequency tuning range depending on contrast. Our results suggest that one of the essential roles of tuning dynamics of area 17 neurons, which have been observed for various visual features, is to adjust tunings depending on contrast.

Response-locked classification image analysis of perceptual decision making in contrast detection

In many situations, humans make decisions based on serially sampled information through the observation of visual stimuli. To quantify the critical information used by the observer in such dynamic decision making, we here applied a classification image (CI) analysis locked to the observer's reaction time (RT) in a simple detection task for a luminance target that gradually appeared in dynamic noise. We found that the response-locked CI shows a spatiotemporally biphasic weighting profile that peaked about 300 ms before the response, but this profile substantially varied depending on RT; positive weights dominated at short RTs and negative weights at long RTs. We show that these diverse results are explained by a simple perceptual decision mechanism that accumulates the output of the perceptual process as modelled by a spatiotemporal contrast detector. We discuss possible applications and the limitations of the response-locked CI analysis.

Transsaccadic integration of visual information is predictive, attention-based, and spatially precise

Eye movements produce shifts in the positions of objects in the retinal image, but observers are able to integrate these shifting retinal images into a coherent representation of visual space. This ability is thought to be mediated by attention-dependent saccade-related neural activity that is used by the visual system to anticipate the retinal consequences of impending eye movements. Previous investigations of the perceptual consequences of this predictive activity typically infer attentional allocation using indirect measures such as accuracy or reaction time. Here, we investigated the perceptual consequences of saccades using an objective measure of attentional allocation, reverse correlation. Human observers executed a saccade while monitoring a flickering target object flanked by flickering distractors and reported whether the average luminance of the target was lighter or darker than the background. Successful task performance required subjects to integrate visual information across the saccade. A reverse correlation analysis yielded a spatiotemporal "psychophysical kernel" characterizing how different parts of the stimulus contributed to the luminance decision throughout each trial. Just before the saccade, observers integrated luminance information from a distractor located at the post-saccadic retinal position of the target, indicating a predictive perceptual updating of the target. Observers did not integrate information from distractors placed in alternative locations, even when they were nearer to the target object. We also observed simultaneous predictive perceptual updating for two spatially distinct targets. These findings suggest both that shifting neural representations mediate the coherent representation of visual space, and that these shifts have significant consequences for transsaccadic perception.

Serial dependence revealed in history-dependent perceptual templates

In any given perceptual task, the visual system selectively weighs or filters incoming information. The particular set of weights or filters form a kind of template, which reveals the regions or types of information that are particularly useful for a given perceptual decision.1,2 Unfortunately, sensory input is noisy and ever changing. To compensate for these fluctuations, the visual system could adopt a strategy of biasing the templates such that they reflect a temporal smoothing of input, which would be a form of serial dependence.3-5 Here, we demonstrate that perceptual templates are, in fact, altered by serial dependence. Using a simple orientation detection task and classification-image technique, we found that perceptual templates are systematically biased toward previously seen, task-irrelevant orientations. The results of an orientation discrimination task suggest that this shift in perceptual template derives from a change in the perceptual appearance of orientation. Our study reveals how serial dependence biases internal templates of orientation and suggests that the sensitivity of classification-image techniques in general could be improved by taking into account history-dependent fluctuations in templates.

Local organization of spatial frequency tuning dynamics in the cat visual areas 17 and 18

Spatial frequency (SF) is a prominent feature to which most neurons in cat areas 17 and 18 (area 17/18) exhibit tuning selectivity. Previous studies have shown that neurons with similar SF tunings are locally clustered into SF preference domains. However, the functional organization of SF tuning remains not fully understood. Neurons in these areas show a variety of SF tuning dynamics; however, it is unknown how neurons with diverse dynamics are locally organized to form the population dynamics of the domains. The laminar organization of SF dynamics is also unknown, knowledge of which may be useful for determining how SF tuning dynamics of cat area 17/18 neurons arise in cortical circuits. To address these issues, we recorded the activities of multiple neurons in the cat area 17/18 using microelectrode arrays and characterized the time courses of the SF tunings of these neurons by a subspace reverse correlation. A wide range of SF dynamics was already present in the input layer, suggesting that intracortical mechanisms contribute to generating SF dynamics inside this layer but do not further shape it outside this layer. Local neuronal pools with similar SF tunings contained diverse SF dynamics. The average preferred SF of a pool similarly increased with response time. Moreover, the range of single-neuron preferred SFs in a pool tended to increase with time. Our results suggest that, in the presence of organized tuning diversity within an SF domain, the cortical SF organization remains stable during response time in cat area 17/18.NEW & NOTEWORTHY In cat area 17/18, we found that a local pool of neurons with similar spatial frequency (SF) tunings shows diverse but organized dynamics. Our results suggest that, in the presence of organized tuning diversity within an SF domain, the cortical SF organization remains stable over response time in these areas. Laminar analysis suggests that intracortical mechanisms contribute to generating SF dynamics inside the input layer but do not further shape it outside this layer.

Voluntary control of illusory contour formation

The extent to which visual inference is shaped by attentional goals is unclear. Voluntary attention may simply modulate the priority with which information is accessed by the higher cognitive functions involved in perceptual decision making. Alternatively, voluntary attention may influence fundamental visual processes, such as those involved in segmenting an incoming retinal signal into a structured scene of coherent objects, thereby determining perceptual organization. Here we tested whether the segmentation and integration of visual form can be determined by an observer's goals, by exploiting a novel variant of the classical Kanizsa figure. We generated predictions about the influence of attention with a machine classifier and tested these predictions with a psychophysical response classification technique. Despite seeing the same image on each trial, observers' perception of illusory spatial structure depended on their attentional goals. These attention-contingent illusory contours directly conflicted with other, equally plausible visual forms implied by the geometry of the stimulus, revealing that attentional selection can determine the perceived layout of a fragmented scene. Attentional goals, therefore, not only select precomputed features or regions of space for prioritized processing, but under certain conditions also greatly influence perceptual organization, and thus visual appearance.

Psychophysical reverse correlation reflects both sensory and decision-making processes

Goal-directed behavior depends on both sensory mechanisms that gather information from the outside world and decision-making mechanisms that select appropriate behavior based on that sensory information. Psychophysical reverse correlation is commonly used to quantify how fluctuations of sensory stimuli influence behavior and is generally believed to uncover the spatiotemporal weighting functions of sensory processes. Here we show that reverse correlations also reflect decision-making processes and can deviate significantly from the true sensory filters. Specifically, changes of decision bound and mechanisms of evidence integration systematically alter psychophysical reverse correlations. Similarly, trial-to-trial variability of sensory and motor delays and decision times causes systematic distortions in psychophysical kernels that should not be attributed to sensory mechanisms. We show that ignoring details of the decision-making process results in misinterpretation of reverse correlations, but proper use of these details turns reverse correlation into a powerful method for studying both sensory and decision-making mechanisms.

Spatial limitations in averaging social cues

The direction of social attention from groups provides stronger cueing than from an individual. It has previously been shown that both basic visual features such as size or orientation and more complex features such as face emotion and identity can be averaged across multiple elements. Here we used an equivalent noise procedure to compare observers' ability to average social cues with their averaging of a non-social cue. Estimates of observers' internal noise (uncertainty associated with processing any individual) and sample-size (the effective number of gaze-directions pooled) were derived by fitting equivalent noise functions to discrimination thresholds. We also used reverse correlation analysis to estimate the spatial distribution of samples used by participants. Averaging of head-rotation and cone-rotation was less noisy and more efficient than averaging of gaze direction, though presenting only the eye region of faces at a larger size improved gaze averaging performance. The reverse correlation analysis revealed greater sampling areas for head rotation compared to gaze. We attribute these differences in averaging between gaze and head cues to poorer visual processing of faces in the periphery. The similarity between head and cone averaging are examined within the framework of a general mechanism for averaging of object rotation.

Dynamics of unconscious contextual effects in orientation processing

Contextual effects abound in the real world; how we perceive an object depends on what surrounds it. A classic example of this is the tilt illusion (TI) whereby the presence of a surround shifts the perceived orientation of a target. Surprisingly, the magnitude and direction of this shift depend on the orientation difference between the target and surround: when their orientations are similar, the perceived difference is amplified and the target appears repelled in orientation from the surround (i.e., the TI). However, when their orientations are close to perpendicular, the difference is decreased and the target appears attracted in orientation toward the surround (i.e., the indirect TI). These misperceptions of orientation have revealed much about the underlying detectors involved in visual processing and how they interact with each other. What remains at stake are the levels of processing involved. To examine this, we designed a reverse-correlation technique whereby observers are blind to the orientation of the surround. We find that the TI and indirect TI occur reliably and over a similar time course, supporting the role of a single mechanism underlying orientation biases that operates in the early stages of visual processing before the conscious extraction of the surround orientation.

Coarse to fine dynamics of monocular and binocular processing in human pattern vision

Biological image processing has been hypothesized to adopt a coarse to fine strategy: the image is initially analyzed at a coarse spatial scale, and this analysis is then used to guide subsequent inspection at a finer scale. Neurons in visual cortex often display response characteristics that are consistent with this hypothesis for both monocular and binocular signals. Puzzlingly, measurements in human observers have failed to expose similar coarse to fine dynamics for human pattern vision, questioning the applicability of direct parallels between single neurons and perception. We performed a series of measurements using experimental protocols that were specifically designed to examine this question in more detail. We were able to confirm that, when the analysis is restricted to the linear properties of the perceptual process, no coarse to fine dynamics were evident in the data. However, when the analysis was extended to nonlinear descriptors, a clear coarse to fine structure emerged that consisted of two processes: an early nonlinear process operating on a coarse spatial scale followed by a linear process operating on a fine spatial scale. These results potentially serve to reduce the gap between the electrophysiological and behavioral findings.

Reward sharpens orientation coding independently of attention

It has long been known that rewarding improves performance. However it is unclear whether this is due to high level modulations in the output modules of associated neural systems or due to low level mechanisms favoring more “generous” inputs? Some recent studies suggest that primary sensory areas, including V1 and A1, may form part of the circuitry of reward-based modulations, but there is no data indicating whether reward can be dissociated from attention or cross-trial forms of perceptual learning. Here we address this issue with a psychophysical dual task, to control attention, while perceptual performance on oriented targets associated with different levels of reward is assessed by measuring both orientation discrimination thresholds and behavioral tuning functions for tilt values near threshold. We found that reward, at any rate, improved performance. However, higher reward rates showed an improvement of orientation discrimination thresholds by about 50% across conditions and sharpened behavioral tuning functions. Data were unaffected by changing the attentional load and by dissociating the feature of the reward cue from the task-relevant feature. These results suggest that reward may act within the span of a single trial independently of attention by modulating the activity of early sensory stages through a improvement of the signal-to-noise ratio of task-relevant channels.

Change-Based Inference in Attractor Nets: Linear Analysis

One standard interpretation of networks of cortical neurons is that they form dynamical attractors. Computations such as stimulus estimation are performed by mapping inputs to points on the networks' attractive manifolds. These points represent population codes for the stimulus values. However, this standard interpretation is hard to reconcile with the observation that the firing rates of such neurons constantly change following presentation of stimuli. We have recently suggested an alternative interpretation according to which computations are realized by systematic changes in the states of such networks over time. This way of performing computations is fast, accurate, readily learnable, and robust to various forms of noise. Here we analyze the computation of stimulus discrimination in this change-based setting, relating it directly to the computation of stimulus estimation in the conventional attractor-based view. We use a common linear approximation to compare the two methods and show that perfect performance at estimation implies chance performance at discrimination.

Psychophysical reverse correlation with multiple response alternatives

Psychophysical reverse-correlation methods such as the "classification image" technique provide a unique tool to uncover the internal representations and decision strategies of individual participants in perceptual tasks. Over the past 30 years, these techniques have gained increasing popularity among both visual and auditory psychophysicists. However, thus far, principled applications of the psychophysical reverse-correlation approach have been almost exclusively limited to two-alternative decision (detection or discrimination) tasks. Whether and how reverse-correlation methods can be applied to uncover perceptual templates and decision strategies in situations involving more than just two response alternatives remain largely unclear. Here, the authors consider the problem of estimating perceptual templates and decision strategies in stimulus identification tasks with multiple response alternatives. They describe a modified correlational approach, which can be used to solve this problem. The approach is evaluated under a variety of simulated conditions, including different ratios of internal-to-external noise, different degrees of correlations between the sensory observations, and various statistical distributions of stimulus perturbations. The results indicate that the proposed approach is reasonably robust, suggesting that it could be used in future empirical studies.

Contextual effects on decision templates for parafoveal orientation identification

When a peripherally viewed stimulus is presented with flankers, observers' acuity for shape generally decreases. We wondered whether a change in the locus of information accrual accompanied these performance deficits and employed psychophysical reverse correlation to find out. Surrounding the target (a near-vertical Gabor patch) with a vertical grating caused a slight elongation and a rotation in the decision templates for orientation identification. We also found that the contrast required to maintain criterion performance in this condition was actually lower than it was in a target-alone condition. However, this facilitation decreased with practice, due to perceptual learning in the target-alone condition. Unlike a continuous surround, isolated flanks elevated contrast thresholds, but decision templates were similar with both of these contexts. The rotation of decision templates (off-orientation looking) suggests that performance is limited by additive internal noise. We speculate that this noise can be reduced when the target is easily segregated from its surround.