Contour saliency in primary visual cortex

Pattern completion and disruption characterize contextual modulation in the visual cortex

Vision is fundamentally context-dependent, with neuronal responses influenced not just by local features but also by surrounding contextual information. In the visual cortex, studies using simple grating stimuli indicate that congruent stimuli - where the center and surround share the same orientation - are more inhibitory than when orientations are orthogonal, potentially serving redundancy reduction and predictive coding. Understanding these center-surround interactions in relation to natural image statistics is challenging due to the high dimensionality of the stimulus space, yet crucial for deciphering the neuronal code of real-world sensory processing. Utilizing large-scale recordings from mouse V1, we trained convolutional neural networks (CNNs) to predict and synthesize surround patterns that either optimally suppressed or enhanced responses to center stimuli, confirmed by in vivo experiments. Contrary to the notion that congruent stimuli are suppressive, we found that surrounds that completed patterns based on natural image statistics were facilitatory, while disruptive surrounds were suppressive. Applying our CNN image synthesis method in macaque V1, we discovered that pattern completion within the near surround occurred more frequently with excitatory than with inhibitory surrounds, suggesting that our results in mice are conserved in macaques. Further, experiments and model analyses confirmed previous studies reporting the opposite effect with grating stimuli in both species. Using the MICrONS functional connectomics dataset, we observed that neurons with similar feature selectivity formed excitatory connections regardless of their receptive field overlap, aligning with the pattern completion phenomenon observed for excitatory surrounds. Finally, our empirical results emerged in a normative model of perception implementing Bayesian inference, where neuronal responses are modulated by prior knowledge of natural scene statistics. In summary, our findings identify a novel relationship between contextual information and natural scene statistics and provide evidence for a role of contextual modulation in hierarchical inference.

Dependence of Contextual Modulation in Macaque V1 on Interlaminar Signal Flow

In visual cortex, neural correlates of subjective perception can be generated by modulation of activity from beyond the classical receptive field (CRF). In macaque V1, activity generated by nonclassical receptive field (nCRF) stimulation involves different intracortical circuitry than activity generated by CRF stimulation, suggesting that interactions between neurons across V1 layers differ under CRF and nCRF stimulus conditions. We measured border ownership modulation within large populations of V1 neurons. We found that neurons in single columns preferred the same side of objects located outside of the CRF. In addition, we found that interactions between pairs of neurons situated across feedback/horizontal and input layers differed between CRF and nCRF stimulation. Furthermore, the magnitude of border ownership modulation was predicted by greater information flow from feedback/horizontal to input layers. These results demonstrate that the flow of signals between layers covaries with the degree to which neurons integrate information from beyond the CRF.

Top-down generation of low-resolution representations improves visual perception and imagination

Perception or imagination requires top-down signals from high-level cortex to primary visual cortex (V1) to reconstruct or simulate the representations bottom-up stimulated by the seen images. Interestingly, top-down signals in V1 have lower spatial resolution than bottom-up representations. It is unclear why the brain uses low-resolution signals to reconstruct or simulate high-resolution representations. By modeling the top-down pathway of the visual system using the decoder of a variational auto-encoder (VAE), we reveal that low-resolution top-down signals can better reconstruct or simulate the information contained in the sparse activities of V1 simple cells, which facilitates perception and imagination. This advantage of low-resolution generation is related to facilitating high-level cortex to form geometry-respecting representations observed in experiments. Furthermore, we present two findings regarding this phenomenon in the context of AI-generated sketches, a style of drawings made of lines. First, we found that the quality of the generated sketches critically depends on the thickness of the lines in the sketches: thin-line sketches are harder to generate than thick-line sketches. Second, we propose a technique to generate high-quality thin-line sketches: instead of directly using original thin-line sketches, we use blurred sketches to train VAE or GAN (generative adversarial network), and then infer the thin-line sketches from the VAE- or GAN-generated blurred sketches. Collectively, our work suggests that low-resolution top-down generation is a strategy the brain uses to improve visual perception and imagination, which inspires new sketch-generation AI techniques.

Involvement of superior colliculus in complex figure detection of mice

Object detection is an essential function of the visual system. Although the visual cortex plays an important role in object detection, the superior colliculus can support detection when the visual cortex is ablated or silenced. Moreover, it has been shown that superficial layers of mouse SC (sSC) encode visual features of complex objects, and that this code is not inherited from the primary visual cortex. This suggests that mouse sSC may provide a significant contribution to complex object vision. Here, we use optogenetics to show that mouse sSC is involved in figure detection based on differences in figure contrast, orientation, and phase. Additionally, our neural recordings show that in mouse sSC, image elements that belong to a figure elicit stronger activity than those same elements when they are part of the background. The discriminability of this neural code is higher for correct trials than for incorrect trials. Our results provide new insight into the behavioral relevance of the visual processing that takes place in sSC.

Modeling the Role of Contour Integration in Visual Inference

Under difficult viewing conditions, the brain's visual system uses a variety of recurrent modulatory mechanisms to augment feedforward processing. One resulting phenomenon is contour integration, which occurs in the primary visual (V1) cortex and strengthens neural responses to edges if they belong to a larger smooth contour. Computational models have contributed to an understanding of the circuit mechanisms of contour integration, but less is known about its role in visual perception. To address this gap, we embedded a biologically grounded model of contour integration in a task-driven artificial neural network and trained it using a gradient-descent variant. We used this model to explore how brain-like contour integration may be optimized for high-level visual objectives as well as its potential roles in perception. When the model was trained to detect contours in a background of random edges, a task commonly used to examine contour integration in the brain, it closely mirrored the brain in terms of behavior, neural responses, and lateral connection patterns. When trained on natural images, the model enhanced weaker contours and distinguished whether two points lay on the same versus different contours. The model learned robust features that generalized well to out-of-training-distribution stimuli. Surprisingly, and in contrast with the synthetic task, a parameter-matched control network without recurrence performed the same as or better than the model on the natural-image tasks. Thus, a contour integration mechanism is not essential to perform these more naturalistic contour-related tasks. Finally, the best performance in all tasks was achieved by a modified contour integration model that did not distinguish between excitatory and inhibitory neurons.

Gestalt neurons and emergent properties in visual perception: A novel concept for the transformation from local to global processing

Gestalten in visual perception are defined by emergent properties of the whole, which cannot be predicted from the sum of its parts; rather, they arise by virtue of inherent principles, the Laws of Seeing. This review attempts to assign neurophysiological correlates to select emergent properties in motion and contour perception and proposes parallels to the processing of local versus global attributes by classical versus contextual receptive fields. The aim is to identify Gestalt neurons in the visual system to account for the Laws of Seeing in causal terms and to explain "Why do things look as they do" (Koffka, 1935, p. 76).

Context-dependent attentional spotlight in pulvinar-V1 interaction

Spatial attention is often described as a mental spotlight that enhances information processing at the attended location. Using fMRI, we investigated background connectivity between the pulvinar and V1 in relation to focused versus diffused attention allocation, in weak and strong crowding contexts. Our findings revealed that focused attention led to enhanced correlations between the pulvinar and V1. Notably, this modulation was initiated by the pulvinar, and the strength of the modulation was dependent on the saliency of the target. These findings suggest that the pulvinar initiates information reweighting to V1, which underlies attentional selection in cluttered scenes.

Measuring uncertainty in human visual segmentation

Segmenting visual stimuli into distinct groups of features and visual objects is central to visual function. Classical psychophysical methods have helped uncover many rules of human perceptual segmentation, and recent progress in machine learning has produced successful algorithms. Yet, the computational logic of human segmentation remains unclear, partially because we lack well-controlled paradigms to measure perceptual segmentation maps and compare models quantitatively. Here we propose a new, integrated approach: given an image, we measure multiple pixel-based same-different judgments and perform model-based reconstruction of the underlying segmentation map. The reconstruction is robust to several experimental manipulations and captures the variability of individual participants. We demonstrate the validity of the approach on human segmentation of natural images and composite textures. We show that image uncertainty affects measured human variability, and it influences how participants weigh different visual features. Because any putative segmentation algorithm can be inserted to perform the reconstruction, our paradigm affords quantitative tests of theories of perception as well as new benchmarks for segmentation algorithms.

Contour integration deficits at high spatial frequencies in children treated for anisometropic amblyopia

Purpose

This study was conducted to reexamine the question of whether children treated for anisometropic amblyopia have contour integration deficits. To do so, we used psychophysical methods that require global contour processing while minimizing the influence of low-level deficits: visibility, shape perception, and positional uncertainty.

Methods

Thirteen children with anisometropic amblyopia (age: 10.1 ± 1.8 years) and thirteen visually normal children (age: 10.8 ± 2.0 years) participated in this study. The stimuli were closed figures made up of Gabor patches either in noise or on a blank field. The contrast thresholds to detect a circular contour on a blank field, as well as the thresholds of aspect ratio and contour element number to discriminate a circular or elliptical contour in noise, were measured at Gabor spatial frequencies of 1.5, 3, and 6 cpd for amblyopic eyes (AEs), fellow eyes (FEs), and normal control eyes. Visual acuities and contrast sensitivity functions for AEs and FEs and the Randot stereoacuity were measured before testing.

Results

The AEs showed contrast deficits and degraded shape perception compared to the FEs at higher spatial frequencies (6 cpd). When the influence of abnormal contrast sensitivity and shape perception were minimized, the AEs showed contour integration deficits at spatial frequencies 3 and 6 cpd. These deficits were not related to basic losses in contrast sensitivity and acuity, stereoacuity, and visual crowding. Besides, no significant difference was found between the fellow eyes of the amblyopic children and the normal control eyes in the performance of contour integration.

Conclusion

After eliminating or compensating for the low-level deficits, children treated for anisometropic amblyopia still show contour integration deficits, primarily at higher spatial frequencies, which might reflect the deficits in global processing caused by amblyopia. Contour integration deficits are likely independent of spatial vision deficits. Refractive correction and/or occlusion therapies may not be sufficient to fully restore contour integration deficits, which indicates the need for the development of clinical treatments to recover these deficits.

Solving the binding problem: Assemblies form when neurons enhance their firing rate-they don't need to oscillate or synchronize

When we look at an image, its features are represented in our visual system in a highly distributed manner, calling for a mechanism that binds them into coherent object representations. There have been different proposals for the neuronal mechanisms that can mediate binding. One hypothesis is that binding is achieved by oscillations that synchronize neurons representing features of the same perceptual object. This view allows separate communication channels between different brain areas. Another hypothesis is that binding of features that are represented in different brain regions occurs when the neurons in these areas that respond to the same object simultaneously enhance their firing rate, which would correspond to directing object-based attention to these features. This review summarizes evidence in favor of and against these two hypotheses, examining the neuronal correlates of binding and assessing the time course of perceptual grouping. I conclude that enhanced neuronal firing rates bind features into coherent object representations, whereas oscillations and synchrony are unrelated to binding.

The Psychosis Human Connectome Project: Design and rationale for studies of visual neurophysiology

Visual perception is abnormal in psychotic disorders such as schizophrenia. In addition to hallucinations, laboratory tests show differences in fundamental visual processes including contrast sensitivity, center-surround interactions, and perceptual organization. A number of hypotheses have been proposed to explain visual dysfunction in psychotic disorders, including an imbalance between excitation and inhibition. However, the precise neural basis of abnormal visual perception in people with psychotic psychopathology (PwPP) remains unknown. Here, we describe the behavioral and 7 tesla MRI methods we used to interrogate visual neurophysiology in PwPP as part of the Psychosis Human Connectome Project (HCP). In addition to PwPP (n = 66) and healthy controls (n = 43), we also recruited first-degree biological relatives (n = 44) in order to examine the role of genetic liability for psychosis in visual perception. Our visual tasks were designed to assess fundamental visual processes in PwPP, whereas MR spectroscopy enabled us to examine neurochemistry, including excitatory and inhibitory markers. We show that it is feasible to collect high-quality data across multiple psychophysical, functional MRI, and MR spectroscopy experiments with a sizable number of participants at a single research site. These data, in addition to those from our previously described 3 tesla experiments, will be made publicly available in order to facilitate further investigations by other research groups. By combining visual neuroscience techniques and HCP brain imaging methods, our experiments offer new opportunities to investigate the neural basis of abnormal visual perception in PwPP.

Attention along the cortical hierarchy: Development matters

We build on the existing biased competition view to argue that attention is an emergent property of neural computations within and across hierarchically embedded and structurally connected cortical pathways. Critically then, one must ask, what is attention emergent from? Within this framework, developmental changes in the quality of sensory input and feedforward-feedback information flow shape the emergence and efficiency of attention. Several gradients of developing structural and functional cortical architecture across the caudal-to-rostral axis provide the substrate for attention to emerge. Neural activity within visual areas depends on neuronal density, receptive field size, tuning properties of neurons, and the location of and competition between features and objects in the visual field. These visual cortical properties highlight the information processing bottleneck attention needs to resolve. Recurrent feedforward and feedback connections convey sensory information through a series of steps at each level of the cortical hierarchy, integrating sensory information across the entire extent of the cortical hierarchy and linking sensory processing to higher-order brain regions. Higher-order regions concurrently provide input conveying behavioral context and goals. Thus, attention reflects the output of a series of complex biased competition neural computations that occur within and across hierarchically embedded cortical regions. Cortical development proceeds along the caudal-to-rostral axis, mirroring the flow in sensory information from caudal to rostral regions, and visual processing continues to develop into childhood. Examining both typical and atypical development will offer critical mechanistic insight not otherwise available in the adult stable state. This article is categorized under: Psychology > Attention.

Adaptive processing and perceptual learning in visual cortical areas V1 and V4

Neurons in visual cortical areas primary visual cortex (V1) and V4 are adaptive processors, influenced by perceptual task. This is reflected in their ability to segment the visual scene into task-relevant and task-irrelevant stimulus components and by changing their tuning to task-relevant stimulus properties according to the current top-down instruction. Differences between the information represented in each area were seen. While V1 represented detailed stimulus characteristics, V4 filtered the input from V1 to carry the binary information required for the two-alternative judgement task. Neurons in V1 were activated at locations where the behaviorally relevant stimulus was placed well outside the grating-mapped receptive field. By systematically following the development of the task-dependent signals over the course of perceptual learning, we found that neuronal selectivity for task-relevant information was initially seen in V4 and, over a period of weeks, subsequently in V1. Once the learned information was represented in V1, on any given trial, task-relevant information appeared initially in V1 responses, followed by a 12-ms delay in V4. We propose that the shifting representation of learned information constitutes a mechanism for systems consolidation of memory.

Horizontal connectivity in V1: Prediction of coherence in contour and motion integration

This study demonstrates the functional importance of the Surround context relayed laterally in V1 by the horizontal connectivity, in controlling the latency and the gain of the cortical response to the feedforward visual drive. We report here four main findings: 1) a centripetal apparent motion sequence results in a shortening of the spiking latency of V1 cells, when the orientation of the local inducer and the global motion axis are both co-aligned with the RF orientation preference; 2) this contextual effects grows with visual flow speed, peaking at 150–250°/s when it matches the propagation speed of horizontal connectivity (0.15–0.25 mm/ms); 3) For this speed range, the axial sensitivity of V1 cells is tilted by 90° to become co-aligned with the orientation preference axis; 4) the strength of modulation by the surround context correlates with the spatiotemporal coherence of the apparent motion flow. Our results suggest an internally-generated binding process, linking local (orientation /position) and global (motion/direction) features as early as V1. This long-range diffusion process constitutes a plausible substrate in V1 of the human psychophysical bias in speed estimation for collinear motion. Since it is demonstrated in the anesthetized cat, this novel form of contextual control of the cortical gain and phase is a built-in property in V1, whose expression does not require behavioral attention and top-down control from higher cortical areas. We propose that horizontal connectivity participates in the propagation of an internal “prediction” wave, shaped by visual experience, which links contour co-alignment and global axial motion at an apparent speed in the range of saccade-like eye movements.

Attention impedes neural representation of interpolated orientation during perceptual completion

One of the most remarkable functional feats accomplished by visual system is the interpolation of missing retinal inputs based on surrounding information, a process known as perceptual completion. Perceptual completion enables the active construction of coherent, vivid percepts from spatially discontinuous visual information that is prevalent in real-life visual scenes. Despite mounting evidence linking sensory activity enhancement and perceptual completion, surprisingly little is known about whether and how attention, a fundamental modulator of sensory activities, affects perceptual completion. Using EEG-based time-resolved inverted encoding model (IEM), we reconstructed the moment-to-moment representation of the illusory grating that resulted from spatially interpolating the orientation of surrounding inducers. We found that, despite manipulation of observers' attentional focus, the illusory grating representation unfolded in time in a similar manner. Critically, attention to the surrounding inducers simultaneously attenuated the illusory grating representation and delayed its temporal development. Our findings disclosed, for the first time, the suppressive role of selective attention in perceptual completion and were suggestive of a fast, automatic neural machinery that implements the interpolation of missing visual information.

A neuronal basis of iconic memory in macaque primary visual cortex

After a briefly presented visual stimulus disappears, observers retain a detailed representation of this stimulus for a short period of time. This sensory storage is called iconic memory. We measured iconic memory in the perception of monkeys and its neuronal correlates in the primary visual cortex (area V1). We determined how many milliseconds extra viewing time iconic memory is worth and how it decays by varying the duration of a brief stimulus and the timing of a mask. The V1 activity that persists after the disappearance of a stimulus predicted accuracy, with a time course resembling the worth and decay of iconic memory. Finally, we examined how iconic memory interacts with attention. A cue presented after the stimulus disappears boosts attentional influences pertaining to a relevant part of the stimulus but only if it appears before iconic memory decayed. Our results relate iconic memory to neuronal activity in early visual cortex.

New Contour Cue-Based Hybrid Sparse Learning for Salient Object Detection

Saliency detection is a hot topic in recent years and much efforts have been made to address it from different perspectives. However, current saliency models cannot meet the needs for diversified scenes due to their limited generalization capability. To tackle this problem, in this paper, we propose a hybrid saliency model, which can fuse heterogeneous visual cues for robust salient object detection. A new contour cue is first introduced to provide discriminative saliency information for scene description. Its realization is based on a discrete optimization objective and can be solved efficiently with an iterative algorithm. Followed by this, the contour cue is taken as a part of a hybrid sparse learning model, in which cues from different domains can interact and complement with each other for joint saliency fusion. This saliency fusion model is parameter-free and its numerical solution can be obtained using gradient descent methods. Finally, we advance an object proposal-based collaborative filtering strategy to generate high quality saliency maps from the above fusion results. Compared with traditional methods, the proposed saliency model can fuse heterogeneous cues in a unified optimization framework rather than combine them separately. Therefore, it has favorable modeling capability under diversified scenes where the saliency patterns appear quite differently. To verify the effectiveness of the proposed method, we take experiments on four large saliency benchmark datasets and compare it with other 26 state-of-the-art saliency models. Both qualitative and quantitative evaluation results indicate the superiority of our method, especially in challenging situations. Besides, we apply our saliency model to ship detection of radar platforms and promising results are obtained over traditional detectors.

Modulation of the Earliest Component of the Human VEP by Spatial Attention: An Investigation of Task Demands

Spatial attention modulations of initial afferent activity in area V1, indexed by the first component “C1” of the human visual evoked potential, are rarely found. It has thus been suggested that early modulation is induced only by special task conditions, but what these conditions are remains unknown. Recent failed replications—findings of no C1 modulation using a certain task that had previously produced robust modulations—present a strong basis for examining this question. We ran 3 experiments, the first to more exactly replicate the stimulus and behavioral conditions of the original task, and the second and third to manipulate 2 key factors that differed in the failed replication studies: the provision of informative performance feedback, and the degree to which the probed stimulus features matched those facilitating target perception. Although there was an overall significant C1 modulation of 11%, individually, only experiments 1 and 2 showed reliable effects, underlining that the modulations do occur but not consistently. Better feedback induced greater P1, but not C1, modulations. Target-probe feature matching had an inconsistent influence on modulation patterns, with behavioral performance differences and signal-overlap analyses suggesting interference from extrastriate modulations as a potential cause.

The default mode network: where the idiosyncratic self meets the shared social world

The default mode network (DMN) is classically considered an 'intrinsic' system, specializing in internally oriented cognitive processes such as daydreaming, reminiscing and future planning. In this Perspective, we suggest that the DMN is an active and dynamic 'sense-making' network that integrates incoming extrinsic information with prior intrinsic information to form rich, context-dependent models of situations as they unfold over time. We review studies that relied on naturalistic stimuli, such as stories and movies, to demonstrate how an individual's DMN neural responses are influenced both by external information accumulated as events unfold over time and by the individual's idiosyncratic past memories and knowledge. The integration of extrinsic and intrinsic information over long timescales provides a space for negotiating a shared neural code, which is necessary for establishing shared meaning, shared communication tools, shared narratives and, above all, shared communities and social networks.

Visual association learning induces global network reorganization

It has been proposed that visual learning is accomplished not only by neural plasticity in the visual cortex, but also by complex interactions between bottom-up and top-down processes that may induce global network reorganization. Here, we applied a multivariate analysis to functional connectivity (FC) patterns across the brain to investigate how visual association learning was achieved through large-scale network reorganization. Participants were trained to associate a set of artificial line-drawing objects with English letters. After three consecutive days of training, participants underwent a functional magnetic resonance imaging scan in which they were presented with the trained stimuli, untrained stimuli, and English words. By calculating pairwise FC between 189 nodes of 10 well-established networks across the brain, we found that the visual association learning induced changes in the global FC pattern when viewing the trained stimuli, rendering it more similar to the FC pattern when viewing English words. Critically, the learning-induced global FC pattern differences were mainly driven by the FC related to the high-level networks involved in attention and cognitive control, suggesting the modification of top-down processes during learning. In sum, our study provides one of the first evidence revealing global network reorganization induced by visual learning and sheds new light on the network mechanisms of top-down influences in learning.

Integration of contours defined by second-order contrast-modulation of texture

Boundaries in the visual world can be defined by changes in luminance and texture in the input image. A "contour integration" process joins together local changes into percepts of lines or edges. A previous study tested the integration of contours defined by second-order contrast-modulation. Their contours were placed in a background of random wavelets. Participants performed near chance. We re-visited second-order contour integration with a different task. Participants distinguished contours with "good continuation" from distractors. We measured thresholds in different amounts of external orientation or position noise. This gave two noise-masking functions. We also measured thresholds for contours with a baseline curvature to assess performance with more curvy targets. Our participants were able to discriminate the good continuation of second-order contours. Thresholds were higher than for first-order contours. In our modelling, we found this was due to multiple factors. There was a doubling of equivalent internal noise between first- and second-order contour integration. There was also a reduction in efficiency. The efficiency difference was only significant in our orientation noise condition. For both first- and second-order stimuli, participants were also able to perform our task with more curved contours. We conclude that humans can integrate second-order contours, even when they are curved. There is however reduced performance compared to first-order contours. We find both an impaired input to the integrating mechanism, and reduced efficiency seem responsible. Second-order contour integration may be more affected by the noise background used in the previous study. Difficulty segregating that background may explain their result.

Heritability of human visual contour integration-an integrated genomic study

Contour integration, a key visual function to deal with occlusion and discontinuity in natural scenes, is essential to human survival. However, individuals are not equally well equipped with this ability. In particular, contour integration deficiencies are commonly detected in patients with mental disorders, especially schizophrenia. To understand the underlying sources of these individual differences, the current study investigated the genetic basis of contour integration in humans. A total of 2619 normal participants were tested on their ability to detect continuous contours embedded in a cluttered background. Quantitative genomic analysis was performed, involving heritability estimation based on single nucleotide polymorphisms (SNPs) and association testing at SNP, gene, and pathway levels. Heritability estimation showed that common SNPs contributed 49.5% (standard error of the mean = 15.6%) of overall phenotypic variation, indicating moderate heritability of contour integration. Two-stage genome-wide association analysis (GWAS) detected four SNPs reaching genome-wide significance in the discovery test (N = 1931) but not passing the replication test (N = 688). Gene-level analysis further revealed a significant genome-wide association of a microRNA-encoding gene MIR1178 in both the discovery and replication cohorts. Another gene poly(A)-binding protein nuclear 1 like, cytoplasmic (PABPN1L) showed suggestive significance in the discovery cohort (p < 1 × 10-4) and was replicated in the replication cohort (p = 0.009). The pathway analysis did not detect any significant pathway. Taken together, this study identified significant gene associations with contour integration and provided support for a genetic transmission of the ability to perceive continuous contours in the environment.

Spike-field Granger causality for hybrid neural data analysis

Neurotechnological innovations allow for simultaneous recording at various scales, ranging from spiking activity of individual neurons to large neural populations’ local field potentials (LFPs). This capability necessitates developing multiscale analysis of spike-field activity. A joint analysis of the hybrid neural data is crucial for bridging the scales between single neurons and local networks. Granger causality is a fundamental measure to evaluate directional influences among neural signals. However, it is mainly limited to inferring causal influence between the same type of signals—either LFPs or spike trains—and not well developed between two different signal types. Here we propose a model-free, nonparametric spike-field Granger causality measure for hybrid data analysis. Our measure is distinct from existing methods in that we use “binless” spikes (precise spike timing) rather than “binned” spikes (spike counts within small consecutive time windows). The latter clearly distort the information in the mixed analysis of spikes and LFP. Therefore, our spectral estimate of spike trains is directly applied to the neural point process itself, i.e., sequences of spike times rather than spike counts. Our measure is validated by an extensive set of simulated data. When the measure is applied to LFPs and spiking activity simultaneously recorded from visual areas V1 and V4 of monkeys performing a contour detection task, we are able to confirm computationally the long-standing experimental finding of the input-output relationship between LFPs and spikes. Importantly, we demonstrate that spike-field Granger causality can be used to reveal the modulatory effects that are inaccessible by traditional methods, such that spike→LFP Granger causality is modulated by the behavioral task, whereas LFP→spike Granger causality is mainly related to the average synaptic input.

NEW & NOTEWORTHY It is a pressing question to study the directional interactions between local field potential (LFP) and spiking activity. In this report, we propose a model-free, nonparametric spike-field Granger causality measure that can be used to reveal directional influences between spikes and LFPs. This new measure is crucial for bridging the scales between single neurons and neural networks; hence it represents an important step to explicate how the brain orchestrates information processing.

The face-in-the-crowd effect: Threat detection versus iso-feature suppression and collinear facilitation

Are people biologically prepared for the rapid detection of threat posed by an angry facial expression, even when it is conveyed in the form of a schematic line drawing? Based on visual search times, the current literature would suggest that the answer is yes. But are there low-level explanations for this effect? Here, we present visual search results for schematic faces using current best practice, based on a concentric search array and set size manipulation. Using this approach, we replicate the classic search advantage for angry over happy faces. However, we also report a comparable effect when abstract plus- and square-shaped stimuli-derived from the angry and happy schematic faces respectively-are used within the same paradigm. We then go on to demonstrate that, while reduced, the effect remains after removal of the circular surround, bringing us closer to the source of the effect. We explore the possibility that the source of this search asymmetry could be the iso-feature suppression and collinear facilitation model proposed in Li's (1999a, 1999b, and 2002) bottom-up model of saliency. Simulations with this model using the abstract stimuli align with the corresponding behavioral results (i.e., the plus shape was found to be more salient than the square). Given the deliberate similarities between these abstract shapes and the respective face stimuli, we propose that the underlying cause for the asymmetries typically found using schematic faces, may be more related to early visual processing of line orientation than threat detection.

Superior colliculus encodes visual saliency during smooth pursuit eye movements

The saliency map has played a long-standing role in models and theories of visual attention, and it is now supported by neurobiological evidence from several cortical and subcortical brain areas. While visual saliency is computed during moments of active fixation, it is not known whether the same is true while engaged in smooth pursuit of a moving stimulus, which is very common in real-world vision. Here, we examined extrafoveal saliency coding in the superior colliculus, a midbrain area associated with attention and gaze, during smooth pursuit eye movements. We found that SC neurons from the superficial visual layers showed a robust representation of peripheral saliency evoked by a conspicuous stimulus embedded in a wide-field array of goal-irrelevant stimuli. In contrast, visuomotor neurons from the intermediate saccade-related layers showed a poor saliency representation, even though most of these neurons were visually responsive during smooth pursuit. These results confirm and extend previous findings that place the SCs in a unique role as a saliency map that monitors peripheral vision during foveation of stationary and now moving objects.

Cognitive load effects on early visual perceptual processing

Contrast-based early visual processing has largely been considered to involve autonomous processes that do not need the support of cognitive resources. However, as spatial attention is known to modulate early visual perceptual processing, we explored whether cognitive load could similarly impact contrast-based perception. We used a dual-task paradigm to assess the impact of a concurrent working memory task on the performance of three different early visual tasks. The results from Experiment 1 suggest that cognitive load can modulate early visual processing. No effects of cognitive load were seen in Experiments 2 or 3. Together, the findings provide evidence that under some circumstances cognitive load effects can penetrate the early stages of visual processing and that higher cognitive function and early perceptual processing may not be as independent as was once thought.

Mechanisms of Spatiotemporal Selectivity in Cortical Area MT

Cortical sensory neurons are characterized by selectivity to stimulation. This selectivity was originally viewed as a part of the fundamental "receptive field" characteristic of neurons. This view was later challenged by evidence that receptive fields are modulated by stimuli outside of the classical receptive field. Here, we show that even this modified view of selectivity needs revision. We measured spatial frequency selectivity of neurons in cortical area MT of alert monkeys and found that their selectivity strongly depends on luminance contrast, shifting to higher spatial frequencies as contrast increases. The changes of preferred spatial frequency are large at low temporal frequency, and they decrease monotonically as temporal frequency increases. That is, even interactions among basic stimulus dimensions of luminance contrast, spatial frequency, and temporal frequency strongly influence neuronal selectivity. This dynamic nature of neuronal selectivity is inconsistent with the notion of stimulus preference as a stable characteristic of cortical neurons.

Recurrent Processing of Contour Integration in the Human Visual Cortex as Revealed By fMRI-Guided TMS

Contour integration is a critical step in visual perception because it groups discretely local elements into perceptually global contours. Previous investigations have suggested that striate and extrastriate visual areas are involved in this mid-level processing of visual perception. However, the temporal dynamics of these areas in the human brain during contour integration is less understood. The present study used functional magnetic resonance imaging-guided transcranial magnetic stimulation (TMS) to briefly disrupt 1 of 2 visual areas (V1/V2 and V3B) and examined the causal contributions of these areas to contour detection. The results demonstrated that the earliest critical time window at which behavioral detection performance was impaired by TMS pluses differed between V1/V2 and V3B. The first critical window of V3B (90-110 ms after stimulus onset) was earlier than that of V1/V2 (120-140 ms after stimulus onset), thus indicating that feedback connection from higher to lower area was necessary for complete contour integration. These results suggested that the fine processing of contour-related information in V1/V2 follows the generation of a coarse template in the higher visual areas, such as V3B. Our findings provide direct causal evidence that a recurrent mechanism is necessary for the integration of contours from cluttered background in the human brain.

Axonal plasticity associated with perceptual learning in adult macaque primary visual cortex

Perceptual learning is associated with changes in the functional properties of neurons even in primary sensory areas. In macaque monkeys trained to perform a contour detection task, we have observed changes in contour-related facilitation of neuronal responses in primary visual cortex that track their improvement in performance on a contour detection task. We have previously explored the anatomical substrate of experience-dependent changes in the visual cortex based on a retinal lesion model, where we find sprouting and pruning of the axon collaterals in the cortical lesion projection zone. Here, we attempted to determine whether similar changes occur under normal visual experience, such as that associated with perceptual learning. We labeled the long-range horizontal connections in visual cortex by virally mediated transfer of genes expressing fluorescent probes, which enabled us to do longitudinal two-photon imaging of axonal arbors over the period during which animals improve in contour detection performance. We found that there are substantial changes in the axonal arbors of neurons in cortical regions representing the trained part of the visual field, with sprouting of new axon collaterals and pruning of preexisting axon collaterals. Our findings indicate that changes in the structure of axonal arbors are part of the circuit-level mechanism of perceptual learning, and further support the idea that the learned information is encoded at least in part in primary visual cortex.

Bottom-up saliency and top-down learning in the primary visual cortex of monkeys

Early sensory cortex is better known for representing sensory inputs but less for the effect of its responses on behavior. Here we explore the behavioral correlates of neuronal responses in primary visual cortex (V1) in a task to detect a uniquely oriented bar-the orientation singleton-in a background of uniformly oriented bars. This singleton is salient or inconspicuous when the orientation contrast between the singleton and background bars is sufficiently large or small, respectively. Using implanted microelectrodes, we measured V1 activities while monkeys were trained to quickly saccade to the singleton. A neuron's responses to the singleton within its receptive field had an early and a late component, both increased with the orientation contrast. The early component started from the outset of neuronal responses; it remained unchanged before and after training on the singleton detection. The late component started ∼40 ms after the early one; it emerged and evolved with practicing the detection task. Training increased the behavioral accuracy and speed of singleton detection and increased the amount of information in the late response component about a singleton's presence or absence. Furthermore, for a given singleton, faster detection performance was associated with higher V1 responses; training increased this behavioral-neural correlate in the early V1 responses but decreased it in the late V1 responses. Therefore, V1's early responses are directly linked with behavior and represent the bottom-up saliency signals. Learning strengthens this link, likely serving as the basis for making the detection task more reflexive and less top-down driven.

Contour interpolation: A case study in Modularity of Mind

In his monograph Modularity of Mind (1983), philosopher Jerry Fodor argued that mental architecture can be partly decomposed into computational organs termed modules, which were characterized as having nine co-occurring features such as automaticity, domain specificity, and informational encapsulation. Do modules exist? Debates thus far have been framed very generally with few, if any, detailed case studies. The topic is important because it has direct implications on current debates in cognitive science and because it potentially provides a viable framework from which to further understand and make hypotheses about the mind's structure and function. Here, the case is made for the modularity of contour interpolation, which is a perceptual process that represents non-visible edges on the basis of how surrounding visible edges are spatiotemporally configured. There is substantial evidence that interpolation is domain specific, mandatory, fast, and developmentally well-sequenced; that it produces representationally impoverished outputs; that it relies upon a relatively fixed neural architecture that can be selectively impaired; that it is encapsulated from belief and expectation; and that its inner workings cannot be fathomed through conscious introspection. Upon differentiating contour interpolation from a higher-order contour representational ability ("contour abstraction") and upon accommodating seemingly inconsistent experimental results, it is argued that interpolation is modular to the extent that the initiating conditions for interpolation are strong. As interpolated contours become more salient, the modularity features emerge. The empirical data, taken as a whole, show that at least certain parts of the mind are modularly organized.

Uncovering the Spatial Profile of Contour Integration from Fixational Saccades: Evidence for Widespread Processing in V1

During contour integration, neuronal populations in the primary visual cortex (V1) enhance their responses to the contour while suppressing their responses to the noisy background. However, the spatial extent and profile of these responses are not fully understood. To investigate this question, 2 monkeys were trained on a contour detection task while we measured population responses in V1 using voltage-sensitive dyes. During stimulus presentation the animals made few fixational saccades, and we used their changing gaze position to image and analyze neuronal responses from large part of the stimulus, encoding multiple contour/background elements. We found that contour enhancement was present over the entire contour-mapped areas. The background suppression increased with distance from the contour, extending into background-mapped areas remotely located from the contour. The spatial profile of enhancement and suppression fitted well with a Gaussian model. These results imply that the divergent cortical responses to contour integration are modulated independently and extend over large areas in V1.

Nonuniform surround suppression of visual responses in mouse V1

Complex receptive field characteristics, distributed across a population of neurons, are thought to be critical for solving perceptual inference problems that arise during motion and image segmentation. For example, in a class of neurons referred to as "end-stopped," increasing the length of stimuli outside of the bar-responsive region into the surround suppresses responsiveness. It is unknown whether these properties exist for receptive field surrounds in the mouse. We examined surround modulation in layer 2/3 neurons of the primary visual cortex in mice using two-photon calcium imaging. We found that surround suppression was significantly asymmetric in 17% of the visually responsive neurons examined. Furthermore, the magnitude of asymmetry was correlated with orientation selectivity. Our results demonstrate that neurons in mouse primary visual cortex are differentially sensitive to the addition of elements in the surround and that individual neurons can be described as being either uniformly suppressed by the surround, end-stopped, or side-stopped. NEW & NOTEWORTHY Perception of visual scenes requires active integration of both local and global features to successfully segment objects from the background. Although the underlying circuitry and development of perceptual inference is not well understood, converging evidence indicates that asymmetry and diversity in surround modulation are likely fundamental for these computations. We determined that these key features are present in the mouse. Our results support the mouse as a model to explore the neural basis and development of surround modulation as it relates to perceptual inference.

Contour Integration in Dynamic Scenes: Impaired Detection Performance in Extended Presentations

Since scenes in nature are highly dynamic, perception requires an on-going and robust integration of local information into global representations. In vision, contour integration (CI) is one of these tasks, and it is performed by our brain in a seemingly effortless manner. Following the rule of good continuation, oriented line segments are linked into contour percepts, thus supporting important visual computations such as the detection of object boundaries. This process has been studied almost exclusively using static stimuli, raising the question of whether the observed robustness and "pop-out" quality of CI carries over to dynamic scenes. We investigate contour detection in dynamic stimuli where targets appear at random times by Gabor elements aligning themselves to form contours. In briefly presented displays (230 ms), a situation comparable to classical paradigms in CI, performance is about 87%. Surprisingly, we find that detection performance decreases to 67% in extended presentations (about 1.9-3.8 s) for the same target stimuli. In order to observe the same reduction with briefly presented stimuli, presentation time has to be drastically decreased to intervals as short as 50 ms. Cueing a specific contour position or shape helps in partially compensating this deterioration, and only in extended presentations combining a location and a shape cue was more efficient than providing a single cue. Our findings challenge the notion of CI as a mainly stimulus-driven process leading to pop-out percepts, indicating that top-down processes play a much larger role in supporting fundamental integration processes in dynamic scenes than previously thought.

Superior colliculus encodes visual saliency before the primary visual cortex

Models of visual attention postulate the existence of a bottom-up saliency map that is formed early in the visual processing stream. Although studies have reported evidence of a saliency map in various cortical brain areas, determining the contribution of phylogenetically older pathways is crucial to understanding its origin. Here, we compared saliency coding from neurons in two early gateways into the visual system: the primary visual cortex (V1) and the evolutionarily older superior colliculus (SC). We found that, while the response latency to visual stimulus onset was earlier for V1 neurons than superior colliculus superficial visual-layer neurons (SCs), the saliency representation emerged earlier in SCs than in V1. Because the dominant input to the SCs arises from V1, these relative timings are consistent with the hypothesis that SCs neurons pool the inputs from multiple V1 neurons to form a feature-agnostic saliency map, which may then be relayed to other brain areas.

Interactions between feedback and lateral connections in the primary visual cortex

Perceptual grouping of line segments into object contours has been thought to be mediated, in part, by long-range horizontal connectivity intrinsic to the primary visual cortex (V1), with a contribution by top-down feedback projections. To dissect the contributions of intraareal and interareal connections during contour integration, we applied conditional Granger causality analysis to assess directional influences among neural signals simultaneously recorded from visual cortical areas V1 and V4 of monkeys performing a contour detection task. Our results showed that discounting the influences from V4 markedly reduced V1 lateral interactions, indicating dependence on feedback signals of the effective connectivity within V1. On the other hand, the feedback influences were reciprocally dependent on V1 lateral interactions because the modulation strengths from V4 to V1 were greatly reduced after discounting the influences from other V1 neurons. Our findings suggest that feedback and lateral connections closely interact to mediate image grouping and segmentation.

Responses in early visual areas to contour integration are context dependent

It has been shown that early visual areas are involved in contour processing. However, it is not clear how local and global context interact to influence responses in those areas, nor has the interarea coordination that yields coherent structural percepts been fully studied, especially in human observers. In this study, we used functional magnetic resonance imaging (fMRI) to measure activity in early visual cortex while observers performed a contour detection task in which alignment of Gabor elements and background clutter were manipulated. Six regions of interest (two regions, containing either the cortex representing the target or the background clutter, in each of areas V1, V2, and V3) were predefined using separate target versus background functional localizer scans. The first analysis using a general linear model showed that in the presence of background clutter, responses in V1 and V2 target regions of interest were significantly stronger to aligned than unaligned contours, whereas when background clutter was absent, no significant difference was observed. The second analysis using interarea correlations showed that with background clutter, there was an increase in V1–V2 coordination within the target regions when perceiving aligned versus unaligned contours; without clutter, however, correlations between V1 and V2 were similar no matter whether aligned contours were present or not. Both the average response magnitude and the connectivity analysis suggest different mechanisms support contour processing with or without background distractors. Coordination between V1 and V2 may play a major role in coherent structure perception, especially with complex scene organization.

Posterior-Anterior Brain Maturation Reflected in Perceptual, Motor and Cognitive Performance

Based on several postmortem morphometric and in vivo imaging studies it has been postulated that brain maturation roughly follows a caudal to rostral direction. In this study, we linked this maturational pattern to psychological function employing a series of well-established behavioral tasks. We addressed three distinct functions and brain regions with a perceptual (contour integration, CI), motor (finger tapping, FT), and executive control (Navon global–local) task. Our purpose was to investigate basic visual integration functions relying on primary visual cortex (V1) in CI; motor coordination function related to primary motor cortex (M1) in FT, and the executive control component, switching, related to the dorsolateral prefrontal region of the brain in the Navon task. 122 volunteer subjects were recruited to participate in this study between the ages of 10 and 20 (females n = 63, males n = 59). Employing conventional statistical methods, we found that 10 and 12 year olds are performing significantly weaker than 20 year olds in all three tasks. In the CI and Navon global–local tasks, even 14 years old perform poorer than adults. We have also investigated the developmental trajectories by fitting sigmoid curves on our data streams. The analysis of the developmental trajectories of the three tasks showed a posterior to anterior pattern in the emergence of the developmental functions with the earliest development in the visual CI task (V1), followed by motor development in the FT task (M1), and cognitive development as measured in the Navon global–local task (DLPC) being the slowest. Gender difference was also present in FT task showing an earlier maturation for girls in the motor domain.

Contour Integration over Time: Psychophysical and fMRI Evidence

The brain integrates discrete but collinear stimuli to perceive global contours. Previous contour integration (CI) studies mainly focus on integration over space, and CI is attributed to either V1 long-range connections or contour processing in high-visual areas that top-down modulate V1 responses. Here, we show that CI also occurs over time in a design that minimizes the roles of V1 long-range interactions. We use tilted contours embedded in random orientation noise and moving horizontally behind a fixed vertical slit. Individual contour elements traveling up/down within the slit would be encoded over time by parallel, rather than aligned, V1 neurons. However, we find robust contour detection even when the slit permits only one viewable contour element. Similar to CI over space, CI over time also obeys the rule of collinearity. fMRI evidence shows that while CI over space engages visual areas as early as V1, CI over time mainly engages higher dorsal and ventral visual areas involved in shape processing, as well as posterior parietal regions involved in visual memory that can represent the orientation of temporally integrated contours. These results suggest at least partially dissociable mechanisms for implementing the Gestalt rule of continuity in CI over space and time.

Evidence of Stereoscopic Surface Disambiguation in the Responses of V1 Neurons

For the important task of binocular depth perception from complex natural-image stimuli, the neurophysiological basis for disambiguating multiple matches between the eyes across similar features has remained a long-standing problem. Recurrent interactions among binocular disparity-tuned neurons in the primary visual cortex (V1) could play a role in stereoscopic computations by altering responses to favor the most likely depth interpretation for a given image pair. Psychophysical research has shown that binocular disparity stimuli displayed in 1 region of the visual field can be extrapolated into neighboring regions that contain ambiguous depth information. We tested whether neurons in macaque V1 interact in a similar manner and found that unambiguous binocular disparity stimuli displayed in the surrounding visual fields of disparity-selective V1 neurons indeed modified their responses when either bistable stereoscopic or uniform featureless stimuli were presented within their receptive field centers. The delayed timing of the response behavior compared with the timing of classical surround suppression and multiple control experiments suggests that these modulations are carried out by slower disparity-specific recurrent connections among V1 neurons. These results provide explicit evidence that the spatial interactions that are predicted by cooperative algorithms play an important role in solving the stereo correspondence problem.

Superior colliculus neurons encode a visual saliency map during free viewing of natural dynamic video

Models of visual attention postulate the existence of a saliency map whose function is to guide attention and gaze to the most conspicuous regions in a visual scene. Although cortical representations of saliency have been reported, there is mounting evidence for a subcortical saliency mechanism, which pre-dates the evolution of neocortex. Here, we conduct a strong test of the saliency hypothesis by comparing the output of a well-established computational saliency model with the activation of neurons in the primate superior colliculus (SC), a midbrain structure associated with attention and gaze, while monkeys watched video of natural scenes. We find that the activity of SC superficial visual-layer neurons (SCs), specifically, is well-predicted by the model. This saliency representation is unlikely to be inherited from fronto-parietal cortices, which do not project to SCs, but may be computed in SCs and relayed to other areas via tectothalamic pathways.

Oh, the number of things you will process (in parallel)!

We highlight the importance of considering the variance produced during the parallel processing stage in vision and present a case for why it is useful to consider the “item” as a meaningful unit of study when investigating early visual processing in visual search tasks.

Contextual Interactions in Grating Plaid Configurations Are Explained by Natural Image Statistics and Neural Modeling

Processing natural scenes requires the visual system to integrate local features into global object descriptions. To achieve coherent representations, the human brain uses statistical dependencies to guide weighting of local feature conjunctions. Pairwise interactions among feature detectors in early visual areas may form the early substrate of these local feature bindings. To investigate local interaction structures in visual cortex, we combined psychophysical experiments with computational modeling and natural scene analysis. We first measured contrast thresholds for 2 × 2 grating patch arrangements (plaids), which differed in spatial frequency composition (low, high, or mixed), number of grating patch co-alignments (0, 1, or 2), and inter-patch distances (1° and 2° of visual angle). Contrast thresholds for the different configurations were compared to the prediction of probability summation (PS) among detector families tuned to the four retinal positions. For 1° distance the thresholds for all configurations were larger than predicted by PS, indicating inhibitory interactions. For 2° distance, thresholds were significantly lower compared to PS when the plaids were homogeneous in spatial frequency and orientation, but not when spatial frequencies were mixed or there was at least one misalignment. Next, we constructed a neural population model with horizontal laminar structure, which reproduced the detection thresholds after adaptation of connection weights. Consistent with prior work, contextual interactions were medium-range inhibition and long-range, orientation-specific excitation. However, inclusion of orientation-specific, inhibitory interactions between populations with different spatial frequency preferences were crucial for explaining detection thresholds. Finally, for all plaid configurations we computed their likelihood of occurrence in natural images. The likelihoods turned out to be inversely related to the detection thresholds obtained at larger inter-patch distances. However, likelihoods were almost independent of inter-patch distance, implying that natural image statistics could not explain the crowding-like results at short distances. This failure of natural image statistics to resolve the patch distance modulation of plaid visibility remains a challenge to the approach.

A neural mechanism of dynamic gating of task-relevant information by top-down influence in primary visual cortex

Visual recognition involves bidirectional information flow, which consists of bottom-up information coding from retina and top-down information coding from higher visual areas. Recent studies have demonstrated the involvement of early visual areas such as primary visual area (V1) in recognition and memory formation. V1 neurons are not passive transformers of sensory inputs but work as adaptive processor, changing their function according to behavioral context. Top-down signals affect tuning property of V1 neurons and contribute to the gating of sensory information relevant to behavior. However, little is known about the neuronal mechanism underlying the gating of task-relevant information in V1. To address this issue, we focus on task-dependent tuning modulations of V1 neurons in two tasks of perceptual learning. We develop a model of the V1, which receives feedforward input from lateral geniculate nucleus and top-down input from a higher visual area. We show here that the change in a balance between excitation and inhibition in V1 connectivity is necessary for gating task-relevant information in V1. The balance change well accounts for the modulations of tuning characteristic and temporal properties of V1 neuronal responses. We also show that the balance change of V1 connectivity is shaped by top-down signals with temporal correlations reflecting the perceptual strategies of the two tasks. We propose a learning mechanism by which synaptic balance is modulated. To conclude, top-down signal changes the synaptic balance between excitation and inhibition in V1 connectivity, enabling early visual area such as V1 to gate context-dependent information under multiple task performances.