Contextual influences on visual processing

Dependence of Contextual Modulation in Macaque V1 on Interlaminar Signal Flow

In visual cortex, neural correlates of subjective perception can be generated by modulation of activity from beyond the classical receptive field (CRF). In macaque V1, activity generated by nonclassical receptive field (nCRF) stimulation involves different intracortical circuitry than activity generated by CRF stimulation, suggesting that interactions between neurons across V1 layers differ under CRF and nCRF stimulus conditions. Using Neuropixels probes, we measured border ownership modulation within large, local populations of V1 neurons. We found that neurons in single columns preferred the same side of objects located outside of the CRF. In addition, we found that cross-correlations between pairs of neurons situated across feedback/horizontal and input layers differed between CRF and nCRF stimulation. Furthermore, independent of the comparison with CRF stimulation, we observed that the magnitude of border ownership modulation increased with the proportion of information flow from feedback/horizontal layers to input layers. These results demonstrate that the flow of signals between layers covaries with the degree to which neurons integrate information from beyond the CRF.

Transformers and cortical waves: encoders for pulling in context across time

The capabilities of transformer networks such as ChatGPT and other large language models (LLMs) have captured the world's attention. The crucial computational mechanism underlying their performance relies on transforming a complete input sequence - for example, all the words in a sentence - into a long 'encoding vector' that allows transformers to learn long-range temporal dependencies in naturalistic sequences. Specifically, 'self-attention' applied to this encoding vector enhances temporal context in transformers by computing associations between pairs of words in the input sequence. We suggest that waves of neural activity traveling across single cortical areas, or multiple regions on the whole-brain scale, could implement a similar encoding principle. By encapsulating recent input history into a single spatial pattern at each moment in time, cortical waves may enable a temporal context to be extracted from sequences of sensory inputs, the same computational principle as that used in transformers.

Bistable perception of symbolic numbers

Numerals, that is, semantic expressions of numbers, enable us to have an exact representation of the amount of things. Visual processing of numerals plays an indispensable role in the recognition and interpretation of numbers. Here, we investigate how visual information from numerals is processed to achieve semantic understanding. We first found that partial occlusion of some digital numerals introduces bistable interpretations. Next, by using the visual adaptation method, we investigated the origin of this bistability in human participants. We showed that adaptation to digital and normal Arabic numerals, as well as homologous shapes, but not Chinese numerals, biases the interpretation of a partially occluded digital numeral. We suggest that this bistable interpretation is driven by intermediate shape processing stages of vision, that is, by features more complex than local visual orientations, but more basic than the abstract concepts of numerals.

Predicting long-term time to cardiovascular incidents using myocardial perfusion imaging and deep convolutional neural networks

Myocardial perfusion imaging (MPI) is a clinical tool which can assess the heart's perfusion status, thereby revealing impairments in patients' cardiac function. Within the MPI modality, the acquired three-dimensional signals are typically represented as a sequence of two-dimensional grayscale tomographic images. Here, we proposed an end-to-end survival training approach for processing gray-scale MPI tomograms to generate a risk score which reflects subsequent time to cardiovascular incidents, including cardiovascular death, non-fatal myocardial infarction, and non-fatal ischemic stroke (collectively known as Major Adverse Cardiovascular Events; MACE) as well as Congestive Heart Failure (CHF). We recruited a total of 1928 patients who had undergone MPI followed by coronary interventions. Among them, 80% (n = 1540) were randomly reserved for the training and 5- fold cross-validation stage, while 20% (n = 388) were set aside for the testing stage. The end-to-end survival training can converge well in generating effective AI models via the fivefold cross-validation approach with 1540 patients. When a candidate model is evaluated using independent images, the model can stratify patients into below-median-risk (n = 194) and above-median-risk (n = 194) groups, the corresponding survival curves of the two groups have significant difference (P < 0.0001). We further stratify the above-median-risk group to the quartile 3 and 4 group (n = 97 each), and the three patient strata, referred to as the high, intermediate and low risk groups respectively, manifest statistically significant difference. Notably, the 5-year cardiovascular incident rate is less than 5% in the low-risk group (accounting for 50% of all patients), while the rate is nearly 40% in the high-risk group (accounting for 25% of all patients). Evaluation of patient subgroups revealed stronger effect size in patients with three blocked arteries (Hazard ratio [HR]: 18.377, 95% CI 3.719-90.801, p < 0.001), followed by those with two blocked vessels at HR 7.484 (95% CI 1.858-30.150; p = 0.005). Regarding stent placement, patients with a single stent displayed a HR of 4.410 (95% CI 1.399-13.904; p = 0.011). Patients with two stents show a HR of 10.699 (95% CI 2.262-50.601; p = 0.003), escalating notably to a HR of 57.446 (95% CI 1.922-1717.207; p = 0.019) for patients with three or more stents, indicating a substantial relationship between the disease severity and the predictive capability of the AI for subsequent cardiovascular inciidents. The success of the MPI AI model in stratifying patients into subgroups with distinct time-to-cardiovascular incidents demonstrated the feasibility of proposed end-to-end survival training approach.

Nonadditive integration of visual information in ensemble processing

Statistically summarizing information from a stimulus array into an ensemble representation (e.g., the mean) improves the efficiency of visual processing. However, little is known about how the brain computes the ensemble statistics. Here, we propose that ensemble processing is realized by nonadditive integration, rather than linear averaging, of individual items. We used a linear regression model approach to extract EEG responses to three levels of information: the individual items, their local interactions, and their global interaction. The local and global interactions, representing nonadditive integration of individual items, elicited rapid and independent neural responses. Critically, only the neural representation of the global interaction predicted the precision of the ensemble perception at the behavioral level. Furthermore, spreading attention over the global pattern to enhance ensemble processing directly promoted rapid neural representation of the global interaction. Taken together, these findings advocate a global, nonadditive mechanism of ensemble processing in the brain.

Influences of local and global context on local orientation perception

Visual context modulates perception of local orientation attributes. These spatially very localised effects are considered to correspond to specific excitatory-inhibitory connectivity patterns of early visual areas as V1, creating perceptual tilt repulsion and attraction effects. Here, orientation misperception of small Gabor stimuli was used as a probe of this computational structure by sampling a large spatio-orientation space to reveal expected asymmetries due to the underlying neuronal processing. Surprisingly, the results showed a regular iso-orientation pattern of nearby location effects whose reference point was globally modulated by the spatial structure, without any complex interactions between local positions and orientation. This pattern of results was confirmed by the two perceptual parameters of bias and discrimination ability. Furthermore, the response times to stimulus configuration displayed variations that further provided evidence of how multiple early visual stages affect perception of simple stimuli.

Waves traveling over a map of visual space can ignite short-term predictions of sensory input

Recent analyses have found waves of neural activity traveling across entire visual cortical areas in awake animals. These traveling waves modulate the excitability of local networks and perceptual sensitivity. The general computational role of these spatiotemporal patterns in the visual system, however, remains unclear. Here, we hypothesize that traveling waves endow the visual system with the capacity to predict complex and naturalistic inputs. We present a network model whose connections can be rapidly and efficiently trained to predict individual natural movies. After training, a few input frames from a movie trigger complex wave patterns that drive accurate predictions many frames into the future solely from the network's connections. When the recurrent connections that drive waves are randomly shuffled, both traveling waves and the ability to predict are eliminated. These results suggest traveling waves may play an essential computational role in the visual system by embedding continuous spatiotemporal structures over spatial maps.

A Texture Statistics Encoding Model Reveals Hierarchical Feature Selectivity across Human Visual Cortex

Midlevel features, such as contour and texture, provide a computational link between low- and high-level visual representations. Although the nature of midlevel representations in the brain is not fully understood, past work has suggested a texture statistics model, called the P-S model (Portilla and Simoncelli, 2000), is a candidate for predicting neural responses in areas V1-V4 as well as human behavioral data. However, it is not currently known how well this model accounts for the responses of higher visual cortex to natural scene images. To examine this, we constructed single-voxel encoding models based on P-S statistics and fit the models to fMRI data from human subjects (both sexes) from the Natural Scenes Dataset (Allen et al., 2022). We demonstrate that the texture statistics encoding model can predict the held-out responses of individual voxels in early retinotopic areas and higher-level category-selective areas. The ability of the model to reliably predict signal in higher visual cortex suggests that the representation of texture statistics features is widespread throughout the brain. Furthermore, using variance partitioning analyses, we identify which features are most uniquely predictive of brain responses and show that the contributions of higher-order texture features increase from early areas to higher areas on the ventral and lateral surfaces. We also demonstrate that patterns of sensitivity to texture statistics can be used to recover broad organizational axes within visual cortex, including dimensions that capture semantic image content. These results provide a key step forward in characterizing how midlevel feature representations emerge hierarchically across the visual system.SIGNIFICANCE STATEMENT Intermediate visual features, like texture, play an important role in cortical computations and may contribute to tasks like object and scene recognition. Here, we used a texture model proposed in past work to construct encoding models that predict the responses of neural populations in human visual cortex (measured with fMRI) to natural scene stimuli. We show that responses of neural populations at multiple levels of the visual system can be predicted by this model, and that the model is able to reveal an increase in the complexity of feature representations from early retinotopic cortex to higher areas of ventral and lateral visual cortex. These results support the idea that texture-like representations may play a broad underlying role in visual processing.

Rethinking symbolic and visual context in Referring Expression Generation

Situational context is crucial for linguistic reference to visible objects, since the same description can refer unambiguously to an object in one context but be ambiguous or misleading in others. This also applies to Referring Expression Generation (REG), where the production of identifying descriptions is always dependent on a given context. Research in REG has long represented visual domains throughsymbolicinformation about objects and their properties, to determine identifying sets of target features during content determination. In recent years, research invisual REGhas turned to neural modeling and recasted the REG task as an inherently multimodal problem, looking at more natural settings such as generating descriptions for objects in photographs. Characterizing the precise ways in which context influences generation is challenging in both paradigms, as context is notoriously lacking precise definitions and categorization. In multimodal settings, however, these problems are further exacerbated by the increased complexity and low-level representation of perceptual inputs. The main goal of this article is to provide a systematic review of the types and functions of visual context across various approaches to REG so far and to argue for integrating and extending different perspectives on visual context that currently co-exist in research on REG. By analyzing the ways in which symbolic REG integrates context in rule-based approaches, we derive a set of categories of contextual integration, including the distinction betweenpositiveandnegative semantic forcesexerted by context during reference generation. Using this as a framework, we show that so far existing work in visual REG has considered only some of the ways in which visual context can facilitate end-to-end reference generation. Connecting with preceding research in related areas, as possible directions for future research, we highlight some additional ways in which contextual integration can be incorporated into REG and other multimodal generation tasks.

Contextual drive of neuronal responses in mouse V1 in the absence of feedforward input

Neurons in the primary visual cortex (V1) respond to stimuli in their receptive field (RF), which is defined by the feedforward input from the retina. However, V1 neurons are also sensitive to contextual information outside their RF, even if the RF itself is unstimulated. Here, we examined the cortical circuits for V1 contextual responses to gray disks superimposed on different backgrounds. Contextual responses began late and were strongest in the feedback-recipient layers of V1. They differed between the three main classes of inhibitory neurons, with particularly strong contextual drive of VIP neurons, indicating a contribution of disinhibitory circuits to contextual drive. Contextual drive was strongest when the gray disk was perceived as figure, occluding its background, rather than a hole. Our results link contextual drive in V1 to perceptual organization and provide previously unknown insight into how recurrent processing shapes the response of sensory neurons to facilitate figure perception.

Temporal resolution and temporal extent of orientation repulsion

A vertical target is perceived as tilted against a slightly tilted inducer surrounding it. To identify the temporal resolution and temporal extent of this phenomenon of orientation repulsion in the same paradigm, we used an alternating pair of inducer stimuli having complementary orientation distributions and quantified repulsion at various alternation frequencies. The duration of each inducer stimulus was inversely proportional to the frequency. When an orthogonal pair of D2 patterns, a type of grating whose luminance modulation in a particular orientation was the second-order partial derivative of an isotropic 2D-Gaussian, was used as the inducer, repulsion occurred when the duration exceeded 20 ms and leveled off at 30 ms and beyond. When a custom-made texture with a narrowband orientation distribution and another texture with a complementary orientation distribution were alternated as the inducer, repulsion gradually increased until the inducer duration reached 200 ms. The gradual increase in repulsion was observed regardless of whether the orientation of the inducer that appeared simultaneously with the target was discernible. These findings reveal that contextual modulation in orientation occurs at a high temporal resolution and continues to a long temporal extent under optimal conditions.

How to make better forensic decisions

Much of forensic practice today involves human decisions about the origins of patterned sensory evidence, such as tool marks and fingerprints discovered at a crime scene. These decisions are made by trained observers who compare the evidential pattern to an exemplar pattern produced by the suspected source of the evidence. The decision consists of a determination as to whether the two patterns are similar enough to have come from the same source. Although forensic pattern comparison disciplines have for decades played a valued role in criminal investigation and prosecution, the extremely high personal and societal costs of failure-the conviction of innocent people-has elicited calls for caution and for the development of better practices. These calls have been heard by the scientific community involved in the study of human information processing, which has begun to offer much-needed perspectives on sensory measurement, discrimination, and classification in a forensic context. Here I draw from a well-established theoretical and empirical approach in sensory science to illustrate the vulnerabilities of contemporary pattern comparison disciplines and to suggest specific strategies for improvement.

Facial Expression Recognition: One Attention-Modulated Contextual Spatial Information Network

Facial expression recognition (FER) in the wild is a challenging task due to some uncontrolled factors such as occlusion, illumination, and pose variation. The current methods perform well in controlled conditions. However, there are still two issues with the in-the-wild FER task: (i) insufficient descriptions of long-range dependency of expression features in the facial information space and (ii) not finely refining subtle inter-classes distinction from multiple expressions in the wild. To overcome the above issues, an end-to-end model for FER, named attention-modulated contextual spatial information network (ACSI-Net), is presented in this paper, with the manner of embedding coordinate attention (CA) modules into a contextual convolutional residual network (CoResNet). Firstly, CoResNet is constituted by arranging contextual convolution (CoConv) blocks of different levels to integrate facial expression features with long-range dependency, which generates a holistic representation of spatial information on facial expression. Then, the CA modules are inserted into different stages of CoResNet, at each of which the subtle information about facial expression acquired from CoConv blocks is first modulated by the corresponding CA module across channels and spatial locations and then flows into the next layer. Finally, to highlight facial regions related to expression, a CA module located at the end of the whole network, which produces attentional masks to multiply by input feature maps, is utilized to focus on salient regions. Different from other models, the ACSI-Net is capable of exploring intrinsic dependencies between features and yielding a discriminative representation for facial expression classification. Extensive experimental results on AffectNet and RAF_DB datasets demonstrate its effectiveness and competitiveness compared to other FER methods.

Spatially distributed computation in cortical circuits

The traditional view of neural computation in the cerebral cortex holds that sensory neurons are specialized, i.e., selective for certain dimensions of sensory stimuli. This view was challenged by evidence of contextual interactions between stimulus dimensions in which a neuron's response to one dimension strongly depends on other dimensions. Here, we use methods of mathematical modeling, psychophysics, and electrophysiology to address shortcomings of the traditional view. Using a model of a generic cortical circuit, we begin with the simple demonstration that cortical responses are always distributed among neurons, forming characteristic waveforms, which we call neural waves. When stimulated by patterned stimuli, circuit responses arise by interference of neural waves. Results of this process depend on interaction between stimulus dimensions. Comparison of modeled responses with responses of biological vision makes it clear that the framework of neural wave interference provides a useful alternative to the standard concept of neural computation.

Surround Modulation Properties of Tectal Neurons in Pigeons Characterized by Moving and Flashed Stimuli

Simple Summary

Surround modulation is a basic visual attribute of sensory neurons in many species and has been extensively characterized in mammal primary visual cortex, lateral geniculate nucleus, and superior colliculus. Little attention has been paid to birds, which have a highly developed visual system. We undertook a systematic analysis on surround modulation properties of tectal neurons in pigeons (Columba livia). This study complements existing studies on surrounding modulation properties in non-mammalian species and deepens the understanding of mechanisms of figure–background segmentation performed by avians.

Abstract

Surround modulation has been abundantly studied in several mammalian brain areas, including the primary visual cortex, lateral geniculate nucleus, and superior colliculus (SC), but systematic analysis is lacking in the avian optic tectum (OT, homologous to mammal SC). Here, multi-units were recorded from pigeon (Columba livia) OT, and responses to different sizes of moving, flashed squares, and bars were compared. The statistical results showed that most tectal neurons presented suppressed responses to larger stimuli in both moving and flashed paradigms, and suppression induced by flashed squares was comparable with moving ones when the stimuli center crossed the near classical receptive field (CRF) center, which corresponded to the full surrounding condition. Correspondingly, the suppression grew weaker when the stimuli center moved across the CRF border, equivalent to partially surrounding conditions. Similarly, suppression induced by full surrounding flashed squares was more intense than by partially surrounding flashed bars. These results suggest that inhibitions performed on tectal neurons appear to be full surrounding rather than locally lateral. This study enriches the understanding of surround modulation properties of avian tectum neurons and provides possible hypotheses about the arrangement of inhibitions from other nuclei, both of which are important for clarifying the mechanism of target detection against clutter background performed by avians.

Global-local consistency benefits memory-guided tracking of a moving target

INTRODUCTION

Previous findings have demonstrated that several Gestalt principles do facilitate VSTM performance in change detection tasks. However, few studies have investigated the role of and time-course of global-local consistency in motion perception.

METHODS

Participants were required to track a moving target surrounded by three different backgrounds: blank, inconsistent, or consistent. Global-local objects were be bound to move together (covariation). During the PMT, participants had to follow the moving target with their eyes and react as fast as possible when the target had just vanished behind the obstruction or would arrive at a predetermined point of interception. Variable error (VE) and constant error (CE) of estimated time-to-contact (TTC) and gain of smooth pursuit eye movements were calculated in various conditions and analyzed qualitatively.

RESULTS

Experiment 1 established the basic finding that VSTM performance could benefit from global-local consistency. Experiment 2 extended this finding by eye-tracking device. Both in visible phase and in occluded phase, CEs were smaller for the target in a consistent background than for the target in an inconsistent background and for the target in a blank background, with both differences significant (ps < .05). However, the difference in VE among three conditions was not significant. At early stage (100-250 ms), later stage (2750-3000 ms), and termination stage (5750-6000 ms) of smooth pursuit, the velocity gains were higher in the trials with consistent backgrounds than in the trials with inconsistent backgrounds and blank backgrounds (ps < .001). With the exception of 100-250 ms phase, the means did not differ between the inconsistent background and the blank background trials (ps > .1).

CONCLUSIONS

Global-local consistency could be activated within the first few hundred milliseconds to prioritize the deployment of attention and eye movement to component target. Meanwhile, it also removes ambiguity from motion tracking and TTC estimation under some unpredictable conditions, leading to the consistency advantage during smooth-pursuit termination phase. Global-local consistency may act as an important information source to TTC estimation and oculomotor response in PMT.

Reunification of Object and View-Center Background Information in the Primate Medial Temporal Lobe

Recent work has shown that the medial temporal lobe (MTL), including the hippocampus (HPC) and its surrounding limbic cortices, plays a role in scene perception in addition to episodic memory. The two basic factors of scene perception are the object (“what”) and location (“where”). In this review, we first summarize the anatomical knowledge related to visual inputs to the MTL and physiological studies examining object-related information processed along the ventral pathway briefly. Thereafter, we discuss the space-related information, the processing of which was unclear, presumably because of its multiple aspects and a lack of appropriate task paradigm in contrast to object-related information. Based on recent electrophysiological studies using non-human primates and the existing literature, we proposed the “reunification theory,” which explains brain mechanisms which construct object-location signals at each gaze. In this reunification theory, the ventral pathway signals a large-scale background image of the retina at each gaze position. This view-center background signal reflects the first person’s perspective and specifies the allocentric location in the environment by similarity matching between images. The spatially invariant object signal and view-center background signal, both of which are derived from the same retinal image, are integrated again (i.e., reunification) along the ventral pathway-MTL stream, particularly in the perirhinal cortex. The conjunctive signal, which represents a particular object at a particular location, may play a role in scene perception in the HPC as a key constituent element of an entire scene.

Parts and Wholes in Scene Processing

During natural vision, our brains are constantly exposed to complex, but regularly structured environments. Real-world scenes are defined by typical part-whole relationships, where the meaning of the whole scene emerges from configurations of localized information present in individual parts of the scene. Such typical part-whole relationships suggest that information from individual scene parts is not processed independently, but that there are mutual influences between the parts and the whole during scene analysis. Here, we review recent research that used a straightforward, but effective approach to study such mutual influences: By dissecting scenes into multiple arbitrary pieces, these studies provide new insights into how the processing of whole scenes is shaped by their constituent parts and, conversely, how the processing of individual parts is determined by their role within the whole scene. We highlight three facets of this research: First, we discuss studies demonstrating that the spatial configuration of multiple scene parts has a profound impact on the neural processing of the whole scene. Second, we review work showing that cortical responses to individual scene parts are shaped by the context in which these parts typically appear within the environment. Third, we discuss studies demonstrating that missing scene parts are interpolated from the surrounding scene context. Bridging these findings, we argue that efficient scene processing relies on an active use of the scene's part-whole structure, where the visual brain matches scene inputs with internal models of what the world should look like.

How Cortical Circuits Implement Cortical Computations: Mouse Visual Cortex as a Model

The mouse, as a model organism to study the brain, gives us unprecedented experimental access to the mammalian cerebral cortex. By determining the cortex's cellular composition, revealing the interaction between its different components, and systematically perturbing these components, we are obtaining mechanistic insight into some of the most basic properties of cortical function. In this review, we describe recent advances in our understanding of how circuits of cortical neurons implement computations, as revealed by the study of mouse primary visual cortex. Further, we discuss how studying the mouse has broadened our understanding of the range of computations performed by visual cortex. Finally, we address how future approaches will fulfill the promise of the mouse in elucidating fundamental operations of cortex.

Neuronal variability reflects probabilistic inference tuned to natural image statistics

Neuronal activity in sensory cortex fluctuates over time and across repetitions of the same input. This variability is often considered detrimental to neural coding. The theory of neural sampling proposes instead that variability encodes the uncertainty of perceptual inferences. In primary visual cortex (V1), modulation of variability by sensory and non-sensory factors supports this view. However, it is unknown whether V1 variability reflects the statistical structure of visual inputs, as would be required for inferences correctly tuned to the statistics of the natural environment. Here we combine analysis of image statistics and recordings in macaque V1 to show that probabilistic inference tuned to natural image statistics explains the widely observed dependence between spike count variance and mean, and the modulation of V1 activity and variability by spatial context in images. Our results show that the properties of a basic aspect of cortical responses-their variability-can be explained by a probabilistic representation tuned to naturalistic inputs.

The attentional template in high and low similarity search: Optimal tuning or tuning to relations?

The attentional template is often described as the mental representation that drives attentional selection and guidance, for instance, in visual search. Recent research suggests that this template is not a veridical representation of the sought-for target, but instead an altered representation that allows more efficient search. The current paper contrasts two such theories. Firstly, the Optimal Tuning account which posits that the attentional template shifts to an exaggerated target value to maximise the signal-to-noise ratio between similar targets and non-targets. Secondly, the Relational account which states that instead of tuning to feature values, attention is directed to the relative value created by the search context, e.g. all redder items or the reddest item. Both theories are empirically supported, but used different paradigms (perceptual decision tasks vs. visual search), and different attentional measures (probe response accuracy vs. gaze capture). The current design incorporates both paradigms and measures. The results reveal that while Optimal Tuning shifts are observed in probe trials they do not drive early attention or eye- movement behaviour in visual search. Instead, early attention follows the Relational Account, selecting all items with the relative target colour (e.g., redder). This suggests that the masked probe trials used in Optimal Tuning do not probe the attentional template that guides attention. In Experiment 3 we find that optimal tuning shifts correspond in magnitude to purely perceptual shifts created by contrast biases in the visual search arrays. This suggests that the shift in probe responses may in fact be a perceptual artefact rather than a strategic adaptation to optimise the signal-to-noise ratio. These results highlight the distinction between early attentional mechanisms and later, target identification mechanisms. SIGNIFICANCE STATEMENT: Classical theories of attention suggest that attention is guided by a feature-specific target template. In recent designs this has been challenged by an apparent non- veridical tuning of the template in situations where the target stimulus is similar to non-targets. The current studies compare two theories that propose different explanations for non-veridical tuning; the Relational and the Optimal Tuning account. We show that the Relational account describes the mechanism that guides early search behaviour, while the Optimal Tuning account describes perceptual decision-making. Optimal Tuning effects may be due to an artefact that has not been described in visual search before (simultaneous contrast).

Effects of muscarinic and nicotinic receptors on contextual modulation in macaque area V1

Context affects the salience and visibility of image elements in visual scenes. Collinear flankers can enhance or decrease the perceptual and neuronal sensitivity to flanked stimuli. These effects are mediated through lateral interactions between neurons in the primary visual cortex (area V1), in conjunction with feedback from higher visual areas. The strength of lateral interactions is affected by cholinergic neuromodulation. Blockade of muscarinic receptors should increase the strength of lateral intracortical interactions, while nicotinic blockade should reduce thalamocortical feed-forward drive. Here we test this proposal through local iontophoretic application of the muscarinic receptor antagonist scopolamine and the nicotinic receptor antagonist mecamylamine, while recording single cells in parafoveal representations in awake fixating macaque V1. Collinear flankers generally reduced neuronal contrast sensitivity. Muscarinic and nicotinic receptor blockade equally reduced neuronal contrast sensitivity. Contrary to our hypothesis, flanker interactions were not systematically affected by either receptor blockade.

Stream-specific feedback inputs to the primate primary visual cortex

The sensory neocortex consists of hierarchically-organized areas reciprocally connected via feedforward and feedback circuits. Feedforward connections shape the receptive field properties of neurons in higher areas within parallel streams specialized in processing specific stimulus attributes. Feedback connections have been implicated in top-down modulations, such as attention, prediction and sensory context. However, their computational role remains unknown, partly because we lack knowledge about rules of feedback connectivity to constrain models of feedback function. For example, it is unknown whether feedback connections maintain stream-specific segregation, or integrate information across parallel streams. Using viral-mediated labeling of feedback connections arising from specific cytochrome-oxidase stripes of macaque visual area V2, here we show that feedback to the primary visual cortex (V1) is organized into parallel streams resembling the reciprocal feedforward pathways. This suggests that functionally-specialized V2 feedback channels modulate V1 responses to specific stimulus attributes, an organizational principle potentially extending to feedback pathways in other sensory systems.

Inferential affective tracking reveals the remarkable speed of context-based emotion perception

Understanding the emotional states of others is important for social functioning. Recent studies show that context plays an essential role in emotion recognition. However, it remains unclear whether emotion inference from visual scene context is as efficient as emotion recognition from faces. Here, we measured the speed of context-based emotion perception, using Inferential Affective Tracking (IAT) with naturalistic and dynamic videos. Using cross-correlation analyses, we found that inferring affect based on visual context alone is just as fast as tracking affect with all available information including face and body. We further demonstrated that this approach has high precision and sensitivity to sub-second lags. Our results suggest that emotion recognition from dynamic contextual information might be automatic and immediate. Seemingly complex context-based emotion perception is far more efficient than previously assumed.

Adaptive Surround Modulation of MT Neurons: A Computational Model

The classical receptive field (CRF) of a spiking visual neuron is defined as the region in the visual field that can generate spikes when stimulated by a visual stimulus. Many visual neurons also have an extra-classical receptive field (ECRF) that surrounds the CRF. The presence of a stimulus in the ECRF does not generate spikes but rather modulates the response to a stimulus in the neuron's CRF. Neurons in the primate Middle Temporal (MT) area, which is a motion specialist region, can have directionally antagonistic or facilitatory surrounds. The surround's effect switches between directionally antagonistic or facilitatory based on the characteristics of the stimulus, with antagonistic effects when there are directional discontinuities but facilitatory effects when there is directional coherence. Here, we present a computational model of neurons in area MT that replicates this observation and uses computational building blocks that correlate with observed cell types in the visual pathways to explain the mechanism of this modulatory effect. The model shows that the categorization of MT neurons based on the effect of their surround depends on the input stimulus rather than being a property of the neurons. Also, in agreement with neurophysiological findings, the ECRFs of the modeled MT neurons alter their center-surround interactions depending on image contrast.

Adaptation to feedback representation of illusory orientation produced from flash grab effect

Adaptation is a ubiquitous property of sensory systems. It is typically considered that neurons adapt to dominant energy in the ambient environment to function optimally. However, perceptual representation of the stimulus, often modulated by feedback signals, sometimes do not correspond to the input state of the stimulus, which tends to be more linked with feedforward signals. Here we investigated the relative contributions to cortical adaptation from feedforward and feedback signals, taking advantage of a visual illusion, the Flash-Grab Effect, to disassociate the feedforward and feedback representation of an adaptor. Results reveal that orientation adaptation is exclusively dependent on the perceived rather than the retinal orientation of the adaptor. Combined fMRI and EEG measurements demonstrate that the perceived orientation of the Flash-Grab Effect is indeed supported by feedback signals in the cortex. These findings highlight the important contribution of feedback signals for cortical neurons to recalibrate their sensitivity.

Feedforward-feedback signal interactions are common in the brain during sensory information processing. Here, the authors show that feedback-driven representation of perceived orientation dominates visual adaptation, despite the discrepant feedforward representation of input orientation.

Contextual Integration in Cortical and Convolutional Neural Networks

It has been suggested that neurons can represent sensory input using probability distributions and neural circuits can perform probabilistic inference. Lateral connections between neurons have been shown to have non-random connectivity and modulate responses to stimuli within the classical receptive field. Large-scale efforts mapping local cortical connectivity describe cell type specific connections from inhibitory neurons and like-to-like connectivity between excitatory neurons. To relate the observed connectivity to computations, we propose a neuronal network model that approximates Bayesian inference of the probability of different features being present at different image locations. We show that the lateral connections between excitatory neurons in a circuit implementing contextual integration in this should depend on correlations between unit activities, minus a global inhibitory drive. The model naturally suggests the need for two types of inhibitory gates (normalization, surround inhibition). First, using natural scene statistics and classical receptive fields corresponding to simple cells parameterized with data from mouse primary visual cortex, we show that the predicted connectivity qualitatively matches with that measured in mouse cortex: neurons with similar orientation tuning have stronger connectivity, and both excitatory and inhibitory connectivity have a modest spatial extent, comparable to that observed in mouse visual cortex. We incorporate lateral connections learned using this model into convolutional neural networks. Features are defined by supervised learning on the task, and the lateral connections provide an unsupervised learning of feature context in multiple layers. Since the lateral connections provide contextual information when the feedforward input is locally corrupted, we show that incorporating such lateral connections into convolutional neural networks makes them more robust to noise and leads to better performance on noisy versions of the MNIST dataset. Decomposing the predicted lateral connectivity matrices into low-rank and sparse components introduces additional cell types into these networks. We explore effects of cell-type specific perturbations on network computation. Our framework can potentially be applied to networks trained on other tasks, with the learned lateral connections aiding computations implemented by feedforward connections when the input is unreliable and demonstrate the potential usefulness of combining supervised and unsupervised learning techniques in real-world vision tasks.

Improved fidelity of orientation perception: a learning effect dissociable from enhanced discriminability

Visual perception can be influenced by stimulus context, selective attention, and prior experience. Many previous studies have shown complex interactions among these influencing factors, but it remains unclear whether context-induced illusions could be reduced by perceptual training and whether such a change in perceptual fidelity is linked to improved perceptual discriminability. To address this question, we introduced a context-induced tilt illusion into an orientation discrimination training paradigm. This resulted in parallel and long-term improvements in the discriminability and fidelity of orientation perception. The improved discriminability was specific to the task-relevant target stimulus but nonspecific to the task-irrelevant context. By contrast, the improved perceptual fidelity was specific to the task-irrelevant contextual stimulus that induced the illusion, but not specific to the task-relevant target stimulus or task performed on one of its features. These results indicate two dissociable learning effects associated with the same training procedure. Such a dissociation was further supported by the observation that the sizes of the two learning effects were uncorrelated across the subjects. Our findings suggest two parallel learning processes: a task-dependent process giving rise to enhanced discriminability for the task-relevant stimulus attribute, and a context-dependent process leading to improved perceptual fidelity for the attended stimuli.

Image and Sentence Matching via Semantic Concepts and Order Learning

Image and sentence matching has made great progress recently, but it remains challenging due to the existing large visual-semantic discrepancy. This mainly arises from two aspects: 1) images consist of unstructured content which is not semantically abstract as the words in the sentences, so they are not directly comparable, and 2) arranging semantic concepts in different semantic order could lead to quite diverse meanings. The words in the sentences are sequentially arranged in a grammatical manner, while the semantic concepts in the images are usually unorganized. In this work, we propose a semantic concepts and order learning framework for image and sentence matching, which can improve the image representation by first predicting semantic concepts and then organizing them in a correct semantic order. Given an image, we first use a multi-regional multi-label CNN to predict its included semantic concepts in terms of object, property and action. These word-level semantic concepts are directly comparable with the words of noun, adjective and verb in the matched sentence. Then, to organize these concepts and make them express similar meanings as the matched sentence, we use a context-modulated attentional LSTM to learn the semantic order. It regards the predicted semantic concepts and image global scene as context at each timestep, and selectively attends to concept-related image regions by referring to the context in a sequential order. To further enhance the semantic order, we perform additional sentence generation on the image representation, by using the groundtruth order in the matched sentence as supervision. After obtaining the improved image representation, we learn the sentence representation with a conventional LSTM, and then jointly perform image and sentence matching and sentence generation for model learning. Extensive experiments demonstrate the effectiveness of our learned semantic concepts and order, by achieving the state-of-the-art results on two public benchmark datasets.

Combinations of low-level and high-level neural processes account for distinct patterns of context-dependent choice

Context effects have been explained by either low-level neural adjustments or high-level cognitive processes but not their combination. It is currently unclear how these processes interact to shape individuals’ responses to context. Here, we used a large cohort of human subjects in experiments involving choice between two or three gambles in order to study the dependence of context effects on neural adaptation and individuals’ risk attitudes. Our experiments did not provide any evidence that neural adaptation on long timescales (~100 trials) contributes to context effects. Using post-hoc analyses we identified two groups of subjects with distinct patterns of responses to decoys, both of which depended on individuals’ risk aversion. Subjects in the first group exhibited strong, consistent decoy effects and became more risk averse due to decoy presentation. In contrast, subjects in the second group did not show consistent decoy effects and became more risk seeking. The degree of change in risk aversion due to decoy presentation was positively correlated with the original degrees of risk aversion. To explain these results and reveal underlying neural mechanisms, we developed new models incorporating both low- and high-level processes and used these models to fit individuals’ choice behavior. We found that observed distinct patterns of decoy effects can be explained by a combination of adjustments in neural representations and competitive weighting of reward attributes, both of which depend on risk aversion but in opposite directions. Altogether, our results demonstrate how a combination of low- and high-level processes shapes choice behavior in more naturalistic settings, modulates overall risk preference, and explains distinct behavioral phenotypes.

A large body of experimental work has illustrated that the introduction of a new, and often irrelevant, option can influence preference among the existing options, a phenomenon referred to as context or decoy effects. For example, introducing a new option that is worse than one of the two existing options in all its attributes but better than the alternative option in some attributes (and thus should not ever be selected) can increase the preference for the former option. Context effects have been explained by high-level cognitive processes—such as comparisons and competitions between attributes—or low-level adjustments of neural representations. However, it is unclear how these processes interact to shape individuals’ responses to context. Here, we show that both high-level cognitive processes and low-level neural adjustments shift risk preference during choice between multiple risky options but in opposite directions. Moreover, we demonstrate that combinations of these processes can account for distinct patterns of context effects in human subjects.

Neural mechanisms of contextual modulation in the retinal direction selective circuit

Contextual modulation of neuronal responses by surrounding environments is a fundamental attribute of sensory processing. In the mammalian retina, responses of On–Off direction selective ganglion cells (DSGCs) are modulated by motion contexts. However, the underlying mechanisms are unknown. Here, we show that posterior-preferring DSGCs (pDSGCs) are sensitive to discontinuities of moving contours owing to contextually modulated cholinergic excitation from starburst amacrine cells (SACs). Using a combination of synapse-specific genetic manipulations, patch clamp electrophysiology and connectomic analysis, we identified distinct circuit motifs upstream of On and Off SACs that are required for the contextual modulation of pDSGC activity for bright and dark contrasts. Furthermore, our results reveal a class of wide-field amacrine cells (WACs) with straight, unbranching dendrites that function as “continuity detectors” of moving contours. Therefore, divergent circuit motifs in the On and Off pathways extend the information encoding of On-Off DSGCs beyond their direction selectivity during complex stimuli.

The mechanisms of contextual modulation in direction selective ganglion cells in the retina remain unclear. Here, the authors find that that On-Off direction-selective ganglion cells are differentially sensitive to discontinuities of dark and bright moving edges in the visual environment and, using synapse-specific genetic manipulations with functional measurements, reveal the microcircuits underlying this contextual sensitivity.

Mechanisms of Spatiotemporal Selectivity in Cortical Area MT

Cortical sensory neurons are characterized by selectivity to stimulation. This selectivity was originally viewed as a part of the fundamental "receptive field" characteristic of neurons. This view was later challenged by evidence that receptive fields are modulated by stimuli outside of the classical receptive field. Here, we show that even this modified view of selectivity needs revision. We measured spatial frequency selectivity of neurons in cortical area MT of alert monkeys and found that their selectivity strongly depends on luminance contrast, shifting to higher spatial frequencies as contrast increases. The changes of preferred spatial frequency are large at low temporal frequency, and they decrease monotonically as temporal frequency increases. That is, even interactions among basic stimulus dimensions of luminance contrast, spatial frequency, and temporal frequency strongly influence neuronal selectivity. This dynamic nature of neuronal selectivity is inconsistent with the notion of stimulus preference as a stable characteristic of cortical neurons.

Co-circularity opponency in visual texture

It is well known that the human visual system is sensitive to co-circularity among oriented edges, which are ubiquitous features of object contours. Here, we report a novel aftereffect in which the appearance of a texture is dramatically altered after adaptation to a texture composed of elements with co-circular structure. Following prolonged viewing of a texture made of pairs of adjacent Gabor elements arranged to form obtuse angle co-circular pairs, i.e. shallow curves, a subsequently viewed random texture appears to be composed of acute angle, i.e. near-parallel pairs. Conversely, adaptation to a texture made of parallel pairs causes a random texture to appear to be composed of shallow curves. This suggests that mechanisms sensitive to co-circularity are organized in an opponent manner, with one pole sensitive to shallow curves the other parallel shapes. This notion was tested further in a non-adaptation experiment in which co-circular and non-co-circular Gabor pairs were mixed within a single texture. Results revealed summation between pairs that fell on one side of the opponent continuum, and cancellation between pairs that fell on opposite sides of the continuum. Taken together these results support opponent interactions between mechanisms sensitive to pairwise co-circular texture features.

Rolling Motion Along an Incline: Visual Sensitivity to the Relation Between Acceleration and Slope

People easily intercept a ball rolling down an incline, despite its acceleration varies with the slope in a complex manner. Apparently, however, they are poor at detecting anomalies when asked to judge artificial animations of descending motion. Since the perceptual deficiencies have been reported in studies involving a limited visual context, here we tested the hypothesis that judgments of naturalness of rolling motion are consistent with physics when the visual scene incorporates sufficient cues about environmental reference and metric scale, roughly comparable to those present when intercepting a ball. Participants viewed a sphere rolling down an incline located in the median sagittal plane, presented in 3D wide-field virtual reality. In different experiments, either the slope of the plane or the sphere acceleration were changed in arbitrary combinations, resulting in a kinematics that was either consistent or inconsistent with physics. In Experiment 1 (slope adjustment), participants were asked to modify the slope angle until the resulting motion looked natural for a given ball acceleration. In Experiment 2 (acceleration adjustment), instead, they were asked to modify the acceleration until the motion on a given slope looked natural. No feedback about performance was provided. For both experiments, we found that participants were rather accurate at finding the match between slope angle and ball acceleration congruent with physics, but there was a systematic effect of the initial conditions: accuracy was higher when the participants started the exploration from the combination of slope and acceleration corresponding to the congruent conditions than when they started far away from the congruent conditions. In Experiment 3, participants modified the slope angle based on an adaptive staircase, but the target never coincided with the starting condition. Here we found a generally accurate performance, irrespective of the target slope. We suggest that, provided the visual scene includes sufficient cues about environmental reference and metric scale, joint processing of slope and acceleration may facilitate the detection of natural motion. Perception of rolling motion may rely on the kind of approximate, probabilistic simulations of Newtonian mechanics that have previously been called into play to explain complex inferences in rich visual scenes.

Contextual signals in visual cortex

Vision is an active process. What we perceive strongly depends on our actions, intentions and expectations. During visual processing, these internal signals therefore need to be integrated with the visual information from the retina. The mechanisms of how this is achieved by the visual system are still poorly understood. Advances in recording and manipulating neuronal activity in specific cell types and axonal projections together with tools for circuit tracing are beginning to shed light on the neuronal circuit mechanisms of how internal, contextual signals shape sensory representations. Here we review recent work, primarily in mice, that has advanced our understanding of these processes, focusing on contextual signals related to locomotion, behavioural relevance and predictions.

Near- and Far-Surround Suppression in Human Motion Discrimination

The spatial context has strong effects on visual processing. Psychophysics and modeling studies have provided evidence that the surround context can systematically modulate the perception of center stimuli. For motion direction, these center-surround interactions are considered to come from spatio-directional interactions between direction of motion tuned neurons, which are attributed to the middle temporal (MT) area. Here, we investigated through psychophysics experiments on human subjects changes with spatial separation in center-surround inhibition and motion direction interactions. Center-surround motion repulsion effects were measured under near-and far-surround conditions. Using a simple physiological model of the repulsion effect we extracted theoretical population parameters of surround inhibition strength and tuning widths with spatial distance. All 11 subjects showed clear motion repulsion effects under the near-surround condition, while only 10 subjects showed clear motion repulsion effects under the far-surround condition. The model predicted human performance well. Surround inhibition under the near-surround condition was significantly stronger than that under the far-surround condition, and the tuning widths were smaller under the near-surround condition. These results demonstrate that spatial separation can both modulate the surround inhibition strength and surround to center tuning width.

Cortical travelling waves: mechanisms and computational principles

Multichannel recording technologies have revealed travelling waves of neural activity in multiple sensory, motor and cognitive systems. These waves can be spontaneously generated by recurrent circuits or evoked by external stimuli. They travel along brain networks at multiple scales, transiently modulating spiking and excitability as they pass. Here, we review recent experimental findings that have found evidence for travelling waves at single-area (mesoscopic) and whole-brain (macroscopic) scales. We place these findings in the context of the current theoretical understanding of wave generation and propagation in recurrent networks. During the large low-frequency rhythms of sleep or the relatively desynchronized state of the awake cortex, travelling waves may serve a variety of functions, from long-term memory consolidation to processing of dynamic visual stimuli. We explore new avenues for experimental and computational understanding of the role of spatiotemporal activity patterns in the cortex.

Developing a Nonstationary Computational Framework With Application to Modeling Dynamic Modulations in Neural Spiking Responses

OBJECTIVE

This paper aims to develop a computational model that incorporates the functional effects of modulatory covariates (such as context, task, or behavior), which dynamically alter the relationship between the stimulus and the neural response.

METHODS

We develop a general computational approach along with an efficient estimation procedure in the widely used generalized linear model (GLM) framework to characterize such nonstationary dynamics in spiking response and spatiotemporal characteristics of a neuron at the level of individual trials. The model employs a set of modulatory components, which nonlinearly interact with other stimulus-related signals to reproduce such nonstationary effects.

RESULTS

The model is tested for its ability to predict the responses of neurons in the middle temporal cortex of macaque monkeys during an eye movement task. The fitted model proves successful in capturing the fast temporal modulations in the response, reproducing the spike response temporal statistics, and accurately accounting for the neurons' dynamic spatiotemporal sensitivities, during eye movements.

CONCLUSION

The nonstationary GLM framework developed in this study can be used in cases where a time-varying behavioral or cognitive component makes GLM-based models insufficient to describe the dependencies of neural responses on the stimulus-related covariates.

SIGNIFICANCE

In addition to being quite powerful in encoding time-varying response modulations, this general framework also enables a readout of the neural code while dissociating the influence of other nonstimulus covariates. This framework will advance our ability to understand sensory processing in higher brain areas when modulated by several behavioral or cognitive variables.

Circuit Mechanisms Governing Local vs. Global Motion Processing in Mouse Visual Cortex

A withstanding question in neuroscience is how neural circuits encode representations and perceptions of the external world. A particularly well-defined visual computation is the representation of global object motion by pattern direction-selective (PDS) cells from convergence of motion of local components represented by component direction-selective (CDS) cells. However, how PDS and CDS cells develop their distinct response properties is still unresolved. The visual cortex of the mouse is an attractive model for experimentally solving this issue due to the large molecular and genetic toolbox available. Although mouse visual cortex lacks the highly ordered orientation columns of primates, it is organized in functional sub-networks and contains striate- and extrastriate areas like its primate counterparts. In this Perspective article, we provide an overview of the experimental and theoretical literature on global motion processing based on works in primates and mice. Lastly, we propose what types of experiments could illuminate what circuit mechanisms are governing cortical global visual motion processing. We propose that PDS cells in mouse visual cortex appear as the perfect arena for delineating and solving how individual sensory features extracted by neural circuits in peripheral brain areas are integrated to build our rich cohesive sensory experiences.

Attention to distinguishing features in object recognition: An interactive-iterative framework

This article advances a framework that casts object recognition as a process of discrimination between alternative object identities, in which top-down and bottom-up processes interact-iteratively when necessary-with attention to distinguishing features playing a critical role. In two experiments, observers discriminated between different types of artificial fish. In parallel, a secondary, variable-SOA visual-probe detection task was used to examine the dynamics of visual attention. In Experiment 1, the fish varied in three distinguishing features: one indicating the general category (saltwater, freshwater), and one of the two other features indicating the specific type of fish within each category. As predicted, in the course of recognizing each fish, attention was allocated iteratively to the distinguishing features in an optimal manner: first to the general category feature, and then, based on its value, to the second feature that identified the specific fish. In Experiment 2, two types of fish could be discriminated on the basis of either of two distinguishing features, one more visually discriminable than the other. On some of the trials, one of the two alternative distinguishing features was occluded. As predicted, in the course of recognizing each fish, attention was directed initially to the more discriminable distinguishing feature, but when this feature was occluded, it was then redirected to the less discriminable feature. The implications of these findings, and the interactive-iterative framework they support, are discussed with regard to several fundamental issues having a long history in the literatures on object recognition, object categorization, and visual perception in general.

New perspectives on the neurophysiology of primate amygdala emerging from the study of naturalistic social behaviors

A major challenge of primate neurophysiology, particularly in the domain of social neuroscience, is to adopt more natural behaviors without compromising the ability to relate patterns of neural activity to specific actions or sensory inputs. Traditional approaches have identified neural activity patterns in the amygdala in response to simplified versions of social stimuli such as static images of faces. As a departure from this reduced approach, single images of faces were replaced with arrays of images or videos of conspecifics. These stimuli elicited more natural behaviors and new types of neural responses: (1) attention-gated responses to faces, (2) selective responses to eye contact, and (3) selective responses to touch and somatosensory feedback during the production of facial expressions. An additional advance toward more natural social behaviors in the laboratory was the implementation of dyadic social interactions. Under these conditions, neurons encoded similarly rewards that monkeys delivered to self and to their social partner. These findings reinforce the value of bringing natural, ethologically valid, behavioral tasks under neurophysiological scrutiny. WIREs Cogn Sci 2018, 9:e1449. doi: 10.1002/wcs.1449 This article is categorized under: Psychology > Emotion and Motivation Neuroscience > Cognition Neuroscience > Physiology.

Catching the voltage gradient-asymmetric boost of cortical spread generates motion signals across visual cortex: a brief review with special thanks to Amiram Grinvald

Wide-field voltage imaging is unique in its capability to capture snapshots of activity-across the full gradient of average changes in membrane potentials from subthreshold to suprathreshold levels-of hundreds of thousands of superficial cortical neurons that are simultaneously active. Here, I highlight two examples where voltage-sensitive dye imaging (VSDI) was exploited to track gradual space-time changes of activity within milliseconds across several millimeters of cortex at submillimeter resolution: the line-motion condition, measured in Amiram Grinvald's Laboratory more than 10 years ago and-coming full circle running VSDI in my laboratory-another motion-inducing condition, in which two neighboring stimuli counterchange luminance simultaneously. In both examples, cortical spread is asymmetrically boosted, creating suprathreshold activity drawn out over primary visual cortex. These rapidly propagating waves may integrate brain signals that encode motion independent of direction-selective circuits.