Priority coding in the visual system

Coordinated changes in a cortical circuit sculpt effects of novelty on neural dynamics

Recent studies have found dramatic cell-type-specific responses to stimulus novelty, highlighting the importance of analyzing the cortical circuitry at this granularity to understand brain function. Although initial work characterized activity by cell type, the alterations in cortical circuitry due to interacting novelty effects remain unclear. We investigated circuit mechanisms underlying the observed neural dynamics in response to novel stimuli using a large-scale public dataset of electrophysiological recordings in behaving mice and a population network model. The model was constrained by multi-patch synaptic physiology and electron microscopy data. We found generally weaker connections under novel stimuli, with shifts in the balance between somatostatin (SST) and vasoactive intestinal polypeptide (VIP) populations and increased excitatory influences on parvalbumin (PV) and SST populations. These findings systematically characterize how cortical circuits adapt to stimulus novelty.

Predictive filtering of sensory response via orbitofrontal top-down input

Habituation is a crucial sensory filtering mechanism whose dysregulation can lead to a continuously intense world in disorders with sensory overload. While habituation is considered to require top-down predictive signaling to suppress irrelevant inputs, the exact brain loci storing the internal predictive model and the circuit mechanisms of sensory filtering remain unclear. We found that daily neural habituation in the primary auditory cortex (A1) was reversed by inactivation of the orbitofrontal cortex (OFC). Top-down projections from the ventrolateral OFC, but not other frontal areas, carried predictive signals that grew with daily sound experience and suppressed A1 via somatostatin-expressing inhibitory neurons. Thus, prediction signals from the OFC cancel out behaviorally irrelevant anticipated stimuli by generating their "negative images" in sensory cortices.

Manipulating attentional priority creates a trade-off between memory and sensory representations in human visual cortex

People often remember visual information over brief delays while actively engaging with ongoing inputs from the surrounding visual environment. Depending on the situation, one might prioritize mnemonic contents (i.e., remembering details of a past event), or preferentially attend sensory inputs (i.e., minding traffic while crossing a street). Previous fMRI work has shown that early sensory regions can simultaneously represent both mnemonic and passively viewed sensory information. Here we test the limits of such simultaneity by manipulating attention towards sensory distractors during a working memory task performed by human subjects during fMRI scanning. Participants remembered the orientation of a target grating while a distractor grating was shown during the middle portion of the memory delay. Critically, there were several subtle changes in the contrast and the orientation of the distractor, and participants were cued to either ignore the distractor, detect a change in contrast, or detect a change in orientation. Despite sensory stimulation being matched in all three conditions, the fidelity of memory representations in early visual cortex was highest when the distractor was ignored, intermediate when participants attended distractor contrast, and lowest when participants attended the orientation of the distractor during the delay. In contrast, the fidelity of distractor representations was lowest when ignoring the distractor, intermediate when attending distractor-contrast, and highest when attending distractor-orientation. These data suggest a trade-off in early sensory representations when engaging top-down feedback to attend both seen and remembered features and may partially explain memory failures that occur when subjects are distracted by external events.

Cooperative thalamocortical circuit mechanism for sensory prediction errors

The brain functions as a prediction machine, utilizing an internal model of the world to anticipate sensations and the outcomes of our actions. Discrepancies between expected and actual events, referred to as prediction errors, are leveraged to update the internal model and guide our attention towards unexpected events1-10. Despite the importance of prediction-error signals for various neural computations across the brain, surprisingly little is known about the neural circuit mechanisms responsible for their implementation. Here we describe a thalamocortical disinhibitory circuit that is required for generating sensory prediction-error signals in mouse primary visual cortex (V1). We show that violating animals' predictions by an unexpected visual stimulus preferentially boosts responses of the layer 2/3 V1 neurons that are most selective for that stimulus. Prediction errors specifically amplify the unexpected visual input, rather than representing non-specific surprise or difference signals about how the visual input deviates from the animal's predictions. This selective amplification is implemented by a cooperative mechanism requiring thalamic input from the pulvinar and cortical vasoactive-intestinal-peptide-expressing (VIP) inhibitory interneurons. In response to prediction errors, VIP neurons inhibit a specific subpopulation of somatostatin-expressing inhibitory interneurons that gate excitatory pulvinar input to V1, resulting in specific pulvinar-driven response amplification of the most stimulus-selective neurons in V1. Therefore, the brain prioritizes unpredicted sensory information by selectively increasing the salience of unpredicted sensory features through the synergistic interaction of thalamic input and neocortical disinhibitory circuits.

Individual differences in visual search: A systematic review of the link between visual search performance and traits or abilities

Visual search (VS) comprises a class of tasks that we typically perform several times during a day and requires intentionally scanning (with or without moving the eyes) the environment for a specific target (be it an object or a feature) among distractor stimuli. Experimental research in lab-based or real-world settings has offered insight into its underlying neurocognitive mechanisms from a nomothetic point of view. A lesser-known but rapidly growing body of quasi-experimental and correlational research has explored the link between individual differences and VS performance. This combines different research traditions and covers a wide range of individual differences in studies deploying a vast array of VS tasks. As such, it is a challenge to determine whether any associations highlighted in single studies are robust when considering the wider literature. However, clarifying such relationships systematically and comprehensively would help build more accurate models of VS, and it would highlight promising directions for future research. This systematic review provides an up to date and comprehensive synthesis of the existing literature investigating associations between common indices of performance in VS tasks and measures of individual differences mapped onto four categories of cognitive abilities (short-term working memory, fluid reasoning, visual processing and processing speed) and seven categories of traits (Big Five traits, trait anxiety and autistic traits). Consistent associations for both traits (in particular, conscientiousness, autistic traits and trait anxiety - the latter limited to emotional stimuli) and cognitive abilities (particularly visual processing) were identified. Overall, however, informativeness of future studies would benefit from checking and reporting the reliability of all measurement tools, applying multiplicity correction, using complementary techniques, study preregistration and testing why, rather than only if, a robust relation between certain individual differences and VS performance exists.

Predictions enable top-down pattern separation in the macaque face-processing hierarchy

Distinguishing faces requires well distinguishable neural activity patterns. Contextual information may separate neural representations, leading to enhanced identity recognition. Here, we use functional magnetic resonance imaging to investigate how predictions derived from contextual information affect the separability of neural activity patterns in the macaque face-processing system, a 3-level processing hierarchy in ventral visual cortex. We find that in the presence of predictions, early stages of this hierarchy exhibit well separable and high-dimensional neural geometries resembling those at the top of the hierarchy. This is accompanied by a systematic shift of tuning properties from higher to lower areas, endowing lower areas with higher-order, invariant representations instead of their feedforward tuning properties. Thus, top-down signals dynamically transform neural representations of faces into separable and high-dimensional neural geometries. Our results provide evidence how predictive context transforms flexible representational spaces to optimally use the computational resources provided by cortical processing hierarchies for better and faster distinction of facial identities.

Mixed selectivity: Cellular computations for complexity

The property of mixed selectivity has been discussed at a computational level and offers a strategy to maximize computational power by adding versatility to the functional role of each neuron. Here, we offer a biologically grounded implementational-level mechanistic explanation for mixed selectivity in neural circuits. We define pure, linear, and nonlinear mixed selectivity and discuss how these response properties can be obtained in simple neural circuits. Neurons that respond to multiple, statistically independent variables display mixed selectivity. If their activity can be expressed as a weighted sum, then they exhibit linear mixed selectivity; otherwise, they exhibit nonlinear mixed selectivity. Neural representations based on diverse nonlinear mixed selectivity are high dimensional; hence, they confer enormous flexibility to a simple downstream readout neural circuit. However, a simple neural circuit cannot possibly encode all possible mixtures of variables simultaneously, as this would require a combinatorially large number of mixed selectivity neurons. Gating mechanisms like oscillations and neuromodulation can solve this problem by dynamically selecting which variables are mixed and transmitted to the readout.

Memorability shapes perceived time (and vice versa)

Visual stimuli are known to vary in their perceived duration. Some visual stimuli are also known to linger for longer in memory. Yet, whether these two features of visual processing are linked is unknown. Despite early assumptions that time is an extracted or higher-order feature of perception, more recent work over the past two decades has demonstrated that timing may be instantiated within sensory modality circuits. A primary location for many of these studies is the visual system, where duration-sensitive responses have been demonstrated. Furthermore, visual stimulus features have been observed to shift perceived duration. These findings suggest that visual circuits mediate or construct perceived time. Here we present evidence across a series of experiments that perceived time is affected by the image properties of scene size, clutter and memorability. More specifically, we observe that scene size and memorability dilate time, whereas clutter contracts it. Furthermore, the durations of more memorable images are also perceived more precisely. Conversely, the longer the perceived duration of an image, the more memorable it is. To explain these findings, we applied a recurrent convolutional neural network model of the ventral visual system, in which images are progressively processed over time. We find that more memorable images are processed faster, and that this increase in processing speed predicts both the lengthening and the increased precision of perceived durations. These findings provide evidence for a link between image features, time perception and memory that can be further explored with models of visual processing.

Simple synaptic modulations implement diverse novelty computations

Detecting novelty is ethologically useful for an organism's survival. Recent experiments characterize how different types of novelty over timescales from seconds to weeks are reflected in the activity of excitatory and inhibitory neuron types. Here, we introduce a learning mechanism, familiarity-modulated synapses (FMSs), consisting of multiplicative modulations dependent on presynaptic or pre/postsynaptic neuron activity. With FMSs, network responses that encode novelty emerge under unsupervised continual learning and minimal connectivity constraints. Implementing FMSs within an experimentally constrained model of a visual cortical circuit, we demonstrate the generalizability of FMSs by simultaneously fitting absolute, contextual, and omission novelty effects. Our model also reproduces functional diversity within cell subpopulations, leading to experimentally testable predictions about connectivity and synaptic dynamics that can produce both population-level novelty responses and heterogeneous individual neuron signals. Altogether, our findings demonstrate how simple plasticity mechanisms within a cortical circuit structure can produce qualitatively distinct and complex novelty responses.

Orthogonal neural encoding of targets and distractors supports multivariate cognitive control

The complex challenges of our mental life require us to coordinate multiple forms of neural information processing. Recent behavioural studies have found that people can coordinate multiple forms of attention, but the underlying neural control process remains obscure. We hypothesized that the brain implements multivariate control by independently monitoring feature-specific difficulty and independently prioritizing feature-specific processing. During functional MRI, participants performed a parametric conflict task that separately tags target and distractor processing. Consistent with feature-specific monitoring, univariate analyses revealed spatially segregated encoding of target and distractor difficulty in the dorsal anterior cingulate cortex. Consistent with feature-specific attentional priority, our encoding geometry analysis revealed overlapping but orthogonal representations of target and distractor coherence in the intraparietal sulcus. Coherence representations were mediated by control demands and aligned with both performance and frontoparietal activity, consistent with top-down attention. Together, these findings provide evidence for the neural geometry necessary to coordinate multivariate cognitive control.

Representational structures as a unifying framework for attention

Our visual system consciously processes only a subset of the incoming information. Selective attention allows us to prioritize relevant inputs, and can be allocated to features, locations, and objects. Recent advances in feature-based attention suggest that several selection principles are shared across these domains and that many differences between the effects of attention on perceptual processing can be explained by differences in the underlying representational structures. Moving forward, it can thus be useful to assess how attention changes the structure of the representational spaces over which it operates, which include the spatial organization, feature maps, and object-based coding in visual cortex. This will ultimately add to our understanding of how attention changes the flow of visual information processing more broadly.

Brain-wide arousal signals are segregated from movement planning in the superior colliculus

The superior colliculus (SC) is traditionally considered a brain region that functions as an interface between processing visual inputs and generating eye movement outputs. Although its role as a primary reflex center is thought to be conserved across vertebrate species, evidence suggests that the SC has evolved to support higher-order cognitive functions including spatial attention. When it comes to oculomotor areas such as the SC, it is critical that high precision fixation and eye movements are maintained even in the presence of signals related to ongoing changes in cognition and brain state, both of which have the potential to interfere with eye position encoding and movement generation. In this study, we recorded spiking responses of neuronal populations in the SC while monkeys performed a memory-guided saccade task and found that the activity of some of the neurons fluctuated over tens of minutes. By leveraging the statistical power afforded by high-dimensional neuronal recordings, we were able to identify a low-dimensional pattern of activity that was correlated with the subjects' arousal levels. Importantly, we found that the spiking responses of deep-layer SC neurons were less correlated with this brain-wide arousal signal, and that neural activity associated with changes in pupil size and saccade tuning did not overlap in population activity space with movement initiation signals. Taken together, these findings provide a framework for understanding how signals related to cognition and arousal can be embedded in the population activity of oculomotor structures without compromising the fidelity of the motor output.

Coordinated changes in a cortical circuit sculpt effects of novelty on neural dynamics

Recent studies have found dramatic cell-type specific responses to stimulus novelty, highlighting the importance of analyzing the cortical circuitry at the cell-type specific level of granularity to understand brain function. Although initial work classified and characterized activity for each cell type, the specific alterations in cortical circuitry-particularly when multiple novelty effects interact-remain unclear. To address this gap, we employed a large-scale public dataset of electrophysiological recordings in the visual cortex of awake, behaving mice using Neuropixels probes and designed population network models to investigate the observed changes in neural dynamics in response to a combination of distinct forms of novelty. The model parameters were rigorously constrained by publicly available structural datasets, including multi-patch synaptic physiology and electron microscopy data. Our systematic optimization approach identified tens of thousands of model parameter sets that replicate the observed neural activity. Analysis of these solutions revealed generally weaker connections under novel stimuli, as well as a shift in the balance e between SST and VIP populations. Along with this, PV and SST populations experienced overall more excitatory influences compared to excitatory and VIP populations. Our results also highlight the role of VIP neurons in multiple aspects of visual stimulus processing and altering gain and saturation dynamics under novel conditions. In sum, our findings provide a systematic characterization of how the cortical circuit adapts to stimulus novelty by combining multiple rich public datasets.

Co-Occurrence, Extension, and Social Salience: The Emergence of Indexicality in an Artificial Language

We investigated the emergence of sociolinguistic indexicality using an artificial-language-learning paradigm. Sociolinguistic indexicality involves the association of linguistic variants with nonlinguistic social or contextual features. Any linguistic variant can acquire "constellations" of such indexical meanings, though they also exhibit an ordering, with first-order indices associated with particular speaker groups and higher-order indices targeting stereotypical attributes of those speakers. Much natural-language research has been conducted on this phenomenon, but little experimental work has focused on how indexicality emerges. Here, we present three miniature artificial-language experiments designed to break ground on this question. Results show ready formation of first-order indexicality based on co-occurrence alone, with higher-order indexicality emerging as a result of extension to new speaker groups, modulated by the perceived practical importance of the indexed social feature.

Spike rates of frontal eye field neurons predict reaction times in a spatial attention task

Which neuronal signal(s) predict reaction times when subjects respond to a target at covertly attended locations? Although recent studies showed that spike rates are not predictive, it remains a highly contested question. Therefore, we record single-unit activity from frontal eye field (FEF) neurons while macaques are performing a covert spatial attention task. We find that the attentional modulation of spike rates of FEF neurons is strongly correlated with behavioral reaction times. Moreover, this correlation already emerges 1 s before target dimming, which triggers the behavioral responses. This prediction of reaction times by spike rates is found in neurons showing attention-dependent enhanced and suppressed activity for targets and distractors, respectively, yet in varying degrees across subjects. Thus, spike rates of FEF neurons can predict reaction times persistently and well before the operant behavior during selective attention tasks. Such long prediction windows will be useful for developing spike-based brain-machine interfaces.

Changes in behavioral priority influence the accessibility of working memory content

Evolving behavioral goals require the existence of selection mechanisms that prioritize task-relevant working memory (WM) content for action. Selecting an item stored in WM is known to blunt and/or reverse information loss in stimulus-specific representations of that item reconstructed from human brain activity, but extant studies have focused on all-or-none circumstances that allow or disallow an agent to select one of several items stored in WM. Conversely, behavioral studies suggest that humans can flexibly assign different levels of priority to different items stored in WM, but how doing so influences neural representations of WM content is unclear. One possibility is that assigning different levels of priority to items in WM influences the quality of those representations, resulting in more robust neural representations of high- vs. low-priority WM content. A second - and non-exclusive - possibility is that asymmetries in behavioral priority influence how rapidly neural representations of high- vs. low-priority WM content can be selected and reported. We tested these possibilities in two experiments by decoding high- and low-priority WM content from EEG recordings obtained while human volunteers performed a retrospectively cued WM task. Probabilistic changes in the behavioral relevance of a remembered item had no effect on our ability to decode it from EEG signals; instead, these changes influenced the latency at which above-chance decoding performance was reached. Thus, our results indicate that probabilistic changes in the behavioral relevance of WM content influence the ease with which memories can be selected independently of their strength.

Active visual search in naturalistic environments reflects individual differences in classic visual search performance

Visual search is a ubiquitous activity in real-world environments. Yet, traditionally, visual search is investigated in tightly controlled paradigms, where head-restricted participants locate a minimalistic target in a cluttered array that is presented on a computer screen. Do traditional visual search tasks predict performance in naturalistic settings, where participants actively explore complex, real-world scenes? Here, we leverage advances in virtual reality technology to test the degree to which classic and naturalistic search are limited by a common factor, set size, and the degree to which individual differences in classic search behavior predict naturalistic search behavior in a large sample of individuals (N = 75). In a naturalistic search task, participants looked for an object within their environment via a combination of head-turns and eye-movements using a head-mounted display. Then, in a classic search task, participants searched for a target within a simple array of colored letters using only eye-movements. In each task, we found that participants' search performance was impacted by increases in set size-the number of items in the visual display. Critically, we observed that participants' efficiency in classic search tasks-the degree to which set size slowed performance-indeed predicted efficiency in real-world scenes. These results demonstrate that classic, computer-based visual search tasks are excellent models of active, real-world search behavior.

The tricky business of defining brain functions

Neuroscience has a long history of investigating the neural correlates of brain functions. One example is fear, which has been studied intensely in a variety of species. In parallel, unease about definitions of brain functions has existed for over 100 years. Because the translational impact of basic research hinges on how we define these functions, these definitions should be carefully considered.

Unraveling the Entangled Brain: How Do We Go About It?

An impactful understanding of the brain will require entirely new approaches and unprecedented collaborative efforts. The next steps will require brain researchers to develop theoretical frameworks that allow them to tease apart dependencies and causality in complex dynamical systems, as well as the ability to maintain awe while not getting lost in the effort. The outstanding question is: How do we go about it?