Dynamic integration of task-relevant visual features in posterior parietal cortex

Learning attentional templates for value-based decision-making

Attention filters sensory inputs to enhance task-relevant information. It is guided by an "attentional template" that represents the stimulus features that are currently relevant. To understand how the brain learns and uses templates, we trained monkeys to perform a visual search task that required them to repeatedly learn new attentional templates. Neural recordings found that templates were represented across the prefrontal and parietal cortex in a structured manner, such that perceptually neighboring templates had similar neural representations. When the task changed, a new attentional template was learned by incrementally shifting the template toward rewarded features. Finally, we found that attentional templates transformed stimulus features into a common value representation that allowed the same decision-making mechanisms to deploy attention, regardless of the identity of the template. Altogether, our results provide insight into the neural mechanisms by which the brain learns to control attention and how attention can be flexibly deployed across tasks.

Dissociable neuronal substrates of visual feature attention and working memory

Attention and working memory (WM) are distinct cognitive functions, yet given their close interactions, it is often assumed that they share the same neuronal mechanisms. We show that in macaques performing a WM-guided feature attention task, the activity of most neurons in areas middle temporal (MT), medial superior temporal (MST), lateral intraparietal (LIP), and posterior lateral prefrontal cortex (LPFC-p) displays attentional modulation or WM coding and not both. One area thought to play a role in both functions is LPFC-p. To test this, we optogenetically inactivated LPFC-p bilaterally during different task periods. Attention period inactivation reduced attentional modulation in LPFC-p, MST, and LIP neurons and impaired task performance. In contrast, WM period inactivation did not affect attentional modulation or performance and minimally affected WM coding. Our results suggest that feature attention and WM have dissociable neuronal substrates and that LPFC-p plays a critical role in feature attention, but not in WM.

From Multisensory Integration to Multisensory Decision-Making

Organisms live in a dynamic environment in which sensory information from multiple sources is ever changing. A conceptually complex task for the organisms is to accumulate evidence across sensory modalities and over time, a process known as multisensory decision-making. This is a new concept, in terms of that previous researches have been largely conducted in parallel disciplines. That is, much efforts have been put either in sensory integration across modalities using activity summed over a duration of time, or in decision-making with only one sensory modality that evolves over time. Recently, a few studies with neurophysiological measurements emerge to study how different sensory modality information is processed, accumulated, and integrated over time in decision-related areas such as the parietal or frontal lobes in mammals. In this review, we summarize and comment on these studies that combine the long-existed two parallel fields of multisensory integration and decision-making. We show how the new findings provide insight into our understanding about neural mechanisms mediating multisensory information processing in a more complete way.

Information processing in the dorsal pathway: from "where" to "what we do with it"

Visual processing in the brain has been understood as the ventral and dorsal pathways processing "what" and "where" information, respectively. Mocz et al. (Mocz V, Vaziri-Pashkam M, Chun M, Xu Y. J Cogn Neurosci 34: 2406-2435, 2022), however, report that the two pathways code object features in a parallel manner. These results support that information processing in the dorsal pathway is not strictly limited to "where" and that the two pathways work in parallel to process task-relevant information ("what we do with it").

Posterior Parietal Cortex Plays a Causal Role in Abstract Memory-Based Visual Categorical Decisions

Neural activity in the lateral intraparietal cortex (LIP) correlates with both sensory evaluation and motor planning underlying visuomotor decisions. We previously showed that LIP plays a causal role in visually-based perceptual and categorical decisions, and preferentially contributes to evaluating sensory stimuli over motor planning. In that study, however, monkeys reported their decisions with a saccade to a colored target associated with the correct motion category or direction. Since LIP is known to play a role in saccade planning, it remains unclear whether LIP's causal role in such decisions extend to decision-making tasks which do not involve saccades. Here, we employed reversible pharmacological inactivation of LIP neural activity while two male monkeys performed delayed match to category (DMC) and delayed match to sample (DMS) tasks. In both tasks, monkeys needed to maintain gaze fixation throughout the trial and report whether a test stimulus was a categorical match or nonmatch to the previous sample stimulus by releasing a touch bar. LIP inactivation impaired monkeys' behavioral performance in both tasks, with deficits in both accuracy and reaction time (RT). Furthermore, we recorded LIP neural activity in the DMC task targeting the same cortical locations as in the inactivation experiments. We found significant neural encoding of the sample category, which was correlated with monkeys' categorical decisions in the DMC task. Taken together, our results demonstrate that LIP plays a generalized role in visual categorical decisions independent of the task-structure and motor response modality.SIGNIFICANCE STATEMENT Neural activity in the lateral intraparietal cortex (LIP) correlates with perceptual and categorical decisions, in addition to its role in mediating saccadic eye movements. Past work found that LIP is causally involved in visual decisions that are rapidly reported by saccades in a reaction time based decision making task. Here we use reversible inactivation of LIP to test whether LIP is also causally involved in visual decisions when reported by hand movements during delayed matching tasks. Here we show that LIP inactivation impaired monkeys' task performance during both memory-based discrimination and categorization tasks. These results demonstrate that LIP plays a generalized role in visual categorical decisions independent of the task-structure and motor response modality.

Cell-type-specific integration of feedforward and feedback synaptic inputs in the posterior parietal cortex

The integration of feedforward (sensory) and feedback (top-down) neuronal signals is a principal function of the neocortex. Yet, we have limited insight into how these information streams are combined by individual neurons. Using a two-color optogenetic strategy, we found that layer 5 pyramidal neurons in the posterior parietal cortex receive monosynaptic dual innervation, combining sensory inputs with top-down signals. Subclasses of layer 5 pyramidal neurons integrated these synapses with distinct temporal dynamics. Specifically, regular spiking cells exhibited supralinear enhancement of delayed-but not coincident-inputs, while intrinsic burst-firing neurons selectively boosted coincident synaptic events. These subthreshold integration characteristics translated to a nonlinear summation of action potential firing. Complementing electrophysiology with computational modeling, we found that distinct integration profiles arose from a cell-type-specific interaction of ionic mechanisms and feedforward inhibition. These data provide insight into the cellular properties that guide the nonlinear interaction of distinct long-range afferents in the neocortex.

Task Context Modulates Feature-Selective Responses in Area V4

Feature selectivity of visual cortical responses measured during passive fixation provides only a partial view of selectivity because it does not account for the influence of cognitive factors. Here we focus on primate area V4 and ask how neuronal tuning is modulated by task engagement. We investigated whether responses to colored shapes during active shape discrimination are simple, stimulus-agnostic, scaled versions of responses during passive fixation, akin to results from attentional studies. Alternatively, responses could be subject to stimulus-specific scaling, that is, responses to different stimuli are modulated differently, resulting in changes in underlying shape/color selectivity. Among 83 well-isolated V4 neurons in two male macaques, only a minority (16 of 83), which were weakly tuned to both shape and color, displayed responses during fixation and discrimination tasks that could be related by stimulus-agnostic scaling. The majority (67 of 83), which were strongly tuned to shape, color, or both, displayed stimulus-dependent response changes during discrimination. For some of these neurons (39 of 83), the shape or color of the stimulus dictated the magnitude of the change, and for others (28 of 83) it was the combination of stimulus shape and color. Importantly, for neurons with one strong and one weak tuning dimension, stimulus-dependent response changes during discrimination were associated with a relative increase in selectivity along the stronger tuning dimension, without changes in tuning peak. These results reveal that more strongly tuned V4 neurons may also be more flexible in their selectivity, and imbalances in selectivity are amplified during active task contexts.SIGNIFICANCE STATEMENT Tuning for stimulus features is typically characterized by recording responses during passive fixation, but cognitive factors, including attention, influence responses in visual cortex. To determine how behavioral engagement influences neuronal responses in area V4, we compared responses to colored shapes during passive fixation and active behavior. For a large fraction of neurons, differences in responses between passive fixation and active behavior depended on the identity of the visual stimulus. For a subgroup of strongly feature-selective neurons, this response modulation was associated with enhanced selectivity for one feature at the expense of selectivity for the other. Such flexibility in tuning strength could improve performance in tasks requiring active judgment of stimuli.

Activity in the Fronto-Parietal and Visual Cortex Is Modulated by Feature-Based Attentional Weighting

In day-to-day dynamic activities where sensory input is abundant, stimulus representations in the visual cortex are modulated based on their attentional priority. Several studies have established the top-down role of a fronto-parietal dorsal attention network in selective attention. In the current study, we aimed to investigate whether activity of subregions of this network and the visual cortex is modulated by feature-based attentional weighting, and if so, whether their timecourses of activity are correlated. To this end, we analyzed fMRI data of 28 healthy subjects, who performed a feature-based go/no-go task. Participants had to attend to one or two colored streams of sinusoidal gratings and respond to each grating in the task-relevant stream(s) except to a single non-target grating. Univariate and multivariate fMRI results indicated that activity in bilateral fronto-parietal (frontal eye fields, intraparietal sulcus and superior parietal lobe) and visual (V1-V4, lateral occipital cortex and fusiform gyrus) regions was modulated by selecting one instead of attending to two gratings. Functional connectivity was not significantly different between fronto-parietal and visual regions when attending to one as opposed to two gratings. Our study demonstrates that activity in subregions of both the fronto-parietal and visual cortex is modified by feature-based attentional weighting.

The time-course of feature-based attention effects dissociated from temporal expectation and target-related processes

Selective attention prioritises relevant information amongst competing sensory input. Time-resolved electrophysiological studies have shown stronger representation of attended compared to unattended stimuli, which has been interpreted as an effect of attention on information coding. However, because attention is often manipulated by making only the attended stimulus a target to be remembered and/or responded to, many reported attention effects have been confounded with target-related processes such as visual short-term memory or decision-making. In addition, attention effects could be influenced by temporal expectation about when something is likely to happen. The aim of this study was to investigate the dynamic effect of attention on visual processing using multivariate pattern analysis of electroencephalography (EEG) data, while (1) controlling for target-related confounds, and (2) directly investigating the influence of temporal expectation. Participants viewed rapid sequences of overlaid oriented grating pairs while detecting a "target" grating of a particular orientation. We manipulated attention, one grating was attended and the other ignored (cued by colour), and temporal expectation, with stimulus onset timing either predictable or not. We controlled for target-related processing confounds by only analysing non-target trials. Both attended and ignored gratings were initially coded equally in the pattern of responses across EEG sensors. An effect of attention, with preferential coding of the attended stimulus, emerged approximately 230 ms after stimulus onset. This attention effect occurred even when controlling for target-related processing confounds, and regardless of stimulus onset expectation. These results provide insight into the effect of feature-based attention on the dynamic processing of competing visual information.

The role of temporal cortex in the control of attention

Attention is an indispensable component of active vision. Contrary to the widely accepted notion that temporal cortex processing primarily focusses on passive object recognition, a series of very recent studies emphasize the role of temporal cortex structures, specifically the superior temporal sulcus (STS) and inferotemporal (IT) cortex, in guiding attention and implementing cognitive programs relevant for behavioral tasks. The goal of this theoretical paper is to advance the hypothesis that the temporal cortex attention network (TAN) entails necessary components to actively participate in attentional control in a flexible task-dependent manner. First, we will briefly discuss the general architecture of the temporal cortex with a focus on the STS and IT cortex of monkeys and their modulation with attention. Then we will review evidence from behavioral and neurophysiological studies that support their guidance of attention in the presence of cognitive control signals. Next, we propose a mechanistic framework for executive control of attention in the temporal cortex. Finally, we summarize the role of temporal cortex in implementing cognitive programs and discuss how they contribute to the dynamic nature of visual attention to ensure flexible behavior.

Feature-based attention processes in primate prefrontal cortex do not rely on feature similarity

Feature-based attention enables privileged processing of specific visual properties. During feature-based attention, neurons in visual cortices show "gain modulation" by enhancing neuronal responses to the features of attended stimuli due to top-down signals originating from prefrontal cortex (PFC). Attentional modulation in visual cortices requires "feature similarity:" neurons only increase their responses when the attended feature variable and the neurons' preferred feature coincide. However, whether gain modulation based on feature similarity is a general attentional mechanism is currently unknown. To address this issue, we record single-unit activity from PFC of macaques trained to switch attention between two conjunctive feature parameters. We find that PFC neurons experience gain modulation in response to attentional demands. However, this attentional gain modulation in PFC is independent of the feature-tuning preferences of neurons. These findings suggest that feature similarity is not a general mechanism in feature-based attention throughout the cortical processing hierarchy.

Lessons From Deep Neural Networks for Studying the Coding Principles of Biological Neural Networks

One of the central goals in systems neuroscience is to understand how information is encoded in the brain, and the standard approach is to identify the relation between a stimulus and a neural response. However, the feature of a stimulus is typically defined by the researcher's hypothesis, which may cause biases in the research conclusion. To demonstrate potential biases, we simulate four likely scenarios using deep neural networks trained on the image classification dataset CIFAR-10 and demonstrate the possibility of selecting suboptimal/irrelevant features or overestimating the network feature representation/noise correlation. Additionally, we present studies investigating neural coding principles in biological neural networks to which our points can be applied. This study aims to not only highlight the importance of careful assumptions and interpretations regarding the neural response to stimulus features but also suggest that the comparative study between deep and biological neural networks from the perspective of machine learning can be an effective strategy for understanding the coding principles of the brain.

Coding Perceptual Decisions: From Single Units to Emergent Signaling Properties in Cortical Circuits

Spiking activity in single neurons of the primate visual cortex has been tightly linked to perceptual decisions. Any mechanism that reads out these perceptual signals to support behavior must respect the underlying neuroanatomy that shapes the functional properties of sensory neurons. Spatial distribution and timing of inputs to the next processing levels are critical, as conjoint activity of precursor neurons increases the spiking rate of downstream neurons and ultimately drives behavior. I set out how correlated activity might coalesce into a micropool of task-sensitive neurons signaling a particular percept to determine perceptual decision signals locally and for flexible interarea transmission depending on the task context. As data from more and more neurons and their complex interactions are analyzed, the space of computational mechanisms must be constrained based on what is plausible within neurobiological limits. This review outlines experiments to test the new perspectives offered by these extended methods.

The profile of attentional modulation to visual features

Although it is well established that feature-based attention (FBA) can enhance an attended feature, how it modulates unattended features remains less clear. Previous studies have generally supported either a graded profile as predicted by the feature-similarity gain model or a nonmonotonic profile predicted by the surround suppression model. To reconcile these different views, we systematically measured the attentional profile in three basic feature dimensions-orientation, motion direction, and spatial frequency. In three experiments, we instructed participants to detect a coherent feature signal against noise under attentional or neutral condition. Our results support a nonmonotonic hybrid model of attentional modulation consisting of feature-similarity gain and surround suppression for orientation and motion direction. For spatial frequency, we also found a similar nonmonotonic profile for higher frequencies than the attended frequency, but a lack of attentional modulation for lower frequencies than the attended frequency. The current findings can reconcile the discrepancies in the literature and suggest the hybrid model as a new framework for attentional modulation in feature space. In addition, a computational model incorporating known properties of spatial frequency channels and attentional modulations at the neural level reproduced the asymmetric attentional modulation, thus revealing a connection between surround suppression and the basic neural architecture of an early visual system.

Feature-based attention: effects and control

Feature-based attention prioritizes the processing of non-spatial features across the visual field. Classical studies revealed a feature-similarity gain modulation of sensory neuron's activity. While early studies that quantified behavioral performance have provided support for this model, recent studies have revealed a non-monotonic, surround suppression effect in near feature space. The attentional suppression effects may give rise to a highly limited capacity when selecting multiple features, as documented by studies manipulating the number of attended features. These effects of feature-based attention are likely due to attentional control mechanisms exerting top-down modulations, which have been linked to neural signals in the dorsal frontoparietal network. The neural representation of attentional priority at multiple levels of the visual hierarchy thus shape visual perception and behavioral performance.

Attentional amplification of neural codes for number independent of other quantities along the dorsal visual stream

Humans and other animals base important decisions on estimates of number, and intraparietal cortex is thought to provide a crucial substrate of this ability. However, it remains debated whether an independent neuronal processing mechanism underlies this ‘number sense’, or whether number is instead judged indirectly on the basis of other quantitative features. We performed high-resolution 7 Tesla fMRI while adult human volunteers attended either to the numerosity or an orthogonal dimension (average item size) of visual dot arrays. Along the dorsal visual stream, numerosity explained a significant amount of variance in activation patterns, above and beyond non-numerical dimensions. Its representation was selectively amplified and progressively enhanced across the hierarchy when task relevant. Our results reveal a sensory extraction mechanism yielding information on numerosity separable from other dimensions already at early visual stages and suggest that later regions along the dorsal stream are most important for explicit manipulation of numerical quantity.

Numbers and the ability to count and calculate are an essential part of human culture. They are part of everyday life, featuring in calendars, computers or the weekly shop, but also in some of humanity’s biggest achievements: without them the pyramids or space travel would not exist. A precursor of sophisticated mathematical skill could reside in a simpler mental ability: the capacity to assess numerical quantities at a glance. This ‘number sense’ appears in humans in early childhood and it is also present in other animals, but it is still poorly understood.

Brain imaging techniques have identified the parts of the brain that are active when perceiving numbers or making calculations. As techniques have advanced, it has become possible to resolve fine differences in brain activity that occur when people switch their attention between different visual tasks. But how exactly does the human brain process visual information to make sense of numbers? One theory suggests that humans use visual cues, such as the size of a group of objects or how densely packed objects are, to estimate numbers. On the other hand, it is also possible that humans can sense number directly, without reference to other properties of the group being observed.

Castaldi et al. presented twenty adult volunteers with groups of dots and asked them to focus either on the number of dots or on the size of the dots during a brain scan. This approach allowed the separation of brain signals specific to number from signals corresponding to other visual cues, such as size or density of the group. The experiment revealed that brain activity changed depending on the number of dots displayed. The signal related to number became stronger when people focused on the number of dots, while signals related to other properties of the group remained unchanged. Moreover, brain signals for number were observed at the very early stages of visual processing, in the parts of the brain that receive input from the eyes first.

These results suggest that the human visual system perceives number directly, and not by processing information about the size or density of a group of objects. This finding provides insights into how human brains encode numbers, which could be important to understand disorders where number sense can be impaired leading to difficulties learning math and operating with numbers.

An Integrative Framework for Sensory, Motor, and Cognitive Functions of the Posterior Parietal Cortex

Throughout the history of modern neuroscience, the parietal cortex has been associated with a wide array of sensory, motor, and cognitive functions. The use of non-human primates as a model organism has been instrumental in our current understanding of how areas in the posterior parietal cortex (PPC) modulate our perception and influence our behavior. In this Perspective, we highlight a series of influential studies over the last five decades examining the role of the PPC in visual perception and motor planning. We also integrate long-standing views of PPC functions with more recent evidence to propose a more general model framework to explain integrative sensory, motor, and cognitive functions of the PPC.

Continuous and discrete representations of feature-based attentional priority in human frontoparietal network

Previous studies suggest that human frontoparietal network represents feature-based attentional priority, yet the precise nature of the priority signals remains unclear. Here, we examined whether priority signals vary continuously or discretely as a function of feature similarity. In an fMRI experiment, we presented two superimposed dot fields moving along two linear directions (leftward and rightward) while varying the angular separation between the two directions. Subjects were cued to attend to one of the two dot fields and respond to a possible speed-up in the cued direction. We used multivariate analysis to evaluate how priority representation of the attended direction changes with feature similarity. We found that in early visual areas as well as posterior intraparietal sulcus and inferior frontal junction, the patterns of neural activity became more different as the feature similarity decreased, indicating a continuous representation of the attended feature. In contrast, patterns of neural activity in anterior intraparietal sulcus and frontal eye field remained invariant to changes in feature similarity, indicating a discrete representation of the attended feature. Such distinct neural coding of attentional priority across the frontoparietal network may make complementary contributions to enable flexible attentional control.

The role of putative human anterior intraparietal sulcus area in observed manipulative action discrimination

INTRODUCTION

Although it has become widely accepted that the action observation network (AON) includes three levels (occipito-temporal, parietal and premotor), little is known concerning the specific role of these levels within perceptual tasks probing action observation. Recent single cell studies suggest that the parietal level carries the information required to discriminate between two-alternative observed actions, but do not exclude possible contributions from the other two levels.

METHODS

Two functional magnetic resonance imaging experiments used a task-based attentional modulation paradigm in which subjects viewed videos of an actor performing a manipulative action on a coloured object, and discriminated between either two observed manipulative actions, two actors or two colours.

RESULTS

Both experiments demonstrated that relative to actor and colour discrimination, discrimination between observed manipulative actions involved the putative human anterior intraparietal sulcus (phAIP) area in parietal cortex. In one experiment, where the observed actions also differed with regard to effectors, premotor cortex was also specifically recruited.

CONCLUSIONS

Our results highlight the primary role of parietal cortex in discriminating between two-alternative observed manipulative actions, consistent with the view that this level plays a major role in representing the identity of an observed action.

Attention to colors induces surround suppression at category boundaries

We investigated how attention to a visual feature modulates representations of other features. The feature-similarity gain model predicts a graded modulation, whereas an alternative model asserts an inhibitory surround in feature space. Although evidence for both types of modulations can be found, a consensus has not emerged in the literature. Here, we aimed to reconcile these different views by systematically measuring how attention modulates color perception. Based on previous literature, we also predicted that color categories would impact attentional modulation. Our results showed that both surround suppression and feature-similarity gain modulate perception of colors but they operate on different similarity scales. Furthermore, the region of the suppressive surround coincided with the color category boundary, suggesting a categorical sharpening effect. We implemented a neural population coding model to explain the observed behavioral effects, which revealed a hitherto unknown connection between neural tuning shift and surround suppression.

Decomposing Parietal Memory Reactivation to Predict Consequences of Remembering

Memory retrieval can strengthen, but also distort memories. Parietal cortex is a candidate region involved in retrieval-induced memory changes as it reflects retrieval success and represents retrieved content. Here, we conducted an fMRI experiment to test whether different forms of parietal reactivation predict distinct consequences of retrieval. Subjects studied associations between words and pictures of faces, scenes, or objects, and then repeatedly retrieved half of the pictures, reporting the vividness of the retrieved pictures (“retrieval practice”). On the following day, subjects completed a recognition memory test for individual pictures. Critically, the test included lures highly similar to studied pictures. Behaviorally, retrieval practice increased both hit and false alarm (FA) rates to similar lures, confirming a causal influence of retrieval on subsequent memory. Using pattern similarity analyses, we measured two different levels of reactivation during retrieval practice: generic “category-level” reactivation and idiosyncratic “item-level” reactivation. Vivid remembering during retrieval practice was associated with stronger category- and item-level reactivation in parietal cortex. However, these measures differentially predicted subsequent recognition memory performance: whereas higher category-level reactivation tended to predict FAs to lures, item-level reactivation predicted correct rejections. These findings indicate that parietal reactivation can be decomposed to tease apart distinct consequences of memory retrieval.

The Posterior Parietal Cortex in Adaptive Visual Processing

Although the primate posterior parietal cortex (PPC) has been largely associated with space, attention, and action-related processing, a growing number of studies have reported the direct representation of a diverse array of action-independent nonspatial visual information in the PPC during both perception and visual working memory. By describing the distinctions and the close interactions of visual representation with space, attention, and action-related processing in the PPC, here I propose that we may understand these diverse PPC functions together through the unique contribution of the PPC to adaptive visual processing and form a more integrated and structured view of the role of the PPC in vision, cognition, and action.

Parietal Representations of Stimulus Features Are Amplified during Memory Retrieval and Flexibly Aligned with Top-Down Goals

In studies of human episodic memory, the phenomenon of reactivation has traditionally been observed in regions of occipitotemporal cortex (OTC) involved in visual perception. However, reactivation also occurs in lateral parietal cortex (LPC), and recent evidence suggests that stimulus-specific reactivation may be stronger in LPC than in OTC. These observations raise important questions about the nature of memory representations in LPC and their relationship to representations in OTC. Here, we report two fMRI experiments that quantified stimulus feature information (color and object category) within LPC and OTC, separately during perception and memory retrieval, in male and female human subjects. Across both experiments, we observed a clear dissociation between OTC and LPC: while feature information in OTC was relatively stronger during perception than memory, feature information in LPC was relatively stronger during memory than perception. Thus, while OTC and LPC represented common stimulus features in our experiments, they preferentially represented this information during different stages. In LPC, this bias toward mnemonic information co-occurred with stimulus-level reinstatement during memory retrieval. In Experiment 2, we considered whether mnemonic feature information in LPC was flexibly and dynamically shaped by top-down retrieval goals. Indeed, we found that dorsal LPC preferentially represented retrieved feature information that addressed the current goal. In contrast, ventral LPC represented retrieved features independent of the current goal. Collectively, these findings provide insight into the nature and significance of mnemonic representations in LPC and constitute an important bridge between putative mnemonic and control functions of parietal cortex.SIGNIFICANCE STATEMENT When humans remember an event from the past, patterns of sensory activity that were present during the initial event are thought to be reactivated. Here, we investigated the role of lateral parietal cortex (LPC), a high-level region of association cortex, in representing prior visual experiences. We find that LPC contained stronger information about stimulus features during memory retrieval than during perception. We also found that current task goals influenced the strength of stimulus feature information in LPC during memory. These findings suggest that, in addition to early sensory areas, high-level areas of cortex, such as LPC, represent visual information during memory retrieval, and that these areas may play a special role in flexibly aligning memories with current goals.

Target Interneuron Preference in Thalamocortical Pathways Determines the Temporal Structure of Cortical Responses

Sensory processing relies on fast detection of changes in environment, as well as integration of contextual cues over time. The mechanisms by which local circuits of the cerebral cortex simultaneously perform these opposite processes remain obscure. Thalamic “specific” nuclei relay sensory information, whereas “nonspecific” nuclei convey information on the environmental and behavioral contexts. We expressed channelrhodopsin in the ventrobasal specific (sensory) or the rhomboid nonspecific (contextual) thalamic nuclei. By selectively activating each thalamic pathway, we found that nonspecific inputs powerfully activate adapting (slow-responding) interneurons but weakly connect fast-spiking interneurons, whereas specific inputs exhibit opposite interneuron preference. Specific inputs thereby induce rapid feedforward inhibition that limits response duration, whereas, in the same cortical area, nonspecific inputs elicit delayed feedforward inhibition that enables lasting recurrent excitation. Using a mean field model, we confirm that cortical response dynamics depends on the type of interneuron targeted by thalamocortical inputs and show that efficient recruitment of adapting interneurons prolongs the cortical response and allows the summation of sensory and contextual inputs. Hence, target choice between slow- and fast-responding inhibitory neurons endows cortical networks with a simple computational solution to perform both sensory detection and integration.

A Tale of Two Visual Systems: Invariant and Adaptive Visual Information Representations in the Primate Brain

Visual information processing contains two opposite needs. There is both a need to comprehend the richness of the visual world and a need to extract only pertinent visual information to guide thoughts and behavior at a given moment. I argue that these two aspects of visual processing are mediated by two complementary visual systems in the primate brain-specifically, the occipitotemporal cortex (OTC) and the posterior parietal cortex (PPC). The role of OTC in visual processing has been documented extensively by decades of neuroscience research. I review here recent evidence from human imaging and monkey neurophysiology studies to highlight the role of PPC in adaptive visual processing. I first document the diverse array of visual representations found in PPC. I then describe the adaptive nature of visual representation in PPC by contrasting visual processing in OTC and PPC and by showing that visual representations in PPC largely originate from OTC.

The role of inferior frontal junction in controlling the spatially global effect of feature-based attention in human visual areas

Feature-based attention has a spatially global effect, i.e., responses to stimuli that share features with an attended stimulus are enhanced not only at the attended location but throughout the visual field. However, how feature-based attention modulates cortical neural responses at unattended locations remains unclear. Here we used functional magnetic resonance imaging (fMRI) to examine this issue as human participants performed motion- (Experiment 1) and color- (Experiment 2) based attention tasks. Results indicated that, in both experiments, the respective visual processing areas (middle temporal area [MT+] for motion and V4 for color) as well as early visual, parietal, and prefrontal areas all showed the classic feature-based attention effect, with neural responses to the unattended stimulus significantly elevated when it shared the same feature with the attended stimulus. Effective connectivity analysis using dynamic causal modeling (DCM) showed that this spatially global effect in the respective visual processing areas (MT+ for motion and V4 for color), intraparietal sulcus (IPS), frontal eye field (FEF), medial frontal gyrus (mFG), and primary visual cortex (V1) was derived by feedback from the inferior frontal junction (IFJ). Complementary effective connectivity analysis using Granger causality modeling (GCM) confirmed that, in both experiments, the node with the highest outflow and netflow degree was IFJ, which was thus considered to be the source of the network. These results indicate a source for the spatially global effect of feature-based attention in the human prefrontal cortex.

Attentional selection is the mechanism by which relevant sensory information is processed preferentially. Feature-based attention plays a key role in identifying an attentional target in a complex scene, because we often know the features of the target but not its exact location. The ability to quickly select the target is mainly attributed to enhancement of responses to stimuli that share features with an attended stimulus, not only at the attended location but throughout the whole visual field. However, little is known regarding how feature-based attention modulates brain responses at unattended locations. Here we used fMRI and advanced connectivity analyses to examine human subjects as they performed either motion- or color-based attention tasks. Our results indicated that the visual processing areas for motion and color showed the feature-based attention effect. Effective connectivity analysis showed that this feature-based attention effect was derived by feedback from the inferior frontal junction, an area of the posterior lateral prefrontal cortex involved in many different cognitive processes, including spatial attention and working memory. Further modeling confirmed that the inferior frontal junction showed connectivity features supporting its role as the source of the network. Our results support the hypothesis that the inferior frontal junction plays a key role in the spatially global effect of feature-based attention.

Attention Effects on Neural Population Representations for Shape and Location Are Stronger in the Ventral than Dorsal Stream

We examined how attention causes neural population representations of shape and location to change in ventral stream (AIT) and dorsal stream (LIP). Monkeys performed two identical delayed-match-to-sample (DMTS) tasks, attending either to shape or location. In AIT, shapes were more discriminable when directing attention to shape rather than location, measured by an increase in mean distance between population response vectors. In LIP, attending to location rather than shape did not increase the discriminability of different stimulus locations. Even when factoring out the change in mean vector response distance, multidimensional scaling (MDS) still showed a significant task difference in AIT, but not LIP, indicating that beyond increasing discriminability, attention also causes a nonlinear warping of representation space in AIT. Despite single-cell attentional modulations in both areas, our data show that attentional modulations of population representations are weaker in LIP, likely due to a need to maintain veridical representations for visuomotor control.

Pharmacological inactivation does not support a unique causal role for intraparietal sulcus in the discrimination of visual number

The "number sense" describes the intuitive ability to quantify without counting. Single neuron recordings in non-human primates and functional imaging in humans suggest the intraparietal sulcus is an important neuroanatomical locus of numerical estimation. Other lines of inquiry implicate the IPS in numerous other functions, including attention and decision making. Here we provide a direct test of whether IPS has functional specificity for numerosity judgments. We used muscimol to reversibly and independently inactivate the ventral and lateral intraparietal areas in two monkeys performing a numerical discrimination task and a color discrimination task, roughly equilibrated for difficulty. Inactivation of either area caused parallel impairments in both tasks and no evidence of a selective deficit in numerical processing. These findings do not support a causal role for the IPS in numerical discrimination, except insofar as it also has a role in the discrimination of color. We discuss our findings in light of several alternative hypotheses of IPS function, including a role in orienting responses, a general cognitive role in attention and decision making processes and a more specific role in ordinal comparison that encompasses both number and color judgments.

Coding of spatial attention priorities and object features in the macaque lateral intraparietal cortex

Primate posterior parietal cortex (PPC) is known to be involved in controlling spatial attention. Neurons in one part of the PPC, the lateral intraparietal area (LIP), show enhanced responses to objects at attended locations. Although many are selective for object features, such as the orientation of a visual stimulus, it is not clear how LIP circuits integrate feature-selective information when providing attentional feedback about behaviorally relevant locations to the visual cortex. We studied the relationship between object feature and spatial attention properties of LIP cells in two macaques by measuring the cells' orientation selectivity and the degree of attentional enhancement while performing a delayed match-to-sample task. Monkeys had to match both the location and orientation of two visual gratings presented separately in time. We found a wide range in orientation selectivity and degree of attentional enhancement among LIP neurons. However, cells with significant attentional enhancement had much less orientation selectivity in their response than cells which showed no significant modulation by attention. Additionally, orientation-selective cells showed working memory activity for their preferred orientation, whereas cells showing attentional enhancement also synchronized with local neuronal activity. These results are consistent with models of selective attention incorporating two stages, where an initial feature-selective process guides a second stage of focal spatial attention. We suggest that LIP contributes to both stages, where the first stage involves orientation-selective LIP cells that support working memory of the relevant feature, and the second stage involves attention-enhanced LIP cells that synchronize to provide feedback on spatial priorities.

Computing by Robust Transience: How the Fronto-Parietal Network Performs Sequential, Category-Based Decisions

Decision making involves dynamic interplay between internal judgements and external perception, which has been investigated in delayed match-to-category (DMC) experiments. Our analysis of neural recordings shows that, during DMC tasks, LIP and PFC neurons demonstrate mixed, time-varying, and heterogeneous selectivity, but previous theoretical work has not established the link between these neural characteristics and population-level computations. We trained a recurrent network model to perform DMC tasks and found that the model can remarkably reproduce key features of neuronal selectivity at the single-neuron and population levels. Analysis of the trained networks elucidates that robust transient trajectories of the neural population are the key driver of sequential categorical decisions. The directions of trajectories are governed by network self-organized connectivity, defining a "neural landscape" consisting of a task-tailored arrangement of slow states and dynamical tunnels. With this model, we can identify functionally relevant circuit motifs and generalize the framework to solve other categorization tasks.

Object comparison in the lateral intraparietal area

We can search for and locate specific objects in our environment by looking for objects with similar features. Object recognition involves stimulus similarity responses in ventral visual areas and task-related responses in prefrontal cortex. We tested whether neurons in the lateral intraparietal area (LIP) of posterior parietal cortex could form an intermediary representation, collating information from object-specific similarity map representations to allow general decisions about whether a stimulus matches the object being looked for. We hypothesized that responses to stimuli would correlate with how similar they are to a sample stimulus. When animals compared two peripheral stimuli to a sample at their fovea, the response to the matching stimulus was similar, independent of the sample identity, but the response to the nonmatch depended on how similar it was to the sample: the more similar, the greater the response to the nonmatch stimulus. These results could not be explained by task difficulty or confidence. We propose that LIP uses its known mechanistic properties to integrate incoming visual information, including that from the ventral stream about object identity, to create a dynamic representation that is concise, low dimensional, and task relevant and that signifies the choice priorities in mental matching behavior.NEW & NOTEWORTHY Studies in object recognition have focused on the ventral stream, in which neurons respond as a function of how similar a stimulus is to their preferred stimulus, and on prefrontal cortex, where neurons indicate which stimulus is being looked for. We found that parietal area LIP uses its known mechanistic properties to form an intermediary representation in this process. This creates a perceptual similarity map that can be used to guide decisions in prefrontal areas.

Sequential sensory and decision processing in posterior parietal cortex

Decisions about the behavioral significance of sensory stimuli often require comparing sensory inference of what we are looking at to internal models of what we are looking for. Here, we test how neuronal selectivity for visual features is transformed into decision-related signals in posterior parietal cortex (area LIP). Monkeys performed a visual matching task that required them to detect target stimuli composed of conjunctions of color and motion-direction. Neuronal recordings from area LIP revealed two main findings. First, the sequential processing of visual features and the selection of target-stimuli suggest that LIP is involved in transforming sensory information into decision-related signals. Second, the patterns of color and motion selectivity and their impact on decision-related encoding suggest that LIP plays a role in detecting target stimuli by comparing bottom-up sensory inputs (what the monkeys were looking at) and top-down cognitive encoding inputs (what the monkeys were looking for).

DOI:http://dx.doi.org/10.7554/eLife.23743.001

Two Visual Pathways in Primates Based on Sampling of Space: Exploitation and Exploration of Visual Information

Evidence is strong that the visual pathway is segregated into two distinct streams—ventral and dorsal. Two proposals theorize that the pathways are segregated in function: The ventral stream processes information about object identity, whereas the dorsal stream, according to one model, processes information about either object location, and according to another, is responsible in executing movements under visual control. The models are influential; however recent experimental evidence challenges them, e.g., the ventral stream is not solely responsible for object recognition; conversely, its function is not strictly limited to object vision; the dorsal stream is not responsible by itself for spatial vision or visuomotor control; conversely, its function extends beyond vision or visuomotor control. In their place, we suggest a robust dichotomy consisting of a ventral stream selectively sampling high-resolution/focal spaces, and a dorsal stream sampling nearly all of space with reduced foveal bias. The proposal hews closely to the theme of embodied cognition: Function arises as a consequence of an extant sensory underpinning. A continuous, not sharp, segregation based on function emerges, and carries with it an undercurrent of an exploitation-exploration dichotomy. Under this interpretation, cells of the ventral stream, which individually have more punctate receptive fields that generally include the fovea or parafovea, provide detailed information about object shapes and features and lead to the systematic exploitation of said information; cells of the dorsal stream, which individually have large receptive fields, contribute to visuospatial perception, provide information about the presence/absence of salient objects and their locations for novel exploration and subsequent exploitation by the ventral stream or, under certain conditions, the dorsal stream. We leverage the dichotomy to unify neuropsychological cases under a common umbrella, account for the increased prevalence of multisensory integration in the dorsal stream under a Bayesian framework, predict conditions under which object recognition utilizes the ventral or dorsal stream, and explain why cells of the dorsal stream drive sensorimotor control and motion processing and have poorer feature selectivity. Finally, the model speculates on a dynamic interaction between the two streams that underscores a unified, seamless perception. Existing theories are subsumed under our proposal.

Long-Range Attention Networks: Circuit Motifs Underlying Endogenously Controlled Stimulus Selection

Attention networks comprise brain areas whose coordinated activity implements stimulus selection. This selection is reflected in spatially referenced priority maps across frontal-parietal-collicular areas and is controlled through interactions with circuits representing behavioral goals, including prefrontal, cingulate, and striatal circuits, among others. We review how these goal-providing structures control stimulus selection through long-range dynamic projection motifs. These motifs (i) combine feature-tuned subnetworks to a distributed priority map, (ii) establish endogenously controlled, long-range coherent activity at 4-10 Hz theta and 12-30 Hz beta-band frequencies, and (iii) are composed of unique cell types implementing long-range networks through disynaptic disinhibition, dendritic gating, and feedforward inhibitory gain control. This evidence reveals common circuit motifs used to coordinate attentionally selected information across multi-node brain networks during goal-directed behavior.

Interaction between Spatial and Feature Attention in Posterior Parietal Cortex

Lateral intraparietal (LIP) neurons encode a vast array of sensory and cognitive variables. Recently, we proposed that the flexibility of feature representations in LIP reflect the bottom-up integration of sensory signals, modulated by feature-based attention (FBA), from upstream feature-selective cortical neurons. Moreover, LIP activity is also strongly modulated by the position of space-based attention (SBA). However, the mechanisms by which SBA and FBA interact to facilitate the representation of task-relevant spatial and non-spatial features in LIP remain unclear. We recorded from LIP neurons during performance of a task that required monkeys to detect specific conjunctions of color, motion direction, and stimulus position. Here we show that FBA and SBA potentiate each other's effect in a manner consistent with attention gating the flow of visual information along the cortical visual pathway. Our results suggest that linear bottom-up integrative mechanisms allow LIP neurons to emphasize task-relevant spatial and non-spatial features.

Task-specific versus generalized mnemonic representations in parietal and prefrontal cortices

Our ability to learn a wide range of behavioral tasks is essential for responding appropriately to sensory stimuli according to behavioral demands, but the underlying neural mechanism has been rarely examined by neurophysiological recordings in the same subjects across learning. To understand how learning new behavioral tasks affects neuronal representations, we recorded from posterior parietal cortex (PPC) before and after training on a visual motion categorization task. We found that categorization training influenced cognitive encoding in PPC, with a marked enhancement of memory-related delay-period encoding during the categorization task that was absent during a motion discrimination task before categorization training. In contrast, the prefrontal cortex (PFC) exhibited strong delay-period encoding during both discrimination and categorization tasks. This reveals a dissociation between PFC's and PPC's roles in working memory, with general engagement of PFC across multiple tasks, in contrast with more task-specific mnemonic encoding in PPC.

Inferring learning rules from distributions of firing rates in cortical neurons

Information about external stimuli is thought to be stored in cortical circuits through experience-dependent modifications of synaptic connectivity. These modifications of network connectivity should lead to changes in neuronal activity as a particular stimulus is repeatedly encountered. Here we ask what plasticity rules are consistent with the differences in the statistics of the visual response to novel and familiar stimuli in inferior temporal cortex, an area underlying visual object recognition. We introduce a method that allows one to infer the dependence of the presumptive learning rule on postsynaptic firing rate, and we show that the inferred learning rule exhibits depression for low postsynaptic rates and potentiation for high rates. The threshold separating depression from potentiation is strongly correlated with both mean and s.d. of the firing rate distribution. Finally, we show that network models implementing a rule extracted from data show stable learning dynamics and lead to sparser representations of stimuli.

Generalization in category learning: the roles of representational and decisional uncertainty

Effective generalization in a multiple-category situation involves both assessing potential membership in individual categories and resolving conflict between categories while implementing a decision bound. We separated generalization from decision bound implementation using an information integration task in which category exemplars varied over two incommensurable feature dimensions. Human subjects first learned to categorize stimuli within limited training regions, and then, during fMRI scanning, they also categorized transfer stimuli from new regions of perceptual space. Transfer stimuli differed both in distance from the training region prototype and distance from the decision bound, allowing us to independently assess neural systems sensitive to each. Across all stimulus regions, categorization was associated with activity in the extrastriate visual cortex, basal ganglia, and the bilateral intraparietal sulcus. Categorizing stimuli near the decision bound was associated with recruitment of the frontoinsular cortex and medial frontal cortex, regions often associated with conflict and which commonly coactivate within the salience network. Generalization was measured in terms of greater distance from the decision bound and greater distance from the category prototype (average training region stimulus). Distance from the decision bound was associated with activity in the superior parietal lobe, lingual gyri, and anterior hippocampus, whereas distance from the prototype was associated with left intraparietal sulcus activity. The results are interpreted as supporting the existence of different uncertainty resolution mechanisms for uncertainty about category membership (representational uncertainty) and uncertainty about decision bound (decisional uncertainty).