Neuronal activity in the visual cortex reveals the temporal order of cognitive operations

Why motor imagery is not really motoric: towards a re-conceptualization in terms of effect-based action control

Overt and imagined action seem inextricably linked. Both have similar timing, activate shared brain circuits, and motor imagery influences overt action and vice versa. Motor imagery is, therefore, often assumed to recruit the same motor processes that govern action execution, and which allow one to play through or simulate actions offline. Here, we advance a very different conceptualization. Accordingly, the links between imagery and overt action do not arise because action imagery is intrinsically motoric, but because action planning is intrinsically imaginistic and occurs in terms of the perceptual effects one want to achieve. Seen like this, the term 'motor imagery' is a misnomer of what is more appropriately portrayed as 'effect imagery'. In this article, we review the long-standing arguments for effect-based accounts of action, which are often ignored in motor imagery research. We show that such views provide a straightforward account of motor imagery. We review the evidence for imagery-execution overlaps through this new lens and argue that they indeed emerge because every action we execute is planned, initiated and controlled through an imagery-like process. We highlight findings that this new view can now explain and point out open questions.

Recurrent neural networks that learn multi-step visual routines with reinforcement learning

Many cognitive problems can be decomposed into series of subproblems that are solved sequentially by the brain. When subproblems are solved, relevant intermediate results need to be stored by neurons and propagated to the next subproblem, until the overarching goal has been completed. We will here consider visual tasks, which can be decomposed into sequences of elemental visual operations. Experimental evidence suggests that intermediate results of the elemental operations are stored in working memory as an enhancement of neural activity in the visual cortex. The focus of enhanced activity is then available for subsequent operations to act upon. The main question at stake is how the elemental operations and their sequencing can emerge in neural networks that are trained with only rewards, in a reinforcement learning setting. We here propose a new recurrent neural network architecture that can learn composite visual tasks that require the application of successive elemental operations. Specifically, we selected three tasks for which electrophysiological recordings of monkeys' visual cortex are available. To train the networks, we used RELEARNN, a biologically plausible four-factor Hebbian learning rule, which is local both in time and space. We report that networks learn elemental operations, such as contour grouping and visual search, and execute sequences of operations, solely based on the characteristics of the visual stimuli and the reward structure of a task. After training was completed, the activity of the units of the neural network elicited by behaviorally relevant image items was stronger than that elicited by irrelevant ones, just as has been observed in the visual cortex of monkeys solving the same tasks. Relevant information that needed to be exchanged between subroutines was maintained as a focus of enhanced activity and passed on to the subsequent subroutines. Our results demonstrate how a biologically plausible learning rule can train a recurrent neural network on multistep visual tasks.

A selection and targeting framework of cortical locations for line-scanning fMRI

Depth-resolved functional magnetic resonance imaging (fMRI) is an emerging field growing in popularity given the potential of separating signals from different computational processes in cerebral cortex. Conventional acquisition schemes suffer from low spatial and temporal resolutions. Line-scanning methods allow depth-resolved fMRI by sacrificing spatial coverage to sample blood oxygenated level-dependent (BOLD) responses at ultra-high temporal and spatial resolution. For neuroscience applications, it is critical to be able to place the line accurately to (1) sample the right neural population and (2) target that neural population with tailored stimuli or tasks. To this end, we devised a multi-session framework where a target cortical location is selected based on anatomical and functional properties. The line is then positioned according to this information in a separate second session, and we tailor the experiment to focus on the target location. Anatomically, the precision of the line placement was confirmed by projecting a nominal representation of the acquired line back onto the surface. Functional estimates of neural selectivities in the line, as quantified by a visual population-receptive field model, resembled the target selectivities well for most subjects. This functional precision was quantified in detail by estimating the distance between the visual field location of the targeted vertex and the location in visual cortex (V1) that most closely resembled the line-scanning estimates; this distance was on average ~5.5 mm. Given the dimensions of the line, differences in acquisition, session, and stimulus design, this validates that line-scanning can be used to probe local neural sensitivities across sessions. In summary, we present an accurate framework for line-scanning MRI; we believe such a framework is required to harness the full potential of line-scanning and maximize its utility. Furthermore, this approach bridges canonical fMRI experiments with electrophysiological experiments, which in turn allows novel avenues for studying human physiology non-invasively.

Solving the binding problem: Assemblies form when neurons enhance their firing rate-they don't need to oscillate or synchronize

When we look at an image, its features are represented in our visual system in a highly distributed manner, calling for a mechanism that binds them into coherent object representations. There have been different proposals for the neuronal mechanisms that can mediate binding. One hypothesis is that binding is achieved by oscillations that synchronize neurons representing features of the same perceptual object. This view allows separate communication channels between different brain areas. Another hypothesis is that binding of features that are represented in different brain regions occurs when the neurons in these areas that respond to the same object simultaneously enhance their firing rate, which would correspond to directing object-based attention to these features. This review summarizes evidence in favor of and against these two hypotheses, examining the neuronal correlates of binding and assessing the time course of perceptual grouping. I conclude that enhanced neuronal firing rates bind features into coherent object representations, whereas oscillations and synchrony are unrelated to binding.

Inversion of pop-out for a distracting feature dimension in monkey visual cortex

During visual search, it is important to reduce the interference of distracting objects in the scene. The neuronal responses elicited by the search target stimulus are typically enhanced. However, it is equally important to suppress the representations of distracting stimuli, especially if they are salient and capture attention. We trained monkeys to make an eye movement to a unique "pop-out" shape stimulus among an array of distracting stimuli. One of these distractors had a salient color that varied across trials and differed from the color of the other stimuli, causing it to also pop-out. The monkeys were able to select the pop-out shape target with high accuracy and actively avoided the pop-out color distractor. This behavioral pattern was reflected in the activity of neurons in area V4. Responses to the shape targets were enhanced, while the activity evoked by the pop-out color distractor was only briefly enhanced, directly followed by a sustained period of pronounced suppression. These behavioral and neuronal results demonstrate a cortical selection mechanism that rapidly inverts a pop-out signal to "pop-in" for an entire feature dimension thereby facilitating goal-directed visual search in the presence of salient distractors.

Transcranial direct current stimulation of the occipital lobes with adjunct lithium attenuates the progression of cognitive impairment in patients with first episode schizophrenia

BACKGROUND

There is no standard effective treatment for schizophrenia-associated cognitive impairment. Efforts to use non-invasive brain stimulation for this purpose have been focused mostly on the frontal cortex, with little attention being given to the occipital lobe.

MATERIALS AND METHODS

We compared the effects of nine intervention strategies on cognitive performance in psychometric measures and brain connectivity measured obtained from functional magnetic resonance imaging analyses. The strategies consisted of transcranial direct current stimulation (t-DCS) or repetitive transcranial magnetic stimulation (r-TMS) of the frontal lobe or of the occipital alone or with adjunct lithium, or lithium monotherapy. We measured global functional connectivity density (gFCD) voxel-wise.

RESULTS

Although all nine patient groups showed significant improvements in global disability scores (GDSs) following the intervention period (vs. before), the greatest improvement in GDS was observed for the group that received occipital lobe-targeted t-DCS with adjunct lithium therapy. tDCS of the occipital lobe improved gFCD throughout the brain, including in the frontal lobes, whereas stimulation of the frontal lobes had less far-reaching benefits on gFCD in the brain. Adverse secondary effects (ASEs) such as heading, dizziness, and nausea, were commonly experienced by patients treated with t-DCS and r-TMS, with or without lithium, whereas ASEs were rare with lithium alone.

CONCLUSION

The most effective treatment strategy for impacting cognitive impairment and brain communication was t-DCS stimulation of the occipital lobe with adjunct lithium therapy, though patients often experienced headache with dizziness and nausea after treatment sessions.

How biological attention mechanisms improve task performance in a large-scale visual system model

How does attentional modulation of neural activity enhance performance? Here we use a deep convolutional neural network as a large-scale model of the visual system to address this question. We model the feature similarity gain model of attention, in which attentional modulation is applied according to neural stimulus tuning. Using a variety of visual tasks, we show that neural modulations of the kind and magnitude observed experimentally lead to performance changes of the kind and magnitude observed experimentally. We find that, at earlier layers, attention applied according to tuning does not successfully propagate through the network, and has a weaker impact on performance than attention applied according to values computed for optimally modulating higher areas. This raises the question of whether biological attention might be applied at least in part to optimize function rather than strictly according to tuning. We suggest a simple experiment to distinguish these alternatives.

Layer-specificity in the effects of attention and working memory on activity in primary visual cortex

Neuronal activity in early visual cortex depends on attention shifts but the contribution to working memory has remained unclear. Here, we examine neuronal activity in the different layers of the primary visual cortex (V1) in an attention-demanding and a working memory task. A current-source density analysis reveales top-down inputs in the superficial layers and layer 5, and an increase in neuronal firing rates most pronounced in the superficial and deep layers and weaker in input layer 4. This increased activity is strongest in the attention task but it is also highly reliable during working memory delays. A visual mask erases the V1 memory activity, but it reappeares at a later point in time. These results provide new insights in the laminar circuits involved in the top-down modulation of activity in early visual cortex in the presence and absence of visual stimuli.

Noncommutative Biology: Sequential Regulation of Complex Networks

Single-cell variability in gene expression is important for generating distinct cell types, but it is unclear how cells use the same set of regulatory molecules to specifically control similarly regulated genes. While combinatorial binding of transcription factors at promoters has been proposed as a solution for cell-type specific gene expression, we found that such models resulted in substantial information bottlenecks. We sought to understand the consequences of adopting sequential logic wherein the time-ordering of factors informs the final outcome. We showed that with noncommutative control, it is possible to independently control targets that would otherwise be activated simultaneously using combinatorial logic. Consequently, sequential logic overcomes the information bottleneck inherent in complex networks. We derived scaling laws for two noncommutative models of regulation, motivated by phosphorylation/neural networks and chromosome folding, respectively, and showed that they scale super-exponentially in the number of regulators. We also showed that specificity in control is robust to the loss of a regulator. Lastly, we connected these theoretical results to real biological networks that demonstrate specificity in the context of promiscuity. These results show that achieving a desired outcome often necessitates roundabout steps.

DNA is the blueprint of life. Yet the order in which a cell follows these instructions makes it capable of generating thousands of different fates. How this information is extracted from underlying gene regulatory networks is unclear, especially given that biological networks are highly interconnected, and that the number of signaling pathways is relatively small (approximately 5–10). The conventional approach for increasing the information capacity of a limited set of regulators is to use them in combination. Surprisingly, combinatorial logic does not increase the diversity of target configurations or cell fates, but instead causes information bottlenecks. A different approach, called sequential logic, uses noncommutative sequences of a small set of regulators to drive networks to a large number of novel configurations. If certain targets are first protected, then even promiscuous regulators can activate specific subsets of lineage-specific targets. In this paper we show how sequential logic outperforms combinatorial logic, and argue that noncommutative sequences underlie a number of cases of biological regulation, e.g. how a small number of signaling pathways generates a large diversity of cell types in development. In addition to explaining biological networks, sequential logic may be a general experimental design strategy in synthetic and single-cell biology.

Early Visual Cortex as a Multiscale Cognitive Blackboard

Neurons in early visual cortical areas not only represent incoming visual information but are also engaged by higher level cognitive processes, including attention, working memory, imagery, and decision-making. Are these cognitive effects an epiphenomenon or are they functionally relevant for these mental operations? We review evidence supporting the hypothesis that the modulation of activity in early visual areas has a causal role in cognition. The modulatory influences allow the early visual cortex to act as a multiscale cognitive blackboard for read and write operations by higher visual areas, which can thereby efficiently exchange information. This blackboard architecture explains how the activity of neurons in the early visual cortex contributes to scene segmentation and working memory, and relates to the subject's inferences about the visual world. The architecture also has distinct advantages for the processing of visual routines that rely on a number of sequentially executed processing steps.

Learning a New Selection Rule in Visual and Frontal Cortex

How do you make a decision if you do not know the rules of the game? Models of sensory decision-making suggest that choices are slow if evidence is weak, but they may only apply if the subject knows the task rules. Here, we asked how the learning of a new rule influences neuronal activity in the visual (area V1) and frontal cortex (area FEF) of monkeys. We devised a new icon-selection task. On each day, the monkeys saw 2 new icons (small pictures) and learned which one was relevant. We rewarded eye movements to a saccade target connected to the relevant icon with a curve. Neurons in visual and frontal cortex coded the monkey's choice, because the representation of the selected curve was enhanced. Learning delayed the neuronal selection signals and we uncovered the cause of this delay in V1, where learning to select the relevant icon caused an early suppression of surrounding image elements. These results demonstrate that the learning of a new rule causes a transition from fast and random decisions to a more considerate strategy that takes additional time and they reveal the contribution of visual and frontal cortex to the learning process.

Reinforcement Learning of Linking and Tracing Contours in Recurrent Neural Networks

The processing of a visual stimulus can be subdivided into a number of stages. Upon stimulus presentation there is an early phase of feedforward processing where the visual information is propagated from lower to higher visual areas for the extraction of basic and complex stimulus features. This is followed by a later phase where horizontal connections within areas and feedback connections from higher areas back to lower areas come into play. In this later phase, image elements that are behaviorally relevant are grouped by Gestalt grouping rules and are labeled in the cortex with enhanced neuronal activity (object-based attention in psychology). Recent neurophysiological studies revealed that reward-based learning influences these recurrent grouping processes, but it is not well understood how rewards train recurrent circuits for perceptual organization. This paper examines the mechanisms for reward-based learning of new grouping rules. We derive a learning rule that can explain how rewards influence the information flow through feedforward, horizontal and feedback connections. We illustrate the efficiency with two tasks that have been used to study the neuronal correlates of perceptual organization in early visual cortex. The first task is called contour-integration and demands the integration of collinear contour elements into an elongated curve. We show how reward-based learning causes an enhancement of the representation of the to-be-grouped elements at early levels of a recurrent neural network, just as is observed in the visual cortex of monkeys. The second task is curve-tracing where the aim is to determine the endpoint of an elongated curve composed of connected image elements. If trained with the new learning rule, neural networks learn to propagate enhanced activity over the curve, in accordance with neurophysiological data. We close the paper with a number of model predictions that can be tested in future neurophysiological and computational studies.

Our experience with the visual world allows us to group image elements that belong to the same perceptual object and to segregate them from other objects and the background. If subjects learn to group contour elements, this experience influences neuronal activity in early visual cortical areas, including the primary visual cortex (V1). Learning presumably depends on alterations in the pattern of connections within and between areas of the visual cortex. However, the processes that control changes in connectivity are not well understood. Here we present the first computational model that can train a neural network to integrate collinear contour elements into elongated curves and to trace a curve through the visual field. The new learning algorithm trains fully recurrent neural networks, provided the connectivity causes the networks to reach a stable state. The model reproduces the behavioral performance of monkeys trained in these tasks and explains the patterns of neuronal activity in the visual cortex that emerge during learning, which is remarkable because the only feedback for the model is a reward for successful trials. We discuss a number of the model predictions that can be tested in future neuroscientific work.

The Formation of Hierarchical Decisions in the Visual Cortex

Intelligence relies on our ability to find appropriate sequences of decisions in complex problem spaces. The efficiency of a problem solver depends on the speed of its individual decisions and the number of decisions it can explore in parallel. It remains unknown whether the primate brain can consider multiple decisions at the same time. We therefore trained monkeys to navigate through a decision tree with stochastic sensory evidence at multiple branching points and recorded neuronal activity in visual cortical areas V1 and V4. We found a first phase of decision making in which neuronal activity increased in parallel along multiple branches of the decision tree. This was followed by an integration phase where the optimal overall strategy crystallized as the result of interactions between local decisions. The results reveal how sensory evidence is integrated efficiently for hierarchical decisions and contribute to our understanding of the brain mechanisms that implement complex mental programs.

Effects of attention and distractor contrast on the responses of middle temporal area neurons to transient motion direction changes

The ability of primates to detect transient changes in a visual scene can be influenced by the allocation of attention, as well as by the presence of distractors. We investigated the neural substrates of these effects by recording the responses of neurons in the middle temporal area (MT) of two monkeys while they detected a transient motion direction change in a moving target. We found that positioning a distractor near the target impaired the change-detection performance of the animals. This impairment monotonically decreased as the distractor's contrast decreased. A neural correlate of this effect was a decrease in the ability of MT neurons to signal the direction change (detection sensitivity or DS) when a distractor was near the target, both located inside the neuron's receptive field. Moreover, decreasing distractor contrast increased neuronal DS. On the other hand, directing attention away from the target decreased neuronal DS. At the level of individual neurons, we found a negative correlation between the degree of response normalization and the DS. Finally, the intensity of a neuron's response to the change was predictive of the animal's reaction time, suggesting that the activity of our recorded neurons was linked to the animal's detection performance. Our results suggest that the ability of an MT neuron to signal a transient direction change is regulated by the degree of inhibitory drive into the cell. The presence of distractors, their contrast and the allocation of attention influence such inhibitory drive, therefore modulating the ability of the neurons to signal transient changes in stimulus features and consequently behavioral performance.

Simultaneous selection by object-based attention in visual and frontal cortex

Models of visual attention hold that top-down signals from frontal cortex influence information processing in visual cortex. It is unknown whether situations exist in which visual cortex actively participates in attentional selection. To investigate this question, we simultaneously recorded neuronal activity in the frontal eye fields (FEF) and primary visual cortex (V1) during a curve-tracing task in which attention shifts are object-based. We found that accurate performance was associated with similar latencies of attentional selection in both areas and that the latency in both areas increased if the task was made more difficult. The amplitude of the attentional signals in V1 saturated early during a trial, whereas these selection signals kept increasing for a longer time in FEF, until the moment of an eye movement, as if FEF integrated attentional signals present in early visual cortex. In erroneous trials, we observed an interareal latency difference because FEF selected the wrong curve before V1 and imposed its erroneous decision onto visual cortex. The neuronal activity in visual and frontal cortices was correlated across trials, and this trial-to-trial coupling was strongest for the attended curve. These results imply that selective attention relies on reciprocal interactions within a large network of areas that includes V1 and FEF.

The functional link between area MT neural fluctuations and detection of a brief motion stimulus

Fluctuations of neural firing rates in visual cortex are known to be correlated with variations in perceptual performance. It is important to know whether these fluctuations are functionally linked to perception in a causal manner or instead reflect non-causal processes that arise after the perceptual decision is made. We recorded from middle temporal (MT) neurons from monkey subjects while they detected the random occurrence of a brief 50 ms motion pulse that occurred in either of two (or simultaneously in both) random dot patches located in the same hemisphere. The receptive field parameters of the motion pulse were matched to that preferred by each MT neuron under study. This task contained uncertainty in both space and time because, on any given trial, the subjects did not know which patch would contain the motion pulse or when the motion pulse would occur. Covariations between MT activity and behavior began just before the motion pulse onset and peaked at the maximum neural response. These neural-behavioral covariations were strongest when only one patch contained the motion pulse and were still weakly present when a patch did not contain a motion pulse. A feedforward temporal integration model with two independent detector channels captured both the detection performance and evolution of the neural-behavior covariations over time and stimulus condition. The results suggest that, when detecting a brief visual stimulus, there is a causal relationship between fluctuations in neural activity and variations in behavior across trials.

The human Turing machine: a neural framework for mental programs

In recent years much has been learned about how a single computational processing step is implemented in the brain. By contrast, we still have surprisingly little knowledge of the neuronal mechanisms by which multiple such operations are sequentially assembled into mental algorithms. We outline a theory of how individual neural processing steps might be combined into serial programs. We propose a hybrid neuronal device: each step involves massively parallel computation that feeds a slow and serial production system. Production selection is mediated by a system of competing accumulator neurons that extends the role of these neurons beyond the selection of a motor action. Productions change the state of sensory and mnemonic neurons and iteration of such cycles provides a basis for mental programs.

Different states in visual working memory: when it guides attention and when it does not

Recent studies have revealed a strong relationship between visual working memory and selective attention, such that attention is biased by what is currently on our mind. However, other data show that not all memorized items influence the deployment of attention, thus calling for a distinction within working memory: whereas active memory items function as an attentional template and directly affect perception, other, accessory items do not. We review recent evidence that items compete for the status of 'attentional template' that contains only one object at a time. Neurophysiological results provide insight into these different memory states by revealing a more intricate organization of working memory than was previously thought.