Attention-gated reinforcement learning of internal representations for classification

Desiderata for Normative Models of Synaptic Plasticity

Normative models of synaptic plasticity use computational rationales to arrive at predictions of behavioral and network-level adaptive phenomena. In recent years, there has been an explosion of theoretical work in this realm, but experimental confirmation remains limited. In this review, we organize work on normative plasticity models in terms of a set of desiderata that, when satisfied, are designed to ensure that a given model demonstrates a clear link between plasticity and adaptive behavior, is consistent with known biological evidence about neural plasticity and yields specific testable predictions. As a prototype, we include a detailed analysis of the REINFORCE algorithm. We also discuss how new models have begun to improve on the identified criteria and suggest avenues for further development. Overall, we provide a conceptual guide to help develop neural learning theories that are precise, powerful, and experimentally testable.

Recurrent neural networks that learn multi-step visual routines with reinforcement learning

Many cognitive problems can be decomposed into series of subproblems that are solved sequentially by the brain. When subproblems are solved, relevant intermediate results need to be stored by neurons and propagated to the next subproblem, until the overarching goal has been completed. We will here consider visual tasks, which can be decomposed into sequences of elemental visual operations. Experimental evidence suggests that intermediate results of the elemental operations are stored in working memory as an enhancement of neural activity in the visual cortex. The focus of enhanced activity is then available for subsequent operations to act upon. The main question at stake is how the elemental operations and their sequencing can emerge in neural networks that are trained with only rewards, in a reinforcement learning setting. We here propose a new recurrent neural network architecture that can learn composite visual tasks that require the application of successive elemental operations. Specifically, we selected three tasks for which electrophysiological recordings of monkeys' visual cortex are available. To train the networks, we used RELEARNN, a biologically plausible four-factor Hebbian learning rule, which is local both in time and space. We report that networks learn elemental operations, such as contour grouping and visual search, and execute sequences of operations, solely based on the characteristics of the visual stimuli and the reward structure of a task. After training was completed, the activity of the units of the neural network elicited by behaviorally relevant image items was stronger than that elicited by irrelevant ones, just as has been observed in the visual cortex of monkeys solving the same tasks. Relevant information that needed to be exchanged between subroutines was maintained as a focus of enhanced activity and passed on to the subsequent subroutines. Our results demonstrate how a biologically plausible learning rule can train a recurrent neural network on multistep visual tasks.

Profiles of visual perceptual learning in feature space

Visual perceptual learning (VPL), experience-induced gains in discriminating visual features, has been studied extensively and intensively for many years, its profile in feature space, however, remains unclear. Here, human subjects were trained to perform either a simple low-level feature (grating orientation) or a complex high-level object (face view) discrimination task over a long-time course. During, immediately after, and one month after training, all results showed that in feature space VPL in grating orientation discrimination was a center-surround profile; VPL in face view discrimination, however, was a monotonic gradient profile. Importantly, these two profiles can be emerged by a deep convolutional neural network with a modified AlexNet consisted of 7 and 12 layers, respectively. Altogether, our study reveals for the first time a feature hierarchy-dependent profile of VPL in feature space, placing a necessary constraint on our understanding of the neural computation of VPL.

An Agent-Based Model to Reproduce the Boolean Logic Behaviour of Neuronal Self-Organised Communities through Pulse Delay Modulation and Generation of Logic Gates

The human brain is arguably the most complex "machine" to ever exist. Its detailed functioning is yet to be fully understood, let alone modelled. Neurological processes have logical signal-processing and biophysical aspects, and both affect the brain's structure, functioning and adaptation. Mathematical approaches based on both information and graph theory have been extensively used in an attempt to approximate its biological functioning, along with Artificial Intelligence frameworks inspired by its logical functioning. In this article, an approach to model some aspects of the brain learning and signal processing is presented, mimicking the metastability and backpropagation found in the real brain while also accounting for neuroplasticity. Several simulations are carried out with this model to demonstrate how dynamic neuroplasticity, neural inhibition and neuron migration can reshape the brain's logical connectivity to synchronise signal processing and obtain certain target latencies. This work showcases the importance of dynamic logical and biophysical remodelling in brain plasticity. Combining mathematical (agents, graph theory, topology and backpropagation) and biomedical ingredients (metastability, neuroplasticity and migration), these preliminary results prove complex brain phenomena can be reproduced-under pertinent simplifications-via affordable computations, which can be construed as a starting point for more ambitiously accurate simulations.

Silences, spikes and bursts: Three-part knot of the neural code

When a neuron breaks silence, it can emit action potentials in a number of patterns. Some responses are so sudden and intense that electrophysiologists felt the need to single them out, labelling action potentials emitted at a particularly high frequency with a metonym - bursts. Is there more to bursts than a figure of speech? After all, sudden bouts of high-frequency firing are expected to occur whenever inputs surge. The burst coding hypothesis advances that the neural code has three syllables: silences, spikes and bursts. We review evidence supporting this ternary code in terms of devoted mechanisms for burst generation, synaptic transmission and synaptic plasticity. We also review the learning and attention theories for which such a triad is beneficial.

Distinct Patterns of Connectivity between Brain Regions Underlie the Intra-Modal and Cross-Modal Value-Driven Modulations of the Visual Cortex

Past reward associations may be signaled from different sensory modalities; however, it remains unclear how different types of reward-associated stimuli modulate sensory perception. In this human fMRI study (female and male participants), a visual target was simultaneously presented with either an intra- (visual) or a cross-modal (auditory) cue that was previously associated with rewards. We hypothesized that, depending on the sensory modality of the cues, distinct neural mechanisms underlie the value-driven modulation of visual processing. Using a multivariate approach, we confirmed that reward-associated cues enhanced the target representation in early visual areas and identified the brain valuation regions. Then, using an effective connectivity analysis, we tested three possible patterns of connectivity that could underlie the modulation of the visual cortex: a direct pathway from the frontal valuation areas to the visual areas, a mediated pathway through the attention-related areas, and a mediated pathway that additionally involved sensory association areas. We found evidence for the third model demonstrating that the reward-related information in both sensory modalities is communicated across the valuation and attention-related brain regions. Additionally, the superior temporal areas were recruited when reward was cued cross-modally. The strongest dissociation between the intra- and cross-modal reward-driven effects was observed at the level of the feedforward and feedback connections of the visual cortex estimated from the winning model. These results suggest that, in the presence of previously rewarded stimuli from different sensory modalities, a combination of domain-general and domain-specific mechanisms are recruited across the brain to adjust the visual perception.SIGNIFICANCE STATEMENT Reward has a profound effect on perception, but it is not known whether shared or disparate mechanisms underlie the reward-driven effects across sensory modalities. In this human fMRI study, we examined the reward-driven modulation of the visual cortex by visual (intra-modal) and auditory (cross-modal) reward-associated cues. Using a model-based approach to identify the most plausible pattern of inter-regional effective connectivity, we found that higher-order areas involved in the valuation and attentional processing were recruited by both types of rewards. However, the pattern of connectivity between these areas and the early visual cortex was distinct between the intra- and cross-modal rewards. This evidence suggests that, to effectively adapt to the environment, reward signals may recruit both domain-general and domain-specific mechanisms.

Kernel Reinforcement Learning-Assisted Adaptive Decoder Facilitates Stable and Continuous Brain Control Tasks

Brain-Machine Interfaces (BMIs) assist paralyzed people to brain control (BC) the neuro-prosthesis continuously moving in space. During the BC process, the subject imagines the movement of the real limb and adapts the brain activity according to the sensory feedback. The neural adaptation in the closed-loop control results in complex and changing brain signals. Simultaneously, the decoder interprets the time-varying functional mapping between neural activity and continuous trajectory. It is crucial and challenging to accurately and adaptively track the mapping to help the subject accomplish the BC task with a stable performance. Existing Kalman Filter (KF) based decoders achieve continuous trajectory control by linearly interpreting neural firing observations into self-evolving prosthetic states. However, the linear neural-state mapping might not accurately reflect the movement intention of the subject. In this paper, we propose a novel method that allows subjects to achieve continuous brain control efficiently and stably. The proposed method incorporates a kernel reinforcement learning method into a state-observation model to decode the nonlinearly neural observation into a continuous trajectory state. The state transition function ensures the continuity of the prosthetic state. The kernel reinforcement learning allows the quick adaptation of the nonlinear neural-movement mapping during the BC process. The proposed method is tested in an online brain control reaching task for rats. Compared with KF, our method achieved more successful trials, faster response time, shorter inter-trial time, and remained stable over days. These results demonstrate that the proposed method is an efficient tool to assist subjects in brain control tasks.

Audio-induced medial prefrontal cortical dynamics enhances coadaptive learning in brain-machine interfaces

Objectives. Coadaptive brain-machine interfaces (BMIs) allow subjects and external devices to adapt to each other during the closed-loop control, which provides a promising solution for paralyzed individuals. Previous studies have focused on either improving sensory feedback to facilitate subject learning or developing adaptive algorithms to maintain stable decoder performance. In this work, we aim to design an efficient coadaptive BMI framework which not only facilitates the learning of subjects on new tasks with designed sensory feedback, but also improves decoders' learning ability by extracting sensory feedback-induced evaluation information.Approach. We designed dynamic audio feedback during the trial according to the subjects' performance when they were trained to learn a new behavioral task. We compared the learning performance of two groups of Sprague Dawley rats, one with and the other without the designed audio feedback to show whether this audio feedback could facilitate the subjects' learning. Compared with the traditional closed-loop in BMI systems, an additional closed-loop involving medial prefrontal cortex (mPFC) activity was introduced into the coadaptive framework. The neural dynamics of audio-induced mPFC activity was analyzed to investigate whether a significant neural response could be triggered. This audio-induced response was then translated into reward expectation information to guide the learning of decoders on a new task. The multiday decoding performance of the decoders with and without audio-induced reward expectation was compared to investigate whether the extracted information could accelerate decoders to learn a new task.Main results. The behavior performance comparison showed that the average days for rats to achieve 80% well-trained behavioral performance was improved by 26.4% after introducing the designed audio feedback sequence. The analysis of neural dynamics showed that a significant neural response of mPFC activity could be elicited by the audio feedback and the visualization of audio-induced neural patterns was emerged and accompanied by the behavioral improvement of subjects. The multiday decoding performance comparison showed that the decoder taking the reward expectation information could achieve faster task learning by 33.8% on average across subjects.Significance. This study demonstrates that the designed audio feedback could improve the learning of subjects and the mPFC activity induced by audio feedback can be utilized to improve the decoder's learning efficiency on new tasks. The coadaptive framework involving mPFC dynamics in the closed-loop interaction can advance the BMIs into a more adaptive and efficient system with learning ability on new tasks.

Convolutional neural networks for vision neuroscience: significance, developments, and outstanding issues

Convolutional Neural Networks (CNN) are a class of machine learning models predominately used in computer vision tasks and can achieve human-like performance through learning from experience. Their striking similarities to the structural and functional principles of the primate visual system allow for comparisons between these artificial networks and their biological counterparts, enabling exploration of how visual functions and neural representations may emerge in the real brain from a limited set of computational principles. After considering the basic features of CNNs, we discuss the opportunities and challenges of endorsing CNNs as in silico models of the primate visual system. Specifically, we highlight several emerging notions about the anatomical and physiological properties of the visual system that still need to be systematically integrated into current CNN models. These tenets include the implementation of parallel processing pathways from the early stages of retinal input and the reconsideration of several assumptions concerning the serial progression of information flow. We suggest design choices and architectural constraints that could facilitate a closer alignment with biology provide causal evidence of the predictive link between the artificial and biological visual systems. Adopting this principled perspective could potentially lead to new research questions and applications of CNNs beyond modeling object recognition.

Efficient coding theory of dynamic attentional modulation

Activity of sensory neurons is driven not only by external stimuli but also by feedback signals from higher brain areas. Attention is one particularly important internal signal whose presumed role is to modulate sensory representations such that they only encode information currently relevant to the organism at minimal cost. This hypothesis has, however, not yet been expressed in a normative computational framework. Here, by building on normative principles of probabilistic inference and efficient coding, we developed a model of dynamic population coding in the visual cortex. By continuously adapting the sensory code to changing demands of the perceptual observer, an attention-like modulation emerges. This modulation can dramatically reduce the amount of neural activity without deteriorating the accuracy of task-specific inferences. Our results suggest that a range of seemingly disparate cortical phenomena such as intrinsic gain modulation, attention-related tuning modulation, and response variability could be manifestations of the same underlying principles, which combine efficient sensory coding with optimal probabilistic inference in dynamic environments.

Thalamocortical contribution to flexible learning in neural systems

Animal brains evolved to optimize behavior in dynamic environments, flexibly selecting actions that maximize future rewards in different contexts. A large body of experimental work indicates that such optimization changes the wiring of neural circuits, appropriately mapping environmental input onto behavioral outputs. A major unsolved scientific question is how optimal wiring adjustments, which must target the connections responsible for rewards, can be accomplished when the relation between sensory inputs, action taken, and environmental context with rewards is ambiguous. The credit assignment problem can be categorized into context-independent structural credit assignment and context-dependent continual learning. In this perspective, we survey prior approaches to these two problems and advance the notion that the brain's specialized neural architectures provide efficient solutions. Within this framework, the thalamus with its cortical and basal ganglia interactions serves as a systems-level solution to credit assignment. Specifically, we propose that thalamocortical interaction is the locus of meta-learning where the thalamus provides cortical control functions that parametrize the cortical activity association space. By selecting among these control functions, the basal ganglia hierarchically guide thalamocortical plasticity across two timescales to enable meta-learning. The faster timescale establishes contextual associations to enable behavioral flexibility, while the slower one enables generalization to new contexts.

Neural Decoders Using Reinforcement Learning in Brain Machine Interfaces: A Technical Review

Creating flexible and robust brain machine interfaces (BMIs) is currently a popular topic of research that has been explored for decades in medicine, engineering, commercial, and machine-learning communities. In particular, the use of techniques using reinforcement learning (RL) has demonstrated impressive results but is under-represented in the BMI community. To shine more light on this promising relationship, this article aims to provide an exhaustive review of RL's applications to BMIs. Our primary focus in this review is to provide a technical summary of various algorithms used in RL-based BMIs to decode neural intention, without emphasizing preprocessing techniques on the neural signals and reward modeling for RL. We first organize the literature based on the type of RL methods used for neural decoding, and then each algorithm's learning strategy is explained along with its application in BMIs. A comparative analysis highlighting the similarities and uniqueness among neural decoders is provided. Finally, we end this review with a discussion about the current stage of RLBMIs including their limitations and promising directions for future research.

Estimating Reward Function from Medial Prefrontal Cortex Cortical Activity using Inverse Reinforcement Learning

Reinforcement learning (RL)-based brain-machine interfaces (BMIs) learn the mapping from neural signals to subjects' intention using a reward signal. External rewards (water or food) or internal rewards extracted from neural activity are leveraged to update the parameters of decoders in the existing RL-based BMI framework. However, for complex tasks, the design of external reward could be difficult, which may not fully reflect the subject's own evaluation internally. It is important to obtain an internal reward model from neural activity to access subject's internal evaluation when the subject is performing the task through trial and error. In this paper, we propose to use an inverse reinforcement learning (IRL) method to estimate the internal reward function interpreted from the brain to assist the update of the decoders. Specifically, the inverse Q-learning (IQL) algorithm is applied to extract internal reward information from real data collected from medial prefrontal cortex (mPFC) when a rat was learning a two-lever-press discrimination task. Such an internal reward information is validated by checking whether it can guide the training of the RL decoder to complete movement task. Compared with the RL decoder trained with the external reward, our approach achieves a similar decoding performance. This preliminary result validates the effectiveness of using IRL to obtain the internal reward model. It reveals the potential of estimating internal reward model to improve the design of autonomous learning BMIs.

Error-based or target-based? A unified framework for learning in recurrent spiking networks

The field of recurrent neural networks is over-populated by a variety of proposed learning rules and protocols. The scope of this work is to define a generalized framework, to move a step forward towards the unification of this fragmented scenario. In the field of supervised learning, two opposite approaches stand out, error-based and target-based. This duality gave rise to a scientific debate on which learning framework is the most likely to be implemented in biological networks of neurons. Moreover, the existence of spikes raises the question of whether the coding of information is rate-based or spike-based. To face these questions, we proposed a learning model with two main parameters, the rank of the feedback learning matrix R and the tolerance to spike timing τ_⋆. We demonstrate that a low (high) rank R accounts for an error-based (target-based) learning rule, while high (low) tolerance to spike timing promotes rate-based (spike-based) coding. We show that in a store and recall task, high-ranks allow for lower MSE values, while low-ranks enable a faster convergence. Our framework naturally lends itself to Behavioral Cloning and allows for efficiently solving relevant closed-loop tasks, investigating what parameters (R,τ⋆) are optimal to solve a specific task. We found that a high R is essential for tasks that require retaining memory for a long time (Button and Food). On the other hand, this is not relevant for a motor task (the 2D Bipedal Walker). In this case, we find that precise spike-based coding enables optimal performances. Finally, we show that our theoretical formulation allows for defining protocols to estimate the rank of the feedback error in biological networks. We release a PyTorch implementation of our model supporting GPU parallelization.

Learning in biological or artificial networks means changing the laws governing the network dynamics in order to better behave in a specific situation. However, there exists no consensus on what rules regulate learning in biological systems. To face these questions, we propose a novel theoretical formulation for learning with two main parameters, the number of learning constraints (R) and the tolerance to spike timing (τ_⋆). We demonstrate that a low (high) rank R accounts for an error-based (target-based) learning rule, while high (low) tolerance to spike timing τ_⋆ promotes rate-based (spike-based) coding.

Our approach naturally lends itself to Imitation Learning (and Behavioral Cloning in particular) and we apply it to solve relevant closed-loop tasks such as the button-and-food task, and the 2D Bipedal Walker. The button-and-food is a navigation task that requires retaining a long-term memory, and benefits from a high R. On the other hand, the 2D Bipedal Walker is a motor task and benefits from a low τ_⋆.

Finally, we show that our theoretical formulation suggests protocols to deduce the structure of learning feedback in biological networks.

Long-range cortical synchronization supports abrupt visual learning

Visual plasticity declines sharply after the critical period, yet we easily learn to recognize new faces and places, even as adults. Such learning is often characterized by a "moment of insight," an abrupt and dramatic improvement in recognition. The mechanisms that support abrupt learning are unknown, but one hypothesis is that they involve changes in synchronization between brain regions. To test this hypothesis, we used a behavioral task in which non-human primates rapidly learned to recognize novel images and to associate them with specific responses. Simultaneous recordings from inferotemporal and prefrontal cortices revealed a transient synchronization of neural activity between these areas that peaked around the moment of insight. Synchronization was strongest between inferotemporal sites that encoded images and reward-sensitive prefrontal sites. Moreover, its magnitude intensified gradually over image exposures, suggesting that abrupt learning is the culmination of a search for informative signals within a circuit linking sensory information to task demands.

Benefits of Endogenous Spatial Attention During Visual Double-Training in Cortically-Blinded Fields

Recovery of visual discrimination thresholds inside cortically-blinded (CB) fields is most commonly attained at a single, trained location at a time, with iterative progress deeper into the blind field as performance improves over several months. As such, training is slow, inefficient, burdensome, and often frustrating for patients. Here, we investigated whether double-location training, coupled with a covert spatial-attention (SA) pre-cue, could improve the efficiency of training. Nine CB participants completed a randomized, training assignment with either a spatial attention or neutral pre-cue. All trained for a similar length of time on a fine direction discrimination task at two blind field locations simultaneously. Training stimuli and tasks for both cohorts were identical, save for the presence of a central pre-cue, to manipulate endogenous (voluntary) SA, or a Neutral pre-cue. Participants in the SA training cohort demonstrated marked improvements in direction discrimination thresholds, albeit not to normal/intact-field levels; participants in the Neutral training cohort remained impaired. Thus, double-training within cortically blind fields, when coupled with SA pre-cues can significantly improve direction discrimination thresholds at two locations simultaneously, offering a new method to improve performance and reduce the training burden for CB patients. Double-training without SA pre-cues revealed a hitherto unrecognized limitation of cortically-blind visual systems’ ability to improve while processing two stimuli simultaneously. These data could potentially explain why exposure to the typically complex visual environments encountered in everyday life is insufficient to induce visual recovery in CB patients. It is hoped that these new insights will direct both research and therapeutic developments toward methods that can attain better, faster recovery of vision in CB fields.

Value-based cognition and drug dependency

Value-based decision-making is thought to play an important role in drug dependency. Achieving elevated levels of euphoria or ameliorating dysphoria/pain may motivate goal-directed drug consumption in both drug-naïve and long-time users. In other words, drugs become viewed as the preferred means of attaining a desired internal state. The bias towards choosing drugs may affect one's cognition. Observed biases in learning, attention and memory systems within the brain gradually focus one's cognitive functions towards drugs and related cues to the exclusion of other stimuli. In this narrative review, the effects of drug use on learning, attention and memory are discussed with a particular focus on changes across brain-wide functional networks and the subsequent impact on behaviour. These cognitive changes are then incorporated into the cycle of addiction, an established model outlining the transition from casual drug use to chronic dependency. If drug use results in the elevated salience of drugs and their cues, the studies highlighted in this review strongly suggest that this salience biases cognitive systems towards the motivated pursuit of addictive drugs. This bias is observed throughout the cycle of addiction, possibly contributing to the persistent hold that addictive drugs have over the dependent. Taken together, the excessive valuation of drugs as the preferred means of achieving a desired internal state affects more than just decision-making, but also learning, attentional and mnemonic systems. This eventually narrows the focus of one's thoughts towards the pursuit and consumption of addictive drugs.

Reinforcement Learning-based Kalman Filter for Adaptive Brain Control in Brain-Machine Interface. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2021;2021:6619-6622. [PMID: 34892625 DOI: 10.1109/embc46164.2021.9629511] [Citation(s) in RCA: 0] [Impact Index Per Article: 0]

Brain-Machine Interfaces (BMIs) convert paralyzed people's neural signals into the command of the neuro-prosthesis. During the subject's brain control (BC) process, the neural patterns might change across time, making it crucial and challenging for the decoder to co-adapt with the dynamic neural patterns. Kalman Filter (KF) is commonly used for continuous control in BC. However, if the neural patterns become quite different compared with the training data, KF needs a re-calibration session to maintain its performance. On the other hand, Reinforcement Learning (RL) has the advantage of adaptive updating by the reward signal. But it is not very suitable for generating continuous motor states in BC due to the discrete action selection. In this paper, we propose a reinforcement learning-based Kalman filter. We maintain the state transition model of KF for a continuous motor state prediction. At the same time, we use RL to generate the action from the corresponding neural pattern, which is then used as a correction for the state prediction. The RL's parameters are continuously adjusted by the reward signal in BC. In this way, we could achieve a continuous motor state prediction when the neural patterns have drifted across time. The proposed algorithm is tested on a simulated rat lever-pressing experiment, where the rat's neural patterns have drifted across days. Compared with pure KF without re-calibration, our algorithm could follow the neural pattern drift in an online fashion and maintain good performance.Clinical Relevance- The proposed method bridges the gap between the online parameter adaptation and the continuous control of the neuro-prosthesis. It is promising to be used in adaptive brain control applications during clinical usage.

What Is the Role of Thalamostriatal Circuits in Learning Vocal Sequences?

Basal ganglia (BG) circuits integrate sensory and motor-related information from the cortex, thalamus, and midbrain to guide learning and production of motor sequences. Birdsong, like speech, is comprised of precisely sequenced vocal elements. Learning song sequences during development relies on Area X, a vocalization related region in the medial striatum of the songbird BG. Area X receives inputs from cortical-like pallial song circuits and midbrain dopaminergic circuits and sends projections to the thalamus. It has recently been shown that thalamic circuits also send substantial projections back to Area X. Here, we outline a gated-reinforcement learning model for how Area X may use signals conveyed by thalamostriatal inputs to direct song learning. Integrating conceptual advances from recent mammalian and songbird literature, we hypothesize that thalamostriatal pathways convey signals linked to song syllable onsets and offsets and influence striatal circuit plasticity via regulation of cholinergic interneurons (ChIs). We suggest that syllable sequence associated vocal-motor information from the thalamus drive precisely timed pauses in ChIs activity in Area X. When integrated with concurrent corticostriatal and dopaminergic input, this circuit helps regulate plasticity on medium spiny neurons (MSNs) and the learning of syllable sequences. We discuss new approaches that can be applied to test core ideas of this model and how associated insights may provide a framework for understanding the function of BG circuits in learning motor sequences.

Capturing the objects of vision with neural networks

Human visual perception carves a scene at its physical joints, decomposing the world into objects, which are selectively attended, tracked and predicted as we engage our surroundings. Object representations emancipate perception from the sensory input, enabling us to keep in mind that which is out of sight and to use perceptual content as a basis for action and symbolic cognition. Human behavioural studies have documented how object representations emerge through grouping, amodal completion, proto-objects and object files. By contrast, deep neural network models of visual object recognition remain largely tethered to sensory input, despite achieving human-level performance at labelling objects. Here, we review related work in both fields and examine how these fields can help each other. The cognitive literature provides a starting point for the development of new experimental tasks that reveal mechanisms of human object perception and serve as benchmarks driving the development of deep neural network models that will put the object into object recognition.

Strategic Distractor Suppression Improves Selective Control in Human Vision

Our visual environment is complicated, and our cognitive capacity is limited. As a result, we must strategically ignore some stimuli to prioritize others. Common sense suggests that foreknowledge of distractor characteristics, like location or color, might help us ignore these objects. But empirical studies have provided mixed evidence, often showing that knowing about a distractor before it appears counterintuitively leads to its attentional selection. What has looked like strategic distractor suppression in the past is now commonly explained as a product of prior experience and implicit statistical learning, and the long-standing notion the distractor suppression is reflected in α band oscillatory brain activity has been challenged by results appearing to link α to target resolution. Can we strategically, proactively suppress distractors? And, if so, does this involve α? Here, we use the concurrent recording of human EEG and eye movements in optimized experimental designs to identify behavior and brain activity associated with proactive distractor suppression. Results from three experiments show that knowing about distractors before they appear causes a reduction in electrophysiological indices of covert attentional selection of these objects and a reduction in the overt deployment of the eyes to the location of the objects. This control is established before the distractor appears and is predicted by the power of cue-elicited α activity over the visual cortex. Foreknowledge of distractor characteristics therefore leads to improved selective control, and α oscillations in visual cortex reflect the implementation of this strategic, proactive mechanism.SIGNIFICANCE STATEMENT To behave adaptively and achieve goals we often need to ignore visual distraction. Is it easier to ignore distracting objects when we know more about them? We recorded eye movements and electrical brain activity to determine whether foreknowledge of distractor characteristics can be used to limit processing of these objects. Results show that knowing the location or color of a distractor stops us from attentionally selecting it. A neural signature of this inhibition emerges in oscillatory alpha band brain activity, and when this signal is strong, selective processing of the distractor decreases. Knowing about the characteristics of task-irrelevant distractors therefore increases our ability to focus on task-relevant information, in this way gating information processing in the brain.

From perception to inference: Utilization of probabilities as decision weights in children

In a probabilistic inference task (three probabilistic cues predict outcomes for two options), we examined decisions from 233 children (5-6 vs. 9-10 years). Contiguity (low vs. high; i.e., position of probabilistic information far vs. close to options) and demand for selectivity (low vs. high; i.e., showing predictions of desired vs. desired and undesired outcomes) were varied as configural aspects of the presentation format. Probability utilization was measured by the frequency of following the predictions of the highest validity cue in choice. High contiguity and low demand for selectivity strongly and moderately increased probability utilization, respectively. Children are influenced by presentation format when using probabilities as decision weights. They benefit from perception-like presentations that present probabilities and options as compounds.

Human Representation Learning

The central theme of this review is the dynamic interaction between information selection and learning. We pose a fundamental question about this interaction: How do we learn what features of our experiences are worth learning about? In humans, this process depends on attention and memory, two cognitive functions that together constrain representations of the world to features that are relevant for goal attainment. Recent evidence suggests that the representations shaped by attention and memory are themselves inferred from experience with each task. We review this evidence and place it in the context of work that has explicitly characterized representation learning as statistical inference. We discuss how inference can be scaled to real-world decisions by approximating beliefs based on a small number of experiences. Finally, we highlight some implications of this inference process for human decision-making in social environments.

Efficient Spike-Driven Learning With Dendritic Event-Based Processing

A critical challenge in neuromorphic computing is to present computationally efficient algorithms of learning. When implementing gradient-based learning, error information must be routed through the network, such that each neuron knows its contribution to output, and thus how to adjust its weight. This is known as the credit assignment problem. Exactly implementing a solution like backpropagation involves weight sharing, which requires additional bandwidth and computations in a neuromorphic system. Instead, models of learning from neuroscience can provide inspiration for how to communicate error information efficiently, without weight sharing. Here we present a novel dendritic event-based processing (DEP) algorithm, using a two-compartment leaky integrate-and-fire neuron with partially segregated dendrites that effectively solves the credit assignment problem. In order to optimize the proposed algorithm, a dynamic fixed-point representation method and piecewise linear approximation approach are presented, while the synaptic events are binarized during learning. The presented optimization makes the proposed DEP algorithm very suitable for implementation in digital or mixed-signal neuromorphic hardware. The experimental results show that spiking representations can rapidly learn, achieving high performance by using the proposed DEP algorithm. We find the learning capability is affected by the degree of dendritic segregation, and the form of synaptic feedback connections. This study provides a bridge between the biological learning and neuromorphic learning, and is meaningful for the real-time applications in the field of artificial intelligence.

Task Learning Over Multi-Day Recording via Internally Rewarded Reinforcement Learning Based Brain Machine Interfaces

Autonomous brain machine interfaces (BMIs) aim to enable paralyzed people to self-evaluate their movement intention to control external devices. Previous reinforcement learning (RL)-based decoders interpret the mapping between neural activity and movements using the external reward for well-trained subjects, and have not investigated the task learning procedure. The brain has developed a learning mechanism to identify the correct actions that lead to rewards in the new task. This internal guidance can be utilized to replace the external reference to advance BMIs as an autonomous system. In this study, we propose to build an internally rewarded reinforcement learning-based BMI framework using the multi-site recording to demonstrate the autonomous learning ability of the BMI decoder on the new task. We test the model on the neural data collected over multiple days while the rats were learning a new lever discrimination task. The primary motor cortex (M1) and medial prefrontal cortex (mPFC) spikes are interpreted by the proposed RL framework into the discrete lever press actions. The neural activity of the mPFC post the action duration is interpreted as the internal reward information, where a support vector machine is implemented to classify the reward vs. non-reward trials with a high accuracy of 87.5% across subjects. This internal reward is used to replace the external water reward to update the decoder, which is able to adapt to the nonstationary neural activity during subject learning. The multi-cortical recording allows us to take in more cortical recordings as input and uses internal critics to guide the decoder learning. Comparing with the classic decoder using M1 activity as the only input and external guidance, the proposed system with multi-cortical recordings shows a better decoding accuracy. More importantly, our internally rewarded decoder demonstrates the autonomous learning ability on the new task as the decoder successfully addresses the time-variant neural patterns while subjects are learning, and works asymptotically as the subjects' behavioral learning progresses. It reveals the potential of endowing BMIs with autonomous task learning ability in the RL framework.

Flexible Working Memory Through Selective Gating and Attentional Tagging

Working memory is essential: it serves to guide intelligent behavior of humans and nonhuman primates when task-relevant stimuli are no longer present to the senses. Moreover, complex tasks often require that multiple working memory representations can be flexibly and independently maintained, prioritized, and updated according to changing task demands. Thus far, neural network models of working memory have been unable to offer an integrative account of how such control mechanisms can be acquired in a biologically plausible manner. Here, we present WorkMATe, a neural network architecture that models cognitive control over working memory content and learns the appropriate control operations needed to solve complex working memory tasks. Key components of the model include a gated memory circuit that is controlled by internal actions, encoding sensory information through untrained connections, and a neural circuit that matches sensory inputs to memory content. The network is trained by means of a biologically plausible reinforcement learning rule that relies on attentional feedback and reward prediction errors to guide synaptic updates. We demonstrate that the model successfully acquires policies to solve classical working memory tasks, such as delayed recognition and delayed pro-saccade/anti-saccade tasks. In addition, the model solves much more complex tasks, including the hierarchical 12-AX task or the ABAB ordered recognition task, both of which demand an agent to independently store and updated multiple items separately in memory. Furthermore, the control strategies that the model acquires for these tasks subsequently generalize to new task contexts with novel stimuli, thus bringing symbolic production rule qualities to a neural network architecture. As such, WorkMATe provides a new solution for the neural implementation of flexible memory control.

Reinforcement Learning based Decoding Using Internal Reward for Time Delayed Task in Brain Machine Interfaces

Reinforcement learning (RL) algorithm interprets neural signals into movement intentions with the guidance of the reward in Brain-machine interfaces (BMIs). Current RL algorithms generally work for the tasks with immediate rewards delivery, and lack of efficiency in delayed reward task. Prefrontal cortex, including medial prefrontal cortex(mPFC), has been demonstrated to assign credit to intermediate steps, which reinforces preceding action more efficiently. In this paper, we propose to simulate the functionality of mPFC activities as intermediate rewards to train a RL based decoder in a two-step movement task. A support vector machine (SVM) is adopted to verify if the subject expects a reward due to the accomplishment of a subtask from mPFC activity. Then this discrimination result will be utilized to guide the training of the RL decoder for each step respectively. Here, we apply the Sarsa-style attention-gated reinforcement learning (SAGREL) as the decoder to interpret motor cortex(M1) activity to action states. We test on in vivo primary motor cortex (M1) and mPFC data collected from rats, where the rats need to first trigger the start and then press lever for rewards using M1 signals. SAGREL using intermediate rewards from mPFC activities achieves a prediction accuracy of 66.8% ± 2.0.% (mean ± std) %, which is significantly better than the one using the reward by the end of trial (45.9.% ± 1.2%). This reveals the potentials of modelling mPFC activities as intermediate rewards for the delayed reward tasks.

Reinforcement Learning Based Fast Self-Recalibrating Decoder for Intracortical Brain-Machine Interface

BACKGROUND

For the nonstationarity of neural recordings in intracortical brain-machine interfaces, daily retraining in a supervised manner is always required to maintain the performance of the decoder. This problem can be improved by using a reinforcement learning (RL) based self-recalibrating decoder. However, quickly exploring new knowledge while maintaining a good performance remains a challenge in RL-based decoders.

METHODS

To solve this problem, we proposed an attention-gated RL-based algorithm combining transfer learning, mini-batch, and weight updating schemes to accelerate the weight updating and avoid over-fitting. The proposed algorithm was tested on intracortical neural data recorded from two monkeys to decode their reaching positions and grasping gestures.

RESULTS

The decoding results showed that our proposed algorithm achieved an approximate 20% increase in classification accuracy compared to that obtained by the non-retrained classifier and even achieved better classification accuracy than the daily retraining classifier. Moreover, compared with a conventional RL method, our algorithm improved the accuracy by approximately 10% and the online weight updating speed by approximately 70 times.

CONCLUSIONS

This paper proposed a self-recalibrating decoder which achieved a good and robust decoding performance with fast weight updating and might facilitate its application in wearable device and clinical practice.

Self-adaptive shared control with brain state evaluation network for human-wheelchair cooperation

OBJECTIVE

For the shared control systems, how to trade off the control weight between robot autonomy and human operator is an important issue, especially for BCI-based systems. However, most of existing shared controllers have paid less attention to the effects caused by subjects with different levels of brain control ability.

APPROACH

In this paper, a brain state evaluation network, termed BSE-NET, is proposed to evaluate subjects' brain control ability online based on quantized attention-gated kernel reinforcement learning. With the output of BSE-NET (confidence score), a shared controller is designed to dynamically adjust the control weight between robot autonomy and human operator.

MAIN RESULTS

The experimental results show that most of subjects achieved high and stable experimental success rate of approximately 90%. Furthermore, for subjects with different accuracy on EEG decoding, a proper confidence score can be dynamically generated to reflect their levels of brain control ability, and the proposed system can effectively adjust the control weight in all-time shared control.

SIGNIFICANCE

We discuss how our proposed method shows promise for BCI applications that can evaluate subjects' brain control ability online as well as provide a method for the research on self-adaptive shared control to adaptively balance control weight between subject's instruction and robot autonomy.

Dopamine role in learning and action inference

This paper describes a framework for modelling dopamine function in the mammalian brain. It proposes that both learning and action planning involve processes minimizing prediction errors encoded by dopaminergic neurons. In this framework, dopaminergic neurons projecting to different parts of the striatum encode errors in predictions made by the corresponding systems within the basal ganglia. The dopaminergic neurons encode differences between rewards and expectations in the goal-directed system, and differences between the chosen and habitual actions in the habit system. These prediction errors trigger learning about rewards and habit formation, respectively. Additionally, dopaminergic neurons in the goal-directed system play a key role in action planning: They compute the difference between a desired reward and the reward expected from the current motor plan, and they facilitate action planning until this difference diminishes. Presented models account for dopaminergic responses during movements, effects of dopamine depletion on behaviour, and make several experimental predictions.

In the brain, chemicals such as dopamine allow nerve cells to ‘talk’ to each other and to relay information from and to the environment. Dopamine, in particular, is released when pleasant surprises are experienced: this helps the organism to learn about the consequences of certain actions. If a new flavour of ice-cream tastes better than expected, for example, the release of dopamine tells the brain that this flavour is worth choosing again.

However, dopamine has an additional role in controlling movement. When the cells that produce dopamine die, for instance in Parkinson’s disease, individuals may find it difficult to initiate deliberate movements. Here, Rafal Bogacz aimed to develop a comprehensive framework that could reconcile the two seemingly unrelated roles played by dopamine.

The new theory proposes that dopamine is released when an outcome differs from expectations, which helps the organism to adjust and minimise these differences. In the ice-cream example, the difference is between how good the treat is expected to taste, and how tasty it really is. By learning to select the same flavour repeatedly, the brain aligns expectation and the result of the choice. This ability would also apply when movements are planned. In this case, the brain compares the desired reward with the predicted results of the planned actions. For example, while planning to get a spoonful of ice-cream, the brain compares the pleasure expected from the movement that is currently planned, and the pleasure of eating a full spoon of the treat. If the two differ, for example because no movement has been planned yet, the brain releases dopamine to form a better version of the action plan. The theory was then tested using a computer simulation of nerve cells that release dopamine; this showed that the behaviour of the virtual cells closely matched that of their real-life counterparts.

This work offers a comprehensive description of the fundamental role of dopamine in the brain. The model now needs to be verified through experiments on living nerve cells; ultimately, it could help doctors and researchers to develop better treatments for conditions such as Parkinson’s disease or ADHD, which are linked to a lack of dopamine.

Exogenous attention facilitates perceptual learning in visual acuity to untrained stimulus locations and features

Visual perceptual learning (VPL) refers to the improvement in performance on a visual task due to practice. A hallmark of VPL is specificity, as improvements are often confined to the trained retinal locations or stimulus features. We have previously found that exogenous (involuntary, stimulus-driven) and endogenous (voluntary, goal-driven) spatial attention can facilitate the transfer of VPL across locations in orientation discrimination tasks mediated by contrast sensitivity. Here, we investigated whether exogenous spatial attention can facilitate such transfer in acuity tasks that have been associated with higher specificity. We trained observers for 3 days (days 2-4) in a Landolt acuity task (Experiment 1) or a Vernier hyperacuity task (Experiment 2), with either exogenous precues (attention group) or neutral precues (neutral group). Importantly, during pre-tests (day 1) and post-tests (day 5), all observers were tested with neutral precues; thus, groups differed only in their attentional allocation during training. For the Landolt acuity task, we found evidence of location transfer in both the neutral and attention groups, suggesting weak location specificity of VPL. For the Vernier hyperacuity task, we found evidence of location and feature specificity in the neutral group, and learning transfer in the attention group-similar improvement at trained and untrained locations and features. Our results reveal that, when there is specificity in a perceptual acuity task, exogenous spatial attention can overcome that specificity and facilitate learning transfer to both untrained locations and features simultaneously with the same training. Thus, in addition to improving performance, exogenous attention generalizes perceptual learning across locations and features.

A deep learning framework for neuroscience

Systems neuroscience seeks explanations for how the brain implements a wide variety of perceptual, cognitive and motor tasks. Conversely, artificial intelligence attempts to design computational systems based on the tasks they will have to solve. In artificial neural networks, the three components specified by design are the objective functions, the learning rules and the architectures. With the growing success of deep learning, which utilizes brain-inspired architectures, these three designed components have increasingly become central to how we model, engineer and optimize complex artificial learning systems. Here we argue that a greater focus on these components would also benefit systems neuroscience. We give examples of how this optimization-based framework can drive theoretical and experimental progress in neuroscience. We contend that this principled perspective on systems neuroscience will help to generate more rapid progress.

Control of synaptic plasticity in deep cortical networks

Humans and many other animals have an enormous capacity to learn about sensory stimuli and to master new skills. However, many of the mechanisms that enable us to learn remain to be understood. One of the greatest challenges of systems neuroscience is to explain how synaptic connections change to support maximally adaptive behaviour. Here, we provide an overview of factors that determine the change in the strength of synapses, with a focus on synaptic plasticity in sensory cortices. We review the influence of neuromodulators and feedback connections in synaptic plasticity and suggest a specific framework in which these factors can interact to improve the functioning of the entire network.

Clustering Neural Patterns in Kernel Reinforcement Learning Assists Fast Brain Control in Brain-Machine Interfaces

Neuroprosthesis enables the brain control on the external devices purely using neural activity for paralyzed people. Supervised learning decoders recalibrate or re-fit the discrepancy between the desired target and decoder's output, where the correction may over-dominate the user's intention. Reinforcement learning decoder allows users to actively adjust their brain patterns through trial and error, which better represents the subject's motive. The computational challenge is to quickly establish new state-action mapping before the subject becomes frustrated. Recently proposed quantized attention-gated kernel reinforcement learning (QAGKRL) explores the optimal nonlinear neural-action mapping in the Reproducing Kernel Hilbert Space (RKHS). However, considering all past data in RKHS is less efficient and sensitive to detect the new neural patterns emerging in brain control. In this paper, we propose a clustering-based kernel RL algorithm. New neural patterns emerge and are clustered to represent the novel knowledge in brain control. The current neural data only activate the nearest subspace in RKHS for more efficient decoding. The dynamic clustering makes our algorithm more sensitive to new brain patterns. We test our algorithm on both the synthetic and real-world spike data. Compared with QAGKRL, our algorithm can achieve a quicker knowledge adaptation in brain control with less computational complexity.

Selection history in context: Evidence for the role of reinforcement learning in biasing attention

Attention is biased towards learned predictors of reward. The influence of reward history on attentional capture has been shown to be context-specific: When particular stimulus features are associated with reward, these features only capture attention when viewed in the context in which they were rewarded. Selection history can also bias attention, such that prior target features gain priority independently of reward history. The contextual specificity of this influence of selection history on attention has not been examined. In the present study, we demonstrate that the consequences of repetitive selection on attention robustly generalize across context, such that prior target features capture attention even in contexts in which they were never seen previously. Our findings suggest that the learning underlying attention driven by outcome-independent selection history differs qualitatively from the learning underlying value-driven attention, consistent with a distinction between associative and reinforcement learning mechanisms.

On the relationship between value-driven and stimulus-driven attentional capture

Reward history, physical salience, and task relevance all influence the degree to which a stimulus competes for attention, reflecting value-driven, stimulus-driven, and goal-contingent attentional capture, respectively. Theories of value-driven attention have likened reward cues to physically salient stimuli, positing that reward cues are preferentially processed in early visual areas as a result of value-modulated plasticity in the visual system. Such theories predict a strong coupling between value-driven and stimulus-driven attentional capture across individuals. In the present study, we directly test this hypothesis, and demonstrate a robust correlation between value-driven and stimulus-driven attentional capture. Our findings suggest substantive overlap in the mechanisms of competition underlying the attentional priority of reward cues and physically salient stimuli.

Dissociable Components of Experience-Driven Attention

What we pay attention to is influenced by current task goals (goal-directed attention) [1, 2], the physical salience of stimuli (stimulus-driven attention) [3-5], and selection history [6-12]. This third construct, which encompasses reward learning, aversive conditioning, and repetitive orienting behavior [12-18], is often characterized as a unitary mechanism of control that can be contrasted with the other two [12-14]. Here, we present evidence that two different learning processes underlie the influence of selection history on attention, with dissociable consequences for orienting behavior. Human observers performed an antisaccade task in which they were paid for shifting their gaze in the direction opposite one of two color-defined targets. Strikingly, such training resulted in a bias to do the opposite of what observers were motivated and paid to do, with associative learning facilitating orienting toward reward cues. On the other hand, repetitive orienting away from a target produced a bias to repeat this behavior even when it conflicted with current goals, reflecting instrumental conditioning of the orienting response. Our findings challenge the idea that selection history reflects a common mechanism of learning-dependent priority and instead suggest multiple distinct routes by which learning history shapes orienting behavior. We also provide direct evidence for the idea that value-based attention is approach oriented, which limits the effectiveness of attentional bias modification techniques that utilize incentive structures.

Learning to optimize perceptual decisions through suppressive interactions in the human brain

Translating noisy sensory signals to perceptual decisions is critical for successful interactions in complex environments. Learning is known to improve perceptual judgments by filtering external noise and task-irrelevant information. Yet, little is known about the brain mechanisms that mediate learning-dependent suppression. Here, we employ ultra-high field magnetic resonance spectroscopy of GABA to test whether suppressive processing in decision-related and visual areas facilitates perceptual judgments during training. We demonstrate that parietal GABA relates to suppression of task-irrelevant information, while learning-dependent changes in visual GABA relate to enhanced performance in target detection and feature discrimination tasks. Combining GABA measurements with functional brain connectivity demonstrates that training on a target detection task involves local connectivity and disinhibition of visual cortex, while training on a feature discrimination task involves inter-cortical interactions that relate to suppressive visual processing. Our findings provide evidence that learning optimizes perceptual decisions through suppressive interactions in decision-related networks.

Feature-specific prediction errors and surprise across macaque fronto-striatal circuits

To adjust expectations efficiently, prediction errors need to be associated with the precise features that gave rise to the unexpected outcome, but this credit assignment may be problematic if stimuli differ on multiple dimensions and it is ambiguous which feature dimension caused the outcome. Here, we report a potential solution: neurons in four recorded areas of the anterior fronto-striatal networks encode prediction errors that are specific to feature values of different dimensions of attended multidimensional stimuli. The most ubiquitous prediction error occurred for the reward-relevant dimension. Feature-specific prediction error signals a) emerge on average shortly after non-specific prediction error signals, b) arise earliest in the anterior cingulate cortex and later in dorsolateral prefrontal cortex, caudate and ventral striatum, and c) contribute to feature-based stimulus selection after learning. Thus, a widely-distributed feature-specific eligibility trace may be used to update synaptic weights for improved feature-based attention.

In order to adjust expectations efficiently, prediction errors need to be associated with the features that gave rise to the unexpected outcome. Here, the authors show that neurons in anterior fronto-striatal networks encode prediction errors that are specific to feature values of different stimulus dimensions.

Dissociating Reward- and Attention-driven Biasing of Global Feature-based Selection in Human Visual Cortex

Objects that promise rewards are prioritized for visual selection. The way this prioritization shapes sensory processing in visual cortex, however, is debated. It has been suggested that rewards motivate stronger attentional focusing, resulting in a modulation of sensory selection in early visual cortex. An open question is whether those reward-driven modulations would be independent of similar modulations indexing the selection of attended features that are not associated with reward. Here, we use magnetoencephalography in human observers to investigate whether the modulations indexing global color-based selection in visual cortex are separable for target- and (monetary) reward-defining colors. To assess the underlying global color-based activity modulation, we compare the event-related magnetic field response elicited by a color probe in the unattended hemifield drawn either in the target color, the reward color, both colors, or a neutral task-irrelevant color. To test whether target and reward relevance trigger separable modulations, we manipulate attention demands on target selection while keeping reward-defining experimental parameters constant. Replicating previous observations, we find that reward and target relevance produce almost indistinguishable gain modulations in ventral extratriate cortex contralateral to the unattended color probe. Importantly, increasing attention demands on target discrimination increases the response to the target-defining color, whereas the response to the rewarded color remains largely unchanged. These observations indicate that, although task relevance and reward influence the very same feature-selective area in extrastriate visual cortex, the associated modulations are largely independent.

Clustering Based Kernel Reinforcement Learning for Neural Adaptation in Brain-Machine Interfaces

Reinforcement learning (RL) interprets subject's movement intention in Brain Machine Interfaces (BMIs) through trial-and-error with the advantage that it does not need the real limb movements. When the subjects try to control the external devices purely using brain signals without actual movements (brain control), they adjust the neural firing patterns to adapt to device control, which expands the state-action space for the RL decoder to explore. The challenge is to quickly explore the new knowledge in the sizeable state-action space and maintain good performance. Recently quantized attention-gated kernel reinforcement learning (QAGKRL) was proposed to quickly explore the global optimum in Reproducing Kernel Hilbert Space (RKHS). However, its network size will grow large when the new input comes, which makes it computationally inefficient. In addition, the output is generated using the whole input structure without being sensitive to the new knowledge. In this paper, we propose a new kernel based reinforcement learning algorithm that utilizes the clustering technique in the input domain. The similar neural inputs are grouped, and a new input only activates its nearest cluster, which either utilizes an existing sub-network or forms a new one. In this way, we can build the sub-feature space instead of the global mapping to calculate the output, which transfers the old knowledge effectively and also consequently reduces the computational complexity. To evaluate our algorithm, we test on the synthetic spike data, where the subject's task mode switches between manual control and brain control. Compared with QAGKRL, the simulation results show that our algorithm can achieve a faster learning curve, less computational time, and more accuracy. This indicates our algorithm to be a promising method for the online implementation of BMIs.

Neurobiology of value-driven attention

What we pay attention to is influenced by reward learning. Converging evidence points to the idea that associative reward learning changes how visual stimuli are processed in the brain, rendering learned reward cues difficult to ignore. Behavioral evidence distinguishes value-driven attention from other established control mechanisms, suggesting a distinct underlying neurobiological process. Recently, studies have begun to explore the neural substrates of this value-driven attention mechanism. Here, I review the progress that has been made in this area, and synthesize the findings to provide an integrative account of the neurobiology of value-driven attention. The proposed account can explain both attentional capture by previously rewarded targets and the modulatory effect of reward on priming, as well as the decoupling of reward history and prior task relevance in value-driven attention.

Endogenous spatial attention during perceptual learning facilitates location transfer

Covert attention and perceptual learning enhance perceptual performance. The relation between these two mechanisms is largely unknown. Previously, we showed that manipulating involuntary, exogenous spatial attention during training improved performance at trained and untrained locations, thus overcoming the typical location specificity. Notably, attention-induced transfer only occurred for high stimulus contrasts, at the upper asymptote of the psychometric function (i.e., via response gain). Here, we investigated whether and how voluntary, endogenous attention, the top-down and goal-based type of covert visual attention, influences perceptual learning. Twenty-six participants trained in an orientation discrimination task at two locations: half of participants received valid endogenous spatial precues (attention group), while the other half received neutral precues (neutral group). Before and after training, all participants were tested with neutral precues at two trained and two untrained locations. Within each session, stimulus contrast varied on a trial basis from very low (2%) to very high (64%). Performance was fit by a Weibull psychometric function separately for each day and location. Performance improved for both groups at the trained location, and unlike training with exogenous attention, at the threshold level (i.e., via contrast gain). The neutral group exhibited location specificity: Thresholds decreased at the trained locations, but not at the untrained locations. In contrast, participants in the attention group showed significant location transfer: Thresholds decreased to the same extent at both trained and untrained locations. These results indicate that, similar to exogenous spatial attention, endogenous spatial attention induces location transfer, but influences contrast gain instead of response gain.

Dendritic solutions to the credit assignment problem

Guaranteeing that synaptic plasticity leads to effective learning requires a means for assigning credit to each neuron for its contribution to behavior. The 'credit assignment problem' refers to the fact that credit assignment is non-trivial in hierarchical networks with multiple stages of processing. One difficulty is that if credit signals are integrated with other inputs, then it is hard for synaptic plasticity rules to distinguish credit-related activity from non-credit-related activity. A potential solution is to use the spatial layout and non-linear properties of dendrites to distinguish credit signals from other inputs. In cortical pyramidal neurons, evidence hints that top-down feedback signals are integrated in the distal apical dendrites and have a distinct impact on spike-firing and synaptic plasticity. This suggests that the distal apical dendrites of pyramidal neurons help the brain to solve the credit assignment problem.

Cognitive computational neuroscience

To learn how cognition is implemented in the brain, we must build computational models that can perform cognitive tasks, and test such models with brain and behavioral experiments. Cognitive science has developed computational models that decompose cognition into functional components. Computational neuroscience has modeled how interacting neurons can implement elementary components of cognition. It is time to assemble the pieces of the puzzle of brain computation and to better integrate these separate disciplines. Modern technologies enable us to measure and manipulate brain activity in unprecedentedly rich ways in animals and humans. However, experiments will yield theoretical insight only when employed to test brain-computational models. Here we review recent work in the intersection of cognitive science, computational neuroscience and artificial intelligence. Computational models that mimic brain information processing during perceptual, cognitive and control tasks are beginning to be developed and tested with brain and behavioral data.