A neural circuit model of flexible sensorimotor mapping: learning and forgetting on multiple timescales

Encoding and decoding in parietal cortex during sensorimotor decision-making

The lateral intraparietal area (LIP) of macaques has been asserted to play a fundamental role in sensorimotor decision-making. Here we dissect the neural code in LIP at the level of individual trial spike trains using a statistical approach based on generalized linear models. We show that LIP responses reflect a combination of temporally-overlapping task and decision-related signals. Our model accounts for the detailed statistics of LIP spike trains, and accurately predicts spike trains from task events on single trials. Moreover, we derive an optimal decoder for heterogeneous, multiplexed LIP responses that could be implemented in biologically plausible circuits. In contrast to interpretations of LIP as providing an instantaneous code for decision variables, we show that optimal decoding requires integrating LIP spikes over two timescales. These analyses provide a detailed understanding of the neural code in LIP, and a framework for studying the coding of multiplexed signals in higher brain areas.

Inter-synaptic learning of combination rules in a cortical network model

Selecting responses in working memory while processing combinations of stimuli depends strongly on their relations stored in long-term memory. However, the learning of XOR-like combinations of stimuli and responses according to complex rules raises the issue of the non-linear separability of the responses within the space of stimuli. One proposed solution is to add neurons that perform a stage of non-linear processing between the stimuli and responses, at the cost of increasing the network size. Based on the non-linear integration of synaptic inputs within dendritic compartments, we propose here an inter-synaptic (IS) learning algorithm that determines the probability of potentiating/depressing each synapse as a function of the co-activity of the other synapses within the same dendrite. The IS learning is effective with random connectivity and without either a priori wiring or additional neurons. Our results show that IS learning generates efficacy values that are sufficient for the processing of XOR-like combinations, on the basis of the sole correlational structure of the stimuli and responses. We analyze the types of dendrites involved in terms of the number of synapses from pre-synaptic neurons coding for the stimuli and responses. The synaptic efficacy values obtained show that different dendrites specialize in the detection of different combinations of stimuli. The resulting behavior of the cortical network model is analyzed as a function of inter-synaptic vs. Hebbian learning. Combinatorial priming effects show that the retrospective activity of neurons coding for the stimuli trigger XOR-like combination-selective prospective activity of neurons coding for the expected response. The synergistic effects of inter-synaptic learning and of mixed-coding neurons are simulated. The results show that, although each mechanism is sufficient by itself, their combined effects improve the performance of the network.

A tweaking principle for executive control: neuronal circuit mechanism for rule-based task switching and conflict resolution

A hallmark of executive control is the brain's agility to shift between different tasks depending on the behavioral rule currently in play. In this work, we propose a "tweaking hypothesis" for task switching: a weak rule signal provides a small bias that is dramatically amplified by reverberating attractor dynamics in neural circuits for stimulus categorization and action selection, leading to an all-or-none reconfiguration of sensory-motor mapping. Based on this principle, we developed a biologically realistic model with multiple modules for task switching. We found that the model quantitatively accounts for complex task switching behavior: switch cost, congruency effect, and task-response interaction; as well as monkey's single-neuron activity associated with task switching. The model yields several testable predictions, in particular, that category-selective neurons play a key role in resolving sensory-motor conflict. This work represents a neural circuit model for task switching and sheds insights in the brain mechanism of a fundamental cognitive capability.

Bayesian deterministic decision making: a normative account of the operant matching law and heavy-tailed reward history dependency of choices

The decision making behaviors of humans and animals adapt and then satisfy an “operant matching law” in certain type of tasks. This was first pointed out by Herrnstein in his foraging experiments on pigeons. The matching law has been one landmark for elucidating the underlying processes of decision making and its learning in the brain. An interesting question is whether decisions are made deterministically or probabilistically. Conventional learning models of the matching law are based on the latter idea; they assume that subjects learn choice probabilities of respective alternatives and decide stochastically with the probabilities. However, it is unknown whether the matching law can be accounted for by a deterministic strategy or not. To answer this question, we propose several deterministic Bayesian decision making models that have certain incorrect beliefs about an environment. We claim that a simple model produces behavior satisfying the matching law in static settings of a foraging task but not in dynamic settings. We found that the model that has a belief that the environment is volatile works well in the dynamic foraging task and exhibits undermatching, which is a slight deviation from the matching law observed in many experiments. This model also demonstrates the double-exponential reward history dependency of a choice and a heavier-tailed run-length distribution, as has recently been reported in experiments on monkeys.

It's hard to forget: resetting memory in delay-match-to-multiple-image tasks

The Delay-Match-to-Sample (DMS) task has been used in countless studies of memory, undergoing numerous modifications, making the task more and more challenging to participants. The physiological correlate of memory is modified neural activity during the cue-to-match delay period reflecting reverberating attractor activity in multiple interconnected cells. DMS tasks may use a fixed set of well-practiced stimulus images-allowing for creation of attractors-or unlimited novel images, for which no attractor exists. Using well-learned stimuli requires that participants determine if a remembered image was seen in the same or a preceding trial, only responding to the former. Thus, trial-to-trial transitions must include a "reset" mechanism to mark old images as such. We test two groups of monkeys on a delay-match-to-multiple-images task, one with well-trained and one with novel images. Only the first developed a reset mechanism. We then switched tasks between the groups. We find that introducing fixed images initiates development of reset, and once established, switching to novel images does not disable its use. Without reset, memory decays slowly, leaving ~40% recognizable after a minute. Here, presence of reward further enhances memory of previously-seen images.

Dynamical regimes in neural network models of matching behavior

The matching law constitutes a quantitative description of choice behavior that is often observed in foraging tasks. According to the matching law, organisms distribute their behavior across available response alternatives in the same proportion that reinforcers are distributed across those alternatives. Recently a few biophysically plausible neural network models have been proposed to explain the matching behavior observed in the experiments. Here we study systematically the learning dynamics of these networks while performing a matching task on the concurrent variable interval (VI) schedule. We found that the model neural network can operate in one of three qualitatively different regimes depending on the parameters that characterize the synaptic dynamics and the reward schedule: (1) a matching behavior regime, in which the probability of choosing an option is roughly proportional to the baiting fractional probability of that option; (2) a perseverative regime, in which the network tends to make always the same decision; and (3) a tristable regime, in which the network can either perseverate or choose the two targets randomly approximately with the same probability. Different parameters of the synaptic dynamics lead to different types of deviations from the matching law, some of which have been observed experimentally. We show that the performance of the network depends on the number of stable states of each synapse and that bistable synapses perform close to optimal when the proper learning rate is chosen. Because our model provides a link between synaptic dynamics and qualitatively different behaviors, this work provides us with insight into the effects of neuromodulators on adaptive behaviors and psychiatric disorders.

Are fast/slow process in motor adaptation and forward/inverse internal model two sides of the same coin?

Motor adaptation is tuning of motor commands to compensate the disturbances in the outside environment and/or in the sensory-motor system. In spite of various theoretical and empirical studies, mechanism by which the brain learns to adapt has not been clearly understood. Among different computational models, two lines of researches are of interest in this study: first, the models which assume two adaptive processes, i.e. fast and slow, for motor learning, and second, the computational frameworks which assume two types of internal models in the central nervous system (CNS), i.e., forward and inverse models. They explain motor learning by modifying these internal models. Here, we present a hypothesis for a possible relationship between these two viewpoints according to the computational and physiological findings. This hypothesis suggests a direct relationship between the forward (inverse) internal model and the fast (slow) adaptive process. That is, the forward (inverse) model and fast (slow) adaptive process can be two sides of the same coin. Further evaluation of this hypothesis is helpful to achieve a better understanding of motor adaptation mechanism in the brain and also it lends itself to be used in designing therapeutic programs for rehabilitation of certain movement disorders.

Synthesizing cognition in neuromorphic electronic systems

The quest to implement intelligent processing in electronic neuromorphic systems lacks methods for achieving reliable behavioral dynamics on substrates of inherently imprecise and noisy neurons. Here we report a solution to this problem that involves first mapping an unreliable hardware layer of spiking silicon neurons into an abstract computational layer composed of generic reliable subnetworks of model neurons and then composing the target behavioral dynamics as a "soft state machine" running on these reliable subnets. In the first step, the neural networks of the abstract layer are realized on the hardware substrate by mapping the neuron circuit bias voltages to the model parameters. This mapping is obtained by an automatic method in which the electronic circuit biases are calibrated against the model parameters by a series of population activity measurements. The abstract computational layer is formed by configuring neural networks as generic soft winner-take-all subnetworks that provide reliable processing by virtue of their active gain, signal restoration, and multistability. The necessary states and transitions of the desired high-level behavior are then easily embedded in the computational layer by introducing only sparse connections between some neurons of the various subnets. We demonstrate this synthesis method for a neuromorphic sensory agent that performs real-time context-dependent classification of motion patterns observed by a silicon retina.

Synaptic encoding of temporal contiguity

Often we need to perform tasks in an environment that changes stochastically. In these situations it is important to learn the statistics of sequences of events in order to predict the future and the outcome of our actions. The statistical description of many of these sequences can be reduced to the set of probabilities that a particular event follows another event (temporal contiguity). Under these conditions, it is important to encode and store in our memory these transition probabilities. Here we show that for a large class of synaptic plasticity models, the distribution of synaptic strengths encodes transitions probabilities. Specifically, when the synaptic dynamics depend on pairs of contiguous events and the synapses can remember multiple instances of the transitions, then the average synaptic weights are a monotonic function of the transition probabilities. The synaptic weights converge to the distribution encoding the probabilities also when the correlations between consecutive synaptic modifications are considered. We studied how this distribution depends on the number of synaptic states for a specific model of a multi-state synapse with hard bounds. In the case of bistable synapses, the average synaptic weights are a smooth function of the transition probabilities and the accuracy of the encoding depends on the learning rate. As the number of synaptic states increases, the average synaptic weights become a step function of the transition probabilities. We finally show that the information stored in the synaptic weights can be read out by a simple rate-based neural network. Our study shows that synapses encode transition probabilities under general assumptions and this indicates that temporal contiguity is likely to be encoded and harnessed in almost every neural circuit in the brain.

Covariance-based synaptic plasticity in an attractor network model accounts for fast adaptation in free operant learning

In free operant experiments, subjects alternate at will between targets that yield rewards stochastically. Behavior in these experiments is typically characterized by (1) an exponential distribution of stay durations, (2) matching of the relative time spent at a target to its relative share of the total number of rewards, and (3) adaptation after a change in the reward rates that can be very fast. The neural mechanism underlying these regularities is largely unknown. Moreover, current decision-making neural network models typically aim at explaining behavior in discrete-time experiments in which a single decision is made once in every trial, making these models hard to extend to the more natural case of free operant decisions. Here we show that a model based on attractor dynamics, in which transitions are induced by noise and preference is formed via covariance-based synaptic plasticity, can account for the characteristics of behavior in free operant experiments. We compare a specific instance of such a model, in which two recurrently excited populations of neurons compete for higher activity, to the behavior of rats responding on two levers for rewarding brain stimulation on a concurrent variable interval reward schedule (Gallistel et al., 2001). We show that the model is consistent with the rats' behavior, and in particular, with the observed fast adaptation to matching behavior. Further, we show that the neural model can be reduced to a behavioral model, and we use this model to deduce a novel "conservation law," which is consistent with the behavior of the rats.

Sensorimotor learning biases choice behavior: a learning neural field model for decision making

According to a prominent view of sensorimotor processing in primates, selection and specification of possible actions are not sequential operations. Rather, a decision for an action emerges from competition between different movement plans, which are specified and selected in parallel. For action choices which are based on ambiguous sensory input, the frontoparietal sensorimotor areas are considered part of the common underlying neural substrate for selection and specification of action. These areas have been shown capable of encoding alternative spatial motor goals in parallel during movement planning, and show signatures of competitive value-based selection among these goals. Since the same network is also involved in learning sensorimotor associations, competitive action selection (decision making) should not only be driven by the sensory evidence and expected reward in favor of either action, but also by the subject's learning history of different sensorimotor associations. Previous computational models of competitive neural decision making used predefined associations between sensory input and corresponding motor output. Such hard-wiring does not allow modeling of how decisions are influenced by sensorimotor learning or by changing reward contingencies. We present a dynamic neural field model which learns arbitrary sensorimotor associations with a reward-driven Hebbian learning algorithm. We show that the model accurately simulates the dynamics of action selection with different reward contingencies, as observed in monkey cortical recordings, and that it correctly predicted the pattern of choice errors in a control experiment. With our adaptive model we demonstrate how network plasticity, which is required for association learning and adaptation to new reward contingencies, can influence choice behavior. The field model provides an integrated and dynamic account for the operations of sensorimotor integration, working memory and action selection required for decision making in ambiguous choice situations.

Decision making requires the selection between alternative actions. It has been suggested that action selection is not separate from motor preparation of the according actions, but rather that the selection emerges from the competition between different movement plans. We expand on this idea, and ask how action selection mechanisms interact with the learning of new action choices. We present a neurodynamic model that provides an integrated account of action selection and the learning of sensorimotor associations. The model explains recent electrophysiological findings from monkeys' sensorimotor cortex, and correctly predicted a newly described characteristic pattern of their choice errors. Based on the model, we present a theory of how geometrical sensorimotor mapping rules can be learned by association without the need for an explicit representation of the transformation rule, and how the learning history of these associations can have a direct influence on later decision making.

Network Resets in Medial Prefrontal Cortex Mark the Onset of Behavioral Uncertainty

Regions within the prefrontal cortex are thought to process beliefs about the world, but little is known about the circuit dynamics underlying the formation and modification of these beliefs. Using a task that permits dissociation between the activity encoding an animal’s internal state and that encoding aspects of behavior, we found that transient increases in the volatility of activity in the rat medial prefrontal cortex accompany periods when an animal’s belief is modified after an environmental change. Activity across the majority of sampled neurons underwent marked, abrupt, and coordinated changes when prior belief was abandoned in favor of exploration of alternative strategies. These dynamics reflect network switches to a state of instability, which diminishes over the period of exploration as new stable representations are formed.

Statistical Mechanics of Reward-Modulated Learning in Decision-Making Networks

The neural substrates of decision making have been intensively studied using experimental and computational approaches. Alternative-choice tasks accompanying reinforcement have often been employed in investigations into decision making. Choice behavior has been empirically found in many experiments to follow Herrnstein's matching law. A number of theoretical studies have been done on explaining the mechanisms responsible for matching behavior. Various learning rules have been proved in these studies to achieve matching behavior as a steady state of learning processes. The models in the studies have consisted of a few parameters. However, a large number of neurons and synapses are expected to participate in decision making in the brain. We investigated learning behavior in simple but large-scale decision-making networks. We considered the covariance learning rule, which has been demonstrated to achieve matching behavior as a steady state (Loewenstein & Seung, 2006 ). We analyzed model behavior in a thermodynamic limit where the number of plastic synapses went to infinity. By means of techniques of the statistical mechanics, we can derive deterministic differential equations in this limit for the order parameters, which allow an exact calculation of the evolution of choice behavior. As a result, we found that matching behavior cannot be a steady state of learning when the fluctuations in input from individual sensory neurons are so large that they affect the net input to value-encoding neurons. This situation naturally arises when the synaptic strength is sufficiently strong and the excitatory input and the inhibitory input to the value-encoding neurons are balanced. The deviation from matching behavior is caused by increasing variance in the input potential due to the diffusion of synaptic efficacies. This effect causes an undermatching phenomenon, which has been often observed in behavioral experiments.

Robust Working Memory in an Asynchronously Spiking Neural Network Realized with Neuromorphic VLSI

We demonstrate bistable attractor dynamics in a spiking neural network implemented with neuromorphic VLSI hardware. The on-chip network consists of three interacting populations (two excitatory, one inhibitory) of leaky integrate-and-fire (LIF) neurons. One excitatory population is distinguished by strong synaptic self-excitation, which sustains meta-stable states of “high” and “low”-firing activity. Depending on the overall excitability, transitions to the “high” state may be evoked by external stimulation, or may occur spontaneously due to random activity fluctuations. In the former case, the “high” state retains a “working memory” of a stimulus until well after its release. In the latter case, “high” states remain stable for seconds, three orders of magnitude longer than the largest time-scale implemented in the circuitry. Evoked and spontaneous transitions form a continuum and may exhibit a wide range of latencies, depending on the strength of external stimulation and of recurrent synaptic excitation. In addition, we investigated “corrupted” “high” states comprising neurons of both excitatory populations. Within a “basin of attraction,” the network dynamics “corrects” such states and re-establishes the prototypical “high” state. We conclude that, with effective theoretical guidance, full-fledged attractor dynamics can be realized with comparatively small populations of neuromorphic hardware neurons.

Encoding of both positive and negative reward prediction errors by neurons of the primate lateral prefrontal cortex and caudate nucleus

Learning can be motivated by unanticipated success or unexpected failure. The former encourages us to repeat an action or activity, whereas the latter leads us to find an alternative strategy. Understanding the neural representation of these unexpected events is therefore critical to elucidate learning-related circuits. We examined the activity of neurons in the lateral prefrontal cortex (PFC) and caudate nucleus of monkeys as they performed a trial-and-error learning task. Unexpected outcomes were widely represented in both structures, and neurons driven by unexpectedly negative outcomes were as frequent as those activated by unexpectedly positive outcomes. Moreover, both positive and negative reward prediction errors (RPEs) were represented primarily by increases in firing rate, unlike the manner in which dopamine neurons have been observed to reflect these values. Interestingly, positive RPEs tended to appear with shorter latency than negative RPEs, perhaps reflecting the mechanism of their generation. Last, in the PFC but not the caudate, trial-by-trial variations in outcome-related activity were linked to the animals' subsequent behavioral decisions. More broadly, the robustness of RPE signaling by these neurons suggests that actor-critic models of reinforcement learning in which the PFC and particularly the caudate are considered primarily to be "actors" rather than "critics," should be reconsidered to include a prominent evaluative role for these structures.

A mechanism for the formation of hippocampal neuronal firing patterns that represent what happens where

The association of specific events with the context in which they occur is a fundamental feature of episodic memory. However, the underlying network mechanisms generating what-where associations are poorly understood. Recently we reported that some hippocampal principal neurons develop representations of specific events occurring in particular locations (item-position cells). Here, we investigate the emergence of item-position selectivity as rats learn new associations for reward and find that before the animal's performance rises above chance in the task, neurons that will later become item-position cells have a strong selective bias toward one of two behavioral responses, which the animal will subsequently make to that stimulus. This response bias results in an asymmetry of neural activity on correct and error trials that could drive the emergence of particular item specificities based on a simple reward-driven synaptic plasticity mechanism.

Learning with a network of competing synapses

Competition between synapses arises in some forms of correlation-based plasticity. Here we propose a game theory-inspired model of synaptic interactions whose dynamics is driven by competition between synapses in their weak and strong states, which are characterized by different timescales. The learning of inputs and memory are meaningfully definable in an effective description of networked synaptic populations. We study, numerically and analytically, the dynamic responses of the effective system to various signal types, particularly with reference to an existing empirical motor adaptation model. The dependence of the system-level behavior on the synaptic parameters, and the signal strength, is brought out in a clear manner, thus illuminating issues such as those of optimal performance, and the functional role of multiple timescales.

Modeling fast stimulus-response association learning along the occipito-parieto-frontal pathway following rule instructions

On the basis of instructions, humans are able to set up associations between sensory and motor areas of the brain separated by several neuronal relays, within a few seconds. This paper proposes a model of fast learning along the dorsal pathway, from primary visual areas to pre-motor cortex. A new synaptic learning rule is proposed where synaptic efficacies converge rapidly toward a specific value determined by the number of active inputs of a neuron, respecting a principle of resource limitation in terms of total synaptic input efficacy available to a neuron. The efficacies are stable with regards to repeated arrival of spikes in a spike train. This rule reproduces the inverse relationship between initial and final synaptic efficacy observed in long-term potentiation (LTP) experiments. Simulations of learning experiments are conducted in a multilayer network of leaky integrate-and-fire (LIF) spiking neuron models. It is proposed that cortical feedback connections convey a top-down learning-enabling signal that guides bottom-up learning in "hidden" neurons that are not directly exposed to input or output activity. Simulations of repeated presentation of the same stimulus-response pair, show that, under conditions of fast learning with probabilistic synaptic transmission, the networks tend to recruit a new sub-network at each presentation to represent the association, rather than re-using a previously trained one. This increasing allocation of neural resources results in progressively shorter execution times, in line with experimentally observed reduction in response time with practice. This article is part of a Special Issue entitled: Neural Coding.

Prefrontal cortex deactivation in macaques alters activity in the superior colliculus and impairs voluntary control of saccades

The cognitive control of action requires both the suppression of automatic responses to sudden stimuli and the generation of behavior specified by abstract instructions. Though patient, functional imaging and neurophysiological studies have implicated the dorsolateral prefrontal cortex (dlPFC) in these abilities, the mechanism by which the dlPFC exerts this control remains unknown. Here we examined the functional interaction of the dlPFC with the saccade circuitry by deactivating area 46 of the dlPFC and measuring its effects on the activity of single superior colliculus neurons in monkeys performing a cognitive saccade task. Deactivation of the dlPFC reduced preparatory activity and increased stimulus-related activity in these neurons. These changes in neural activity were accompanied by marked decreases in task performance as evidenced by longer reaction times and more task errors. The results suggest that the dlPFC participates in the cognitive control of gaze by suppressing stimulus-evoked automatic saccade programs.

Same or different? A neural circuit mechanism of similarity-based pattern match decision making

The ability to judge whether sensory stimuli match an internally represented pattern is central to many brain functions. To elucidate the underlying mechanism, we developed a neural circuit model for match/nonmatch decision making. At the core of this model is a "comparison circuit" consisting of two distinct neural populations: match enhancement cells show higher firing response for a match than a nonmatch to the target pattern, and match suppression cells exhibit the opposite trend. We propose that these two neural pools emerge from inhibition-dominated recurrent dynamics and heterogeneous top-down excitation from a working memory circuit. A downstream system learns, through plastic synapses, to extract the necessary information to make match/nonmatch decisions. The model accounts for key physiological observations from behaving monkeys in delayed match-to-sample experiments, including tasks that require more than simple feature match (e.g., when BB in ABBA sequence must be ignored). A testable prediction is that magnitudes of match enhancement and suppression neural signals are parametrically tuned to the similarity between compared patterns. Furthermore, the same neural signals from the comparison circuit can be used differently in the decision process for different stimulus statistics or tasks; reward-dependent synaptic plasticity enables decision neurons to flexibly adjust the readout scheme to task demands, whereby the most informative neural signals have the highest impact on the decision.

Expressions of multiple neuronal dynamics during sensorimotor learning in the motor cortex of behaving monkeys

Previous studies support the notion that sensorimotor learning involves multiple processes. We investigated the neuronal basis of these processes by recording single-unit activity in motor cortex of non-human primates (Macaca fascicularis), during adaptation to force-field perturbations. Perturbed trials (reaching to one direction) were practiced along with unperturbed trials (to other directions). The number of perturbed trials relative to the unperturbed ones was either low or high, in two separate practice schedules. Unsurprisingly, practice under high-rate resulted in faster learning with more pronounced generalization, as compared to the low-rate practice. However, generalization and retention of behavioral and neuronal effects following practice in high-rate were less stable; namely, the faster learning was forgotten faster. We examined two subgroups of cells and showed that, during learning, the changes in firing-rate in one subgroup depended on the number of practiced trials, but not on time. In contrast, changes in the second subgroup depended on time and practice; the changes in firing-rate, following the same number of perturbed trials, were larger under high-rate than low-rate learning. After learning, the neuronal changes gradually decayed. In the first subgroup, the decay pace did not depend on the practice rate, whereas in the second subgroup, the decay pace was greater following high-rate practice. This group shows neuronal representation that mirrors the behavioral performance, evolving faster but also decaying faster at learning under high-rate, as compared to low-rate. The results suggest that the stability of a new learned skill and its neuronal representation are affected by the acquisition schedule.

An imperfect dopaminergic error signal can drive temporal-difference learning

An open problem in the field of computational neuroscience is how to link synaptic plasticity to system-level learning. A promising framework in this context is temporal-difference (TD) learning. Experimental evidence that supports the hypothesis that the mammalian brain performs temporal-difference learning includes the resemblance of the phasic activity of the midbrain dopaminergic neurons to the TD error and the discovery that cortico-striatal synaptic plasticity is modulated by dopamine. However, as the phasic dopaminergic signal does not reproduce all the properties of the theoretical TD error, it is unclear whether it is capable of driving behavior adaptation in complex tasks. Here, we present a spiking temporal-difference learning model based on the actor-critic architecture. The model dynamically generates a dopaminergic signal with realistic firing rates and exploits this signal to modulate the plasticity of synapses as a third factor. The predictions of our proposed plasticity dynamics are in good agreement with experimental results with respect to dopamine, pre- and post-synaptic activity. An analytical mapping from the parameters of our proposed plasticity dynamics to those of the classical discrete-time TD algorithm reveals that the biological constraints of the dopaminergic signal entail a modified TD algorithm with self-adapting learning parameters and an adapting offset. We show that the neuronal network is able to learn a task with sparse positive rewards as fast as the corresponding classical discrete-time TD algorithm. However, the performance of the neuronal network is impaired with respect to the traditional algorithm on a task with both positive and negative rewards and breaks down entirely on a task with purely negative rewards. Our model demonstrates that the asymmetry of a realistic dopaminergic signal enables TD learning when learning is driven by positive rewards but not when driven by negative rewards.

What are the physiological changes that take place in the brain when we solve a problem or learn a new skill? It is commonly assumed that behavior adaptations are realized on the microscopic level by changes in synaptic efficacies. However, this is hard to verify experimentally due to the difficulties of identifying the relevant synapses and monitoring them over long periods during a behavioral task. To address this question computationally, we develop a spiking neuronal network model of actor-critic temporal-difference learning, a variant of reinforcement learning for which neural correlates have already been partially established. The network learns a complex task by means of an internally generated reward signal constrained by recent findings on the dopaminergic system. Our model combines top-down and bottom-up modelling approaches to bridge the gap between synaptic plasticity and system-level learning. It paves the way for further investigations of the dopaminergic system in reward learning in the healthy brain and in pathological conditions such as Parkinson's disease, and can be used as a module in functional models based on brain-scale circuitry.

The neuronal basis of long-term sensorimotor learning

The brain has a remarkable ability to learn and adjust behavior. For instance, the brain can adjust muscle activation to cope with changes in the environment. However, the neuronal mechanisms behind this adaptation are not clear. To address this fundamental question, this study examines the neuronal basis of long-term sensorimotor learning by recording neuronal activity in the primary motor cortex of monkeys during a long-term adaptation to a force-field perturbation. For 5 consecutive days, the same perturbation was applied to the monkey's hand when reaching to a single target, whereas movements to all other targets were not perturbed. The gradual improvement in performance over these 5 days was correlated to the evolvement in the population neuronal signal, with two timescales of changes in single-cell activity. Specifically, one subgroup of cells showed a relatively fast increase in activity, whereas the other showed a gradual, slower decrease. These adapted patterns of neuronal activity did not involve changes in directional tuning of single cells, suggesting that adaptation was the result of adjustments of the required motor plan by a population of neurons rather than changes in single-cell properties. Furthermore, generalization was mostly expressed in the direction of the required compensatory force during adaptation. Altogether, the neuronal activity and its generalization accord with the adapted motor plan.

A shared resource between declarative memory and motor memory

The neural systems that support motor adaptation in humans are thought to be distinct from those that support the declarative system. Yet, during motor adaptation changes in motor commands are supported by a fast adaptive process that has important properties (rapid learning, fast decay) that are usually associated with the declarative system. The fast process can be contrasted to a slow adaptive process that also supports motor memory, but learns gradually and shows resistance to forgetting. Here we show that after people stop performing a motor task, the fast motor memory can be disrupted by a task that engages declarative memory, but the slow motor memory is immune from this interference. Furthermore, we find that the fast/declarative component plays a major role in the consolidation of the slow motor memory. Because of the competitive nature of declarative and nondeclarative memory during consolidation, impairment of the fast/declarative component leads to improvements in the slow/nondeclarative component. Therefore, the fast process that supports formation of motor memory is not only neurally distinct from the slow process, but it shares critical resources with the declarative memory system.

A biologically based model for the integration of sensory-motor contingencies in rules and plans: a prefrontal cortex based extension of the Distributed Adaptive Control architecture

Intelligence depends on the ability of the brain to acquire and apply rules and representations. At the neuronal level these properties have been shown to critically depend on the prefrontal cortex. Here we present, in the context of the Distributed Adaptive Control architecture (DAC), a biologically based model for flexible control and planning based on key physiological properties of the prefrontal cortex, i.e. reward modulated sustained activity and plasticity of lateral connectivity. We test the model in a series of pertinent tasks, including multiple T-mazes and the Tower of London that are standard experimental tasks to assess flexible control and planning. We show that the model is both able to acquire and express rules that capture the properties of the task and to quickly adapt to changes. Further, we demonstrate that this biomimetic self-contained cognitive architecture generalizes to planning. In addition, we analyze the extended DAC architecture, called DAC 6, as a model that can be applied for the creation of intelligent and psychologically believable synthetic agents.

Multiple timescales of memory in lateral habenula and dopamine neurons

Midbrain dopamine neurons are thought to signal predictions about future rewards based on the memory of past rewarding experience. Little is known about the source of their reward memory and the factors that control its timescale. Here we recorded from dopamine neurons, as well as one of their sources of input, the lateral habenula, while animals predicted upcoming rewards based on the past reward history. We found that lateral habenula and dopamine neurons accessed two distinct reward memories: a short-timescale memory expressed at the start of the task and a near-optimal long-timescale memory expressed when a future reward outcome was revealed. The short- and long-timescale memories were expressed in different forms of reward-oriented eye movements. Our data show that the habenula-dopamine pathway contains multiple timescales of memory and provide evidence for their role in motivated behavior.

Adaptive, behaviorally gated, persistent encoding of task-relevant auditory information in ferret frontal cortex

Top-down signals from frontal cortex are thought to be important in cognitive control of sensory processing. To explore this interaction, we compared activity in ferret frontal cortex and primary auditory cortex (A1) during auditory and visual tasks requiring discrimination between classes of reference and target stimuli. Frontal cortex responses were behaviorally gated, selectively encoded the timing and invariant behavioral meaning of target stimuli, could be rapid in onset, and sometimes persisted for hours following behavior. These results are consistent with earlier findings in A1 that attention triggered rapid, selective, persistent, task-related changes in spectrotemporal receptive fields. Simultaneously recorded local field potentials revealed behaviorally gated changes in inter-areal coherence that were selectively modulated between frontal cortex and focal regions of A1 that were responsive to target sounds. These results suggest that A1 and frontal cortex dynamically establish a functional connection during auditory behavior that shapes the flow of sensory information and maintains a persistent trace of recent task-relevant stimulus features.

Fractals in the nervous system: conceptual implications for theoretical neuroscience

This essay is presented with two principal objectives in mind: first, to document the prevalence of fractals at all levels of the nervous system, giving credence to the notion of their functional relevance; and second, to draw attention to the as yet still unresolved issues of the detailed relationships among power-law scaling, self-similarity, and self-organized criticality. As regards criticality, I will document that it has become a pivotal reference point in Neurodynamics. Furthermore, I will emphasize the not yet fully appreciated significance of allometric control processes. For dynamic fractals, I will assemble reasons for attributing to them the capacity to adapt task execution to contextual changes across a range of scales. The final Section consists of general reflections on the implications of the reviewed data, and identifies what appear to be issues of fundamental importance for future research in the rapidly evolving topic of this review.

Learning Spike-Based Population Codes by Reward and Population Feedback

We investigate a recently proposed model for decision learning in a population of spiking neurons where synaptic plasticity is modulated by a population signal in addition to reward feedback. For the basic model, binary population decision making based on spike/no-spike coding, a detailed computational analysis is given about how learning performance depends on population size and task complexity. Next, we extend the basic model to [Formula: see text]-ary decision making and show that it can also be used in conjunction with other population codes such as rate or even latency coding.

Determinants of multiple semantic priming: a meta-analysis and spike frequency adaptive model of a cortical network

Recall and language comprehension while processing sequences of words involves multiple semantic priming between several related and/or unrelated words. Accounting for multiple and interacting priming effects in terms of underlying neuronal structure and dynamics is a challenge for current models of semantic priming. Further elaboration of current models requires a quantifiable and reliable account of the simplest case of multiple priming resulting from two primes on a target. The meta-analytic approach offers a better understanding of the experimental data from studies on multiple priming regarding the additivity pattern of priming. The meta-analysis points to the effects of prime-target stimuli onset asynchronies on the pattern of underadditivity, overadditivity, or strict additivity of converging activation from multiple primes. The modeling approach is then constrained by results of the meta-analysis. We propose a model of a cortical network embedding spike frequency adaptation, which allows frequency and time-dependent modulation of neural activity. Model results give a comprehensive understanding of the meta-analysis results in terms of dynamics of neuron populations. They also give predictions regarding how stimuli intensities, association strength, and spike frequency adaptation influence multiple priming effects.

The brain's router: a cortical network model of serial processing in the primate brain

The human brain efficiently solves certain operations such as object recognition and categorization through a massively parallel network of dedicated processors. However, human cognition also relies on the ability to perform an arbitrarily large set of tasks by flexibly recombining different processors into a novel chain. This flexibility comes at the cost of a severe slowing down and a seriality of operations (100-500 ms per step). A limit on parallel processing is demonstrated in experimental setups such as the psychological refractory period (PRP) and the attentional blink (AB) in which the processing of an element either significantly delays (PRP) or impedes conscious access (AB) of a second, rapidly presented element. Here we present a spiking-neuron implementation of a cognitive architecture where a large number of local parallel processors assemble together to produce goal-driven behavior. The precise mapping of incoming sensory stimuli onto motor representations relies on a "router" network capable of flexibly interconnecting processors and rapidly changing its configuration from one task to another. Simulations show that, when presented with dual-task stimuli, the network exhibits parallel processing at peripheral sensory levels, a memory buffer capable of keeping the result of sensory processing on hold, and a slow serial performance at the router stage, resulting in a performance bottleneck. The network captures the detailed dynamics of human behavior during dual-task-performance, including both mean RTs and RT distributions, and establishes concrete predictions on neuronal dynamics during dual-task experiments in humans and non-human primates.

Temporal context and conditional associative learning

BACKGROUND

We investigated how temporal context affects the learning of arbitrary visuo-motor associations. Human observers viewed highly distinguishable, fractal objects and learned to choose for each object the one motor response (of four) that was rewarded. Some objects were consistently preceded by specific other objects, while other objects lacked this task-irrelevant but predictive context.

RESULTS

The results of five experiments showed that predictive context consistently and significantly accelerated associative learning. A simple model of reinforcement learning, in which three successive objects informed response selection, reproduced our behavioral results.

CONCLUSIONS

Our results imply that not just the representation of a current event, but also the representations of past events, are reinforced during conditional associative learning. In addition, these findings are broadly consistent with the prediction of attractor network models of associative learning and their prophecy of a persistent representation of past objects.

Attractor concretion as a mechanism for the formation of context representations

Complex tasks often require the memory of recent events, the knowledge about the context in which they occur, and the goals we intend to reach. All this information is stored in our mental states. Given a set of mental states, reinforcement learning (RL) algorithms predict the optimal policy that maximizes future reward. RL algorithms assign a value to each already-known state so that discovering the optimal policy reduces to selecting the action leading to the state with the highest value. But how does the brain create representations of these mental states in the first place? We propose a mechanism for the creation of mental states that contain information about the temporal statistics of the events in a particular context. We suggest that the mental states are represented by stable patterns of reverberating activity, which are attractors of the neural dynamics. These representations are built from neurons that are selective to specific combinations of external events (e.g. sensory stimuli) and pre-existent mental states. Consistent with this notion, we find that neurons in the amygdala and in orbitofrontal cortex (OFC) often exhibit this form of mixed selectivity. We propose that activating different mixed selectivity neurons in a fixed temporal order modifies synaptic connections so that conjunctions of events and mental states merge into a single pattern of reverberating activity. This process corresponds to the birth of a new, different mental state that encodes a different temporal context. The concretion process depends on temporal contiguity, i.e. on the probability that a combination of an event and mental states follows or precedes the events and states that define a certain context. The information contained in the context thereby allows an animal to assign unambiguously a value to the events that initially appeared in different situations with different meanings.

Attractors and noise: twin drivers of decisions and multistability

Perceptual decisions are made not only during goal-directed behavior such as choice tasks, but also occur spontaneously while multistable stimuli are being viewed. In both contexts, the formation of a perceptual decision is best captured by noisy attractor dynamics. Noise-driven attractor transitions can accommodate a wide range of timescales and a hierarchical arrangement with "nested attractors" harbors even more dynamical possibilities. The attractor framework seems particularly promising for understanding higher-level mental states that combine heterogeneous information from a distributed set of brain areas.

Emotion, cognition, and mental state representation in amygdala and prefrontal cortex

Neuroscientists have often described cognition and emotion as separable processes implemented by different regions of the brain, such as the amygdala for emotion and the prefrontal cortex for cognition. In this framework, functional interactions between the amygdala and prefrontal cortex mediate emotional influences on cognitive processes such as decision-making, as well as the cognitive regulation of emotion. However, neurons in these structures often have entangled representations, whereby single neurons encode multiple cognitive and emotional variables. Here we review studies using anatomical, lesion, and neurophysiological approaches to investigate the representation and utilization of cognitive and emotional parameters. We propose that these mental state parameters are inextricably linked and represented in dynamic neural networks composed of interconnected prefrontal and limbic brain structures. Future theoretical and experimental work is required to understand how these mental state representations form and how shifts between mental states occur, a critical feature of adaptive cognitive and emotional behavior.

Synaptic computation underlying probabilistic inference

We propose that synapses may be the workhorse of the neuronal computations that underlie probabilistic reasoning. We built a neural circuit model for probabilistic inference in which information provided by different sensory cues must be integrated and the predictive powers of individual cues about an outcome are deduced through experience. We found that bounded synapses naturally compute, through reward-dependent plasticity, the posterior probability that a choice alternative is correct given that a cue is presented. Furthermore, a decision circuit endowed with such synapses makes choices on the basis of the summed log posterior odds and performs near-optimal cue combination. The model was validated by reproducing salient observations of, and provides insights into, a monkey experiment using a categorization task. Our model thus suggests a biophysical instantiation of the Bayesian decision rule, while predicting important deviations from it similar to the 'base-rate neglect' observed in human studies when alternatives have unequal prior probabilities.

Dual adaptation supports a parallel architecture of motor memory

Although our understanding of the mechanisms underlying motor adaptation has greatly benefited from previous computational models, the architecture of motor memory is still uncertain. On one hand, two-state models that contain both a fast-learning-fast-forgetting process and a slow-learning-slow-forgetting process explain a wide range of data on motor adaptation, but cannot differentiate whether the fast and slow processes are arranged serially or in parallel and cannot account for learning multiple tasks simultaneously. On the other hand, multiple parallel-state models learn multiple tasks simultaneously but cannot account for a number of motor adaptation data. Here, we investigated the architecture of human motor memory by systematically testing possible architectures via a combination of simulations and a dual visuomotor adaptation experimental paradigm. We found that only one parsimonious model can account for both previous motor adaptation data and our dual-task adaptation data: a fast process that contains a single state is arranged in parallel with a slow process that contains multiple states switched via contextual cues. Our result suggests that during motor adaptation, fast and slow processes are updated simultaneously from the same motor learning errors.

Shared mechanisms of perceptual learning and decision making

Perceptual decisions require the brain to weigh noisy evidence from sensory neurons to form categorical judgments that guide behavior. Here we review behavioral and neurophysiological findings suggesting that at least some forms of perceptual learning do not appear to affect the response properties of neurons that represent the sensory evidence. Instead, improved perceptual performance results from changes in how the sensory evidence is selected and weighed to form the decision. We discuss the implications of this idea for possible sites and mechanisms of training-induced improvements in perceptual processing in the brain.

Proactive inhibitory control and attractor dynamics in countermanding action: a spiking neural circuit model

Flexible behavior depends on the brain's ability to suppress a habitual response or to cancel a planned movement whenever needed. Such inhibitory control has been studied using the countermanding paradigm in which subjects are required to withhold an imminent movement when a stop signal appears infrequently in a fraction of trials. To elucidate the circuit mechanism of inhibitory control of action, we developed a recurrent network model consisting of spiking movement (GO) neurons and fixation (STOP) neurons, based on neurophysiological observations in the frontal eye field and superior colliculus of behaving monkeys. The model places a premium on the network dynamics before the onset of a stop signal, especially the experimentally observed high baseline activity of fixation neurons, which is assumed to be modulated by a persistent top-down control signal, and their synaptic interaction with movement neurons. The model simulated observed neural activity and fit behavioral performance quantitatively. In contrast to a race model in which the STOP process is initiated at the onset of a stop signal, in our model whether a movement will eventually be canceled is determined largely by the proactive top-down control and the stochastic network dynamics, even before the appearance of the stop signal. A prediction about the correlation between the fixation neural activity and the behavioral outcome was verified in the neurophysiological data recorded from behaving monkeys. The proposed mechanism for adjusting control through tonically active neurons that inhibit movement-producing neurons has significant implications for exploring the basis of impulsivity associated with psychiatric disorders.

Learning substrates in the primate prefrontal cortex and striatum: sustained activity related to successful actions

Learning from experience requires knowing whether a past action resulted in a desired outcome. The prefrontal cortex and basal ganglia are thought to play key roles in such learning of arbitrary stimulus-response associations. Previous studies have found neural activity in these areas, similar to dopaminergic neurons' signals, that transiently reflect whether a response is correct or incorrect. However, it is unclear how this transient activity, which fades in under a second, influences actions that occur much later. Here, we report that single neurons in both areas show sustained, persistent outcome-related responses. Moreover, single behavioral outcomes influence future neural activity and behavior: behavioral responses are more often correct and single neurons more accurately discriminate between the possible responses when the previous response was correct. These long-lasting signals about trial outcome provide a way to link one action to the next and may allow reward signals to be combined over time to implement successful learning.

Monkey prefrontal cortical pyramidal and putative interneurons exhibit differential patterns of activity between prosaccade and antisaccade tasks

Previous studies have shown that prefrontal cortex (PFC) neurons carry task-related activity; however, it is largely unknown how this selectivity is implemented in PFC microcircuitry. Here, we exploited known differences in extracellular action potential waveforms, and antidromic identification, to classify PFC neurons as putative pyramidal or interneurons, and investigate their relative contributions to task-selectivity. We recorded the activity of prefrontal neurons while monkeys performed a blocked pro/antisaccade task in which they were required to look either toward or away from a peripheral visual stimulus. We found systematic differences in activity between neuron classes. Putative pyramidal neurons had higher stimulus-related activity on antisaccade trials, whereas putative interneurons exhibited greater activity for prosaccades. These findings suggest that task-selectivity in the PFC may be shaped by interactions between these neuronal classes. They are also consistent with the robust deficits in antisaccade performance frequently observed in disease states associated with PFC dysfunction.

Dynamic readout of behaviorally relevant signals from area MT during task switching

The processes underlying dynamic changes in human behavior during real situations contain much irrelevant information and represent a key issue facing neuroscientists. Although the roles played by the frontal cortex in this switching behavior have been well documented, little is known regarding how neural pathways governing sensorimotor associations accomplish such a switch. We addressed this question by recording activities of middle temporal (MT) neurons in monkeys switching between direction versus depth discrimination tasks. Although the monkeys successfully switched between the tasks, neural sensitivity did not change as a function of task. More importantly, neurons that signaled the same motor output showed trial-to-trial covariation between neuronal responses and perceptual judgments during both tasks, whereas neurons that signaled the opposite motor output showed no covariation in either task. These results suggest that task switching is accomplished via communication from distinct populations of neurons when sensorimotor associations switch within a short time period.

Optimality and robustness of a biophysical decision-making model under norepinephrine modulation

The locus ceruleus (LC) can exhibit tonic or phasic activity and release norepinephrine (NE) throughout the cortex, modulating cellular excitability and synaptic efficacy and thus influencing behavioral performance. We study the effects of LC-NE modulation on decision making in two-alternative forced-choice tasks by changing conductances in a biophysical neural network model, and we investigate how it affects performance measured in terms of reward rate. We find that low tonic NE levels result in unmotivated behavior and high levels in impulsive, inaccurate choices, but that near-optimal performance can occur over a broad middle range. Robustness is greatest when pyramidal cells are less strongly modulated than interneurons, and superior performance can be achieved with phasic NE release, provided only glutamatergic synapses are modulated. We also show that network functions such as sensory information accumulation and short-term memory can be modulated by tonic NE levels, and that previously observed diverse evoked cell responses may be due to network effects.

Dynamic encoding of action selection by the medial striatum

Successful foragers respond flexibly to environmental stimuli. Behavioral flexibility depends on a number of brain areas that send convergent projections to the medial striatum, such as the medial prefrontal cortex, orbital frontal cortex, and amygdala. Here, we tested the hypothesis that neurons in the medial striatum are involved in flexible action selection, by representing changes in stimulus-reward contingencies. Using a novel Go/No-go reaction-time task, we changed the reward value of individual stimuli within single experimental sessions. We simultaneously recorded neuronal activity in the medial and ventral parts of the striatum of rats. The rats modified their actions in the task after the changes in stimulus-reward contingencies. This was preceded by dynamic modulations of spike activity in the medial, but not the ventral, striatum. Our results suggest that the medial striatum biases animals to collect rewards to potentially valuable stimuli and can rapidly influence flexible behavior.

Neurophysiological bases of exponential sensory decay and top-down memory retrieval: a model

Behavioral observations suggest that multiple sensory elements can be maintained for a short time, forming a perceptual buffer which fades after a few hundred milliseconds. Only a subset of this perceptual buffer can be accessed under top-down control and broadcasted to working memory and consciousness. In turn, single-cell studies in awake-behaving monkeys have identified two distinct waves of response to a sensory stimulus: a first transient response largely determined by stimulus properties and a second wave dependent on behavioral relevance, context and learning. Here we propose a simple biophysical scheme which bridges these observations and establishes concrete predictions for neurophsyiological experiments in which the temporal interval between stimulus presentation and top-down allocation is controlled experimentally. Inspired in single-cell observations, the model involves a first transient response and a second stage of amplification and retrieval, which are implemented biophysically by distinct operational modes of the same circuit, regulated by external currents. We explicitly investigated the neuronal dynamics, the memory trace of a presented stimulus and the probability of correct retrieval, when these two stages were bracketed by a temporal gap. The model predicts correctly the dependence of performance with response times in interference experiments suggesting that sensory buffering does not require a specific dedicated mechanism and establishing a direct link between biophysical manipulations and behavioral observations leading to concrete predictions.

Learning flexible sensori-motor mappings in a complex network

Given the complex structure of the brain, how can synaptic plasticity explain the learning and forgetting of associations when these are continuously changing? We address this question by studying different reinforcement learning rules in a multilayer network in order to reproduce monkey behavior in a visuomotor association task. Our model can only reproduce the learning performance of the monkey if the synaptic modifications depend on the pre- and postsynaptic activity, and if the intrinsic level of stochasticity is low. This favored learning rule is based on reward modulated Hebbian synaptic plasticity and shows the interesting feature that the learning performance does not substantially degrade when adding layers to the network, even for a complex problem.