Covariance-based synaptic plasticity in an attractor network model accounts for fast adaptation in free operant learning

Fast adaptation to rule switching using neuronal surprise

In humans and animals, surprise is a physiological reaction to an unexpected event, but how surprise can be linked to plausible models of neuronal activity is an open problem. We propose a self-supervised spiking neural network model where a surprise signal is extracted from an increase in neural activity after an imbalance of excitation and inhibition. The surprise signal modulates synaptic plasticity via a three-factor learning rule which increases plasticity at moments of surprise. The surprise signal remains small when transitions between sensory events follow a previously learned rule but increases immediately after rule switching. In a spiking network with several modules, previously learned rules are protected against overwriting, as long as the number of modules is larger than the total number of rules-making a step towards solving the stability-plasticity dilemma in neuroscience. Our model relates the subjective notion of surprise to specific predictions on the circuit level.

Matching Behaviours and Rewards

Matching describes how behaviour is related to rewards. The matching law holds when the ratio of an individual's behaviours equals the ratio of the rewards obtained. From its origins in the study of pigeons working for food in the laboratory, the law has been applied to a range of species, both in the laboratory and outside it (e.g., human sporting decisions). Probability matching occurs when the probability of a behaviour equals the probability of being rewarded. Input matching predicts the distribution of individuals across habitats. We evaluate the rationality of the matching law and probability matching, expose the logic of matching in real-world cases, review how recent neuroscience findings relate to matching, and suggest future research directions.

Deviation from the matching law reflects an optimal strategy involving learning over multiple timescales

Behavior deviating from our normative expectations often appears irrational. For example, even though behavior following the so-called matching law can maximize reward in a stationary foraging task, actual behavior commonly deviates from matching. Such behavioral deviations are interpreted as a failure of the subject; however, here we instead suggest that they reflect an adaptive strategy, suitable for uncertain, non-stationary environments. To prove it, we analyzed the behavior of primates that perform a dynamic foraging task. In such nonstationary environment, learning on both fast and slow timescales is beneficial: fast learning allows the animal to react to sudden changes, at the price of large fluctuations (variance) in the estimates of task relevant variables. Slow learning reduces the fluctuations but costs a bias that causes systematic behavioral deviations. Our behavioral analysis shows that the animals solved this bias-variance tradeoff by combining learning on both fast and slow timescales, suggesting that learning on multiple timescales can be a biologically plausible mechanism for optimizing decisions under uncertainty.

Recent experience can only provide limited information to guide decisions in a volatile environment. Here, the authors report that the choices made by a monkey in a dynamic foraging task can be better explained by a model that combines learning on both fast and slow timescales.

Striatal action-value neurons reconsidered

It is generally believed that during economic decisions, striatal neurons represent the values associated with different actions. This hypothesis is based on studies, in which the activity of striatal neurons was measured while the subject was learning to prefer the more rewarding action. Here we show that these publications are subject to at least one of two critical confounds. First, we show that even weak temporal correlations in the neuronal data may result in an erroneous identification of action-value representations. Second, we show that experiments and analyses designed to dissociate action-value representation from the representation of other decision variables cannot do so. We suggest solutions to identifying action-value representation that are not subject to these confounds. Applying one solution to previously identified action-value neurons in the basal ganglia we fail to detect action-value representations. We conclude that the claim that striatal neurons encode action-values must await new experiments and analyses.

Adaptive learning and decision-making under uncertainty by metaplastic synapses guided by a surprise detection system

Recent experiments have shown that animals and humans have a remarkable ability to adapt their learning rate according to the volatility of the environment. Yet the neural mechanism responsible for such adaptive learning has remained unclear. To fill this gap, we investigated a biophysically inspired, metaplastic synaptic model within the context of a well-studied decision-making network, in which synapses can change their rate of plasticity in addition to their efficacy according to a reward-based learning rule. We found that our model, which assumes that synaptic plasticity is guided by a novel surprise detection system, captures a wide range of key experimental findings and performs as well as a Bayes optimal model, with remarkably little parameter tuning. Our results further demonstrate the computational power of synaptic plasticity, and provide insights into the circuit-level computation which underlies adaptive decision-making.

Frustrated expected reward induces differential transcriptional changes in the mouse brain

Frustration represents a particular aspect of the addictive process that is related to loss of control when the expected reward is not obtained. We aim to study the consequences of frustrated expected reward on gene expression in the mouse brain. For this purpose, we used an operant model of frustration using palatable food as reward combined with microarrays. Transcriptomic profiles of frontal cortex, ventral striatum and hippocampus were analysed in five groups of mice: (1) positive control receiving palatable food and the cue light as conditioned stimulus; (2) frustrated group only receiving the cue light; (3) extinction learning group that did not receive palatable food nor the light; (4) negative control that never received the reinforcer nor the light during the whole experiment; and (5) yoked that received palatable food passively. Gene expression changes produced by frustration were revealed in the frontal cortex and ventral striatum, but not in the hippocampus. Most of the changes, such as the modification of the dopamine-DARPP-32 signalling pathway, were common in both areas and estimated to have neuronal origin. Extinction learning induced transcriptional changes only in the ventral striatum, with most genes showing down-regulation and without alteration in the dopamine-DARPP-32 signalling pathway. Active palatable food-seeking behaviour induced changes in gene expression in ventral striatum mainly affecting cell communication. In conclusion, frustration behaviour-induced changes in frontal cortex and ventral striatum mainly related to dopamine-DARPP-32 signalling that could play an important role in the loss of behavioural control during the addictive processes.

Spatial generalization in operant learning: lessons from professional basketball

In operant learning, behaviors are reinforced or inhibited in response to the consequences of similar actions taken in the past. However, because in natural environments the “same” situation never recurs, it is essential for the learner to decide what “similar” is so that he can generalize from experience in one state of the world to future actions in different states of the world. The computational principles underlying this generalization are poorly understood, in particular because natural environments are typically too complex to study quantitatively. In this paper we study the principles underlying generalization in operant learning of professional basketball players. In particular, we utilize detailed information about the spatial organization of shot locations to study how players adapt their attacking strategy in real time according to recent events in the game. To quantify this learning, we study how a make \ miss from one location in the court affects the probabilities of shooting from different locations. We show that generalization is not a spatially-local process, nor is governed by the difficulty of the shot. Rather, to a first approximation, players use a simplified binary representation of the court into 2 pt and 3 pt zones. This result indicates that rather than using low-level features, generalization is determined by high-level cognitive processes that incorporate the abstract rules of the game.

According to the law of effect, formulated a century ago by Edward Thorndike, actions which are rewarded in a particular situation are more likely to be executed when that same situation recurs. However, in natural settings the same situation never recurs and therefore, generalization from one state of the world to other states is an essential part of the process of learning. In this paper we utilize basketball statistics to study the computational principles underlying generalization in operant learning of professional basketball players. We show that players are more likely to attempt a field goal from the vicinity of a previously made shot than they are from the vicinity of a missed shot, as expected from the law of effect. However, the outcome of a shot can also affect the likelihood of attempting another shot at a different location. Using hierarchical clustering we characterize the spatial pattern of generalization and show that generalization is primarily determined by the type of shot, 3 pt vs. 2 pt. This result indicates that rather than using low-level features, generalization is determined by high-level cognitive processes that incorporate the abstract rules of the game.

Vague-to-crisp dynamics of percept formation modeled as operant (selectionist) process

We model the vague-to-crisp dynamics of forming percepts in the brain by combining two methodologies: dynamic logic (DL) and operant learning process. Forming percepts upon the presentation of visual inputs is likened to model selection based on sampled evidence. Our framework utilizes the DL in selecting the correct "percept" among competing ones, but uses an intrinsic reward mechanism to allow stochastic online update in lieu of performing the optimization step of the DL framework. We discuss the connection of our framework with cognitive processing and the intentional neurodynamic cycle.

Dynamical regimes in neural network models of matching behavior

The matching law constitutes a quantitative description of choice behavior that is often observed in foraging tasks. According to the matching law, organisms distribute their behavior across available response alternatives in the same proportion that reinforcers are distributed across those alternatives. Recently a few biophysically plausible neural network models have been proposed to explain the matching behavior observed in the experiments. Here we study systematically the learning dynamics of these networks while performing a matching task on the concurrent variable interval (VI) schedule. We found that the model neural network can operate in one of three qualitatively different regimes depending on the parameters that characterize the synaptic dynamics and the reward schedule: (1) a matching behavior regime, in which the probability of choosing an option is roughly proportional to the baiting fractional probability of that option; (2) a perseverative regime, in which the network tends to make always the same decision; and (3) a tristable regime, in which the network can either perseverate or choose the two targets randomly approximately with the same probability. Different parameters of the synaptic dynamics lead to different types of deviations from the matching law, some of which have been observed experimentally. We show that the performance of the network depends on the number of stable states of each synapse and that bistable synapses perform close to optimal when the proper learning rate is chosen. Because our model provides a link between synaptic dynamics and qualitatively different behaviors, this work provides us with insight into the effects of neuromodulators on adaptive behaviors and psychiatric disorders.

Stochasticity, bistability and the wisdom of crowds: a model for associative learning in genetic regulatory networks

It is generally believed that associative memory in the brain depends on multistable synaptic dynamics, which enable the synapses to maintain their value for extended periods of time. However, multistable dynamics are not restricted to synapses. In particular, the dynamics of some genetic regulatory networks are multistable, raising the possibility that even single cells, in the absence of a nervous system, are capable of learning associations. Here we study a standard genetic regulatory network model with bistable elements and stochastic dynamics. We demonstrate that such a genetic regulatory network model is capable of learning multiple, general, overlapping associations. The capacity of the network, defined as the number of associations that can be simultaneously stored and retrieved, is proportional to the square root of the number of bistable elements in the genetic regulatory network. Moreover, we compute the capacity of a clonal population of cells, such as in a colony of bacteria or a tissue, to store associations. We show that even if the cells do not interact, the capacity of the population to store associations substantially exceeds that of a single cell and is proportional to the number of bistable elements. Thus, we show that even single cells are endowed with the computational power to learn associations, a power that is substantially enhanced when these cells form a population.

Psychobiology of cocaine addiction: Contribution of a multi-symptomatic animal model of loss of control

Transition to addiction is the shift from controlled to uncontrolled drug use that occurs after prolonged drug intake in a limited number of drug users. A major challenge of addiction research in recent years has been to develop models for studying this pathological transition. Toward this goal, a DSM-IV/5-based multi-symptomatic model of cocaine addiction has been developed in the rat. It is based on an operational translation of the main features of the disease. 1. Addiction is not just taking drug; it is a non-adaptive drug use: The procedure models addiction in relation to its clinical definition. 2. All drug users do not face the same individual risk of developing addiction: The model includes an individual-based approach. 3. Addiction develops after protracted periods of controlled drug use: This procedure allows for the study of the long-term shift from controlled drug use to addiction. We describe this model in detail and show how it can contribute to our understanding of the pathophysiology of cocaine addiction. This article is part of a Special Issue entitled 'NIDA 40th Anniversary Issue'.

A multistep general theory of transition to addiction

BACKGROUND

Several theories propose alternative explanations for drug addiction.

OBJECTIVES

We propose a general theory of transition to addiction that synthesizes knowledge generated in the field of addiction into a unitary explanatory frame.

MAJOR PRINCIPLES OF THE THEORY

Transition to addiction results from a sequential three-step interaction between: (1) individual vulnerability; (2) degree/amount of drug exposure. The first step, sporadic recreational drug use is a learning process mediated by overactivation of neurobiological substrates of natural rewards that allows most individuals to perceive drugs as highly rewarding stimuli. The second, intensified, sustained, escalated drug use occurs in some vulnerable individuals who have a hyperactive dopaminergic system and impaired prefrontal cortex function. Sustained and prolonged drug use induces incentive sensitization and an allostatic state that makes drugs strongly wanted and needed. Habit formation can also contribute to stabilizing sustained drug use. The last step, loss of control of drug intake and full addiction, is due to a second vulnerable phenotype. This loss-of-control-prone phenotype is triggered by long-term drug exposure and characterized by long-lasting loss of synaptic plasticity in reward areas in the brain that induce a form of behavioral crystallization resulting in loss of control of drug intake. Because of behavioral crystallization, drugs are now not only wanted and needed but also pathologically mourned when absent.

CONCLUSIONS

This general theory demonstrates that drug addiction is a true psychiatric disease caused by a three-step interaction between vulnerable individuals and amount/duration of drug exposure.