Reward, motivation, and reinforcement learning

Dual reward prediction components yield Pavlovian sign- and goal-tracking

Reinforcement learning (RL) has become a dominant paradigm for understanding animal behaviors and neural correlates of decision-making, in part because of its ability to explain Pavlovian conditioned behaviors and the role of midbrain dopamine activity as reward prediction error (RPE). However, recent experimental findings indicate that dopamine activity, contrary to the RL hypothesis, may not signal RPE and differs based on the type of Pavlovian response (e.g. sign- and goal-tracking responses). In this study, we address this discrepancy by introducing a new neural correlate for learning reward predictions; the correlate is called "cue-evoked reward". It refers to a recall of reward evoked by the cue that is learned through simple cue-reward associations. We introduce a temporal difference learning model, in which neural correlates of the cue itself and cue-evoked reward underlie learning of reward predictions. The animal's reward prediction supported by these two correlates is divided into sign and goal components respectively. We relate the sign and goal components to approach responses towards the cue (i.e. sign-tracking) and the food-tray (i.e. goal-tracking) respectively. We found a number of correspondences between simulated models and the experimental findings (i.e. behavior and neural responses). First, the development of modeled responses is consistent with those observed in the experimental task. Second, the model's RPEs were similar to dopamine activity in respective response groups. Finally, goal-tracking, but not sign-tracking, responses rapidly emerged when RPE was restored in the simulated models, similar to experiments with recovery from dopamine-antagonist. These results suggest two complementary neural correlates, corresponding to the cue and its evoked reward, form the basis for learning reward predictions in the sign- and goal-tracking rats.

States of curiosity modulate hippocampus-dependent learning via the dopaminergic circuit

People find it easier to learn about topics that interest them, but little is known about the mechanisms by which intrinsic motivational states affect learning. We used functional magnetic resonance imaging to investigate how curiosity (intrinsic motivation to learn) influences memory. In both immediate and one-day-delayed memory tests, participants showed improved memory for information that they were curious about and for incidental material learned during states of high curiosity. Functional magnetic resonance imaging results revealed that activity in the midbrain and the nucleus accumbens was enhanced during states of high curiosity. Importantly, individual variability in curiosity-driven memory benefits for incidental material was supported by anticipatory activity in the midbrain and hippocampus and by functional connectivity between these regions. These findings suggest a link between the mechanisms supporting extrinsic reward motivation and intrinsic curiosity and highlight the importance of stimulating curiosity to create more effective learning experiences.

The brain correlates of the effects of monetary and verbal rewards on intrinsic motivation

Apart from everyday duties, such as doing the laundry or cleaning the house, there are tasks we do for pleasure and enjoyment. We do such tasks, like solving crossword puzzles or reading novels, without any external pressure or force; instead, we are intrinsically motivated: we do the tasks because we enjoy doing them. Previous studies suggest that external rewards, i.e., rewards from the outside, affect the intrinsic motivation to engage in a task: while performance-based monetary rewards are perceived as controlling and induce a business-contract framing, verbal rewards praising one's competence can enhance the perceived self-determination. Accordingly, the former have been shown to decrease intrinsic motivation, whereas the latter have been shown to increase intrinsic motivation. The present study investigated the neural processes underlying the effects of monetary and verbal rewards on intrinsic motivation in a group of 64 subjects applying functional magnetic resonance imaging (fMRI). We found that, when participants received positive performance feedback, activation in the anterior striatum and midbrain was affected by the nature of the reward; compared to a non-rewarded control group, activation was higher while monetary rewards were administered. However, we did not find a decrease in activation after reward withdrawal. In contrast, we found an increase in activation for verbal rewards: after verbal rewards had been withdrawn, participants showed a higher activation in the aforementioned brain areas when they received success compared to failure feedback. We further found that, while participants worked on the task, activation in the lateral prefrontal cortex was enhanced after the verbal rewards were administered and withdrawn.

Reward components of feeding behavior are preserved during mouse aging

Eating behavior depends on associations between the sensory and energetic properties of foods. Healthful balance of these factors is a challenge for industrialized societies that have an abundance of food, food choices and food-related cues. Here, we were interested in whether appetitive conditioning changes as a function of age. Operant and pavlovian conditioning experiments (rewarding stimulus was a palatable food) in male mice (aged 3, 6, and 15 months) showed that implicit (non-declarative) memory remains intact during aging. Two other essential components of eating behavior, motivation and hedonic preference for rewarding foods, were also found not to be altered in aging mice. Specifically, hedonic responding by satiated mice to isocaloric foods of differing sensory properties (sucrose, milk) was similar in all age groups; importantly, however, this paradigm disclosed that older animals adjust their energy intake according to energetic need. Based on the assumption that the mechanisms that control feeding are conserved across species, it would appear that overeating and obesity in humans reflects a mismatch between ancient physiological mechanisms and today's cue-laden environment. The implication of the present results showing that aging does not impair the ability to learn stimulus-food associations is that the risk of overeating in response to food cues is maintained through to old age.

Transcranial direct current stimulation over the left prefrontal cortex increases randomness of choice in instrumental learning

INTRODUCTION

There is growing evidence from neuro-computational studies that instrumental learning involves the dynamic interaction of a computationally rigid, low-level striatal and a more flexible, high-level prefrontal component.

METHODS

To evaluate the role of the prefrontal cortex in instrumental learning, we applied anodal transcranial direct current stimulation (tDCS) optimized for the left dorsolateral prefrontal cortex, by using realistic MR-derived finite element model-based electric field simulations. In a study with a double-blind, sham-controlled, repeated-measures design, sixteen male participants performed a probabilistic learning task while receiving anodal and sham tDCS in a counterbalanced order.

RESULTS

Compared to sham tDCS, anodal tDCS significantly increased the amount of maladaptive shifting behavior after optimal outcomes during learning when reward probabilities were highly dissociable. Derived parameters of the Q-learning computational model further revealed a significantly increased model parameter that was sensitive to random action selection in the anodal compared to the sham tDCS session, whereas the learning rate parameter was not influenced significantly by tDCS.

CONCLUSION

These results congruently indicate that prefrontal tDCS during instrumental learning increased randomness of choice, possibly reflecting the influence of the cognitive prefrontal component.

Intentional change, intrinsic motivations, and goal generation

Wilson et al. draw our attention to the problem of a science of intentional change. We stress the connection between their approach and existing paradigms for learning and goal generation that have been developed in machine learning, artificial intelligence, and psychology. These paradigms outline the structural principles of a domain-general and teleologically open agent.

Absence of "Warm-Up" during Active Avoidance Learning in a Rat Model of Anxiety Vulnerability: Insights from Computational Modeling

Avoidance behaviors, in which a learned response causes omission of an upcoming punisher, are a core feature of many psychiatric disorders. While reinforcement learning (RL) models have been widely used to study the development of appetitive behaviors, less attention has been paid to avoidance. Here, we present a RL model of lever-press avoidance learning in Sprague-Dawley (SD) rats and in the inbred Wistar Kyoto (WKY) rat, which has been proposed as a model of anxiety vulnerability. We focus on “warm-up,” transiently decreased avoidance responding at the start of a testing session, which is shown by SD but not WKY rats. We first show that a RL model can correctly simulate key aspects of acquisition, extinction, and warm-up in SD rats; we then show that WKY behavior can be simulated by altering three model parameters, which respectively govern the tendency to explore new behaviors vs. exploit previously reinforced ones, the tendency to repeat previous behaviors regardless of reinforcement, and the learning rate for predicting future outcomes. This suggests that several, dissociable mechanisms may contribute independently to strain differences in behavior. The model predicts that, if the “standard” inter-session interval is shortened from 48 to 24 h, SD rats (but not WKY) will continue to show warm-up; we confirm this prediction in an empirical study with SD and WKY rats. The model further predicts that SD rats will continue to show warm-up with inter-session intervals as short as a few minutes, while WKY rats will not show warm-up, even with inter-session intervals as long as a month. Together, the modeling and empirical data indicate that strain differences in warm-up are qualitative rather than just the result of differential sensitivity to task variables. Understanding the mechanisms that govern expression of warm-up behavior in avoidance may lead to better understanding of pathological avoidance, and potential pathways to modify these processes.

Bursts and heavy tails in temporal and sequential dynamics of foraging decisions

A fundamental understanding of behavior requires predicting when and what an individual will choose. However, the actual temporal and sequential dynamics of successive choices made among multiple alternatives remain unclear. In the current study, we tested the hypothesis that there is a general bursting property in both the timing and sequential patterns of foraging decisions. We conducted a foraging experiment in which rats chose among four different foods over a continuous two-week time period. Regarding when choices were made, we found bursts of rapidly occurring actions, separated by time-varying inactive periods, partially based on a circadian rhythm. Regarding what was chosen, we found sequential dynamics in affective choices characterized by two key features: (a) a highly biased choice distribution; and (b) preferential attachment, in which the animals were more likely to choose what they had previously chosen. To capture the temporal dynamics, we propose a dual-state model consisting of active and inactive states. We also introduce a satiation-attainment process for bursty activity, and a non-homogeneous Poisson process for longer inactivity between bursts. For the sequential dynamics, we propose a dual-control model consisting of goal-directed and habit systems, based on outcome valuation and choice history, respectively. This study provides insights into how the bursty nature of behavior emerges from the interaction of different underlying systems, leading to heavy tails in the distribution of behavior over time and choices.

To understand spontaneous animal behavior, two key elements must be explained: when an action is made and what is chosen. Here, we conducted a foraging experiment in which rats chose among four different foods over a continuous two-week time period. With respect to when, we found bursts of rapidly occurring responses separated by long inactive periods. With respect to what, we found biased choice behavior toward the favorite items as well as repetitive behavior, reflecting goal-directed and habitual responding, respectively. We account for the when and what components with two distinct computational mechanisms, each composed of two processes: (a) active and inactive states for the temporal dynamics, and (b) goal-directed and habitual control for the sequential dynamics. This study provides behavioral and computational insights into the dynamical properties of decision-making that determine both when an animal will act and what the animal will choose. Our findings provide an integrated framework for describing the temporal and sequential structure of everyday choices among, for example, food, music, books, brands, web-browsing and social interaction.

Impulse control disorders in Parkinson's disease are associated with dysfunction in stimulus valuation but not action valuation

A substantial subset of Parkinson's disease (PD) patients suffers from impulse control disorders (ICDs), which are side effects of dopaminergic medication. Dopamine plays a key role in reinforcement learning processes. One class of reinforcement learning models, known as the actor-critic model, suggests that two components are involved in these reinforcement learning processes: a critic, which estimates values of stimuli and calculates prediction errors, and an actor, which estimates values of potential actions. To understand the information processing mechanism underlying impulsive behavior, we investigated stimulus and action value learning from reward and punishment in four groups of participants: on-medication PD patients with ICD, on-medication PD patients without ICD, off-medication PD patients without ICD, and healthy controls. Analysis of responses suggested that participants used an actor-critic learning strategy and computed prediction errors based on stimulus values rather than action values. Quantitative model fits also revealed that an actor-critic model of the basal ganglia with different learning rates for positive and negative prediction errors best matched the choice data. Moreover, whereas ICDs were associated with model parameters related to stimulus valuation (critic), PD was associated with parameters related to action valuation (actor). Specifically, PD patients with ICD exhibited lower learning from negative prediction errors in the critic, resulting in an underestimation of adverse consequences associated with stimuli. These findings offer a specific neurocomputational account of the nature of compulsive behaviors induced by dopaminergic drugs.

Laboratory studies of imitation/field studies of tradition: towards a synthesis in animal social learning

Here I discuss: (1) historical precedents that have resulted in comparative psychologists accepting the two-action method as the "gold standard" in laboratory investigations of imitation learning, (2) evidence suggesting that the two-action procedure may not be adequate to answer questions concerning the role of imitation in the development of traditional behaviors of animals living in natural habitat, and (3) an alternative approach to the laboratory study of imitation that might increase the relevance of laboratory studies of imitation to the work of behavioral ecologists/primatologists interested in animal traditions and their relationship to human cumulative culture. This article is part of a Special Issue entitled: Tribute to Tom Zentall.

Predictive codes of familiarity and context during the perceptual learning of facial identities

Face recognition is a key component of successful social behaviour. However, the computational processes that underpin perceptual learning and recognition as faces transition from unfamiliar to familiar are poorly understood. In predictive coding, learning occurs through prediction errors that update stimulus familiarity, but recognition is a function of both stimulus and contextual familiarity. Here we show that behavioural responses on a two-option face recognition task can be predicted by the level of contextual and facial familiarity in a computational model derived from predictive-coding principles. Using fMRI, we show that activity in the superior temporal sulcus varies with the contextual familiarity in the model, whereas activity in the fusiform face area covaries with the prediction error parameter that updated facial familiarity. Our results characterize the key computations underpinning the perceptual learning of faces, highlighting that the functional properties of face-processing areas conform to the principles of predictive coding.

Demonstrator skill modulates observational aversive learning

Learning to avoid danger by observing others can be relatively safe, because it does not incur the potential costs of individual trial and error. However, information gained through social observation might be less reliable than information gained through individual experiences, underscoring the need to apply observational learning critically. In order for observational learning to be adaptive it should be modulated by the skill of the observed person, the demonstrator. To address this issue, we used a probabilistic two-choice task where participants learned to minimize the number of electric shocks through individual learning and by observing a demonstrator performing the same task. By manipulating the demonstrator's skill we varied how useful the observable information was; the demonstrator either learned the task quickly or did not learn it at all (random choices). To investigate the modulatory effect in detail, the task was performed under three conditions of available observable information; no observable information, observation of choices only, and observation of both the choices and their consequences. As predicted, our results showed that observable information can improve performance compared to individual learning, both when the demonstrator is skilled and unskilled; observation of consequences improved performance for both groups while observation of choices only improved performance for the group observing the skilled demonstrator. Reinforcement learning modeling showed that demonstrator skill modulated observational learning from the demonstrator's choices, but not their consequences, by increasing the degree of imitation over time for the group that observed a fast learner. Our results show that humans can adaptively modulate observational learning in response to the usefulness of observable information.

Delegation to automaticity: the driving force for cognitive evolution?

The ability to delegate control over repetitive tasks from higher to lower neural centers may be a fundamental innovation in human cognition. Plausibly, the massive neurocomputational challenges associated with the mastery of balance during the evolution of bipedality in proto-humans provided a strong selective advantage to individuals with brains capable of efficiently transferring tasks in this way. Thus, the shift from quadrupedal to bipedal locomotion may have driven the rapid evolution of distinctive features of human neuronal functioning. We review recent studies of functional neuroanatomy that bear upon this hypothesis, and identify ways to test our ideas.

Impairments in goal-directed actions predict treatment response to cognitive-behavioral therapy in social anxiety disorder

Social anxiety disorder is characterized by excessive fear and habitual avoidance of social situations. Decision-making models suggest that patients with anxiety disorders may fail to exhibit goal-directed control over actions. We therefore investigated whether such biases may also be associated with social anxiety and to examine the relationship between such behavior with outcomes from cognitive-behavioral therapy. Patients diagnosed with social anxiety and controls completed an instrumental learning task in which two actions were performed to earn food outcomes. After outcome devaluation, where one outcome was consumed to satiety, participants were re-tested in extinction. Results indicated that, as expected, controls were goal-directed, selectively reducing responding on the action that previously delivered the devalued outcome. Patients with social anxiety, however, exhibited no difference in responding on either action. This loss of a devaluation effect was associated with greater symptom severity and poorer response to therapy. These findings indicate that variations in goal-directed control in social anxiety may represent both a behavioral endophenotype and may be used to predict individuals who will respond to learning-based therapies.

Surprise beyond prediction error

Surprise drives learning. Various neural “prediction error” signals are believed to underpin surprise‐based reinforcement learning. Here, we report a surprise signal that reflects reinforcement learning but is neither un/signed reward prediction error (RPE) nor un/signed state prediction error (SPE). To exclude these alternatives, we measured surprise responses in the absence of RPE and accounted for a host of potential SPE confounds. This new surprise signal was evident in ventral striatum, primary sensory cortex, frontal poles, and amygdala. We interpret these findings via a normative model of surprise. Hum Brain Mapp 35:4805–4814, 2014. © 2014 The Authors. Human Brain Mapping Published by Wiley Periodicals, Inc.

Exploration and learning in capuchin monkeys (Sapajus spp.): the role of action-outcome contingencies

Animals have a strong propensity to explore the environment. Spontaneous exploration has a great biological significance since it allows animals to discover and learn the relation between specific behaviours and their consequences. The role of the contingency between action and outcome for learning has been mainly investigated in instrumental learning settings and much less in free exploration contexts. We tested 16 capuchin monkeys (Sapajus spp.) with a mechatronic platform that allowed complex modules to be manipulated and to produce different outcomes. Experimental subjects could manipulate the modules and discover the contingencies between their own specific actions and the outcomes produced (i.e., the opening and lighting of a box). By contrast, Control subjects could operate on the modules, but the outcomes experienced were those performed by their paired Experimental subjects ("yoked-control" paradigm). In the exploration phase, in which no food reward was present, Experimental subjects spent more time on the board and manipulated the modules more than Yoked subjects. Experimental subjects outperformed Yoked subjects in the following test phase, where success required recalling the effective action so to open the box, now baited with food. These findings demonstrate that the opportunity to experience action-outcome contingencies in the absence of extrinsic rewards promotes capuchins' exploration and facilitates learning processes. Thus, this intrinsically motivated learning represents a powerful mechanism allowing the acquisition of skills and cognitive competence that the individual can later exploit for adaptive purposes.

Prosocial peer affiliation suppresses genetic influences on non-aggressive antisocial behaviors during childhood

BACKGROUND

Available research has suggested that affiliation with prosocial peers reduces child and adolescent antisocial behavior. However, the etiologic mechanisms driving this association remain unclear. The current study sought to evaluate whether this association takes the form of a gene-environment interaction (G × E) in which prosocial peer affiliation acts to reduce the consequences of genetic risk for non-aggressive antisocial behavior during childhood.

METHOD

Our sample consisted of 500 twin pairs aged 6-10 years from the Michigan State University Twin Registry (MSUTR).

RESULTS

The results robustly support moderation by prosocial peer affiliation. Genetic influences on non-aggressive antisocial behavior were observed to be several times larger in those with lower levels of prosocial peer affiliation than in those with higher levels of prosocial peer affiliation. This pattern of results persisted even after controlling for gene-environment correlations and deviant peer affiliation, and when restricting our analyses to those twins who shared all or nearly all of their friends.

CONCLUSIONS

Such findings not only suggest that prosocial peer affiliation moderates genetic influences on non-aggressive antisocial behaviors during childhood but also provide support for the theoretical notion that protective environmental experiences may exert their influence by promoting resilience to genetic risk.

Modelling individual differences in the form of Pavlovian conditioned approach responses: a dual learning systems approach with factored representations

Reinforcement Learning has greatly influenced models of conditioning, providing powerful explanations of acquired behaviour and underlying physiological observations. However, in recent autoshaping experiments in rats, variation in the form of Pavlovian conditioned responses (CRs) and associated dopamine activity, have questioned the classical hypothesis that phasic dopamine activity corresponds to a reward prediction error-like signal arising from a classical Model-Free system, necessary for Pavlovian conditioning. Over the course of Pavlovian conditioning using food as the unconditioned stimulus (US), some rats (sign-trackers) come to approach and engage the conditioned stimulus (CS) itself - a lever - more and more avidly, whereas other rats (goal-trackers) learn to approach the location of food delivery upon CS presentation. Importantly, although both sign-trackers and goal-trackers learn the CS-US association equally well, only in sign-trackers does phasic dopamine activity show classical reward prediction error-like bursts. Furthermore, neither the acquisition nor the expression of a goal-tracking CR is dopamine-dependent. Here we present a computational model that can account for such individual variations. We show that a combination of a Model-Based system and a revised Model-Free system can account for the development of distinct CRs in rats. Moreover, we show that revising a classical Model-Free system to individually process stimuli by using factored representations can explain why classical dopaminergic patterns may be observed for some rats and not for others depending on the CR they develop. In addition, the model can account for other behavioural and pharmacological results obtained using the same, or similar, autoshaping procedures. Finally, the model makes it possible to draw a set of experimental predictions that may be verified in a modified experimental protocol. We suggest that further investigation of factored representations in computational neuroscience studies may be useful.

Racial bias shapes social reinforcement learning

Both emotional facial expressions and markers of racial-group belonging are ubiquitous signals in social interaction, but little is known about how these signals together affect future behavior through learning. To address this issue, we investigated how emotional (threatening or friendly) in-group and out-group faces reinforced behavior in a reinforcement-learning task. We asked whether reinforcement learning would be modulated by intergroup attitudes (i.e., racial bias). The results showed that individual differences in racial bias critically modulated reinforcement learning. As predicted, racial bias was associated with more efficiently learned avoidance of threatening out-group individuals. We used computational modeling analysis to quantitatively delimit the underlying processes affected by social reinforcement. These analyses showed that racial bias modulates the rate at which exposure to threatening out-group individuals is transformed into future avoidance behavior. In concert, these results shed new light on the learning processes underlying social interaction with racial-in-group and out-group individuals.

Adenosine receptor control of cognition in normal and disease

Adenosine and adenosine receptors (ARs) are increasingly recognized as important therapeutic targets for controlling cognition under normal and disease conditions for its dual roles of neuromodulation as well as of homeostatic function in the brain. This chapter first presents the unique ability of adenosine, by acting on the inhibitory A1 and facilitating A2A receptor, to integrate dopamine, glutamate, and BNDF signaling and to modulate synaptic plasticity (e.g., long-term potentiation and long-term depression) in brain regions relevant to learning and memory, providing the molecular and cellular bases for adenosine receptor (AR) control of cognition. This led to the demonstration of AR modulation of social recognition memory, working memory, reference memory, reversal learning, goal-directed behavior/habit formation, Pavlovian fear conditioning, and effort-related behavior. Furthermore, human and animal studies support that AR activity can also, through cognitive enhancement and neuroprotection, reverse cognitive impairments in animal models of Alzheimer's disease (AD), Parkinson's disease (PD), Huntington's disease, and schizophrenia. Lastly, epidemiological evidence indicates that regular human consumption of caffeine, the most widely used psychoactive drug and nonselective AR antagonists, is associated with the reduced cognitive decline in aging and AD patients, and with the reduced risk in developing PD. Thus, there is a convergence of the molecular studies revealing AR as molecular targets for integrating neurotransmitter signaling and controlling synaptic plasticity, with animal studies demonstrating the strong procognitive impact upon AR antagonism in normal and disease brains and with epidemiological and clinical evidences in support of caffeine and AR drugs for therapeutic modulation of cognition. Since some of adenosine A2A receptor antagonists are already in phase III clinical trials for motor benefits in PD patients with remarkable safety profiles, additional animal and human studies to better understand the mechanism underlying the AR-mediated control of cognition under normal and disease conditions will provide the required rationale to stimulate the necessary clinical investigation to rapidly translate adenosine and AR drug as a novel strategy to control memory impairment in neuropsychiatric disorders.

Stress-protective neural circuits: not all roads lead through the prefrontal cortex

Exposure to an uncontrollable stressor elicits a constellation of physiological and behavioral sequel in laboratory rats that often reflect aspects of anxiety and other emotional disruptions. We review evidence suggesting that plasticity within the serotonergic dorsal raphe nucleus (DRN) is critical to the expression of uncontrollable stressor-induced anxiety. Specifically, after uncontrollable stressor exposure subsequent anxiogenic stimuli evoke greater 5-HT release in DRN terminal regions including the amygdala and striatum; and pharmacological blockade of postsynaptic 5-HT(2C) receptors in these regions prevents expression of stressor-induced anxiety. Importantly, the controllability of stress, the presence of safety signals, and a history of exercise mitigate the expression of stressor-induced anxiety. These stress-protective factors appear to involve distinct neural substrates; with stressor controllability requiring the medial prefrontal cortex, safety signals the insular cortex and exercise affecting the 5-HT system directly. Knowledge of the distinct yet converging mechanisms underlying these stress-protective factors could provide insight into novel strategies for the treatment and prevention of stress-related psychiatric disorders.

Fatal attraction: ventral striatum predicts costly choice errors in humans

Animals approach rewards and cues associated with reward, even when this behavior is irrelevant or detrimental to the attainment of these rewards. Motivated by these findings we study the biology of financially-costly approach behavior in humans. Our subjects passively learned to predict the occurrence of erotic rewards. We show that neuronal responses in ventral striatum during this Pavlovian learning task stably predict an individual's general tendency towards financially-costly approach behavior in an active choice task several months later. Our data suggest that approach behavior may prevent some individuals from acting in their own interests.

Dopamine and extinction: a convergence of theory with fear and reward circuitry

Research on dopamine lies at the intersection of sophisticated theoretical and neurobiological approaches to learning and memory. Dopamine has been shown to be critical for many processes that drive learning and memory, including motivation, prediction error, incentive salience, memory consolidation, and response output. Theories of dopamine's function in these processes have, for the most part, been developed from behavioral approaches that examine learning mechanisms in reward-related tasks. A parallel and growing literature indicates that dopamine is involved in fear conditioning and extinction. These studies are consistent with long-standing ideas about appetitive-aversive interactions in learning theory and they speak to the general nature of cellular and molecular processes that underlie behavior. We review the behavioral and neurobiological literature showing a role for dopamine in fear conditioning and extinction. At a cellular level, we review dopamine signaling and receptor pharmacology, cellular and molecular events that follow dopamine receptor activation, and brain systems in which dopamine functions. At a behavioral level, we describe theories of learning and dopamine function that could describe the fundamental rules underlying how dopamine modulates different aspects of learning and memory processes.

Linear combination of one-step predictive information with an external reward in an episodic policy gradient setting: a critical analysis

One of the main challenges in the field of embodied artificial intelligence is the open-ended autonomous learning of complex behaviors. Our approach is to use task-independent, information-driven intrinsic motivation(s) to support task-dependent learning. The work presented here is a preliminary step in which we investigate the predictive information (the mutual information of the past and future of the sensor stream) as an intrinsic drive, ideally supporting any kind of task acquisition. Previous experiments have shown that the predictive information (PI) is a good candidate to support autonomous, open-ended learning of complex behaviors, because a maximization of the PI corresponds to an exploration of morphology- and environment-dependent behavioral regularities. The idea is that these regularities can then be exploited in order to solve any given task. Three different experiments are presented and their results lead to the conclusion that the linear combination of the one-step PI with an external reward function is not generally recommended in an episodic policy gradient setting. Only for hard tasks a great speed-up can be achieved at the cost of an asymptotic performance lost.

The neurobiology of psychopathic traits in youths

Conduct disorder is a childhood behaviour disorder that is characterized by persistent aggressive or antisocial behaviour that disrupts the child's environment and impairs his or her functioning. A proportion of children with conduct disorder have psychopathic traits. Psychopathic traits consist of a callous-unemotional component and an impulsive-antisocial component, which are associated with two core impairments. The first is a reduced empathic response to the distress of other individuals, which primarily reflects reduced amygdala responsiveness to distress cues; the second is deficits in decision making and in reinforcement learning, which reflects dysfunction in the ventromedial prefrontal cortex and striatum. Genetic and prenatal factors contribute to the abnormal development of these neural systems, and social-environmental variables that affect motivation influence the probability that antisocial behaviour will be subsequently displayed.

A motivation model for interaction between parent and child based on the need for relatedness

In parent-child communication, emotions are evoked by various types of intrinsic and extrinsic motivation. Those emotions encourage actions that promote more interactions. We present a motivation model of infant-caregiver interactions, in which relatedness, one of the most important basic psychological needs, is a variable that increases with experiences of emotion sharing. Besides being an important factor of pleasure, relatedness is a meta-factor that affects other factors such as stress and emotional mirroring. The proposed model is implemented in an artificial agent equipped with a system to recognize gestures and facial expressions. The baby-like agent successfully interacts with an actual human and adversely reacts when the caregiver suddenly ceases facial expressions, similar to the “still-face paradigm” demonstrated by infants in psychological experiments. In the simulation experiment, two agents, each controlled by the proposed motivation model, show relatedness-dependent emotional communication that mimics actual human communication.

In vivo two-photon Ca2+ imaging reveals selective reward effects on stimulus-specific assemblies in mouse visual cortex

Experiences can alter functional properties of neurons in primary sensory neocortex but it is poorly understood how stimulus-reward associations contribute to these changes. Using in vivo two-photon calcium imaging in mouse primary visual cortex (V1), we show that association of a directional visual stimulus with reward results in broadened orientation tuning and sharpened direction tuning in a stimulus-selective subpopulation of V1 neurons. Neurons with preferred orientations similar, but not identical to, the CS+ selectively increased their tuning curve bandwidth and thereby exhibited an increased response amplitude at the CS+ orientation. The increase in response amplitude was observed for a small range of orientations around the CS+ orientation. A nonuniform spatial distribution of reward effects across the cortical surface was observed, as the spatial distance between pairs of CS+ tuned neurons was reduced compared with pairs of CS- tuned neurons and pairs of control directions or orientations. These data show that, in primary visual cortex, formation of a stimulus-reward association results in selective alterations in stimulus-specific assemblies rather than population-wide effects.

Why skill matters

Maximizing rewards per unit time is ideal for success and survival in humans and animals. This goal can be approached by speeding up behavior aiming at rewards and this is done most efficiently by acquiring skills. Importantly, reward-directed skills consist of two components: finding a good object (i.e., object skill) and acting on the object (i.e., action skill), which occur sequentially. Recent studies suggest that object skill is based on high-capacity memory for object-value associations. When a learned object is encountered the corresponding memory is quickly expressed as a value-based gaze bias, leading to the automatic acquisition or avoidance of the object. Object skill thus plays a crucial role in increasing rewards per unit time.

Distinct basal ganglia circuits controlling behaviors guided by flexible and stable values

Choosing valuable objects is critical for survival, but their values may change flexibly or remain stable. Therefore, animals should be able to update the object values flexibly by recent experiences and retain them stably by long-term experiences. However, it is unclear how the brain encodes the two conflicting forms of values and controls behavior accordingly. We found that distinct circuits of the primate caudate nucleus control behavior selectively in the flexible and stable value conditions. Single caudate neurons encoded the values of visual objects in a regionally distinct manner: flexible value coding in the caudate head and stable value coding in the caudate tail. Monkeys adapted in both conditions by looking at objects with higher values. Importantly, inactivation of each caudate subregion disrupted the high-low value discrimination selectively in the flexible or stable context. This parallel complementary mechanism enables animals to choose valuable objects in both flexible and stable conditions.

Frontal theta overrides pavlovian learning biases

Pavlovian biases influence learning and decision making by intricately coupling reward seeking with action invigoration and punishment avoidance with action suppression. This bias is not always adaptive-it can often interfere with instrumental requirements. The prefrontal cortex is thought to help resolve such conflict between motivational systems, but the nature of this control process remains unknown. EEG recordings of midfrontal theta band power are sensitive to conflict and predictive of adaptive control over behavior, but it is not clear whether this signal reflects control over conflict between motivational systems. Here we used a task that orthogonalized action requirements and outcome valence while recording concurrent EEG in human participants. By applying a computational model of task performance, we derived parameters reflective of the latent influence of Pavlovian bias and how it was modulated by midfrontal theta power during motivational conflict. Between subjects, those who performed better under Pavlovian conflict exhibited higher midfrontal theta power. Within subjects, trial-to-trial variance in theta power was predictive of ability to overcome the influence of the Pavlovian bias, and this effect was most pronounced in subjects with higher midfrontal theta to conflict. These findings demonstrate that midfrontal theta is not only a sensitive index of prefrontal control, but it can also reflect the application of top-down control over instrumental processes.

Dopaminergic control of motivation and reinforcement learning: a closed-circuit account for reward-oriented behavior

Humans and animals take actions quickly when they expect that the actions lead to reward, reflecting their motivation. Injection of dopamine receptor antagonists into the striatum has been shown to slow such reward-seeking behavior, suggesting that dopamine is involved in the control of motivational processes. Meanwhile, neurophysiological studies have revealed that phasic response of dopamine neurons appears to represent reward prediction error, indicating that dopamine plays central roles in reinforcement learning. However, previous attempts to elucidate the mechanisms of these dopaminergic controls have not fully explained how the motivational and learning aspects are related and whether they can be understood by the way the activity of dopamine neurons itself is controlled by their upstream circuitries. To address this issue, we constructed a closed-circuit model of the corticobasal ganglia system based on recent findings regarding intracortical and corticostriatal circuit architectures. Simulations show that the model could reproduce the observed distinct motivational effects of D1- and D2-type dopamine receptor antagonists. Simultaneously, our model successfully explains the dopaminergic representation of reward prediction error as observed in behaving animals during learning tasks and could also explain distinct choice biases induced by optogenetic stimulation of the D1 and D2 receptor-expressing striatal neurons. These results indicate that the suggested roles of dopamine in motivational control and reinforcement learning can be understood in a unified manner through a notion that the indirect pathway of the basal ganglia represents the value of states/actions at a previous time point, an empirically driven key assumption of our model.

Strategic cognitive sequencing: a computational cognitive neuroscience approach

We address strategic cognitive sequencing, the "outer loop" of human cognition: how the brain decides what cognitive process to apply at a given moment to solve complex, multistep cognitive tasks. We argue that this topic has been neglected relative to its importance for systematic reasons but that recent work on how individual brain systems accomplish their computations has set the stage for productively addressing how brain regions coordinate over time to accomplish our most impressive thinking. We present four preliminary neural network models. The first addresses how the prefrontal cortex (PFC) and basal ganglia (BG) cooperate to perform trial-and-error learning of short sequences; the next, how several areas of PFC learn to make predictions of likely reward, and how this contributes to the BG making decisions at the level of strategies. The third models address how PFC, BG, parietal cortex, and hippocampus can work together to memorize sequences of cognitive actions from instruction (or "self-instruction"). The last shows how a constraint satisfaction process can find useful plans. The PFC maintains current and goal states and associates from both of these to find a "bridging" state, an abstract plan. We discuss how these processes could work together to produce strategic cognitive sequencing and discuss future directions in this area.

Modafinil increases reward salience in a slot machine game in low and high impulsivity pathological gamblers

This study examined the effects of modafinil (200 mg) on slot machine betting profiles from a previous sample of low and high impulsivity (LI/HI) pathological gamblers (10/Group; Zack and Poulos, 2009). Hierarchical regression assessed the prospective relationship between Payoff and Bet Size on consecutive trials, along with moderating effects of Group, Cumulative Winnings (low/high) and Phase of game (early/late) under drug and placebo. Y intercepts for the simple regressions of Bet Size on Payoff indexed overall motivation to bet. Under placebo, both groups gauged their bets less closely to the preceding Payoff as trials continued when Winnings were low but not high. Under modafinil, both groups gauged their bets more closely to the preceding Payoff when Winnings were low but gauged their bets less closely to the previous Payoff when Winnings were high. The tendency to gauge bets closely to the previous Payoff coincided with a bias toward low overall Bet Size, and modafinil accentuated this relationship, in LI but not HI subjects. Results suggest that modafinil increases the salience of environmental rewards, leading to more tightly calibrated responses to individual rewards when resources are low, but progressively loosens reward-response calibration when resources are high. Increased relative impact of phasic vs. tonic dopamine signals may account for patterns seen at low vs. high Winnings, respectively, under the drug. Clinically, modafinil may deter pathological gamblers from chasing losses but also encourage them to continue betting rather than quit while they are ahead. Whether low-dose modafinil confers more uniform benefits deserves investigation.

Functional circuits and anatomical distribution of response properties in the primate amygdala

Recent electrophysiological studies on the primate amygdala have advanced our understanding of how individual neurons encode information relevant to emotional processes, but it remains unclear how these neurons are functionally and anatomically organized. To address this, we analyzed cross-correlograms of amygdala spike trains recorded during a task in which monkeys learned to associate novel images with rewarding and aversive outcomes. Using this task, we have recently described two populations of amygdala neurons: one that responds more strongly to images predicting reward (positive value-coding), and another that responds more strongly to images predicting an aversive stimulus (negative value-coding). Here, we report that these neural populations are organized into distinct, but anatomically intermingled, appetitive and aversive functional circuits, which are dynamically modulated as animals used the images to predict outcomes. Furthermore, we report that responses to sensory stimuli are prevalent in the lateral amygdala, and are also prevalent in the medial amygdala for sensory stimuli that are emotionally significant. The circuits identified here could potentially mediate valence-specific emotional behaviors thought to involve the amygdala.

Attenuated neural response to gamble outcomes in drug-naive patients with Parkinson's disease

Parkinson's disease results from the degeneration of dopaminergic neurons in the substantia nigra, manifesting as a spectrum of motor, cognitive and affective deficits. Parkinson's disease also affects reward processing, but disease-related deficits in reinforcement learning are thought to emerge at a slower pace than motor symptoms as the degeneration progresses from dorsal to ventral striatum. Dysfunctions in reward processing are difficult to study in Parkinson's disease as most patients have been treated with dopaminergic drugs, which sensitize reward responses in the ventral striatum, commonly resulting in impulse control disorders. To circumvent this treatment confound, we assayed the neural basis of reward processing in a group of newly diagnosed patients with Parkinson's disease that had never been treated with dopaminergic drugs. Thirteen drug-naive patients with Parkinson's disease and 12 healthy age-matched control subjects underwent whole-brain functional magnetic resonance imaging while they performed a simple two-choice gambling task resulting in stochastic and parametrically variable monetary gains and losses. In patients with Parkinson's disease, the neural response to reward outcome (as reflected by the blood oxygen level-dependent signal) was attenuated in a large group of mesolimbic and mesocortical regions, comprising the ventral putamen, ventral tegmental area, thalamus and hippocampus. Although these regions showed a linear response to reward outcome in healthy individuals, this response was either markedly reduced or undetectable in drug-naive patients with Parkinson's disease. The results show that the core regions of the meso-cortico-limbic dopaminergic system, including the ventral tegmental area, ventral striatum, and medial orbitofrontal cortex, are already significantly compromised in the early stages of the disease and that these deficits cannot be attributed to the contaminating effect of dopaminergic treatment.

Habits, action sequences and reinforcement learning

It is now widely accepted that instrumental actions can be either goal-directed or habitual; whereas the former are rapidly acquired and regulated by their outcome, the latter are reflexive, elicited by antecedent stimuli rather than their consequences. Model-based reinforcement learning (RL) provides an elegant description of goal-directed action. Through exposure to states, actions and rewards, the agent rapidly constructs a model of the world and can choose an appropriate action based on quite abstract changes in environmental and evaluative demands. This model is powerful but has a problem explaining the development of habitual actions. To account for habits, theorists have argued that another action controller is required, called model-free RL, that does not form a model of the world but rather caches action values within states allowing a state to select an action based on its reward history rather than its consequences. Nevertheless, there are persistent problems with important predictions from the model; most notably the failure of model-free RL correctly to predict the insensitivity of habitual actions to changes in the action-reward contingency. Here, we suggest that introducing model-free RL in instrumental conditioning is unnecessary, and demonstrate that reconceptualizing habits as action sequences allows model-based RL to be applied to both goal-directed and habitual actions in a manner consistent with what real animals do. This approach has significant implications for the way habits are currently investigated and generates new experimental predictions.

From prediction error to incentive salience: mesolimbic computation of reward motivation

Reward contains separable psychological components of learning, incentive motivation and pleasure. Most computational models have focused only on the learning component of reward, but the motivational component is equally important in reward circuitry, and even more directly controls behavior. Modeling the motivational component requires recognition of additional control factors besides learning. Here I discuss how mesocorticolimbic mechanisms generate the motivation component of incentive salience. Incentive salience takes Pavlovian learning and memory as one input and as an equally important input takes neurobiological state factors (e.g. drug states, appetite states, satiety states) that can vary independently of learning. Neurobiological state changes can produce unlearned fluctuations or even reversals in the ability of a previously learned reward cue to trigger motivation. Such fluctuations in cue-triggered motivation can dramatically depart from all previously learned values about the associated reward outcome. Thus, one consequence of the difference between incentive salience and learning can be to decouple cue-triggered motivation of the moment from previously learned values of how good the associated reward has been in the past. Another consequence can be to produce irrationally strong motivation urges that are not justified by any memories of previous reward values (and without distorting associative predictions of future reward value). Such irrationally strong motivation may be especially problematic in addiction. To understand these phenomena, future models of mesocorticolimbic reward function should address the neurobiological state factors that participate to control generation of incentive salience.

Cognition in insects

A traditional view of cognition is that it involves an internal process that represents, tracks or predicts an external process. This is not a general characteristic of all complex neural processing or feedback control, but rather implies specific forms of processing giving rise to specific behavioural capabilities. In this paper, I will review the evidence for such capabilities in insect navigation and learning. Do insects know where they are, or do they only know what to do? Do they learn what stimuli mean, or do they only learn how to behave?

Functional roles of neurotransmitters and neuromodulators in the dorsal striatum

The dorsal striatum, with its functional microcircuits galore, serves as the primary gateway of the basal ganglia and is known to play a key role in implicit learning. Initially, excitatory inputs from the cortex and thalamus arrive on the direct and indirect pathways, where the precise flow of information is then regulated by local GABAergic interneurons. The balance of excitatory and inhibitory transmission in the dorsal striatum is modulated by neuromodulators such as dopamine and acetylcholine. Under pathophysiological states in the dorsal striatum, an alteration in excitatory and inhibitory transmission may underlie dysfunctional motor control. Here, we review the cellular connections and modulation of striatal microcircuits and propose that modulating the excitatory and inhibitory balance in synaptic transmission of the dorsal striatum is important for regulating locomotion.

Integrating cortico-limbic-basal ganglia architectures for learning model-based and model-free navigation strategies

Behavior in spatial navigation is often organized into map-based (place-driven) vs. map-free (cue-driven) strategies; behavior in operant conditioning research is often organized into goal-directed vs. habitual strategies. Here we attempt to unify the two. We review one powerful theory for distinct forms of learning during instrumental conditioning, namely model-based (maintaining a representation of the world) and model-free (reacting to immediate stimuli) learning algorithms. We extend these lines of argument to propose an alternative taxonomy for spatial navigation, showing how various previously identified strategies can be distinguished as “model-based” or “model-free” depending on the usage of information and not on the type of information (e.g., cue vs. place). We argue that identifying “model-free” learning with dorsolateral striatum and “model-based” learning with dorsomedial striatum could reconcile numerous conflicting results in the spatial navigation literature. From this perspective, we further propose that the ventral striatum plays key roles in the model-building process. We propose that the core of the ventral striatum is positioned to learn the probability of action selection for every transition between states of the world. We further review suggestions that the ventral striatal core and shell are positioned to act as “critics” contributing to the computation of a reward prediction error for model-free and model-based systems, respectively.

The effect of ratio and interval training on Pavlovian-instrumental transfer in mice

Conditional stimuli (CS) that are paired with reward can be used to motivate instrumental responses. This process is called Pavlovian-instrumental transfer (PIT). A recent study in rats suggested that habitual responses are particularly sensitive to the motivational effects of reward cues. The current experiments examined this idea using ratio and interval training in mice. Two groups of animals were trained to lever press for food pellets that were delivered on random ratio or random interval schedules. Devaluation tests revealed that interval training led to habitual responding while ratio training produced goal-directed actions. The presentation of CSs paired with reward led to positive transfer in both groups, however, the size of this effect was much larger in mice that were trained on interval schedules. This result suggests that habitual responses are more sensitive to the motivational influence of reward cues than goal-directed actions. The implications for neurobiological models of motivation and drug seeking behaviors are discussed.

Layered reward signalling through octopamine and dopamine in Drosophila

Dopamine (DA) is synonymous with reward and motivation in mammals^1,2. However, only recently has dopamine been linked to motivated behavior and rewarding reinforcement in fruit flies^3,4. Instead octopamine (OA) has historically been considered the signal for reward in insects^5-7. Here we show using temporal control of neural function in Drosophila that only short-term appetitive memory is reinforced by OA. Moreover, OA-dependent memory formation requires signaling through DA neurons. Part of the OA signal requires the α-adrenergic like OAMB receptor in an identified subset of mushroom body (MB)-targeted DA neurons. OA triggers an increase in intracellular calcium in these DA neurons and their direct activation can substitute for sugar to form appetitive memory, even in flies lacking OA. Analysis of the β-adrenergic like Octβ2R receptor reveals that OA-dependent reinforcement also requires an interaction with DA neurons that control appetitive motivation. These data suggest that sweet taste engages a distributed OA signal that reinforces memory through discrete subsets of MB-targeted DA neurons. In addition, they reconcile prior findings with OA and DA and suggest that reinforcement systems in flies are more similar to mammals than previously envisaged.

Pause and rebound: sensory control of cholinergic signaling in the striatum

Cholinergic interneurons have emerged as one of the key players controlling network functions in the striatum. Extracellularly recorded cholinergic interneurons acquire characteristic responses to sensory stimuli during reward-related learning, including a pause and subsequent rebound in spiking. However, the precise underlying cellular mechanisms have remained elusive. Here, we review recent advances in our understanding of the regulation of cholinergic interneuron activity. We discuss evidence of mechanisms that have been proposed to underlie sensory responses, including antagonistic actions by dopamine, recurrent inhibition via local interneurons, and an intrinsically generated membrane hyperpolarization in response to excitatory inputs. The review highlights outstanding questions and concludes with a model of the sensory responses and their downstream effects through dynamic acetylcholine receptor activation.