Reward, motivation, and reinforcement learning

Unraveling the influence of Pavlovian cues on decision-making: A pre-registered meta-analysis on Pavlovian-to-instrumental transfer

Amidst the replicability crisis, promoting transparency and rigor in research becomes imperative. The Pavlovian-to-Instrumental Transfer (PIT) paradigm is increasingly used in human studies to offer insights into how Pavlovian cues, by anticipating rewards or punishments, influence decision-making and potentially contribute to the development of clinical conditions. However, research on this topic faces challenges, including methodological variability and the need for standardized approaches, which can undermine the quality and robustness of experimental findings. Hence, we conducted a meta-analysis to unravel the methodological, task-related, individual, training, and learning factors that can modulate PIT. By scrutinizing these factors, the present meta-analysis reviews the current literature on human PIT, provides practical guidelines for future research to enhance study outcomes and refine methodologies, and identifies knowledge gaps that can serve as a direction for future studies aiming to advance the comprehension of how Pavlovian cues shape decision-making.

Internally Formed Preferences for Options only Influence Initial Decisions in Gambling Tasks, while the Gambling Outcomes do not Alter these Preferences

All humans must engage in decision-making. Decision-making processes can be broadly classified into internally guided decision-making (IDM), which is determined by individuals' internal value criteria, such as preference, or externally guided decision-making (EDM), which is determined by environmental external value criteria, such as monetary rewards. However, real-life decisions are never made simply using one kind of decision-making, and the relationship between IDM and EDM remains unclear. This study had individuals perform gambling tasks requiring the EDM using stimuli that formed preferences through the preference judgment task as the IDM. Computational model analysis revealed that strong preferences in the IDM affected initial choice behavior in the EDM. Moreover, through the analysis of the subjective preference evaluation after the gambling tasks, we found that even when stimuli that were preferred in the IDM were perceived as less valuable in the EDM, the preference for IDM was maintained after EDM. These results indicate that although internal criteria, such as preferences, influence EDM, the results show that internal and external criteria differ.

Topographically selective motor inhibition under threat of pain

ABSTRACT

Pain-related motor adaptations may be enacted predictively at the mere threat of pain, before pain occurrence. Yet, in humans, the neurophysiological mechanisms underlying motor adaptations in anticipation of pain remain poorly understood. We tracked the evolution of changes in corticospinal excitability (CSE) as healthy adults learned to anticipate the occurrence of lateralized, muscle-specific pain to the upper limb. Using a Pavlovian threat conditioning task, different visual stimuli predicted pain to the right or left forearm (experiment 1) or hand (experiment 2). During stimuli presentation before pain occurrence, single-pulse transcranial magnetic stimulation was applied over the left primary motor cortex to probe CSE and elicit motor evoked potentials from target right forearm and hand muscles. The correlation between participants' trait anxiety and CSE was also assessed. Results showed that threat of pain triggered corticospinal inhibition specifically in the limb where pain was expected. In addition, corticospinal inhibition was modulated relative to the threatened muscle, with threat of pain to the forearm inhibiting the forearm and hand muscles, whereas threat of pain to the hand inhibited the hand muscle only. Finally, stronger corticospinal inhibition correlated with greater trait anxiety. These results advance the mechanistic understanding of pain processes showing that pain-related motor adaptations are enacted at the mere threat of pain, as sets of anticipatory, topographically organized motor changes that are associated with the expected pain and are shaped by individual anxiety levels. Including such anticipatory motor changes into models of pain may lead to new treatments for pain-related disorders.

A transdisciplinary view on curiosity beyond linguistic humans: animals, infants, and artificial intelligence

Curiosity is a core driver for life-long learning, problem-solving and decision-making. In a broad sense, curiosity is defined as the intrinsically motivated acquisition of novel information. Despite a decades-long history of curiosity research and the earliest human theories arising from studies of laboratory rodents, curiosity has mainly been considered in two camps: 'linguistic human' and 'other'. This is despite psychology being heritable, and there are many continuities in cognitive capacities across the animal kingdom. Boundary-pushing cross-disciplinary debates on curiosity are lacking, and the relative exclusion of pre-linguistic infants and non-human animals has led to a scientific impasse which more broadly impedes the development of artificially intelligent systems modelled on curiosity in natural agents. In this review, we synthesize literature across multiple disciplines that have studied curiosity in non-verbal systems. By highlighting how similar findings have been produced across the separate disciplines of animal behaviour, developmental psychology, neuroscience, and computational cognition, we discuss how this can be used to advance our understanding of curiosity. We propose, for the first time, how features of curiosity could be quantified and therefore studied more operationally across systems: across different species, developmental stages, and natural or artificial agents.

Conditional deletion of neurexin-2 impaired behavioral flexibility to alterations in action-outcome contingency

Neurexins (Nrxns) are critical for synapse organization and their mutations have been documented in autism spectrum disorder, schizophrenia, and epilepsy. We recently reported that conditional deletion of Nrxn2, under the control of Emx1Cre promoter, predominately expressed in the neocortex and hippocampus (Emx1-Nrxn2 cKO mice) induced stereotyped patterns of behavior in mice, suggesting behavioral inflexibility. In this study, we investigated the effects of Nrxn2 deletion through two different conditional approaches targeting presynaptic cortical neurons projecting to dorsomedial striatum on the flexibility between goal-directed and habitual actions in response to devaluation of action-outcome (A-O) contingencies in an instrumental learning paradigm or upon reversal of A-O contingencies in a water T-maze paradigm. Nrxn2 deletion through both the conditional approaches induced an inability of mice to discriminate between goal-directed and habitual action strategies in their response to devaluation of A-O contingency. Emx1-Nrxn2 cKO mice exhibited reversal learning deficits, indicating their inability to adopt new action strategies. Overall, our studies showed that Nrxn2 deletion through two distinct conditional deletion approaches impaired flexibility in response to alterations in A-O contingencies. These investigations can lay the foundation for identification of novel genetic factors underlying behavioral inflexibility.

Distributional coding of associative learning in discrete populations of midbrain dopamine neurons

Midbrain dopamine neurons are thought to play key roles in learning by conveying the difference between expected and actual outcomes. Recent evidence suggests diversity in dopamine signaling, yet it remains poorly understood how heterogeneous signals might be organized to facilitate the role of downstream circuits mediating distinct aspects of behavior. Here, we investigated the organizational logic of dopaminergic signaling by recording and labeling individual midbrain dopamine neurons during associative behavior. Our findings show that reward information and behavioral parameters are not only heterogeneously encoded but also differentially distributed across populations of dopamine neurons. Retrograde tracing and fiber photometry suggest that populations of dopamine neurons projecting to different striatal regions convey distinct signals. These data, supported by computational modeling, indicate that such distributional coding can maximize dynamic range and tailor dopamine signals to facilitate specialized roles of different striatal regions.

Stimulus valence moderates self-learning

Self-relevance has been demonstrated to impair instrumental learning. Compared to unfamiliar symbols associated with a friend, analogous stimuli linked with the self are learned more slowly. What is not yet understood, however, is whether this effect extends beyond arbitrary stimuli to material with intrinsically meaningful properties. Take, for example, stimulus valence an established moderator of self-bias. Does the desirability of to-be-learned material influence self-learning? Here, in conjunction with computational modelling (i.e. Reinforcement Learning Drift Diffusion Model analysis), a probabilistic selection task was used to establish if and how stimulus valence (i.e. desirable/undesirable posters) impacts the acquisition of knowledge relating to object-ownership (i.e. owned-by-self vs. owned-by-friend). Several interesting results were observed. First, undesirable posters were learned more rapidly for self compared to friend, an effect that was reversed for desirable posters. Second, learning rates were accompanied by associated differences in reward sensitivity toward desirable and undesirable choice selections as a function of ownership. Third, decisional caution was greater for self-relevant (vs. friend relevant) responses. Collectively, these findings inform understanding of self-function and how valence and stimulus relevance mutually influence probabilistic learning.

Multiple and subject-specific roles of uncertainty in reward-guided decision-making

Decision-making in noisy, changing, and partially observable environments entails a basic tradeoff between immediate reward and longer-term information gain, known as the exploration-exploitation dilemma. Computationally, an effective way to balance this tradeoff is by leveraging uncertainty to guide exploration. Yet, in humans, empirical findings are mixed, from suggesting uncertainty-seeking to indifference and avoidance. In a novel bandit task that better captures uncertainty-driven behavior, we find multiple roles for uncertainty in human choices. First, stable and psychologically meaningful individual differences in uncertainty preferences actually range from seeking to avoidance, which can manifest as null group-level effects. Second, uncertainty modulates the use of basic decision heuristics that imperfectly exploit immediate rewards: a repetition bias and win-stay-lose-shift heuristic. These heuristics interact with uncertainty, favoring heuristic choices under higher uncertainty. These results, highlighting the rich and varied structure of reward-based choice, are a step to understanding its functional basis and dysfunction in psychopathology.

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants-even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.

The role of prospective contingency in the control of behavior and dopamine signals during associative learning

Associative learning depends on contingency, the degree to which a stimulus predicts an outcome. Despite its importance, the neural mechanisms linking contingency to behavior remain elusive. Here we examined the dopamine activity in the ventral striatum - a signal implicated in associative learning - in a Pavlovian contingency degradation task in mice. We show that both anticipatory licking and dopamine responses to a conditioned stimulus decreased when additional rewards were delivered uncued, but remained unchanged if additional rewards were cued. These results conflict with contingency-based accounts using a traditional definition of contingency or a novel causal learning model (ANCCR), but can be explained by temporal difference (TD) learning models equipped with an appropriate inter-trial-interval (ITI) state representation. Recurrent neural networks trained within a TD framework develop state representations like our best 'handcrafted' model. Our findings suggest that the TD error can be a measure that describes both contingency and dopaminergic activity.

Craving money? Evidence from the laboratory and the field

Continuing to gamble despite harmful consequences has plagued human life in many ways, from loss-chasing in problem gamblers to reckless investing during stock market bubbles. Here, we propose that these anomalies in human behavior can sometimes reflect Pavlovian perturbations on instrumental behavior. To show this, we combined key elements of Pavlovian psychology literature and standard economic theory into a single model. In it, when a gambling cue such as a gaming machine or a financial asset repeatedly delivers a good outcome, the agent may start engaging with the cue even when the expected value is negative. Next, we transported the theoretical framework into an experimental task and found that participants behaved like the agent in our model. Last, we applied the model to the domain of real-world financial trading and discovered an asset-pricing anomaly suggesting that market participants are susceptible to the purported Pavlovian bias.

Positive correlation between measures of habitual responding and motivated responding in mice

Habit and motivation are thought to be separate processes, with motivated behavior often considered to be goal directed, whereas habits are defined by the absence of goal-directed control over behavior. However, there has been increasing interrogation of the binary nature of habitual versus goal-directed behavior. Furthermore, although drug and alcohol exposure can promote the formation of habits, drug seeking itself can also be highly flexible, pointing toward the need for complex consideration of the parallel processes that drive behavior. The goal of the current study was to determine whether there was a relation between motivation-as measured by progressive ratio-and habit-as measured by contingency degradation-and whether this relation was affected by ethanol exposure history and sex. The results showed that these measures were positively correlated such that greater contingency insensitivity was associated with achieving higher break points on the progressive-ratio task. However, this relation depended on reinforcement schedule history, ethanol exposure history, and sex. These point to potential relations between measures of habit and motivation and stress the importance of carefully parsing behavioral findings and assays. These findings are also expected to inform future substance use research, as drug history may affect these relations.

The effects of Lysergic Acid Diethylamide (LSD) on the Positive Valence Systems: A Research Domain Criteria (RDoC)-Informed Systematic Review

BACKGROUND AND OBJECTIVES

The renewed interest in psychedelic research provides growing evidence of potentially unique effects on various aspects of reward processing systems. Using the Research Domain Criteria (RDoC) framework, as proposed by the National Institute of Mental Health, we aim to synthesize the existing literature concerning the impact of lysergic acid diethylamide (LSD) on the RDoC's Positive Valence Systems (PVS) domain, and to identify potential avenues for further research.

METHODS

Two LSD-related terms (lysergic acid diethylamide and LSD) and 13 PVS-related terms (reward, happiness, bliss, motivation, reinforcement learning, operant, conditioning, satisfaction, decision making, habit, valence, affect, mood) were used to search electronic databases such as PubMed, Scopus, PsychINFO, and Web of Science for relevant articles. A manual search of the reference list resulted in nine additional articles. After screening, articles and data were evaluated and included based on their relevance to the objective of investigating the effects of LSD on the PVS. Articles and data were excluded if they did not provide information about the PVS, were observational in nature, lacked comparators or reference groups, or were duplicates. A risk of bias assessment was performed using the National Toxicology Program's Office of Health Assessment and Translation (NTP OHAT) risk of bias (RoB) tool. Data from the included articles were collected and structured based on the RDoC bio-behavioral matrix, specifically focusing on the PVS domain and its three constituent constructs: reward responsiveness, reward learning, and reward valuation.

RESULTS

We reviewed 28 clinical studies with 477 participants. Lysergic acid diethylamide, assessed at self-report (23 studies), molecular (5 studies), circuit (4 studies), and paradigm (3 studies) levels, exhibited dose-dependent mood improvement (20 short-term and 3 long-term studies). The subjective and neural effects of LSD were linked to the 5-HT2A receptor (molecular). Animal studies (14 studies) suggested LSD could mildly reinforce conditioned place preference without aversion and reduce responsiveness to other rewards. Findings on reward learning were inconsistent but hinted at potential associative learning enhancements. Reward valuation measures indicated potential reductions in effort expenditure for other reinforcers.

CONCLUSION

Our findings are consistent with our previous work, which indicated classical psychedelics, primarily serotonin 2A receptor agonists, enhanced reward responsiveness in healthy individuals and patient populations. Lysergic acid diethylamide exhibits a unique profile in the reward learning and valuation constructs. Using the RDoC-based framework, we identified areas for future research, enhancing our understanding of the impact of LSD on reward processing. However, applying RDoC to psychedelic research faces limitations due to diverse study designs that were not initially RDoC-oriented. Limitations include subjective outcome measure selection aligned with RDoC constructs and potential bias in synthesizing varied studies. Additionally, some human studies were open-label, introducing potential bias compared to randomized, blinded studies.

Differential effects of reward and punishment on reinforcement-based motor learning and generalization

Reward and punishment have long been recognized as potent modulators of human behavior. Although reinforcement learning is a significant motor learning process, the exact mechanisms underlying how the brain learns movements through reward and punishment are not yet fully understood. Beyond the memory of specific examples, investigating the ability to generalize to new situations offers a better understanding of motor learning. This study hypothesizes that reward and punishment engage qualitatively different motivational systems with different neurochemical and neuroanatomical substrates, which would have differential effects on reinforcement-based motor learning and generalization. To test this hypothesis, two groups of participants learn a motor task in one direction and then relearn the same task in a new direction, receiving only performance-based reward or punishment score feedback. Our findings support our hypothesis, showing that reward led to slower learning but promoted generalization. On the other hand, punishment led to faster learning but impaired generalization. These behavioral differences may be due to different tendencies of movement variability in each group. The punishment group tended to explore more actively than the reward group during the initial learning phase, possibly due to loss aversion. In contrast, the reward group tended to explore more actively than the initial learning phase during the generalization test phase, seemingly recalling the strategy that led to the reward. These results suggest that reward and punishment may engage different neural mechanisms during reinforcement-based motor learning and generalization, with important implications for practical applications such as sports training and motor rehabilitation.NEW & NOTEWORTHY Although reinforcement learning is a significant motor learning process, the mechanisms underlying how the brain learns movements through reward and punishment are not fully understood. We modified a well-established motor adaptation task and used savings (faster relearning) to measure generalization. We found reward led to slower learning but promoted generalization, whereas punishment led to faster learning but impaired generalization, suggesting that reward and punishment may engage different neural mechanisms during reinforcement-based motor learning and generalization.

Inhibition of Dopamine Neurons Prevents Incentive Value Encoding of a Reward Cue: With Revelations from Deep Phenotyping

The survival of an organism is dependent on its ability to respond to cues in the environment. Such cues can attain control over behavior as a function of the value ascribed to them. Some individuals have an inherent tendency to attribute reward-paired cues with incentive motivational value, or incentive salience. For these individuals, termed sign-trackers, a discrete cue that precedes reward delivery becomes attractive and desirable in its own right. Prior work suggests that the behavior of sign-trackers is dopamine-dependent, and cue-elicited dopamine in the NAc is believed to encode the incentive value of reward cues. Here we exploited the temporal resolution of optogenetics to determine whether selective inhibition of ventral tegmental area (VTA) dopamine neurons during cue presentation attenuates the propensity to sign-track. Using male tyrosine hydroxylase (TH)-Cre Long Evans rats, it was found that, under baseline conditions, ∼84% of TH-Cre rats tend to sign-track. Laser-induced inhibition of VTA dopamine neurons during cue presentation prevented the development of sign-tracking behavior, without affecting goal-tracking behavior. When laser inhibition was terminated, these same rats developed a sign-tracking response. Video analysis using DeepLabCutTM revealed that, relative to rats that received laser inhibition, rats in the control group spent more time near the location of the reward cue even when it was not present and were more likely to orient toward and approach the cue during its presentation. These findings demonstrate that cue-elicited dopamine release is critical for the attribution of incentive salience to reward cues.SIGNIFICANCE STATEMENT Activity of dopamine neurons in the ventral tegmental area (VTA) during cue presentation is necessary for the development of a sign-tracking, but not a goal-tracking, conditioned response in a Pavlovian task. We capitalized on the temporal precision of optogenetics to pair cue presentation with inhibition of VTA dopamine neurons. A detailed behavioral analysis with DeepLabCutTM revealed that cue-directed behaviors do not emerge without dopamine neuron activity in the VTA. Importantly, however, when optogenetic inhibition is lifted, cue-directed behaviors increase, and a sign-tracking response develops. These findings confirm the necessity of dopamine neuron activity in the VTA during cue presentation to encode the incentive value of reward cues.

Aquaporin 4 beyond a water channel; participation in motor, sensory, cognitive and psychological performances, a comprehensive review

Aquaporin 4 (AQP4) is a protein highly expressed in the central nervous system (CNS) and peripheral nervous system (PNS) as well as various other organs, whose different sites of action indicate its importance in various functions. AQP4 has a variety of essential roles beyond water homeostasis. In this article, we have for the first time summarized different roles of AQP4 in motor and sensory functions, besides cognitive and psychological performances, and most importantly, possible physiological mechanisms by which AQP4 can exert its effects. Furthermore, we demonstrated that AQP4 participates in pathology of different neurological disorders, various effects depending on the disease type. Since neurological diseases involve a spectrum of dysfunctions and due to the difficulty of obtaining a treatment that can simultaneously affect these deficits, it is therefore suggested that future studies consider the role of this protein in different functional impairments related to neurological disorders simultaneously or separately by targeting AQP4 expression and/or polarity modulation.

Computational mechanisms underlying the dynamics of physical and cognitive fatigue

The willingness to exert effort for reward is essential but comes at the cost of fatigue. Theories suggest fatigue increases after both physical and cognitive exertion, subsequently reducing the motivation to exert effort. Yet a mechanistic understanding of how this happens on a moment-to-moment basis, and whether mechanisms are common to both mental and physical effort, is lacking. In two studies, participants reported momentary (trial-by-trial) ratings of fatigue during an effort-based decision-making task requiring either physical (grip-force) or cognitive (mental arithmetic) effort. Using a novel computational model, we show that fatigue fluctuates from trial-to-trial as a function of exerted effort and predicts subsequent choices. This mechanism was shared across the domains. Selective to the cognitive domain, committing errors also induced momentary increases in feelings of fatigue. These findings provide insight into the computations underlying the influence of effortful exertion on fatigue and motivation, in both physical and cognitive domains.

Cortical excitability in human somatosensory and visual cortex: implications for plasticity and learning - a minireview

The balance of excitation and inhibition plays a key role in plasticity and learning. A frequently used, reliable approach to assess intracortical inhibition relies on measuring paired-pulse behavior. Moreover, recent developments of magnetic resonance spectroscopy allows measuring GABA and glutamate concentrations. We give an overview about approaches employed to obtain information about excitatory states in human participants and discuss their putative relation. We summarize paired-pulse techniques and basic findings characterizing paired-pulse suppression in somatosensory (SI) and (VI) visual areas. Paired-pulse suppression describes the effect of paired sensory stimulation at short interstimulus intervals where the cortical response to the second stimulus is significantly suppressed. Simultaneous assessments of paired-pulse suppression in SI and VI indicated that cortical excitability is not a global phenomenon, but instead reflects the properties of local sensory processing. We review studies using non-invasive brain stimulation and perceptual learning experiments that assessed both perceptual changes and accompanying changes of cortical excitability in parallel. Independent of the nature of the excitation/inhibition marker used these data imply a close relationship between altered excitability and altered performance. These results suggest a framework where increased or decreased excitability is linked with improved or impaired perceptual performance. Recent findings have expanded the potential role of cortical excitability by demonstrating that inhibition markers such as GABA concentrations, paired-pulse suppression or alpha power predict to a substantial degree subsequent perceptual learning outcome. This opens the door for a targeted intervention where subsequent plasticity and learning processes are enhanced by altering prior baseline states of excitability.

The behavioral signature of stepwise learning strategy in male rats and its neural correlate in the basal forebrain

Studies of associative learning have commonly focused on how rewarding outcomes are predicted by either sensory stimuli or animals' actions. However, in many learning scenarios, reward delivery requires the occurrence of both sensory stimuli and animals' actions in a specific order, in the form of behavioral sequences. How such behavioral sequences are learned is much less understood. Here we provide behavioral and neurophysiological evidence to show that behavioral sequences are learned using a stepwise strategy. In male rats learning a new association, learning started from the behavioral event closest to the reward and sequentially incorporated earlier events. This led to the sequential refinement of reward-seeking behaviors, which was characterized by the stepwise elimination of ineffective and non-rewarded behavioral sequences. At the neuronal level, this stepwise learning process was mirrored by the sequential emergence of basal forebrain neuronal responses toward each event, which quantitatively conveyed a reward prediction error signal and promoted reward-seeking behaviors. Together, these behavioral and neural signatures revealed how behavioral sequences were learned in discrete steps and when each learning step took place.

Dissociable control of motivation and reinforcement by distinct ventral striatal dopamine receptors

Dopamine release in striatal circuits, including the nucleus accumbens (NAc), tracks separable features of reward such as motivation and reinforcement. However, the cellular and circuit mechanisms by which dopamine receptors transform dopamine release into distinct constructs of reward remain unclear. Here, we show that dopamine D3 receptor (D3R) signaling in the NAc drives motivated behavior by regulating local NAc microcircuits. Furthermore, D3Rs co-express with dopamine D1 receptors (D1Rs), which regulate reinforcement, but not motivation. Paralleling dissociable roles in reward function, we report non-overlapping physiological actions of D3R and D1R signaling in NAc neurons. Our results establish a novel cellular framework wherein dopamine signaling within the same NAc cell type is physiologically compartmentalized via actions on distinct dopamine receptors. This structural and functional organization provides neurons in a limbic circuit with the unique ability to orchestrate dissociable aspects of reward-related behaviors that are relevant to the etiology of neuropsychiatric disorders.

Anxiety as a disorder of uncertainty: implications for understanding maladaptive anxiety, anxious avoidance, and exposure therapy

In cognitive-behavioral conceptualizations of anxiety, exaggerated threat expectancies underlie maladaptive anxiety. This view has led to successful treatments, notably exposure therapy, but is not consistent with the empirical literature on learning and choice alterations in anxiety. Empirically, anxiety is better described as a disorder of uncertainty learning. How disruptions in uncertainty lead to impairing avoidance and are treated with exposure-based methods, however, is unclear. Here, we integrate concepts from neurocomputational learning models with clinical literature on exposure therapy to propose a new framework for understanding maladaptive uncertainty functioning in anxiety. Specifically, we propose that anxiety disorders are fundamentally disorders of uncertainty learning and that successful treatments, particularly exposure therapy, work by remediating maladaptive avoidance from dysfunctional explore/exploit decisions in uncertain, potentially aversive situations. This framework reconciles several inconsistencies in the literature and provides a path forward to better understand and treat anxiety.

Inhibition of dopamine neurons prevents incentive value encoding of a reward cue: With revelations from deep phenotyping

The survival of an organism is dependent on their ability to respond to cues in the environment. Such cues can attain control over behavior as a function of the value ascribed to them. Some individuals have an inherent tendency to attribute reward-paired cues with incentive motivational value, or incentive salience. For these individuals, termed sign-trackers, a discrete cue that precedes reward delivery becomes attractive and desirable in its own right. Prior work suggests that the behavior of sign-trackers is dopamine-dependent, and cue-elicited dopamine in the nucleus accumbens is believed to encode the incentive value of reward cues. Here we exploited the temporal resolution of optogenetics to determine whether selective inhibition of ventral tegmental area (VTA) dopamine neurons during cue presentation attenuates the propensity to sign-track. Using male tyrosine hydroxylase (TH)-Cre Long Evans rats it was found that, under baseline conditions, ∼84% of TH-Cre rats tend to sign-track. Laser-induced inhibition of VTA dopamine neurons during cue presentation prevented the development of sign-tracking behavior, without affecting goal-tracking behavior. When laser inhibition was terminated, these same rats developed a sign-tracking response. Video analysis using DeepLabCut revealed that, relative to rats that received laser inhibition, rats in the control group spent more time near the location of the reward cue even when it was not present and were more likely to orient towards and approach the cue during its presentation. These findings demonstrate that cue-elicited dopamine release is critical for the attribution of incentive salience to reward cues.

Significance Statement

Activity of dopamine neurons in the ventral tegmental area (VTA) during cue presentation is necessary for the development of a sign-tracking, but not a goal-tracking, conditioned response in a Pavlovian task. We capitalized on the temporal precision of optogenetics to pair cue presentation with inhibition of VTA dopamine neurons. A detailed behavioral analysis with DeepLabCut revealed that cue-directed behaviors do not emerge without VTA dopamine. Importantly, however, when optogenetic inhibition is lifted, cue-directed behaviors increase, and a sign-tracking response develops. These findings confirm the necessity of VTA dopamine during cue presentation to encode the incentive value of reward cues.

Hierarchical categorization learning is associated with representational changes in the dorsal striatum and posterior frontal and parietal cortex

Learning and recognition can be improved by sorting novel items into categories and subcategories. Such hierarchical categorization is easy when it can be performed according to learned rules (e.g., "if car, then automatic or stick shift" or "if boat, then motor or sail"). Here, we present results showing that human participants acquire categorization rules for new visual hierarchies rapidly, and that, as they do, corresponding hierarchical representations of the categorized stimuli emerge in patterns of neural activation in the dorsal striatum and in posterior frontal and parietal cortex. Participants learned to categorize novel visual objects into a hierarchy with superordinate and subordinate levels based on the objects' shape features, without having been told the categorization rules for doing so. On each trial, participants were asked to report the category and subcategory of the object, after which they received feedback about the correctness of their categorization responses. Participants trained over the course of a one-hour-long session while their brain activation was measured using functional magnetic resonance imaging. Over the course of training, significant hierarchy learning took place as participants discovered the nested categorization rules, as evidenced by the occurrence of a learning trial, after which performance suddenly increased. This learning was associated with increased representational strength of the newly acquired hierarchical rules in a corticostriatal network including the posterior frontal and parietal cortex and the dorsal striatum. We also found evidence suggesting that reinforcement learning in the dorsal striatum contributed to hierarchical rule learning.

Research on online teachers’ training based on the gamification design: A survey analysis of primary and secondary school teachers

The current form of teacher education and training has gradually shifted to distance training, but the most prominent performance of the participants in network training is the lack of motivation and enthusiasm for participation, and the participants participate in the training with the mentality of coping with the assessment, and the training effect is naturally not very good. At this point, the concept of gamification learning has become popular. Gamification applies game thinking and mechanics to non-game areas to guide user interaction and use. Based on Professor Kevin Werbach's gamification design framework, this study developed a case study of gamification for online teacher training, implemented in a professional development program for primary and secondary school teachers in Zhejiang Province, China. A self-administered questionnaire was then surveyed in three teaching classes, and the results found that: Firstly The participants' satisfaction in response to the gamification case was at a moderate to high level (M > 4.0, Tot = 5.0); Secondly, demographic factors did not significantly influence this (p > 0.05); Thirdly, among the gamification design influencing factors, presentation and feedback were the most influential (b = 0.536,p = 0.000), goals and rules (b = 0.167, p = 0.012), communication and interaction (b = 0.155, p = 0.029), and personalization (b = 0.154, p = 0.020) in decreasing order; Fourthly, The effect of case implementation significantly impacted participants' intention to sustain participation (b = 0.874, p = 0.000). Finally, we distilled the study's shortcomings and the prospects for future research.

Intelligent Method to Optimize the Frequency Modulation for Beam Pumping System Based on Deep Reinforcement Learning

A mathematical simulation model of a beam pumping system with frequency conversion control is established, considering the influence of the real-time frequency variation on the motion law of a pumping unit, the longitudinal vibration of a sucker rod string, the crankshaft torque, and the motor power. On this basis, the key links such as state space, action space, and reward function are defined by using deep reinforcement learning theory, and an intelligent model to optimize the frequency modulation for a beam pumping system based on deep reinforcement learning is constructed. The simulation and field application results show that the frequency optimization model can significantly reduce the fluctuation amplitude of the polished rod load, crankshaft torque, motor power, and input power of the system, making the operation of the pumping system more stable and energy-saving. More importantly, the model can realize the independent learning and control of the corresponding parameters without manual intervention to ensure the normal operation of the system and improve the level of information and intelligent management of oil wells.

A quadruple dissociation of reward-related behaviour in mice across excitatory inputs to the nucleus accumbens shell

The nucleus accumbens shell (NAcSh) is critically important for reward valuations, yet it remains unclear how valuation information is integrated in this region to drive behaviour during reinforcement learning. Using an optogenetic spatial self-stimulation task in mice, here we show that contingent activation of different excitatory inputs to the NAcSh change expression of different reward-related behaviours. Our data indicate that medial prefrontal inputs support place preference via repeated actions, ventral hippocampal inputs consistently promote place preferences, basolateral amygdala inputs produce modest place preferences but as a byproduct of increased sensitivity to time investments, and paraventricular inputs reduce place preferences yet do not produce full avoidance behaviour. These findings suggest that each excitatory input provides distinct information to the NAcSh, and we propose that this reflects the reinforcement of different credit assignment functions. Our finding of a quadruple dissociation of NAcSh input-specific behaviours provides insights into how types of information carried by distinct inputs to the NAcSh could be integrated to help drive reinforcement learning and situationally appropriate behavioural responses.

Prefrontal modulation of anxiety through a lens of noradrenergic signaling

Anxiety disorders are the most common class of mental illness in the U.S., affecting 40 million individuals annually. Anxiety is an adaptive response to a stressful or unpredictable life event. Though evolutionarily thought to aid in survival, excess intensity or duration of anxiogenic response can lead to a plethora of adverse symptoms and cognitive dysfunction. A wealth of data has implicated the medial prefrontal cortex (mPFC) in the regulation of anxiety. Norepinephrine (NE) is a crucial neuromodulator of arousal and vigilance believed to be responsible for many of the symptoms of anxiety disorders. NE is synthesized in the locus coeruleus (LC), which sends major noradrenergic inputs to the mPFC. Given the unique properties of LC-mPFC connections and the heterogeneous subpopulation of prefrontal neurons known to be involved in regulating anxiety-like behaviors, NE likely modulates PFC function in a cell-type and circuit-specific manner. In working memory and stress response, NE follows an inverted-U model, where an overly high or low release of NE is associated with sub-optimal neural functioning. In contrast, based on current literature review of the individual contributions of NE and the PFC in anxiety disorders, we propose a model of NE level- and adrenergic receptor-dependent, circuit-specific NE-PFC modulation of anxiety disorders. Further, the advent of new techniques to measure NE in the PFC with unprecedented spatial and temporal resolution will significantly help us understand how NE modulates PFC function in anxiety disorders.

Protective effects of ginsenosides Rg1 and Rb1 against cognitive impairment induced by simulated microgravity in rats

Microgravity experienced during space flight is known to exert several negative effects on the learning ability and memory of astronauts. Few effective strategies are currently available to counteract these effects. Rg1 and Rb1, the major steroidal components of ginseng, have shown potent neuroprotective effects with a high safety profile. The present study aimed to investigate the effects of Rg1 and Rb1 on simulated microgravity-induced learning and memory dysfunction and its underlying mechanism in the hindlimb suspension (HLS) rat model. Administration of Rg1 (30 and 60 μmol/kg) and Rb1 (30 and 60 μmol/kg) for 2 weeks resulted in a significant amelioration of impaired spatial and associative learning and memory caused by 4-week HLS exposure, measured using the Morris water maze and Reward operating conditioning reflex (ROCR) tests, respectively. Furthermore, Rg1 and Rb1 administration alleviated reactive oxygen species production and enhanced antioxidant enzyme activities in the prefrontal cortex (PFC). Rg1 and Rb1 also assisted in the recovery of mitochondrial complex I (NADH dehydrogenase) activities, increased the expression of Mfn2 and decreased the fission marker dynamin-related protein (Drp)-1expression. Additionally, Rg1 and Rb1 treatment increased the SYN, and PSD95 protein expressions and decreased the ratio of Bax:Bcl-2 and reduced the expression of cleaved caspase-3 and cytochrome C. Besides these, the BDNF-TrkB/PI3K-Akt pathway was also activated by Rg1 and Rb1 treatment. Altogether, Rg1 and Rb1 treatment attenuated cognitive deficits induced by HLS, mitigated mitochondrial dysfunction, attenuated oxidative stress, inhibited apoptosis, increased synaptic plasticity, and restored BDNF-TrkB/PI3K-Akt signaling.

Rethinking statistical learning as a continuous dynamic stochastic process, from the motor systems perspective

The brain integrates streams of sensory input and builds accurate predictions, while arriving at stable percepts under disparate time scales. This stochastic process bears different unfolding dynamics for different people, yet statistical learning (SL) currently averages out, as noise, individual fluctuations in data streams registered from the brain as the person learns. We here adopt a new analytical approach that instead of averaging out fluctuations in continuous electroencephalographic (EEG)-based data streams, takes these gross data as the important signals. Our new approach reassesses how individuals dynamically learn predictive information in stable and unstable environments. We find neural correlates for two types of learners in a visuomotor task: narrow-variance learners, who retain explicit knowledge of the regularity embedded in the stimuli. They seem to use an error-correction strategy steadily present in both stable and unstable environments. This strategy can be captured by current optimization-based computational frameworks. In contrast, broad-variance learners emerge only in the unstable environment. Local analyses of the moment-by-moment fluctuations, naïve to the overall outcome, reveal an initial period of memoryless learning, well characterized by a continuous gamma process starting out exponentially distributed whereby all future events are equally probable, with high signal (mean) to noise (variance) ratio. The empirically derived continuous Gamma process smoothly converges to predictive Gaussian signatures comparable to those observed for the error-corrective mode that is captured by current optimization-driven computational models. We coin this initially seemingly purposeless stage exploratory. Globally, we examine a posteriori the fluctuations in distributions' shapes over the empirically estimated stochastic signatures. We then confirm that the exploratory mode of those learners, free of expectation, random and memoryless, but with high signal, precedes the acquisition of the error-correction mode boasting smooth transition from exponential to symmetric distributions' shapes. This early naïve phase of the learning process has been overlooked by current models driven by expected, predictive information and error-based learning. Our work demonstrates that (statistical) learning is a highly dynamic and stochastic process, unfolding at different time scales, and evolving distinct learning strategies on demand.

The effects of induced positive and negative affect on Pavlovian-instrumental interactions

Across species, animals have an intrinsic drive to approach appetitive stimuli and to withdraw from aversive stimuli. In affective science, influential theories of emotion link positive affect with strengthened behavioural approach and negative affect with avoidance. Based on these theories, we predicted that individuals' positive and negative affect levels should particularly influence their behaviour when innate Pavlovian approach/avoidance tendencies conflict with learned instrumental behaviours. Here, across two experiments - exploratory Experiment 1 (N = 91) and a preregistered confirmatory Experiment 2 (N = 335) - we assessed how induced positive and negative affect influenced Pavlovian-instrumental interactions in a reward/punishment Go/No-Go task. Contrary to our hypotheses, we found no evidence for a main effect of positive/negative affect on either approach/avoidance behaviour or Pavlovian-instrumental interactions. However, we did find evidence that the effects of induced affect on behaviour were moderated by individual differences in self-reported behavioural inhibition and gender. Exploratory computational modelling analyses explained these demographic moderating effects as arising from positive correlations between demographic factors and individual differences in the strength of Pavlovian-instrumental interactions. These findings serve to sharpen our understanding of the effects of positive and negative affect on instrumental behaviour.

Cognitive [Computational] Neuroscience Test Reliability and Clinical Applications for Serious Mental Illness (CNTRaCS) Consortium: Progress and Future Directions

The development of treatments for impaired cognition in schizophrenia has been characterized as the most important challenge facing psychiatry at the beginning of the twenty-first century. The Cognitive Neuroscience Treatment Research to Improve Cognition in Schizophrenia (CNTRICS) project was designed to build on the potential benefits of using tasks and tools from cognitive neuroscience to better understanding and treat cognitive impairments in psychosis. These benefits include: (1) the use of fine-grained tasks that measure discrete cognitive processes; (2) the ability to design tasks that distinguish between specific cognitive domain deficits and poor performance due to generalized deficits resulting from sedation, low motivation, poor test taking skills, etc.; and (3) the ability to link cognitive deficits to specific neural systems, using animal models, neuropsychology, and functional imaging. CNTRICS convened a series of meetings to identify paradigms from cognitive neuroscience that maximize these benefits and identified the steps need for translation into use in clinical populations. The Cognitive Neuroscience Test Reliability and Clinical Applications for Schizophrenia (CNTRaCS) Consortium was developed to help carry out these steps. CNTRaCS consists of investigators at five different sites across the country with diverse expertise relevant to a wide range of the cognitive systems identified as critical as part of CNTRICs. This work reports on the progress and current directions in the evaluation and optimization carried out by CNTRaCS of the tasks identified as part of the original CNTRICs process, as well as subsequent extensions into the Positive Valence systems domain of Research Domain Criteria (RDoC). We also describe the current focus of CNTRaCS, which involves taking a computational psychiatry approach to measuring cognitive and motivational function across the spectrum of psychosis. Specifically, the current iteration of CNTRaCS is using computational modeling to isolate parameters reflecting potentially more specific cognitive and visual processes that may provide greater interpretability in understanding shared and distinct impairments across psychiatric disorders.

The Arousal-motor Hypothesis of Dopamine Function: Evidence that Dopamine Facilitates Reward Seeking in Part by Maintaining Arousal

Dopamine facilitates approach to reward via its actions on dopamine receptors in the nucleus accumbens. For example, blocking either D1 or D2 dopamine receptors in the accumbens reduces the proportion of reward-predictive cues to which rats respond with cued approach. Recent evidence indicates that accumbens dopamine also promotes wakefulness and arousal, but the relationship between dopamine's roles in arousal and reward seeking remains unexplored. Here, we show that the ability of systemic or intra-accumbens injections of the D1 antagonist SCH23390 to reduce cued approach to reward depends on the animal's state of arousal. Handling the animal, a manipulation known to increase arousal, was sufficient to reverse the behavioral effects of the antagonist. In addition, SCH23390 reduced spontaneous locomotion and increased time spent in sleep postures, both consistent with reduced arousal, but also increased time spent immobile in postures inconsistent with sleep. In contrast, the ability of the D2 antagonist haloperidol to reduce cued approach was not reversible by handling. Haloperidol reduced spontaneous locomotion but did not increase sleep postures, instead increasing immobility in non-sleep postures. We place these results in the context of the extensive literature on dopamine's contributions to behavior, and propose the arousal-motor hypothesis. This novel synthesis, which proposes that two main functions of dopamine are to promote arousal and facilitate motor behavior, accounts both for our findings and many previous behavioral observations that have led to disparate and conflicting conclusions.

Need for cognition is associated with the interaction of reward and task-load on effort: A verification and extension study

Here, we work to provide nuance around the assumption that people will work for rewards. We examine whether individuals' inherent tendency to mobilize cognitive effort (need for cognition, NFC) moderates this effect. We re-analyzed our existing data to verify an effect reported by Sandra and Otto (2018) regarding the association between NFC and reward-induced cognitive effort expenditure, using a more ecological cognitive task design and adding a psychophysiological measure of effort. Specifically, distinct from their short time course visual task-switching paradigm, we used a relatively long course auditory comprehension task paradigm. We found that, consistent with the original study, increased cognitive effort in response to incentive reward depends on individual differences in cognitive motivation (need for cognition). We also found that, to observe consistent phenomena, different indices of effort (behavioral and psychophysiological) need to be considered when evaluating the relationship between the effort expenditure and cognitive motivation. Pupil dilation showed an advantage over reaction time in revealing mental effort mobilized over a prolonged cognitive task. Our results suggest that assessing cognitive motivation when planning a behavior-change program involving reward feedback for positive performance could help to optimize individuals' effort investment.

Vagus nerve stimulation drives selective circuit modulation through cholinergic reinforcement

Vagus nerve stimulation (VNS) is a neuromodulation therapy for a broad and expanding set of neurologic conditions. However, the mechanism through which VNS influences central nervous system circuitry is not well described, limiting therapeutic optimization. VNS leads to widespread brain activation, but the effects on behavior are remarkably specific, indicating plasticity unique to behaviorally engaged neural circuits. To understand how VNS can lead to specific circuit modulation, we leveraged genetic tools including optogenetics and in vivo calcium imaging in mice learning a skilled reach task. We find that VNS enhances skilled motor learning in healthy animals via a cholinergic reinforcement mechanism, producing a rapid consolidation of an expert reach trajectory. In primary motor cortex (M1), VNS drives precise temporal modulation of neurons that respond to behavioral outcome. This suggests that VNS may accelerate motor refinement in M1 via cholinergic signaling, opening new avenues for optimizing VNS to target specific disease-relevant circuitry.

Sticky me: Self-relevance slows reinforcement learning

A prominent facet of social-cognitive functioning is that self-relevant information is prioritized in perception, attention, and memory. What is not yet understood, however, is whether similar effects arise during learning. In particular, compared to other people (e.g., best friend) is information about the self acquired more rapidly? To explore this matter, here we used a probabilistic selection task in combination with computational modeling (i.e., Reinforcement Learning Drift Diffusion Model analysis) to establish how self-relevance influences learning under conditions of uncertainty (i.e., choices are based on the perceived likelihood of positive and negative outcomes). Across two experiments, a consistent pattern of effects was observed. First, learning rates for both positive and negative prediction errors were slower for self-relevant compared to friend-relevant associations. Second, self-relevant (vs. friend-relevant) learning was characterized by the exploitation (vs. exploration) of choice selections. That is, in a complex (i.e., probabilistic) decision-making environment, previously rewarded self-related outcomes were selected more often than novel - but potentially riskier - alternatives. The implications of these findings for accounts of self-function are considered.

Different Patterns of Neural Activity Characterize Motor Skill Performance During Acquisition and Retention

Motor performance and learning have distinct behavioral and neural signatures and can be uniquely modulated by various informational and motivational factors. Contemporary frameworks describe four different motor learning mechanisms mapped onto specific neural regions which are key for motor skill acquisition: error-based learning (cerebellum), reinforcement learning (basal ganglia), cognitive strategies (prefrontal cortex), and use-dependent learning (motor cortex). However, little is known about the neural circuits engaged during skill acquisition that are modulated specifically by practice-based performance improvement and those that predict recall performance. Based on previous work, we hypothesize that brain activity during practice in primary motor cortex and basal ganglia (1) is associated with trial-by-trial practice performance and (2) is predictive of immediate recall performance. Leveraging the contemporary framework, we use a well-known task paradigm that primarily relies upon cognitive strategy, reinforcement, and use-based learning mechanisms to test our hypotheses. Forty neurotypical young adults were asked to practice a pinch force tracking task. Participants received performance feedback after each trial during practice. We used whole brain analysis of functional magnetic resonance imaging (fMRI) and behavioral performance measures (i.e., time-on-target and self-efficacy) during the practice phase to determine which brain activation patterns are (1) associated with trial-by-trial tracking performance and (2) predictive of immediate no-feedback retention performance. We observed brain activations in the frontal orbital cortex, putamen, amygdala, and insula correlated with tracking performance improvement during practice. In contrast, a different set of performance-related activated regions were observed that were associated with immediate retention performance that included the primary motor cortex, superior frontal gyrus, somatosensory cortex, angular gyrus, and parietal gyrus. Our findings demonstrate that improved practice performance and recall of a sensorimotor skill are correlated with distinct neural activity patterns during acquisition, drawing on different motor learning mechanisms during encoding. While motor performance improvements depend on both cortical and subcortical regions, motor skill recall depends primarily on prefrontal and motor cortices. We discuss possible interpretations for why our hypothesis regarding basal ganglia activity and retention performance was not supported. Understanding the different neural mechanisms engaged in motor performance and learning may inform novel interventions to enhance motor skill learning.

Activation of the Nucleus Taeniae of the Amygdala by Umami Taste in Domestic Chicks (Gallus gallus)

In chickens, the sense of taste plays an important role in detecting nutrients and choosing feed. The molecular mechanisms underlying the taste-sensing system of chickens are well studied, but the neural mechanisms underlying taste reactivity have received less attention. Here we report the short-term taste behaviour of chickens towards umami and bitter (quinine) taste solutions and the associated neural activity in the nucleus taeniae of the amygdala, nucleus accumbens and lateral septum. We found that chickens had more contact with and drank greater volumes of umami than bitter or a water control, and that chicks displayed increased head shaking in response to bitter compared to the other tastes. We found that there was a higher neural activity, measured as c-Fos activation, in response to umami taste in the right hemisphere of the nucleus taeniae of the amygdala. In the left hemisphere, there was a higher c-Fos activation of the nucleus taeniae of the amygdala in response to bitter than in the right hemisphere. Our findings provide clear evidence that chickens respond differently to umami and bitter tastes, that there is a lateralised response to tastes at the neural level, and reveals a new function of the avian nucleus taeniae of the amygdala as a region processing reward information.

A unidirectional but not uniform striatal landscape of dopamine signaling for motivational stimuli

SignificanceAlthough it is undisputed that striatal dopamine plays a prominent role in motivated behavior and learning, the precise information conveyed by dopamine signals as such is under active debate. For a long time, the idea dominated that dopamine encodes a reward prediction error and that this signal is broadcast uniformly throughout the brain. However, here, we capture dopamine dynamics across many striatal regions and demonstrate that dopamine release is, regionally, extremely heterogeneous and that a reward prediction error-like signal is predominantly found in the relatively small limbic domain of the striatum. Another striking organizing principle is that stimulus valence directs dopamine concentration homogeneously across all regions (i.e., appetitive stimuli increase dopamine and aversive stimuli decrease dopamine).

Neuropeptide Y interaction with dopaminergic and serotonergic pathways: interlinked neurocircuits modulating hedonic eating behaviours

Independent from homeostatic needs, the consumption of foods originating from hyperpalatable diets is defined as hedonic eating. Hedonic eating can be observed in many forms of eating phenotypes, such as compulsive eating and stress-eating, heightening the risk of obesity development. For instance, stress can trigger the consumption of palatable foods as a type of coping strategy, which can become compulsive, particularly when developed as a habit. Although eating for pleasure is observed in multiple maladaptive eating behaviours, the current understanding of the neurobiology underlying hedonic eating remains deficient. Intriguingly, the combined orexigenic, anxiolytic and reward-seeking properties of Neuropeptide Y (NPY) ignited great interest and has positioned NPY as one of the core neuromodulators operating hedonic eating behaviours. While extensive literature exists exploring the homeostatic orexigenic and anxiolytic properties of NPY, the rewarding effects of NPY continue to be investigated. As deduced from a series of behavioural and molecular-based studies, NPY appears to motivate the consumption and enhancement of food-rewards. As a possible mechanism, NPY may modulate reward-associated monoaminergic pathways, such as the dopaminergic and serotoninergic neural networks, to modulate hedonic eating behaviours. Furthermore, potential direct and indirect NPYergic neurocircuitries connecting classical homeostatic and hedonic neuropathways may also exist involving the anti-reward centre the lateral habenula. Therefore, this review investigates the participation of NPY in orchestrating hedonic eating behaviours through the modulation of monoaminergic pathways.

Stock Market Prediction Based on Big Data Using Deep Reinforcement Long Short-Term Memory Model

In this modern era, the stock market is one of the important platforms for several business developments to ensure their growth for upcoming years. Big data is another technical aspect of the stock market to import large amounts of stocks. Deep Reinforcement Long Short Term Memory (DRLSTM) model is proposed for achieving better prediction rate for stock market trends based on the technical indicators. Three most popular banking organizations data is obtained in real-time live stocks from the NIFTY-50 market data. The data is enclosed based on trading days from 2000 to 2020. The bid data approach known to be a Hadoop framework is used simultaneously to handle large amounts of data for processing through distributed storage. The experimental results are performed based on the mean squared error (MSE) for the proposed model which obtained a low error rate of 0.017% for SBI, 0.014% for HDFC, and 0.018% for BajajFin. The proposed model is evaluated and results are compared with other existing techniques which the RLSTM outperforms by obtaining a high accuracy rate.

Aversive motivation and cognitive control

Aversive motivation plays a prominent role in driving individuals to exert cognitive control. However, the complexity of behavioral responses attributed to aversive incentives creates significant challenges for developing a clear understanding of the neural mechanisms of this motivation-control interaction. We review the animal learning, systems neuroscience, and computational literatures to highlight the importance of experimental paradigms that incorporate both motivational context manipulations and mixed motivational components (e.g., bundling of appetitive and aversive incentives). Specifically, we postulate that to understand aversive incentive effects on cognitive control allocation, a critical contextual factor is whether such incentives are associated with negative reinforcement or punishment. We further illustrate how the inclusion of mixed motivational components in experimental paradigms enables increased precision in the measurement of aversive influences on cognitive control. A sharpened experimental and theoretical focus regarding the manipulation and assessment of distinct motivational dimensions promises to advance understanding of the neural, monoaminergic, and computational mechanisms that underlie the interaction of motivation and cognitive control.

The short neuropeptide F (sNPF) promotes the formation of appetitive visual memories in honey bees

Motivation can critically influence learning and memory. Multiple neural mechanisms regulate motivational states, among which signalling via specific neuropeptides, such as NPY in vertebrates and NPF and its short variant sNPF in invertebrates, plays an essential role. The honey bee (Apis mellifera) is a privileged model for the study of appetitive learning and memory. Bees learn and memorize sensory cues associated with nectar reward while foraging, and their learning is affected by their feeding state. However, the neural underpinnings of their motivational states remain poorly known. Here we focused on the short neuropeptide F (sNPF) and studied if it modulates the acquisition and formation of colour memories. Artificially increasing sNPF levels in partially fed foragers with a reduced motivation to learn colours resulted in significant colour learning and memory above the levels exhibited by starved foragers. Our results thus identify sNPF as a critical component of motivational processes involved in foraging and in the cognitive processes associated with this activity in honey bees.

Human Variation in Error-Based and Reinforcement Motor Learning Is Associated With Entorhinal Volume

Error-based and reward-based processes are critical for motor learning and are thought to be mediated via distinct neural pathways. However, recent behavioral work in humans suggests that both learning processes can be bolstered by the use of cognitive strategies, which may mediate individual differences in motor learning ability. It has been speculated that medial temporal lobe regions, which have been shown to support motor sequence learning, also support the use of cognitive strategies in error-based and reinforcement motor learning. However, direct evidence in support of this idea remains sparse. Here we first show that better overall learning during error-based visuomotor adaptation is associated with better overall learning during the reward-based shaping of reaching movements. Given the cognitive contribution to learning in both of these tasks, these results support the notion that strategic processes, associated with better performance, drive intersubject variation in both error-based and reinforcement motor learning. Furthermore, we show that entorhinal cortex volume is larger in better learning individuals-characterized across both motor learning tasks-compared with their poorer learning counterparts. These results suggest that individual differences in learning performance during error and reinforcement learning are related to neuroanatomical differences in entorhinal cortex.

Adolescent Dopamine Neurons Represent Reward Differently during Action and State Guided Learning

Neuronal underpinning of learning cause-and-effect associations in the adolescent brain remains poorly understood. Two fundamental forms of associative learning are Pavlovian (classical) conditioning, where a stimulus is followed by an outcome, and operant (instrumental) conditioning, where outcome is contingent on action execution. Both forms of learning, when associated with a rewarding outcome, rely on midbrain dopamine neurons in the ventral tegmental area (VTA) and substantia nigra (SN). We find that, in adolescent male rats, reward-guided associative learning is encoded differently by midbrain dopamine neurons in each conditioning paradigm. Whereas simultaneously recorded VTA and SN adult neurons have a similar phasic response to reward delivery during both forms of conditioning, adolescent neurons display a muted reward response during operant but a profoundly larger reward response during Pavlovian conditioning. These results suggest that adolescent neurons assign a different value to reward when it is not gated by action. The learning rate of adolescents and adults during both forms of conditioning was similar, supporting the notion that differences in reward response in each paradigm may be because of differences in motivation and independent of state versus action value learning. Static characteristics of dopamine neurons, such as dopamine cell number and size, were similar in the VTA and SN of both ages, but there were age-related differences in stimulated dopamine release and correlated spike activity, suggesting that differences in reward responsiveness by adolescent dopamine neurons are not because of differences in intrinsic properties of these neurons but engagement of different dopaminergic networks.SIGNIFICANCE STATEMENT Reckless behavior and impulsive decision-making by adolescents suggest that motivated behavioral states are encoded differently by the adolescent brain. Motivated behavior, which is dependent on the function of the dopamine system, follows learning of cause-and-effect associations in the environment. We find that dopamine neurons in adolescents encode reward differently depending on the cause-and-effect relationship of the means to receive that reward. Compared with adults, reward contingent on action led to a muted response, whereas reward that followed a cue but was not gated by action produced an augmented phasic response. These data demonstrate an age-related difference in dopamine neuron response to reward that is not uniform and is guided by processes that differentiate between state and action values.

Dopamine release in the nucleus accumbens core signals perceived saliency

A large body of work has aimed to define the precise information encoded by dopaminergic projections innervating the nucleus accumbens (NAc). Prevailing models are based on reward prediction error (RPE) theory, in which dopamine updates associations between rewards and predictive cues by encoding perceived errors between predictions and outcomes. However, RPE cannot describe multiple phenomena to which dopamine is inextricably linked, such as behavior driven by aversive and neutral stimuli. We combined a series of behavioral tasks with direct, subsecond dopamine monitoring in the NAc of mice, machine learning, computational modeling, and optogenetic manipulations to describe behavior and related dopamine release patterns across multiple contingencies reinforced by differentially valenced outcomes. We show that dopamine release only conforms to RPE predictions in a subset of learning scenarios but fits valence-independent perceived saliency encoding across conditions. Here, we provide an extended, comprehensive framework for accumbal dopamine release in behavioral control.

Context-dependent representations of movement in Drosophila dopaminergic reinforcement pathways

Dopamine plays a central role in motivating and modifying behavior, serving to invigorate current behavioral performance and guide future actions through learning. Here we examine how this single neuromodulator can contribute to such diverse forms of behavioral modulation. By recording from the dopaminergic reinforcement pathways of the Drosophila mushroom body during active odor navigation, we reveal how their ongoing motor-associated activity relates to goal-directed behavior. We found that dopaminergic neurons correlate with different behavioral variables depending on the specific navigational strategy of an animal, such that the activity of these neurons preferentially reflects the actions most relevant to odor pursuit. Furthermore, we show that these motor correlates are translated to ongoing dopamine release, and acutely perturbing dopaminergic signaling alters the strength of odor tracking. Context-dependent representations of movement and reinforcement cues are thus multiplexed within the mushroom body dopaminergic pathways, enabling them to coordinately influence both ongoing and future behavior.

Modeling incentive salience in Pavlovian learning more parsimoniously using a multiple attribute model

We present a multi-attribute incentive salience (MAIS) model as a computational account of incentive salience in model-based Pavlovian learning. A model of incentive salience as a joint function of reward value and physiological state has been previously proposed by Zhang et al. (2009). In that model, the function takes additive or multiplicative forms depending on whether a preference shifts from positive to negative or vice versa. We demonstrate that arbitrarily varying this function is unnecessary to explain observed data. A multiplicative function is sufficient if one takes into account empirical data suggesting the incentive salience function for an incentive is comprised of multiple physiological signals. We compare our model to the previously proposed model on two datasets. We find the MAIS model predicts the outcomes equally well, fits empirical data describing multiple sensory representations of a single stimulus, better approximates the dual-structure appetitive-aversive nature of the reward system, is compatible with existing knowledge about incentive salience in Pavlovian learning, and better describes revaluation in Pavlovian learning. This model addresses a call (Dayan & Berridge, 2014) for algorithmic and computational models of model-based Pavlovian learning that consistently and fully explain empirical observations. Because a multi-attribute model is relevant even for simple Pavlovian associations, it should be useful in a wide variety of decision-making contexts, including agent modeling and addiction research.