Motivational control of sign-tracking behaviour: A theoretical framework

The role of reinforcement learning and value-based decision-making frameworks in understanding food choice and eating behaviors

The obesogenic food environment includes easy access to highly-palatable, energy-dense, "ultra-processed" foods that are heavily marketed to consumers; therefore, it is critical to understand the neurocognitive processes the underlie overeating in response to environmental food-cues (e.g., food images, food branding/advertisements). Eating habits are learned through reinforcement, which is the process through which environmental food cues become valued and influence behavior. This process is supported by multiple behavioral control systems (e.g., Pavlovian, Habitual, Goal-Directed). Therefore, using neurocognitive frameworks for reinforcement learning and value-based decision-making can improve our understanding of food-choice and eating behaviors. Specifically, the role of reinforcement learning in eating behaviors was considered using the frameworks of (1) Sign-versus Goal-Tracking Phenotypes; (2) Model-Free versus Model-Based; and (3) the Utility or Value-Based Model. The sign-and goal-tracking phenotypes may contribute a mechanistic insight on the role of food-cue incentive salience in two prevailing models of overconsumption-the Extended Behavioral Susceptibility Theory and the Reactivity to Embedded Food Cues in Advertising Model. Similarly, the model-free versus model-based framework may contribute insight to the Extended Behavioral Susceptibility Theory and the Healthy Food Promotion Model. Finally, the value-based model provides a framework for understanding how all three learning systems are integrated to influence food choice. Together, these frameworks can provide mechanistic insight to existing models of food choice and overconsumption and may contribute to the development of future prevention and treatment efforts.

Evolved Features of Artistic Motivation: Analyzing a Brazilian Database Spanning Three Decades

Darwin explored the evolutionary processes underlying artistic propensities in humans. He stressed the universality of the human mind by pointing to the shared pleasure which all populations take in dancing, engaging in music, acting, painting, tattooing, and self-decorating. Artistic motivation drives/reinforces individuals to engage in aesthetically oriented activities. As curiosity/play, artistic behavior is hypothesized as a functionally autonomous activity motivated intrinsically through an evolved, specific, and stable aesthetic motivational system. The author tested whether artistic motivation is rather intrinsically sourced, domain-specific, and temporally stable using a large decades-long real-life public Brazilian database of university applications. In Study I, the author analyzed reasons for career-choice responded to by 403,832 late-adolescent applicants (48.84% women), between 1987 and 1998. In Study II, the author analyzed another career-choice reason question responded to by 1,703,916 late-adolescent applicants (51.02% women), between 1987 and 2020. Music, Dance, Scenic Arts, Visual Arts, and Literary Studies, in combination, presented a higher percentage of individuals reporting intrinsic factors (e.g., personal taste/aptitude/fulfillment) and the lower proportion reporting extrinsic motives (e.g., the influence of media/teacher/family, salary, social contribution/prestige) than other career groups. If artistic motivation were a recent by-product of general curiosity or status-seeking, artistic and non-artistic careers would not differ. Overall, intrinsic motives were 2.60–6.35 times higher than extrinsic factors; among artistic applicants’ were 10.81–28.38 times higher, suggesting domain-specificity. Intrinsic motivation did not differ among artistic careers and remained stable throughout the periods. Converging results corroborated a specific, stable, and intrinsically sourced artistic motivation consistent with its possible evolutionary origins.

Quantifying the instrumental and noninstrumental underpinnings of Pavlovian responding with the Price equation

The Price equation is a mathematical expression of selectionist and non-selectionist pressures on biological, cultural, and behavioral change. We use it here to specify instrumental and noninstrumental behaviors as they arise within the context of the Pavlovian autoshaping procedure, for rats trained under reward certainty and reward uncertainty. The point of departure for this endeavor is that some portion of autoshaped behavior referred to as goal-tracking appears instrumental-a function of resource attainment (the individual approaches the location where the unconditioned stimulus is to be delivered). By contrast, some other portion of autoshaped behavior referred to as sign-tracking is noninstrumental-irrelevant to making contact with the to-be-delivered unconditioned stimulus. A Price equation model is proposed that unifies our understanding of Pavlovian autoshaping behavior by isolating operant and respondent influences on goal-tracking (instrumental) and sign-tracking (noninstrumental) behavior.

Effort-motivated behavior resolves paradoxes in appetitive conditioning

Motivated behavior has long been studied by psychologists, ethologists, and neuroscientists. To date, many scientists agree with the view that cue and reward attraction is the product of a dopamine-dependent unconscious process called incentive salience or "wanting". This process allows the influence of multiple factors such as hunger and odors on motivational attraction. In some cases, however, the resulting motivated behavior differs from what the incentive salience hypothesis would predict. I argue that seeking behavior under reward uncertainty illustrates this situation: Organisms do not just "want" (appetite-based attraction) cues that are inconsistent or associated with reward occasionally, they "hope" that those cues will consistently predict reward procurement in the ongoing trial. Said otherwise, they become motivated to invest time and energy to find consistent cue-reward associations despite no guarantee of success (effort-based attraction). A multi-test comparison of performance between individuals trained under uncertainty and certainty reveals behavioral paradoxes suggesting that the concept of incentive salience cannot fully account for responding to inconsistent cues. A mathematical model explains how appetite-based and effort-based attractions might combine their effects.

Value, drug addiction and the brain

Over the years, various models have been proposed to explain the psychology and biology of drug addiction, built primarily around the habit and compulsion models. Recent research indicates drug addiction may be goal-directed, motivated by excessive valuation of drugs. Drug consumption may initially occur for the sake of pleasure but may transition to a means of escaping withdrawal, stress and negative emotions. In this hypothetical paper, we propose a value-based neurobiological model for drug addiction. We posit that during dependency, the value-based decision-making system in the brain is not inactive but has instead prioritized drugs as the reward of choice. In support of this model, we consider the role of valuation in choice, its influence on pleasure and punishment, and how valuation is contrasted in impulsive and compulsive behaviours. We then discuss the neurobiology of value, beginning with the dopaminergic system and its relationship with incentive salience before moving to brain-wide networks involved in valuation, control and prospection. These value-based neurobiological components are then integrated into the cycle of addiction as we consider the development of drug dependency from a valuation perspective. We conclude with a discussion of cognitive interventions utilizing value-based decision-making, highlighting not just advances in recalibrating the valuation system to focus on non-drug rewards, but also areas for improvement in refining this approach.

Reliability assessment of temporal discounting measures in virtual reality environments

In recent years the emergence of high-performance virtual reality (VR) technology has opened up new possibilities for the examination of context effects in psychological studies. The opportunity to create ecologically valid stimulation in a highly controlled lab environment is especially relevant for studies of psychiatric disorders, where it can be problematic to confront participants with certain stimuli in real life. However, before VR can be confidently applied widely it is important to establish that commonly used behavioral tasks generate reliable data within a VR surrounding. One field of research that could benefit greatly from VR-applications are studies assessing the reactivity to addiction related cues (cue-reactivity) in participants suffering from gambling disorder. Here we tested the reliability of a commonly used temporal discounting task in a novel VR set-up designed for the concurrent assessment of behavioral and psychophysiological cue-reactivity in gambling disorder. On 2 days, thirty-four healthy non-gambling participants explored two rich and navigable VR-environments (neutral: café vs. gambling-related: casino and sports-betting facility), while their electrodermal activity was measured using remote sensors. In addition, participants completed the temporal discounting task implemented in each VR environment. On a third day, participants performed the task in a standard lab testing context. We then used comprehensive computational modeling using both standard softmax and drift diffusion model (DDM) choice rules to assess the reliability of discounting model parameters assessed in VR. Test-retest reliability estimates were good to excellent for the discount rate log(k), whereas they were poor to moderate for additional DDM parameters. Differences in model parameters between standard lab testing and VR, reflecting reactivity to the different environments, were mostly numerically small and of inconclusive directionality. Finally, while exposure to VR generally increased tonic skin conductance, this effect was not modulated by the neutral versus gambling-related VR-environment. Taken together this proof-of-concept study in non-gambling participants demonstrates that temporal discounting measures obtained in VR are reliable, suggesting that VR is a promising tool for applications in computational psychiatry, including studies on cue-reactivity in addiction.

Learning to lose control: A process-based account of behavioral addiction

Learning psycho(bio)logy has developed a solid corpus of evidence and theory regarding behavior control modes. The present article briefly reviews that literature and its influence on recent models in which the transition from goal-directed to compulsive behavior is identified as the main process underlying substance use disorders. This literature is also relevant to non-substance addictive disorders, and serves as basis to propose a restricted definition of behavioral addiction relying on the presence of behavior-specific compulsivity. Complementarily, we consider whether some activities can become disordered while remaining mostly goal-driven. Based on reinforcement learning models, relative outcome utility computation is proposed as an alternative mechanism through which dysfunctional behaviors (even not qualifying as addictive) can override adaptive ones, causing functional impairment. Beyond issues of conceptual delimitation, recommendations are made regarding the importance of identifying individual etiological pathways to dysregulated behavior, the necessity of accurately profiling at-risk individuals, and the potential hazards of symptom-based diagnosis. In our view, the validity of these recommendations does not depend on the position one takes in the nosological debate.

A computational formulation of the behavior systems account of the temporal organization of motivated behavior

The behavior systems framework suggests that motivated behavior-e.g., seeking food and mates, avoiding predators-consists of sequences of actions organized within nested behavioral states. This framework has bridged behavioral ecology and experimental psychology, providing key insights into critical behavioral processes. In particular, the behavior systems framework entails a particular organization of behavior over time. The present paper examines whether such organization emerges from a generic Markov process, where the current behavioral state determines the probability distribution of subsequent behavioral states. This proposition is developed as a systematic examination of increasingly complex Markov models, seeking a computational formulation that balances adherence to the behavior systems approach, parsimony, and conformity to data. As a result of this exercise, a nonstationary partially hidden Markov model is selected as a computational formulation of the predatory subsystem. It is noted that the temporal distribution of discrete responses may further unveil the structure and parameters of the model but, without proper mathematical modeling, these discrete responses may be misleading. Opportunities for further elaboration of the proposed computational formulation are identified, including developments in its architecture, extensions to defensive and reproductive subsystems, and methodological refinements.

Culture and Child Attachment Patterns: a Behavioral Systems Synthesis

We propose that the two dominant culture institutions (individualist and collectivist) are neither learned nor cognitively represented by the people who practice them. Instead, they exist as group-level payoff structures that reflect differential distributions of child attachment patterns within a society. Individualist societies reflect an overrepresentation of insecure-avoidant attachments and collectivist societies reflect an overrepresentation of insecure-anxious attachments. Moreover, attachment patterns are embodied rather than representational-schedule-induced rather than incrementally shaped or verbally learned. If attachment patterns are schedule-induced, the prospects are poorer for effecting cultural change through economic incentives or informational campaigns (top-down). Rather, cultural practices will be responsive to changes in family practices-to the extent they affect attachment patterns (bottom-up). For example, if breastfeeding rates decline or the workforce participation of women increases, a society will become more individualist and less collectivist. That is because those practices increase avoidant as compared to anxious attachments. Moreover, because insecurely attached children are behaviorally less flexible than are securely attached children, the former have a greater impact on cultural practices than do the latter.

Rewards interact with repetition-dependent learning to enhance long-term retention of motor memories

The combination of behavioral experiences that enhance long-term retention remains largely unknown. Informed by neurophysiological lines of work, this study tested the hypothesis that performance-contingent monetary rewards potentiate repetition-dependent forms of learning, as induced by extensive practice at asymptote, to enhance long-term retention of motor memories. To this end, six groups of 14 participants (n = 84) acquired novel motor behaviors by adapting to a gradual visuomotor rotation while these factors were manipulated. Retention was assessed 24 h later. While all groups similarly acquired the novel motor behaviors, results from the retention session revealed an interaction indicating that rewards enhanced long-term retention, but only when practice was extended to asymptote. Specifically, the interaction indicated that this effect selectively occurred when rewards were intermittently available (i.e., 50%), but not when they were absent (i.e., 0%) or continuously available (i.e., 100%) during acquisition. This suggests that the influence of rewards on extensive practice and long-term retention is nonlinear, as continuous rewards did not further enhance retention as compared with intermittent rewards. One possibility is that rewards' intermittent availability allows to maintain their subjective value during acquisition, which may be key to potentiate long-term retention.

Incentive hope: A default psychological response to multiple forms of uncertainty

Our target article proposes that a new concept - incentive hope - is necessary in the behavioral sciences to explain animal foraging under harsh environmental conditions. Incentive hope refers to a specific motivational mechanism in the brain - considered only in mammals and birds. But it can also be understood at a functional level, as an adaptive behavioral strategy that contributes to improve survival. Thus, this concept is an attempt to bridge across different research fields such as behavioral psychology, reward neuroscience, and behavioral ecology. Many commentaries suggest that incentive hope even could help understand phenomena beyond these research fields, including food wasting and food sharing, mental energy conservation, diverse psychopathologies, irrational decisions in invertebrates, and some aspects of evolution by means of sexual selection. We are favorable to such extensions because incentive hope denotes an unconscious process capable of working against many forms of adversity; organisms do not need to hope as a subjective feeling, but to behave as if they had this feeling. In our response, we carefully discuss each suggestion and criticism and reiterate the importance of having a theory accounting for motivation under reward uncertainty.

A glucocorticoid receptor antagonist reduces sign-tracking behavior in male Japanese quail

Addiction is characterized as a chronic debilitating disease. One devastating feature of addiction is the susceptibility of relapse (40-60%) after stretches of abstinence. One theory that may account for relapse suggests that drug cues (e.g., paraphernalia) may increase stress hormones, and this may prompt relapse. Repeatedly pairing a neutral cue with a reward is commonly utilized to measure what subjects learn about a cue that is predictive of reward. Research has shown that animals that attend to a cue more than to the reward (sign trackers) may be more vulnerable to drug addiction. Additionally, research has shown that sign tracking is associated with an increase in corticosterone, a primary stress hormone. PT150 is a novel glucocorticoid receptor antagonist that moderates the release of corticosterone. In the current experiment, it was hypothesized that subjects given repeated administration of PT150 would reduce sign tracking compared to subjects given placebo. Time spent (in seconds) near a cue that predicts reward (conditional stimulus) served as a measure of sign tracking, and PT150 or placebo was administered following sign tracking. An independent-samples t test revealed that subjects that received PT150 had reduced time spent near the conditioned stimulus compared to controls. Given the devastating effects of drug addiction, identification of a potential pharmacological intervention in the reduction of relapse would be of great value. Therefore, future research is needed to validate the use of PT150 in reducing behaviors associated with drug addiction. (PsycINFO Database Record

How foraging works: Uncertainty magnifies food-seeking motivation

Food uncertainty has the effect of invigorating food-related responses. Psychologists have noted that mammals and birds respond more to a conditioned stimulus that unreliably predicts food delivery, and ecologists have shown that animals (especially small passerines) consume and/or hoard more food and can get fatter when access to that resource is unpredictable. Are these phenomena related? We think they are. Psychologists have proposed several mechanistic interpretations, while ecologists have suggested a functional interpretation: The effect of unpredictability on fat reserves and hoarding behavior is an evolutionary strategy acting against the risk of starvation when food is in short supply. Both perspectives are complementary, and we argue that the psychology of incentive motivational processes can shed some light on the causal mechanisms leading animals to seek and consume more food under uncertainty in the wild. Our theoretical approach is in agreement with neuroscientific data relating to the role of dopamine, a neurotransmitter strongly involved in incentive motivation, and its plausibility has received some explanatory and predictive value with respect to Pavlovian phenomena. Overall, we argue that the occasional and unavoidable absence of food rewards has motivational effects (called incentive hope) that facilitate foraging effort. We show that this hypothesis is computationally tenable, leading foragers in an unpredictable environment to consume more food items and to have higher long-term energy storage than foragers in a predictable environment.

Disconnection of basolateral amygdala and insular cortex disrupts conditioned approach in Pavlovian lever autoshaping

Previously established individual differences in appetitive approach and devaluation sensitivity observed in goal- and sign-trackers may be attributed to differences in the acquisition, modification, or use of associative information in basolateral amygdala (BLA) pathways. Here, we sought to determine the extent to which communication of associative information between BLA and anterior portions of insular cortex (IC) supports ongoing Pavlovian conditioned approach behaviors in sign- and goal-tracking rats, in the absence of manipulations to outcome value. We hypothesized that the BLA mediates goal-, but not sign- tracking approach through interactions with the IC, a brain region involved in supporting flexible behavior. We first trained rats in Pavlovian lever autoshaping to determine their sign- or goal-tracking tendency. During alternating test sessions, we gave unilateral intracranial injections of vehicle or a cocktail of gamma-aminobutyric acid (GABA) receptor agonists, baclofen and muscimol, unilaterally into the BLA and contralaterally or ipsilaterally into the IC prior to reinforced lever autoshaping sessions. Consistent with our hypothesis we found that contralateral inactivation of BLA and IC increased the latency to approach the food cup and decreased the number of food cup contacts in goal-trackers. While contralateral inactivation of BLA and IC did not affect the total number of lever contacts in sign-trackers, this manipulation increased the latency to approach the lever. Ipsilateral inactivation of BLA and IC did not impact approach behaviors in Pavlovian lever autoshaping. These findings, contrary to our hypothesis, suggest that communication between BLA and IC maintains a representation of initially learned appetitive associations that commonly support the initiation of Pavlovian conditioned approach behavior regardless of whether it is directed at the cue or the location of reward delivery.

Foraging motivation favors the occurrence of Lévy walks

Lévy walks are a property of random movements often observed among foraging animals (and humans), and they might confer some advantages for survival in an unpredictable environment, in comparison with Brownian walks. In animals with a nervous system, specific neurotransmitters associated with some psychological states could play a crucial role in controlling the occurrence of Lévy walks. We argue that incentive motivation, a dopamine-dependent process that in vertebrates makes rewards and their predictive conditioned stimuli attractive, has behavioral effects that may favor their occurrence: incentive motivation is higher when food is unpredictable and it strongly underpins foraging activity. An individual-based computer model is used to determine whether changes in incentive motivation can influence the probability that Lévy walks occur among foraging agents. Our results suggest that they are produced more often under an unpredictable than a predictable food access, and more often in strongly rather than weakly motivated foragers exposed to an unpredictable food access. Also, our motivational framework indicates that the occurrence of Lévy walks are correlated with, but not causally linked to, the number of food items consumed and the ability to store fat reserves. We conclude that Lévy walks can confer some advantages for survival in an unpredictable environment, provided that they appear in foragers with a high motivation to seek food.

Frameworking memory and serotonergic markers

The evidence for neural markers and memory is continuously being revised, and as evidence continues to accumulate, herein, we frame earlier and new evidence. Hence, in this work, the aim is to provide an appropriate conceptual framework of serotonergic markers associated with neural activity and memory. Serotonin (5-hydroxytryptamine [5-HT]) has multiple pharmacological tools, well-characterized downstream signaling in mammals’ species, and established 5-HT neural markers showing new insights about memory functions and dysfunctions, including receptors (5-HT1A/1B/1D, 5-HT2A/2B/2C, and 5-HT3-7), transporter (serotonin transporter [SERT]) and volume transmission present in brain areas involved in memory. Bidirectional influence occurs between 5-HT markers and memory/amnesia. A growing number of researchers report that memory, amnesia, or forgetting modifies neural markers. Diverse approaches support the translatability of using neural markers and cerebral functions/dysfunctions, including memory formation and amnesia. At least, 5-HT1A, 5-HT4, 5-HT6, and 5-HT7receptors and SERT seem to be useful neural markers and therapeutic targets. Hence, several mechanisms cooperate to achieve synaptic plasticity or memory, including changes in the expression of neurotransmitter receptors and transporters.

Suboptimal choice in rats: Incentive salience attribution promotes maladaptive decision-making

Stimuli that are more predictive of subsequent reward also function as better conditioned reinforcers. Moreover, stimuli attributed with incentive salience function as more robust conditioned reinforcers. Some theories have suggested that conditioned reinforcement plays an important role in promoting suboptimal choice behavior, like gambling. The present experiments examined how different stimuli, those attributed with incentive salience versus those without, can function in tandem with stimulus-reward predictive utility to promote maladaptive decision-making in rats. One group of rats had lights associated with goal-tracking as the reward-predictive stimuli and another had levers associated with sign-tracking as the reward-predictive stimuli. All rats were first trained on a choice procedure in which the expected value across both alternatives was equivalent but differed in their stimulus-reward predictive utility. Next, the expected value across both alternatives was systematically changed so that the alternative with greater stimulus-reward predictive utility was suboptimal in regard to primary reinforcement. The results demonstrate that in order to obtain suboptimal choice behavior, incentive salience alongside strong stimulus-reward predictive utility may be necessary; thus, maladaptive decision-making can be driven more by the value attributed to stimuli imbued with incentive salience that reliably predict a reward rather than the reward itself.

Liking, wanting, and the incentive-sensitization theory of addiction

Rewards are both "liked" and "wanted," and those 2 words seem almost interchangeable. However, the brain circuitry that mediates the psychological process of "wanting" a particular reward is dissociable from circuitry that mediates the degree to which it is "liked." Incentive salience or "wanting," a form of motivation, is generated by large and robust neural systems that include mesolimbic dopamine. By comparison, "liking," or the actual pleasurable impact of reward consumption, is mediated by smaller and fragile neural systems, and is not dependent on dopamine. The incentive-sensitization theory posits the essence of drug addiction to be excessive amplification specifically of psychological "wanting," especially triggered by cues, without necessarily an amplification of "liking." This is because of long-lasting changes in dopamine-related motivation systems of susceptible individuals, called "neural sensitization." A quarter-century after its proposal, evidence has continued to grow in support the incentive-sensitization theory. Further, its scope is now expanding to include diverse behavioral addictions and other psychopathologies. (PsycINFO Database Record