Time discounting and time preference in animals: A critical review

Hungry pigeons prefer sooner rare food over later likely food or faster information

Introduction

Making decisions and investing effort to obtain rewards may depend on various factors, such as the delay to reward, the probability of its occurrence, and the information that can be collected about it. As predicted by various theories, pigeons and other animals indeed mind these factors when deciding.

Methods

We now implemented a task in which pigeons were allowed to choose among three options and to peck at the chosen key to improve the conditions of reward delivery. Pecking more at a first color reduced the 12-s delay before food was delivered with a 33.3% chance, pecking more at a second color increased the initial 33.3% chance of food delivery but did not reduce the 12-s delay, and pecking more at a third color reduced the delay before information was provided whether the trial will be rewarded with a 33.3% chance after 12 s.

Results

Pigeons' preference (delay vs. probability, delay vs. information, and probability vs. information), as well as their pecking effort for the chosen option, were analyzed. Our results indicate that hungry pigeons preferred to peck for delay reduction but did not work more for that option than for probability increase, which was the most profitable alternative and did not induce more pecking effort. In this task, information was the least preferred and induced the lowest level of effort. Refed pigeons showed no preference for any option but did not drastically reduce the average amounts of effort invested.

Discussion

These results are discussed in the context of species-specific ecological conditions that could constrain current foraging theories.

Prior cocaine self-administration does not impair the ability to delay gratification in rats during diminishing returns

Previous exposure to drugs of abuse produces impairments in studies of reversal learning, delay discounting and response inhibition tasks. While these studies contribute to the understanding of normal decision-making and how it is impaired by drugs of abuse, they do not fully capture how decision-making impacts the ability to delay gratification for greater long-term benefit. To address this issue, we used a diminishing returns task to study decision-making in rats that had previously self-administered cocaine. This task was designed to test the ability of the rat to choose to delay gratification in the short-term to obtain more reward over the course of the entire behavioral session. Rats were presented with two choices. One choice had a fixed amount of time delay needed to obtain reward [i.e. fixed delay (FD)], while the other choice had a progressive delay (PD) that started at 0 s and progressively increased by 1 s each time the PD option was selected. During the 'reset' variation of the task, rats could choose the FD option to reset the time delay associated with the PD option. Consistent with previous results, we found that prior cocaine exposure reduced rats' overall preference for the PD option in post-task reversal testing during 'no-reset' sessions, suggesting that cocaine exposure made rats more sensitive to the increasing delay of the PD option. Surprisingly, however, we found that rats that had self-administered cocaine 1-month prior, adapted behavior during 'reset' sessions by delaying gratification to obtain more reward in the long run similar to control rats.

Environmental enrichment promotes adaptive responding during tests of behavioral regulation in male heterogeneous stock rats

Organisms must regulate their behavior flexibly in the face of environmental challenges. Failure can lead to a host of maladaptive behavioral traits associated with a range of neuropsychiatric disorders, including attention deficit hyperactivity disorder, autism, and substance use disorders. This maladaptive dysregulation of behavior is influenced by genetic and environmental factors. For example, environmental enrichment produces beneficial neurobehavioral effects in animal models of such disorders. The present study determined the effects of environmental enrichment on a range of measures related to behavioral regulation using a large cohort of male, outbred heterogeneous stock (HS) rats as subjects. Subjects were reared from late adolescence onwards either in pairs in standard housing with minimal enrichment (n = 200) or in groups of 16 in a highly enriched environment consisting of a large multi-level cage filled with toys, running wheels, and shelters (n = 64). Rats were subjected to a battery of tests, including: (i) locomotor response to novelty, (ii) light reinforcement, (iii) social reinforcement, (iv) reaction time, (v) a patch-depletion foraging test, (vi) Pavlovian conditioned approach, (vii) conditioned reinforcement, and (viii) cocaine conditioned cue preference. Results indicated that rats housed in the enriched environment were able to filter out irrelevant stimuli more effectively and thereby regulate their behavior more efficiently than standard-housing rats. The dramatic impact of environmental enrichment suggests that behavioral studies using standard housing conditions may not generalize to more complex environments that may be more ethologically relevant.

Using rodent data to elucidate dopaminergic mechanisms of ADHD: Implications for human personality

An altered behavioral response to positive reinforcement has been proposed to be a core deficit in attention deficit hyperactivity disorder (ADHD). The spontaneously hypertensive rat (SHR), a congenic animal strain, displays a similarly altered response to reinforcement. The presence of this genetically determined phenotype in a rodent model allows experimental investigation of underlying neural mechanisms. Behaviorally, the SHR displays increased preference for immediate reinforcement, increased sensitivity to individual instances of reinforcement relative to integrated reinforcement history, and a steeper delay of reinforcement gradient compared to other rat strains. The SHR also shows less development of incentive to approach sensory stimuli, or cues, that predict reward after repeated cue-reward pairing. We consider the underlying neural mechanisms for these characteristics. It is well known that midbrain dopamine neurons are initially activated by unexpected reward and gradually transfer their responses to reward-predicting cues. This finding has inspired the dopamine transfer deficit (DTD) hypothesis, which predicts certain behavioral effects that would arise from a deficient transfer of dopamine responses from actual rewards to reward-predicting cues. We argue that the DTD predicts the altered responses to reinforcement seen in the SHR and individuals with ADHD. These altered responses to reinforcement in turn predict core symptoms of ADHD. We also suggest that variations in the degree of dopamine transfer may underlie variations in personality dimensions related to altered reinforcement sensitivity. In doing so, we highlight the value of rodent models to the study of human personality.

Environmental enrichment promotes adaptive responding during tests of behavioral regulation in male heterogeneous stock rats

Organisms must regulate their behavior flexibly in the face of environmental challenges. Failure can lead to a host of maladaptive behavioral traits associated with a range of neuropsychiatric disorders, including attention deficit hyperactivity disorder, autism, and substance use disorders. This maladaptive dysregulation of behavior is influenced by genetic and environmental factors. For example, environmental enrichment produces beneficial neurobehavioral effects in animal models of such disorders. The present study determined the effects of environmental enrichment on a range of measures related to behavioral regulation using a large cohort of male, outbred heterogeneous stock (HS) rats as subjects to mimic the genetic variability found in the human population. Subjects were reared from late adolescence onwards either in pairs in standard housing with minimal enrichment (n=200) or in groups of 16 in a highly enriched environment consisting of a large multi-level cage filled with toys, running wheels, and shelters (n=64). Rats were subjected to a battery of tests, including: (i) locomotor response to novelty, (iI) light reinforcement, (iii) social reinforcement, (iv) reaction time, (v) a patch-depletion foraging test, (vi) Pavlovian conditioned approach, (vii) conditioned reinforcement, and (viii) cocaine conditioned cue preference. Results indicated that rats housed in the enriched environment were able to filter out irrelevant stimuli more effectively and thereby regulate their behavior more efficiently than standard-housing rats. The dramatic impact of environmental enrichment suggests that behavioral studies using standard housing conditions may not generalize to more complex environments that may be more ethologically relevant.

Overharvesting in human patch foraging reflects rational structure learning and adaptive planning

Patch foraging presents a sequential decision-making problem widely studied across organisms-stay with a current option or leave it in search of a better alternative? Behavioral ecology has identified an optimal strategy for these decisions, but, across species, foragers systematically deviate from it, staying too long with an option or "overharvesting" relative to this optimum. Despite the ubiquity of this behavior, the mechanism underlying it remains unclear and an object of extensive investigation. Here, we address this gap by approaching foraging as both a decision-making and learning problem. Specifically, we propose a model in which foragers 1) rationally infer the structure of their environment and 2) use their uncertainty over the inferred structure representation to adaptively discount future rewards. We find that overharvesting can emerge from this rational statistical inference and uncertainty adaptation process. In a patch-leaving task, we show that human participants adapt their foraging to the richness and dynamics of the environment in ways consistent with our model. These findings suggest that definitions of optimal foraging could be extended by considering how foragers reduce and adapt to uncertainty over representations of their environment.

Can we tackle climate change by behavioral hacking of the dopaminergic system?

Climate change is an undeniable fact that will certainly affect millions of people in the following decades. Despite this danger threatening our economies, wellbeing and our lives in general, there is a lack of immediate response at both the institutional and individual level. How can it be that the human brain cannot interpret this threat and act against it to avoid the immense negative consequences that may ensue? Here we argue that this paradox could be explained by the fact that some key brain mechanisms are potentially poorly tuned to take action against a threat that would take full effect only in the long-term. We present neuro-behavioral evidence in favor of this proposal and discuss the role of the dopaminergic (DA) system in learning accurate prediction of the value of an outcome, and its consequences regarding the climate issue. We discuss how this system discounts the value of delayed outcomes and, consequently, does not favor action against the climate crisis. Finally, according to this framework, we suggest that this view may be reconsidered and, on the contrary, that the DA reinforcement learning system could be a powerful ally if adapted to short-term incentives which promote climate-friendly behaviors. Additionally, the DA system interacts with multiple brain systems, in particular those related to higher cognitive functions, which can adjust its functions depending on psychological, social, or other complex contextual information. Thus, we propose several generic action plans that could help to hack these neuro-behavioral processes to promote climate-friendly actions.

Automated pose estimation in primates

Understanding the behavior of primates is important for primatology, for psychology, and for biology more broadly. It is also important for biomedicine, where primates are an important model organism, and whose behavior is often an important variable of interest. Our ability to rigorously quantify behavior has, however, long been limited. On one hand, we can rigorously quantify low-information measures like preference, looking time, and reaction time; on the other, we can use more gestalt measures like behavioral categories tracked via ethogram, but at high cost and with high variability. Recent technological advances have led to a major revolution in behavioral measurement that offers affordable and scalable rigor. Specifically, digital video cameras and automated pose tracking software can provide measures of full-body position (i.e., pose) of primates over time (i.e., behavior) with high spatial and temporal resolution. Pose-tracking technology in turn can be used to infer behavioral states, such as eating, sleeping, and mating. We call this technological approach behavioral imaging. In this review, we situate the behavioral imaging revolution in the history of the study of behavior, argue for investment in and development of analytical and research techniques that can profit from the advent of the era of big behavior, and propose that primate centers and zoos will take on a more central role in relevant fields of research than they have in the past.

Effect of Financial Incentives on Hypertension Control: A Multicenter Randomized Controlled Trial in China

Poorly controlled hypertension is a great challenge to global public health. Incentive approaches, based on behavioral and economic concepts, may improve patients’ adherence to treatment.

OXTR Gene DNA Methylation Levels Are Associated with Discounting Behavior with Untrustworthy Proposers

Individual differences in temporal and probabilistic discounting are associated with a wide range of life outcomes in literature. Traditional approaches have focused on impulsiveness and cognitive control skills, on goal-oriented personality traits as well as on the psychological perception of time. More recently, literature started to consider the role of social and contextual factors in discounting behavior. Between others, higher generalized trust in human beings and specific trust in people who will deliver the future/probabilistic rewards have been related to a stronger willingness to wait and to assume risk. Moreover, the tendency to trust others has been associated with the oxytocin receptor gene regulation that can be modified by life experiences. In this perspective, we hypothesized that differences in the tendency to wait and to take risks for a more desirable reward according to the proposer’s trustworthiness could be related to a different level of DNA methylation at the oxytocin receptor gene. Findings confirmed that participants are less willing to wait and to risk when the proposer is considered highly untrustworthy and revealed how higher oxytocin receptor gene DNA methylation is associated with a stronger effect due to the presence of an untrustworthy proposer. Limits and future directions are outlined.

Perceptions of nurses and physicians on pay-for-performance in hospital: a systematic review of qualitative studies

AIMS

To systematically examine perceptions of nurses and physicians on pay-for-performance in hospital.

BACKGROUND

Pay-for-performance projects have proliferated over the past two decades, most systematic reviews of which solely focused on its effectiveness in primary healthcare and the physicians' or nurses' attitudes. However, systematic reviews of qualitative approaches for better examining perceptions of both nurses and physicians in hospital are lacking.

EVALUATION

Electronic databases were systematic searched with date from its inception to December 31, 2020. Meta-aggregation synthesis methodology and the conceptual framework of the Theory of Planned Behavior were used to summarize findings.

KEY ISSUES

A total of nine studies were included. Three major synthesized themes were identified: (1) perceptions of the motivation effects and positive outcomes (2) perceptions about the design defects and negative effects (3) perceptions of the obstacles in the implementation process.

CONCLUSION

To maximize the intended positive effects, nurses' and physicians' perceptions should be considered and incorporated into the project design and implementation stage.

IMPLICATIONS FOR NURSING MANAGEMENT AND RESEARCH

The paper gives enlightenment to nurse managers on improving and advancing the cause of nurses when planning for or evaluating their institutions' policies on pay-for-performance in the future research.

Context-sensitive valuation and learning

A variety of behavioral and neural phenomena suggest that organisms evaluate outcomes not on an absolute utility scale, but relative to some dynamic and context-sensitive reference or scale. Sometimes, as in foraging tasks, this results in sensible choices; in other situations, like choosing between options learned in different contexts, irrational choices can result. We argue that what unites and demystifies these various phenomena is that the brain's goal is not assessing utility as an end in itself, but rather comparing different options to choose the better one. In the presence of uncertainty, noise, or costly computation, adjusting options to the context can produce more accurate choices.

Rats delay gratification during a time-based diminishing returns task

The rat is a common animal model used to uncover the neural underpinnings of decision making and their disruption in psychiatric illness. Here, we ask if rats can perform a decision-making task that assesses self-control by delayed gratification in the context of diminishing returns. In this task, rats could choose to press one of two levers. One lever was associated with a fixed delay (FD) schedule that delivered reward after a fixed time delay (10 s). The other lever was associated with a progressive delay (PD) schedule; the delay increased by a fixed amount of time (1 s) after each PD lever press. Rats were tested under two conditions: a reset condition where rats could reset the PD schedule back to its initial 0-s delay by pressing the FD lever and a no-reset condition in which resetting the PD schedule was unavailable. We found that rats adapted behavior within reset sessions by delaying gratification to obtain more reward in the long run. That is, they selected the FD lever with the longer delay to reset the PD delay back to zero prior to the equality point, thus achieving more reward over the course of the session. These results are consistent with other species, demonstrating that rats can also maximize the net rate of reward by selecting an option that is not immediately beneficial. Moreover, use of this task in rodents might provide insights into how the brain governs normal and abnormal behavior, as well as treatments that can improve self-control. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

The irrelevancy of the inter-trial interval in delay-discounting experiments on an animal model of ADHD

Delay discounting involves choosing between a small, immediate reward, and a larger but delayed one. As the delay between choice and large reward gets longer, people with ADHD tend to become impulsive faster than controls, indicated by a switch in preference from the large to the smaller reward. Choosing the smaller reward when the larger is considered reward maximizing is labeled impulsive behaviour. It is well documented that increased delays between choice and reward affects choice preference in both humans and other animals. Other variables such as the inter-trial interval or trial length are observed to have an effect on human discounting, but their effect on discounting in other animals is largely assumed rather than tested. In the current experiment, we tested this assumption. One group of rats was exposed to increasing delays between choosing the large reward and receiving it, while another group experienced longer inter-trial intervals that were equal in length to the delays in the other group. This ensured that trial length was controlled for in delay discounting, but that the delay function and inter-trial intervals could be manipulated and measured separately. Results showed that while the delay between choice and reward caused impulsive behaviour in rats, the length of the inter-trial interval (and by extension trial length) had no impact on choice behaviour. A follow-up experiment found this to be the case even if the length of the inter-trial interval was signaled with audio cues. These results suggest that rats, and possibly animals in general, are insensitive to time between trials, and therefore cannot easily represent human counterparts on the task.

Continuous decisions

Humans and other animals evolved to make decisions that extend over time with continuous and ever-changing options. Nonetheless, the academic study of decision-making is mostly limited to the simple case of choice between two options. Here, we advocate that the study of choice should expand to include continuous decisions. Continuous decisions, by our definition, involve a continuum of possible responses and take place over an extended period of time during which the response is continuously subject to modification. In most continuous decisions, the range of options can fluctuate and is affected by recent responses, making consideration of reciprocal feedback between choices and the environment essential. The study of continuous decisions raises new questions, such as how abstract processes of valuation and comparison are co-implemented with action planning and execution, how we simulate the large number of possible futures our choices lead to, and how our brains employ hierarchical structure to make choices more efficiently. While microeconomic theory has proven invaluable for discrete decisions, we propose that engineering control theory may serve as a better foundation for continuous ones. And while the concept of value has proven foundational for discrete decisions, goal states and policies may prove more useful for continuous ones. This article is part of the theme issue 'Existence and prevalence of economic behaviours among non-human primates'.

Dopaminergic modulation of reward discounting in healthy rats: a systematic review and meta-analysis

RATIONALE

Although numerous studies have suggested that pharmacological alteration of the dopamine (DA) system modulates reward discounting, these studies have produced inconsistent findings.

OBJECTIVES

Here, we conducted a systematic review and pre-registered meta-analysis to evaluate DA drug-mediated effects on reward discounting of time, probability, and effort costs in studies of healthy rats. This produced a total of 1343 articles to screen for inclusion/exclusion. From the literature, we identified 117 effects from approximately 1549 individual rats.

METHODS

Using random effects with maximum-likelihood estimation, we meta-analyzed placebo-controlled drug effects for (1) DA D1-like receptor agonists and (2) antagonists, (3) D2-like agonists and (4) antagonists, and (5) DA transporter-modulating drugs.

RESULTS

Meta-analytic effects showed that DAT-modulating drugs decreased reward discounting. While D1-like and D2-like antagonists both increased discounting, agonist drugs for those receptors had no significant effect on discounting behavior. A number of these effects appear contingent on study design features like cost type, rat strain, and microinfusion location.

CONCLUSIONS

These findings suggest a nuanced relationship between DA and discounting behavior and urge caution when drawing generalizations about the effects of pharmacologically manipulating dopamine on reward-based decision-making.

Some theoretical notes on spatial discounting

Spatial discounting is a largely underexplored area of decision-making research, both theoretically and empirically, especially when compared to intertemporal choice, which has received significant attention in psychology and animal behaviour. Spatial decision problems seem to share some of the same features of a temporal decision problem (namely, the risk of reward objects disappearing and the opportunity cost of waiting), but there are several additional factors that affect the appropriate discount function for distant rewards. These include more significant opportunity costs, changes in the distances to all the other available opportunities, the post-reward costs of getting back home, the complex energetics associated with locomotion and all the additional risks faced by travelling itself. This paper organises and explores these factors and suggests some normative models that should predict the adaptive behaviour of animals and humans.

Effects of a high-fat diet on impulsive choice in rats

INTRODUCTION

Obesity and binge eating disorder are associated with high levels of impulsivity, but the causal role of eating and palatable food in these associations is unclear. Studies in rodents show that a high-fat diet can increase one aspect of impulsivity (impulsive action); it is less clear, however, whether a dissociable aspect of impulsivity (impulsive choice) is similarly affected. Hence, the aim of this study was to ascertain whether chronic exposure to a high-fat diet would alter impulsive choice.

METHODS

Male rats were maintained on either a high-fat or control chow diet for two weeks ad libitum. They then underwent equi-caloric food restriction for the duration of the experiment, with each group maintained on their respective diet. To measure impulsive choice, rats were trained on a delay discounting task (DDT) in which they made discrete choices between a lever that delivered a small food reward immediately and a lever that delivered a large food reward accompanied by systematically increasing delays. Upon reaching stable performance on the DDT, rats were given acute systemic injections of amphetamine prior to testing in the DDT to determine whether increased monoamine transmission affected impulsive choice differently in the two diet groups. Lastly, subjects were tested on a progressive ratio schedule of reinforcement to assess motivation for a sucrose reward.

RESULTS

There was no significant effect of the high-fat diet on impulsive choice. Further, amphetamine decreased choice of the large, delayed reward (increased impulsive choice) to the same extent in both groups. Exposure to the high-fat diet did, however, increase motivation to obtain a sucrose reward.

CONCLUSIONS

These experiments reveal that, under conditions that do not promote weight gain, a chronic high-fat diet does not affect impulsive choice in a delay discounting task. The data are surprising in light of findings showing that this same diet alters impulsive action, and highlight the necessity of further research to elucidate relationships between palatable food consumption and impulsivity.

Are capuchin monkeys (Sapajus spp.) sensitive to lost opportunities? The role of opportunity costs in intertemporal choice

Principles of economics predict that the costs associated with obtaining rewards can influence choice. When individuals face choices between a smaller, immediate option and a larger, later option, they often experience opportunity costs associated with waiting for delayed rewards because they must forego the opportunity to make other choices. We evaluated how reducing opportunity costs affects delay tolerance in capuchin monkeys. After choosing the larger option, in the High cost condition, subjects had to wait for the delay to expire, whereas in the Low cost different and Low cost same conditions, they could perform a new choice during the delay. To control for the effect of intake rate on choices, the Low cost same condition had the same intake rate ratio as the High cost condition. We found that capuchins attended both to intake rates and to opportunity costs. They chose the larger option more often in the Low cost different and Low cost same conditions than in the High cost condition, and more often in the Low cost different condition than in the Low cost same condition. Understanding how non-human primates represent and use costs in making decisions not only helps to develop theoretical frameworks to explain their choices but also addresses similarities with and differences from human decision-making. These outcomes provide insights into the origins of human economic behaviour. This article is part of the theme issue 'Existence and prevalence of economic behaviours among non-human primates'.

Trail Pheromone Does Not Modulate Subjective Reward Evaluation in Lasius niger Ants

Comparing the value of options is at the heart of economic decision-making. While an option may have an absolute quality (e.g. a food source has a fixed energy content), the perceived value of the option may be malleable. The factors affecting the perceived value of an option may thus strongly influence which option is ultimately chosen. Expectations have been shown to be a strong driver of perceived value in both humans and social insects, causing an undervaluation of a given option if a better option was expected, and an overvaluation if a poorer one was expected. In humans, perceived value can be strongly affected by social information. Value perception in some insects has also been shown to be affected by social information, showing conformism as in humans and other animals. Here, over a series of experiments, we tested whether pheromone trail presence, a social information source, influenced the perceived value of a food source in the ant Lasius niger. We found that the presence of pheromone trails leading to a sucrose solution does not influence food acceptance, pheromone deposition when returning from a food source, drinking time, or frequency of U-turns on return from the food. Two further assays for measuring changes in food acceptance, designed to increase sensitivity by avoiding ceiling effects, also showed no effect of pheromone presence on food acceptance. In a separate study, L. niger have also been found to show no preference for, or avoidance of, odors associated with foods found in the presence of pheromone. We are thus confident that trail pheromone presence does not affect the perceived value of a food source in these ants.

Choosing fast and simply: Construction of preferences by starlings through parallel option valuation

The integration of normative and descriptive analyses of decision processes in humans struggles with the fact that measuring preferences by different procedures yields different rankings and that humans appear irrationally impulsive (namely, show maladaptive preference for immediacy). Failure of procedure invariance has led to the widespread hypothesis that preferences are constructed "on the spot" by cognitive evaluations performed at choice time, implying that choices should take extra time in order to perform the necessary comparisons. We examine this issue in experiments with starlings (Sturnus vulgaris) and show that integrating normative and descriptive arguments is possible and may help reinterpreting human decision results. Our main findings are that (1) ranking alternatives through direct rating (response time) accurately predicts preference in choice, overcoming failures of procedure invariance; (2) preference is not constructed at choice time nor does it involve extra time (we show that the opposite is true); and (3) starlings' choices are not irrationally impulsive but are instead directly interpretable in terms of profitability ranking. Like all nonhuman research, our protocols examine decisions by experience rather than by description, and hence support the conjecture that irrationalities that prevail in research with humans may not be observed in decisions by experience protocols.

Neural correlates of delay discount alterations in addiction and psychiatric disorders: A systematic review of magnetic resonance imaging studies

Delay discounting (DD) represents decreased subjective value for delayed reward relative to the same reward at present. The concept of DD has been applied for pathophysiology of addiction and psychiatric disorders. However, the detailed neuroimaging correlates of DD underlying pathophysiology still remain unclear. Thus, we conducted a systematic review to investigate neural correlates of DD on magnetic resonance imaging studies among addiction and psychiatric disorders. Specific search terms were set on PubMed to identify relevant articles. Initial search identified 551 records and 31 studies met the inclusion criteria. The present review revealed that greater DD was correlated with increased activity in areas related to reward evaluation and prediction as well as decreased activity in areas related to cognitive control. Healthy controls showed smaller changes in activities of these areas associated with DD when compared to patient groups. As the neural basis related to DD, three neural networks have been proposed that are associated with the actions of short-term interests and long-term benefits. Among the three potential neural networks on DD, the first one included the ventromedial prefrontal cortex and ventral striatum and implicated in evaluating reward values, the second network included the anterior cingulate cortex and linked to cognitive control, and the third network included the middle temporal gyrus and was involved in predictions and affection. This review generated consistent findings on the neural basis of DD among patients with addiction and psychiatric disorders, which may represent the pathophysiology related to DD and impulsivity of mental illness.

A Critical Review of Behavioral and Emotional Disinhibition

The theory of "disinhibition" has been very influential in psychiatry and neurology for over a century. Disinhibition has been used to explain clinical findings in many neurological and psychiatric disorders including dementia, traumatic brain injury, attention deficit hyperactive disorder, substance abuse, impulsivity in personality disorders, and neurodevelopmental disorders. In addition, disinhibition has been used as a unifying theory to link clinical observations with cognitive findings, and even cellular findings. This review discusses the origins and history of the theory of disinhibition and its strengths and weaknesses in four domains: face validity, consistency with other brain mechanisms, consistency with evolutionary mechanisms, and empiric support. I assert that the vagueness of the theory, inconsistency with other brain mechanisms, and lack of empiric support limit the usefulness of this theory. Alternative approaches, based on findings in other motor, language, and cognitive functions, are discussed.

Self-control in crows, parrots and nonhuman primates

Self-control is critical for both humans and nonhuman animals because it underlies complex cognitive abilities, such as decision-making and future planning, enabling goal-directed behavior. For instance, it is positively associated with social competence and life success measures in humans. We present the first review of delay of gratification as a measure of self-control in nonhuman primates, corvids (crow family) and psittacines (parrot order): disparate groups that show comparable advanced cognitive abilities and similar socio-ecological factors. We compare delay of gratification performance and identify key issues and outstanding areas for future research, including finding the best measures and drivers of delayed gratification. Our review therefore contributes to our understanding of both delayed gratification as a measure of self-control and of complex cognition in animals. This article is categorized under: Cognitive Biology > Evolutionary Roots of Cognition Psychology > Comparative Psychology.

Flexible combination of reward information across primates

A fundamental but rarely contested assumption in economics and neuroeconomics is that decision-makers compute subjective values of risky options by multiplying functions of reward probability and magnitude. By contrast, an additive strategy for valuation allows flexible combination of reward information required in uncertain or changing environments. We hypothesized that the level of uncertainty in the reward environment should determine the strategy used for valuation and choice. To test this hypothesis, we examined choice between risky options in humans and rhesus macaques across three tasks with different levels of uncertainty. We found that whereas humans and monkeys adopted a multiplicative strategy under risk when probabilities are known, both species spontaneously adopted an additive strategy under uncertainty when probabilities must be learned. Additionally, the level of volatility influenced relative weighting of certain and uncertain reward information, and this was reflected in the encoding of reward magnitude by neurons in the dorsolateral prefrontal cortex.

Foraging as an evidence accumulation process

The patch-leaving problem is a canonical foraging task, in which a forager must decide to leave a current resource in search for another. Theoretical work has derived optimal strategies for when to leave a patch, and experiments have tested for conditions where animals do or do not follow an optimal strategy. Nevertheless, models of patch-leaving decisions do not consider the imperfect and noisy sampling process through which an animal gathers information, and how this process is constrained by neurobiological mechanisms. In this theoretical study, we formulate an evidence accumulation model of patch-leaving decisions where the animal averages over noisy measurements to estimate the state of the current patch and the overall environment. We solve the model for conditions where foraging decisions are optimal and equivalent to the marginal value theorem, and perform simulations to analyze deviations from optimal when these conditions are not met. By adjusting the drift rate and decision threshold, the model can represent different “strategies”, for example an incremental, decremental, or counting strategy. These strategies yield identical decisions in the limiting case but differ in how patch residence times adapt when the foraging environment is uncertain. To describe sub-optimal decisions, we introduce an energy-dependent marginal utility function that predicts longer than optimal patch residence times when food is plentiful. Our model provides a quantitative connection between ecological models of foraging behavior and evidence accumulation models of decision making. Moreover, it provides a theoretical framework for potential experiments which seek to identify neural circuits underlying patch-leaving decisions.

Foraging is a ubiquitous animal behavior, performed by organisms as different as worms, birds, rats, and humans. Although the behavior has been extensively studied, it is not known how the brain processes information obtained during foraging activity to make subsequent foraging decisions. We form an evidence accumulation model of foraging decisions that describes the process through which an animal gathers information and uses it to make foraging decisions. By building on studies of the neural decision mechanisms within systems neuroscience, this model connects the foraging decision process with ecological models of patch-leaving decisions, such as the marginal value theorem. The model suggests the existence of different foraging strategies, which optimize for different environmental conditions and their potential implementation by neural decision making circuits. The model also shows how state-dependence, such as satiation level, can affect evidence accumulation to lead to sub-optimal foraging decisions. Our model provides a framework for future experimental studies which seek to elucidate how neural decision making mechanisms have been shaped by evolutionary forces in an animal’s surrounding environment.

Ventromedial Prefrontal Cortex Tracks Multiple Environmental Variables during Search

To make efficient foraging decisions, we must combine information about the values of available options with nonvalue information. Some accounts of ventromedial PFC (vmPFC) suggest that it has a narrow role limited to evaluating immediately available options. We examined responses of neurons in area 14 (a putative macaque homolog of human vmPFC) as 2 male macaques performed a novel foraging search task. Although many neurons encoded the values of immediately available offers, they also independently encoded several other variables that influence choice, but that are conceptually distinct from offer value. These variables include average reward rate, number of offers viewed per trial, previous offer values, previous outcome sizes, and the locations of the currently attended offer. We conclude that, rather than serving as specialized economic value center, vmPFC plays a broad role in integrating relevant environmental information to drive foraging decisions.SIGNIFICANCE STATEMENT Decision makers must often choose whether to take an immediately available option or continue to search for a better one. We hypothesized that this process, which is integral to foraging theory, leaves neural signatures in the brain region ventromedial PFC. Subjects performed a novel foraging task in which they searched through differently valued options and attempted to balance their reward threshold with various time costs. We found that neurons not only encode the values of immediately available offers, but multiplexed these with environmental variables, including reward rate, number of offers viewed, previous offer values, and spatial information. We conclude that vmPFC plays a rich role in encoding and integrating multiple foraging-related variables during economic decisions.

Estimating Scale-Invariant Future in Continuous Time

Natural learners must compute an estimate of future outcomes that follow from a stimulus in continuous time. Widely used reinforcement learning algorithms discretize continuous time and estimate either transition functions from one step to the next (model-based algorithms) or a scalar value of exponentially discounted future reward using the Bellman equation (model-free algorithms). An important drawback of model-based algorithms is that computational cost grows linearly with the amount of time to be simulated. An important drawback of model-free algorithms is the need to select a timescale required for exponential discounting. We present a computational mechanism, developed based on work in psychology and neuroscience, for computing a scale-invariant timeline of future outcomes. This mechanism efficiently computes an estimate of inputs as a function of future time on a logarithmically compressed scale and can be used to generate a scale-invariant power-law-discounted estimate of expected future reward. The representation of future time retains information about what will happen when. The entire timeline can be constructed in a single parallel operation that generates concrete behavioral and neural predictions. This computational mechanism could be incorporated into future reinforcement learning algorithms.

Monkeys are curious about counterfactual outcomes

Many non-human animals show exploratory behaviors. It remains unclear whether any possess human-like curiosity. We previously proposed three criteria for applying the term curiosity to animal behavior: (1) the subject is willing to sacrifice reward to obtain information, (2) the information provides no immediate instrumental or strategic benefit, and (3) the amount the subject is willing to pay depends systematically on the amount of information available. In previous work on information-seeking in animals, information generally predicts upcoming rewards, and animals' decisions may therefore be a byproduct of reinforcement processes. Here we get around this potential confound by taking advantage of macaques' ability to reason counterfactually (that is, about outcomes that could have occurred had the subject chosen differently). Specifically, macaques sacrificed fluid reward to obtain information about counterfactual outcomes. Moreover, their willingness to pay scaled with the information (Shannon entropy) offered by the counterfactual option. These results demonstrate the existence of human-like curiosity in non-human primates according to our criteria, which circumvent several confounds associated with less stringent criteria.

Why has evolution not selected for perfect self-control?

Self-control refers to the ability to deliberately reject tempting options and instead select ones that produce greater long-term benefits. Although some apparent failures of self-control are, on closer inspection, reward maximizing, at least some self-control failures are clearly disadvantageous and non-strategic. The existence of poor self-control presents an important evolutionary puzzle because there is no obvious reason why good self-control should be more costly than poor self-control. After all, a rock is infinitely patient. I propose that self-control failures result from cases in which well-learned (and thus routinized) decision-making strategies yield suboptimal choices. These mappings persist in the decision-makers' repertoire because they result from learning processes that are adaptive in the broader context, either on the timescale of learning or of evolution. Self-control, then, is a form of cognitive control and the subjective feeling of effort likely reflects the true costs of cognitive control. Poor self-control, in this view, is ultimately a result of bounded optimality. This article is part of the theme issue 'Risk taking and impulsive behaviour: fundamental discoveries, theoretical perspectives and clinical implications.

Can delay discounting deliver on the promise of RDoC?

The National Institute of Mental Health launched the Research Domain Criteria (RDoC) initiative to better understand dimensions of behavior and identify targets for treatment. Examining dimensions across psychiatric illnesses has proven challenging, as reliable behavioral paradigms that are known to engage specific neural circuits and translate across diagnostic populations are scarce. Delay discounting paradigms seem to be an exception: they are useful for understanding links between neural systems and behavior in healthy individuals, with potential for assessing how these mechanisms go awry in psychiatric illnesses. This article reviews relevant literature on delay discounting (or the rate at which the value of a reward decreases as the delay to receipt increases) in humans, including methods for examining it, its putative neural mechanisms, and its application in psychiatric research. There exist rigorous and reproducible paradigms to evaluate delay discounting, standard methods for calculating discount rate, and known neural systems probed by these paradigms. Abnormalities in discounting have been associated with psychopathology ranging from addiction (with steep discount rates indicating relative preference for immediate rewards) to anorexia nervosa (with shallow discount rates indicating preference for future rewards). The latest research suggests that delay discounting can be manipulated in the laboratory. Extensively studied in cognitive neuroscience, delay discounting assesses a dimension of behavior that is important for decision-making and is linked to neural substrates and to psychopathology. The question now is whether manipulating delay discounting can yield clinically significant changes in behavior that promote health. If so, then delay discounting could deliver on the RDoC promise.

Local field potentials in dorsal anterior cingulate sulcus reflect rewards but not travel time costs during foraging

To maximise long-term reward rates, foragers deciding when to leave a patch must compute a decision variable that reflects both the immediately available reward and the time costs associated with travelling to the next patch. Identifying the mechanisms that mediate this computation is central to understanding how brains implement foraging decisions. We previously showed that firing rates of dorsal anterior cingulate sulcus neurons incorporate both variables. This result does not provide information about whether integration of information reflected in dorsal anterior cingulate sulcus spiking activity arises locally or whether it is inherited from upstream structures. Here, we examined local field potentials gathered simultaneously with our earlier recordings. In the majority of recording sites, local field potential spectral bands - specifically theta, beta, and gamma frequency ranges - encoded immediately available rewards but not time costs. The disjunction between information contained in spiking and local field potentials can constrain models of foraging-related processing. In particular, given the proposed link between local field potentials and inputs to a brain area, it raises the possibility that local processing within dorsal anterior cingulate sulcus serves to more fully bind immediate reward and time costs into a single decision variable.

The role of the dorsal striatum in choice impulsivity

It has long been recognized that the dorsal striatum is an essential brain region for control of action selection based on action-outcome contingency learning, particularly when the available actions are bound to rewarding outcomes. In principle, intertemporal choice in the delay-discounting task-a validated measure of choice impulsivity-involves reward-associated actions that require the recruitment of the dorsal striatum. Here, we conjecture about ways the dorsal striatum is involved in choice impulsivity. Based on a selective body of studies, we begin with a brief history of research on choice impulsivity and the dorsal striatum, and then provide a comprehensive summary of contemporary studies utilizing human neuroimaging and animal models to search for links between choice impulsivity and the dorsal striatum. In particular, we discuss in-depth the converging evidence for the associations of choice impulsivity with the reward valuation coded by the caudate, a ventral-to-dorsal gradient in the dorsal striatum, the origins of striatal afferents, and developmental maturation of frontostriatal connectivity during adolescence.

Opposing pupil responses to offered and anticipated reward values

Previous studies have shown that the pupils dilate more in anticipation of larger rewards. This finding raises the possibility of a more general association between reward amount and pupil size. We tested this idea by characterizing macaque pupil responses to offered rewards during evaluation and comparison in a binary choice task. To control attention, we made use of a design in which offers occurred in sequence. By looking at pupil responses after choice but before reward, we confirmed the previously observed positive association between pupil size and anticipated reward values. Surprisingly, however, we find that pupil size is negatively correlated with the value of offered gambles before choice, during both evaluation and comparison stages of the task. These results demonstrate a functional distinction between offered and anticipated rewards and present evidence against a narrow version of the simulation hypothesis; the idea that we represent offers by reactivating states associated with anticipating them. They also suggest that pupil size is correlated with relative, not absolute, values of offers, suggestive of an accept-reject model of comparison.

Individual ant workers show self-control

Often, the first option is not the best. Self-control can allow humans and animals to improve resource intake under such conditions. Self-control in animals is often investigated using intertemporal choice tasks-choosing a smaller reward immediately or a larger reward after a delay. However, intertemporal choice tasks may underestimate self-control, as test subjects may not fully understand the task. Vertebrates show much greater apparent self-control in more natural foraging contexts and spatial discounting tasks than in intertemporal choice tasks. However, little is still known about self-control in invertebrates. Here, we investigate self-control in the black garden ant Lasius niger We confront individual workers with a spatial discounting task, offering a high-quality reward far from the nest and a poor-quality reward closer to the nest. Most ants (69%) successfully ignored the closer, poorer reward in favour of the further, better one. However, when both the far and the close rewards were of the same quality, most ants (83%) chose the closer feeder, indicating that the ants were indeed exercising self-control, as opposed to a fixation on an already known food source.

Are Parents Patient? The Influence of Parenting Role Salience and Parental Status on Impatience

In the classic intertemporal discounting task (Thaler, 1981), individuals make tradeoff decisions between smaller-sooner and larger-later monetary rewards. We explored how parental role salience and parental status influences individual's choice between smaller-sooner and larger-later choices. Parental role salience is manipulated among both parents and non-parents in this research. Our results show a significant interaction between parental status and manipulated parental role salience. Specifically, we found that parents are more impatient than non-parents. Additionally, non-parents become more impatient after parental role salience manipulation, similar to parents. Theoretical implications of our findings are discussed.

Economic Choice as an Untangling of Options into Actions

We propose that economic choice can be understood as a gradual transformation from a domain of options to one of the actions. We draw an analogy with the idea of untangling information in the form vision system and propose that form vision and economic choice may be two aspects of a larger process that sculpts actions based on sensory inputs. From this viewpoint, choice results from the accumulated effect of repetitions of simple computations. These may consist primarily of relative valuations (evaluations relative to the value of rejection, perhaps in a manner akin to divisive normalization) applied to individual offers. With regard to economic choice, cortical brain regions differ primarily in their position and in what information they prioritize, and do not-with a few exceptions-have categorically distinct roles. Each region's specific contribution is determined largely by its inputs; thus, understanding connectivity is crucial for understanding choice. This view suggests that there is no single site of choice, that there is no meaningful distinction between pre- and post-decisionality, and that there is no explicit representation of value in the brain.

Cultural Similarities and Differences in Social Discounting: The Mediating Role of Harmony-Seeking

One's generosity to others declines as a function of social distance, which is known as social discounting. We examined cultural similarities and differences in social discounting and the mediating roles of the two aspects of interdependence (self-expression and distinctiveness of the self) as well as the two aspects of independence (harmony-seeking and rejection avoidance). Using the same procedure that previous researchers used to test North Americans, Study 1 showed that compared to North Americans, Japanese discount more steeply a partner's outcomes compared to their own future outcomes, whereas the decrease in the subjective value of the partner's outcomes accelerates less as a function of social distance. To examine the cultural similarities and differences in social discounting in more detail, Study 2 tested Japanese and Germans and found that the hyperbolic with exponent model fitted the participants' discounting behaviors better than the other models, except for the loss condition in Germans where the utility of the q-exponential model was indicated. Moreover, although the social discounting rate was higher in Japanese than in Germans, the cultural difference was limited to the gain frame. However, the decline in a person's generosity accelerated less as a function of social distance in Japanese than in Germans. Furthermore, the cultural difference in the social discounting in gains was mediated by the level of harmony-seeking, which was higher in Germans than in Japanese. Implications for individuals' generosity against the backdrop of cultural characteristics are discussed.

Future and Present Hedonistic Time Perspectives and the Propensity to Take Investment Risks: The Interplay Between Induced and Chronic Time Perspectives

Willingness to take risk is one of the most important aspects of personal financial decisions, especially those regarding investments. Recent studies show that one's perception of time, specifically the individual level of Present Hedonistic and Future Time Perspectives (TPs), influence risky financial choices. This was demonstrated for both, Time Perspective treated as an individual trait and for experimentally induced Time Perspectives. However, on occasion, people might find themselves under the joint influence of both, chronic and situational Time Perspectives and little is known about interactions between them. The paper focuses on the interplay between chronic and induced levels of Future and Present Hedonistic TPs in explaining people's propensity to take investment risks. An experimental study using a Polish national random-quota sample was conducted. The results showed that situationally induced Future TP lowered the preferred level of portfolio riskiness while situationally induced Present Hedonistic TPs resulted in exactly the opposite effect, and that the higher level of chronic Present Hedonistic TP was linked to higher investment risk preferences. The role of the chronic Present Hedonistic TP was moderated by the situationally induced Future (approaching significance) and Present Hedonistic TPs. The induction of these TPs resulted in reduction of the propensity to take investment risks. The study adds to the literature on psychological factors influencing the propensity to take financial risk. The results are also important for researchers who experimentally manipulate variables that might be also considered as chronic traits. They indicate that whether the manipulation is congruent with one's natural tendencies may have a differential impact on subsequent measures.

On the Flexibility of Basic Risk Attitudes in Monkeys

Monkeys and other animals appear to share with humans two risk attitudes predicted by prospect theory: an inverse-S-shaped probability-weighting (PW) function and a steeper utility curve for losses than for gains. These findings suggest that such preferences are stable traits with common neural substrates. We hypothesized instead that animals tailor their preferences to subtle changes in task contexts, making risk attitudes flexible. Previous studies used a limited number of outcomes, trial types, and contexts. To gain a broader perspective, we examined two large datasets of male macaques' risky choices: one from a task with real (juice) gains and another from a token task with gains and losses. In contrast to previous findings, monkeys were risk seeking for both gains and losses (i.e., lacked a reflection effect) and showed steeper gain than loss curves (loss seeking). Utility curves for gains were substantially different in the two tasks. Monkeys showed nearly linear PWs in one task and S-shaped ones in the other; neither task produced a consistent inverse-S-shaped curve. To account for these observations, we developed and tested various computational models of the processes involved in the construction of reward value. We found that adaptive differential weighting of prospective gamble outcomes could partially account for the observed differences in the utility functions across the two experiments and thus provide a plausible mechanism underlying flexible risk attitudes. Together, our results support the idea that risky choices are constructed flexibly at the time of elicitation and place important constraints on neural models of economic choice.SIGNIFICANCE STATEMENT We respond in reliable ways to risk, but are our risk preferences stable traits or ephemeral states? Using various computational models, we examined two large datasets of macaque risky choices in two different tasks. We observed several deviations from "classic" risk preferences seen in humans and monkeys: no reflection effect, loss seeking as opposed to loss aversion, and linear and S-shaped, as opposed to inverse-S-shaped, probability distortion. These results challenge the idea that our risk attitudes are evolved traits shared with the last common ancestor of macaques and humans, suggesting instead that behavioral flexibility is the hallmark of risky choice in primates. We show how this flexibility can emerge partly as a result of interactions between attentional and reward systems.

Using a Simple Neural Network to Delineate Some Principles of Distributed Economic Choice

The brain uses a mixture of distributed and modular organization to perform computations and generate appropriate actions. While the principles under which the brain might perform computations using modular systems have been more amenable to modeling, the principles by which the brain might make choices using distributed principles have not been explored. Our goal in this perspective is to delineate some of those distributed principles using a neural network method and use its results as a lens through which to reconsider some previously published neurophysiological data. To allow for direct comparison with our own data, we trained the neural network to perform binary risky choices. We find that value correlates are ubiquitous and are always accompanied by non-value information, including spatial information (i.e., no pure value signals). Evaluation, comparison, and selection were not distinct processes; indeed, value signals even in the earliest stages contributed directly, albeit weakly, to action selection. There was no place, other than at the level of action selection, at which dimensions were fully integrated. No units were specialized for specific offers; rather, all units encoded the values of both offers in an anti-correlated format, thus contributing to comparison. Individual network layers corresponded to stages in a continuous rotation from input to output space rather than to functionally distinct modules. While our network is likely to not be a direct reflection of brain processes, we propose that these principles should serve as hypotheses to be tested and evaluated for future studies.

Cognitive Science: Persistent Apes Are Intelligent Apes

In humans, self-control is correlated with general intelligence; a new study finds that this correlation extends to chimpanzees as well. The new results highlight the cognitive bases of self-control and suggest a common evolutionary history for human and primate self-control.

The effects of acute stress on the calibration of persistence

People frequently fail to wait for delayed rewards after choosing them. These preference reversals are sometimes thought to reflect self-control failure. Other times, however, continuing to wait for a delayed reward may be counterproductive (e.g., when reward timing uncertainty is high). Research has demonstrated that people can calibrate how long to wait for rewards in a given environment. Thus, the role of self-control might be to integrate information about the environment to flexibly adapt behavior, not merely to promote waiting. Here we tested effects of acute stress, which has been shown to tax control processes, on persistence, and the calibration of persistence, in young adult human participants. Half the participants (n = 60) performed a task in which persistence was optimal, and the other half (n = 60) performed a task in which it was optimal to quit waiting for reward soon after each trial began. Each participant completed the task either after cold pressor stress or no stress. Stress did not influence persistence or optimal calibration of persistence. Nevertheless, an exploratory analysis revealed an "inverted-U" relationship between cortisol increase and performance in the stress groups, suggesting that choosing the adaptive waiting policy may be facilitated with some stress and impaired with severe stress.