Reduced access to reinforcement drives delay discounting during experienced delays

Rewards lose value as a function of delay. Previous studies suggest that delays have a bigger effect on reward value when people must wait during the delay. However, whether delays involve waiting or postponing has often been confounded with whether choices are about hypothetical or real rewards. The current study characterized the effects of waiting and postponing in hypothetical and experiential choice contexts separately. In Experiment 1 we observed steeper delay discounting for waiting than for postponing in choices about both hypothetical money and about experienced computer game points. Two factors potentially contributing to steeper discounting in choices about waiting are reduced access to other rewards and direct costs of waiting. In Experiment 2, we adapted the experiential delay-discounting task to manipulate each factor separately. Reduced access to other reinforcers had a bigger effect on delay discounting than direct costs of waiting. These results underscore the importance of considering the unique influence of waiting and associated opportunity costs in both basic delay-discounting research and in applied contexts.

A new paradigm for addictions

AIM

To suggest a new paradigm for addictions.

METHODS

Consideration of relevant research findings and thought experiments.

RESULTS

Common mental motors leading to addictions are pleasure-seeking and hyperbolic discounting. The important point of the latter is that given two choices of future rewards, commonly one initially prefers the larger one available after a longer waiting time but despite this the smaller and sooner reward will be chosen when it becomes available. These are general biological properties, found at least in human beings, the rat, and the pigeon. If this continues it may create an unconscious habit, difficult to change. Several other risk factors for addictions are known, notably both externalizing and internalizing mental problems. Predisposing factors are likely to interact.

CONCLUSIONS

The above suggests a new paradigm for addictions. Pleasure provides temptations, hyperbolic discounting weakens the will. Habits emerge. Addictions seem to be a group of problems of its own kind, not diseases, because diseases do not bring about pleasure, and are not sought for pleasure.

Cultural differences in intertemporal decision making: A comparison between Chile and China

A cross-cultural comparison is made of delay discounting in samples of participants from Chile and China. Comparisons are made based on previous literature that suggests that individuals from an Asian culture should be willing to postpone delayed rewards more than are individuals from a Latin American culture. To test the cross-cultural validity of a hyperbolic discounting model, the model was fitted to both data sets. Additionally, a self-enhancement measure was evaluated as a potential mediator between culture of origin and delay discounting. Seventy-eight college students from China and 120 college students from Chile, with similar demographic backgrounds, discounted hypothetical monetary outcomes using an adjusting-amount titration procedure. Additionally, participants completed a self-enhancement measure. Age, academic major, gender, and grade point average were controlled. Chilean participants discounted much more steeply than Chinese nationals did. No support was obtained for the mediation of self-enhancement between culture of origin and degree of delay discounting. In both samples, delay discounting was better described by a hyperboloid than an exponential function, the only exception being the $10,000 condition in which the medians for Chilean participants' present subjective value were equally well explained by a hyperboloid and an exponential function.

Impulsivity and Alcohol-Drinking Behavior: Evidence from Japan

Despite the attempt by the Japanese government to reduce alcohol consumption, reduction of alcohol consumption requires improvement. We explore this issue from the impulsivity perspective and investigate whether a causal relationship exists between impulsivity and drinking behavior. We used data from the Preference Parameter Study of Osaka University to capture respondents' drinking status. Our probit regression showed that procrastination, a proxy measure of impulsivity, was significantly associated with drinking behavior, while hyperbolic discounting, a direct measure of impulsivity, was insignificant. Our findings suggest that impulsive people will discount their health in the future; thus, the government should consider impulsivity in policymaking. For example, awareness programs should focus more on future healthcare costs from alcohol-related problems so that impulsive drinkers can understand how much they may need to spend in the future compared to current satisfaction with alcohol drinking.

Hyperbolic discounting and exponentiated demand: Modeling demand for cigarettes in three dimensions

Behavioral economics has been a fruitful area of research in substance use. Mathematical descriptions of how individuals temporally discount the value of a commodity have been correlated with substance use and mathematical descriptions of drug consumption decreasing as a function of price (i.e., demand) predict maladaptive substance use. While there is a logical assumption that temporal factors affect demand for a drug, little has been done to merge these models. Thus, the purpose of this study was to combine models of discounting and demand, extending Howard Rachlin's work and contributions to novel areas of study. Data from 85 participants from Amazon Mechanical Turk (MTurk) who completed a hypothetical cigarette purchase task that included price of and delay to cigarettes were analyzed. Multilevel modeling was used to determine descriptive accuracy of combined additive and multiplicative models of discounting and demand. Of the discounting models used in conjunction with the exponentiated demand equation, the Rachlin hyperboloid best described the delay dimension of consumption. The multiplicative version of the Rachlin equation applied to both delay and price outperformed other models tested. Therefore, existing models of discounting and demand can be extended to modeling consumption data from complex multidimensional experimental arrangements.

Hierarchies improve individual assessment of temporal discounting behavior

Delay discounting behavior has proven useful in assessing impulsivity across a wide range of populations. As such, accurate estimation of the shape of each individual's temporal discounting profile is paramount when drawing conclusions about how impulsivity relates to clinical and health outcomes such as gambling, addiction, and obesity. Here, we identify an estimation problem with current methods of assessing temporal discounting behavior, and propose a simple solution. First, through a simulation study we identify types of temporal discounting profiles that cannot reliably be estimated. Second, we show how imposing constraints through hierarchical modeling ameliorates these recovery problems. Finally, we apply our solution to a large data set from a temporal discounting task, and illustrate the importance of reliable estimation within patient populations. We conclude with a brief discussion on how hierarchical Bayesian methods can aid in model estimation, compensate for small samples, and improve predictions of externalizing psychopathology.

Subjective value representations during effort, probability and time discounting across adulthood

Every day, humans make countless decisions that require the integration of information about potential benefits (i.e. rewards) with other decision features (i.e. effort required, probability of an outcome or time delays). Here, we examine the overlap and dissociation of behavioral preferences and neural representations of subjective value in the context of three different decision features (physical effort, probability and time delays) in a healthy adult life span sample. While undergoing functional neuroimaging, participants (N = 75) made incentive compatible choices between a smaller monetary reward with lower physical effort, higher probability, or a shorter time delay versus a larger monetary reward with higher physical effort, lower probability, or a longer time delay. Behavioral preferences were estimated from observed choices, and subjective values were computed using individual hyperbolic discount functions. We found that discount rates were uncorrelated across tasks. Despite this apparent behavioral dissociation between preferences, we found overlapping subjective value-related activity in the medial prefrontal cortex across all three tasks. We found no consistent evidence for age differences in either preferences or the neural representations of subjective value across adulthood. These results suggest that while the tolerance of decision features is behaviorally dissociable, subjective value signals share a common representation across adulthood.

Observed and Normative Discount Functions in Addiction and other Diseases

The aim of this paper is to find a suitable discount function able to describe the progression of a certain addiction or disease under treatment as a discounting process. In effect, a certain indicator related to a disease decays over time in a manner which is mathematically similar to the way in which discounting has been modeled. We analyze the discount functions observed in experiments which study addictive and other problematic behaviors as well as some alternative hyperbola-like discount functions in order to fit the patience exhibited by the subject after receiving the treatment. Additionally, it has been experimentally found that people with addiction display high rates of discount (impatience) and preference reversals (dynamic inconsistency). This excessive discounting must be correctly modeled by a suitable discount function, otherwise, it can become a trans-disease process underlying addiction and other disorders. The (generalized) exponentiated hyperbolic discount function is proposed to describe the progression of a disease with respect to the treatment, since it maintains the property of inconsistency by exhibiting a decreasing discount rate after an initial period in which the opposite occurs.

A generalization of Fatou's lemma for extended real-valued functions on σ-finite measure spaces: with an application to infinite-horizon optimization in discrete time

Given a sequence [Formula: see text] of measurable functions on a σ-finite measure space such that the integral of each [Formula: see text] as well as that of [Formula: see text] exists in [Formula: see text], we provide a sufficient condition for the following inequality to hold: [Formula: see text] Our condition is considerably weaker than sufficient conditions known in the literature such as uniform integrability (in the case of a finite measure) and equi-integrability. As an application, we obtain a new result on the existence of an optimal path for deterministic infinite-horizon optimization problems in discrete time.

Probability Weighting Functions Derived from Hyperbolic Time Discounting: Psychophysical Models and Their Individual Level Testing

A probability weighting function (w(p)) is considered to be a nonlinear function of probability (p) in behavioral decision theory. This study proposes a psychophysical model of probability weighting functions derived from a hyperbolic time discounting model and a geometric distribution. The aim of the study is to show probability weighting functions from the point of view of waiting time for a decision maker. Since the expected value of a geometrically distributed random variable X is 1/p, we formulized the probability weighting function of the expected value model for hyperbolic time discounting as w(p) = (1 − k log p)⁻¹. Moreover, the probability weighting function is derived from Loewenstein and Prelec's (1992) generalized hyperbolic time discounting model. The latter model is proved to be equivalent to the hyperbolic-logarithmic weighting function considered by Prelec (1998) and Luce (2001). In this study, we derive a model from the generalized hyperbolic time discounting model assuming Fechner's (1860) psychophysical law of time and a geometric distribution of trials. In addition, we develop median models of hyperbolic time discounting and generalized hyperbolic time discounting. To illustrate the fitness of each model, a psychological experiment was conducted to assess the probability weighting and value functions at the level of the individual participant. The participants were 50 university students. The results of individual analysis indicated that the expected value model of generalized hyperbolic discounting fitted better than previous probability weighting decision-making models. The theoretical implications of this finding are discussed.

Behavioral Economics of Self-Control Failure

The main idea in this article is that addiction is a consequence of falling victim to decision failures that lead to preference for the addictive behaviors. Addiction is viewed as valuation disease, where the nervous system overvalues cues associated with drugs or drug-taking. Thus, addiction can be viewed as a diminished capacity to choose. Addicted individuals assign lower values to delayed rewards than to immediate ones. The preference for immediate gratification leads to self-control problems. This article highlights a number of motivational forces that can generate self-control failure.

Time discounting and smoking behavior: evidence from a panel survey(*)

By using a panel survey of Japanese adults, we show that smoking behavior is associated with personal time discounting and its biases, such as hyperbolic discounting and the sign effect, in the way that theory predicts: smoking depends positively on the discount rate and the degree of hyperbolic discounting and negatively on the presence of the sign effect. Positive effects of hyperbolic discounting on smoking are salient for naïve people, who are not aware of their self-control problem. By estimating smoking participation and smokers' cigarette consumption in Cragg's two-part model, we find that the two smoking decisions depend on different sets of time-discounting variables. Particularly, smoking participation is affected by being a naïve hyperbolic discounter, whereas the discount rate, the presence of the sign effect, and a hyperbolic discounting proxy constructed from procrastination behavior vis-à-vis doing homework assignments affect both types of decision making. The panel data enable us to analyze the over-time instability of elicited discount rates. The instability is shown to come from measurement errors, rather than preference shocks on time preference. Several evidences indicate that the detected associations between time preferences and smoking behavior are interpersonal one, rather than within-personal one.

Rats' choices with token stimuli in concurrent variable-interval schedules

Four rats responded on concurrent variable-interval schedules that delivered token stimuli (stimulus lights arranged vertically above each of two side levers). During exchange periods, each token could be exchanged for one food pellet by responding on a center lever, with one response required for each pellet delivery. In different conditions, the exchange requirements (number of tokens that had to be earned before they could be exchanged for food) varied between one and four for the two response levers. The experiments were closely patterned after research with pigeons by Mazur and Biondi (2013), and the results from the rats in the present experiment were similar. Response percentages on the two levers changed as each additional token was earned, and these patterns indicated that choice was controlled by both the time to the exchange periods and the number of food pellets that were delivered in the exchange period. In some conditions, the exchange requirement was three tokens for each lever, but the token lights were not turned on as they were earned for one of the two keys. The rats showed a slight preference for the lever without the token lights, which may indicate that the token lights were not serving as conditioned reinforcers (a result also found by Mazur and Biondi with pigeons). Overall, these results suggest that, in this choice procedure, the token stimuli served primarily as discriminative stimuli that signaled the temporal proximity and quantity of the primary reinforcer, food.

Discounting of reward sequences: a test of competing formal models of hyperbolic discounting

Humans are known to discount future rewards hyperbolically in time. Nevertheless, a formal recursive model of hyperbolic discounting has been elusive until recently, with the introduction of the hyperbolically discounted temporal difference (HDTD) model. Prior to that, models of learning (especially reinforcement learning) have relied on exponential discounting, which generally provides poorer fits to behavioral data. Recently, it has been shown that hyperbolic discounting can also be approximated by a summed distribution of exponentially discounted values, instantiated in the μAgents model. The HDTD model and the μAgents model differ in one key respect, namely how they treat sequences of rewards. The μAgents model is a particular implementation of a Parallel discounting model, which values sequences based on the summed value of the individual rewards whereas the HDTD model contains a non-linear interaction. To discriminate among these models, we observed how subjects discounted a sequence of three rewards, and then we tested how well each candidate model fit the subject data. The results show that the Parallel model generally provides a better fit to the human data.

Pigeons' choices with token stimuli in concurrent variable-interval schedules

Twelve pigeons responded on concurrent variable-interval schedules that delivered token stimuli (stimulus lights for some pigeons, and white circles on the response keys for others). During exchange periods, each token could be exchanged for food on a fixed-ratio 1 schedule. Across conditions, the exchange requirements (number of tokens that had to be earned before they could be exchanged for food) varied between one and four for the two response keys. The main findings were that the pigeons' response percentages varied as a function of the number of tokens earned at any given moment, and they were determined by both the delays to food and by the number of food deliveries in the exchange periods. In some conditions, tokens had to be earned but were not visible during the variable-interval schedules for one or both keys. When one key had visible tokens and the other did not, the pigeons showed a preference for the key without visible tokens. A model based on the matching law and a hyperbolic delay-discounting equation could account for the main patterns of choice responding, and for how response percentages changed as successive tokens were earned. The results are consistent with the view that the token stimuli served as discriminative stimuli that signaled the current delays to food.

Positive temporal dependence of the biological clock implies hyperbolic discounting

Temporal preferences of animals and humans often exhibit inconsistencies, whereby an earlier, smaller reward may be preferred when it occurs immediately but not when it is delayed. Such choices reflect hyperbolic discounting of future rewards, rather than the exponential discounting required for temporal consistency. Simultaneously, however, evidence has emerged that suggests that animals and humans have an internal representation of time that often differs from the calendar time used in detection of temporal inconsistencies. Here, we prove that temporal inconsistencies emerge if fixed durations in calendar time are experienced as positively related (positive quadrant dependent). Hence, what are time-consistent choices within the time framework of the decision maker appear as time-inconsistent to an outsider who analyzes choices in calendar time. As the biological clock becomes more variable, the fit of the hyperbolic discounting model improves. A recent alternative explanation for temporal choice inconsistencies builds on persistent under-estimation of the length of distant time intervals. By increasing the expected speed of our stochastic biological clock for time farther into the future, we can emulate this explanation. Ours is therefore an encompassing theoretical framework that predicts context-dependent degrees of intertemporal choice inconsistencies, to the extent that context can generate changes in autocorrelation, variability, and expected speed of the biological clock. Our finding should lead to novel experiments that will clarify the role of time perception in impulsivity, with critical implications for, among others, our understanding of aging, drug abuse, and pathological gambling.

A reinforcement learning model of precommitment in decision making

Addiction and many other disorders are linked to impulsivity, where a suboptimal choice is preferred when it is immediately available. One solution to impulsivity is precommitment: constraining one's future to avoid being offered a suboptimal choice. A form of impulsivity can be measured experimentally by offering a choice between a smaller reward delivered sooner and a larger reward delivered later. Impulsive subjects are more likely to select the smaller-sooner choice; however, when offered an option to precommit, even impulsive subjects can precommit to the larger-later choice. To precommit or not is a decision between two conditions: (A) the original choice (smaller-sooner vs. larger-later), and (B) a new condition with only larger-later available. It has been observed that precommitment appears as a consequence of the preference reversal inherent in non-exponential delay-discounting. Here we show that most models of hyperbolic discounting cannot precommit, but a distributed model of hyperbolic discounting does precommit. Using this model, we find (1) faster discounters may be more or less likely than slow discounters to precommit, depending on the precommitment delay, (2) for a constant smaller-sooner vs. larger-later preference, a higher ratio of larger reward to smaller reward increases the probability of precommitment, and (3) precommitment is highly sensitive to the shape of the discount curve. These predictions imply that manipulations that alter the discount curve, such as diet or context, may qualitatively affect precommitment.

Delay-amount tradeoffs in choices by pigeons and rats: hyperbolic versus exponential discounting

An adjusting-delay procedure was used to study the choices of pigeons and rats when both delay and amount of reinforcement were varied. In different conditions, the choice alternatives included one versus two reinforcers, one versus three reinforcers, and three versus two reinforcers. The delay to one alternative (the standard alternative) was kept constant in a condition, and the delay to the other (the adjusting alternative) was increased or decreased many times a session so as to estimate an indifference point--a delay at which the two alternatives were chosen about equally often. Indifference functions were constructed by plotting the adjusting delay as a function of the standard delay for each pair of reinforcer amounts. The experiments were designed to test the prediction of a hyperbolic decay equation that the slopes of the indifference functions should increase as the ratio of the two reinforcer amounts increased. Consistent with the hyperbolic equation, the slopes of the indifference functions depended on the ratios of the two reinforcer amounts for both pigeons and rats. These results were not compatible with an exponential decay equation, which predicts slopes of 1 regardless of the reinforcer amounts. Combined with other data, these findings provide further evidence that delay discounting is well described by a hyperbolic equation for both species, but not by an exponential equation. Quantitative differences in the y-intercepts of the indifference functions from the two species suggested that the rate at which reinforcer strength decreases with increasing delay may be four or five times slower for rats than for pigeons.

An algorithm for identifying nonsystematic delay-discounting data

Several [corrected] discounting studies have use the R2 measure to identify data [corrected] with poor fits to a mathematical discounting model as nonsystematic data to be eliminated [corrected] Data from three previous delay-discounting studies (six separate groups, with a total of 161 individuals) were used to demonstrate why using R2 to assess the fits of discounting data is problematic. A significant, positive correlation between discounting rate parameter and R2 was found in most groups, showing that R2 is more stringent as a measure of fit for low discounting rates than for high discounting rates. Furthermore, it is suggested that identifying nonsystematic data based on any measure of fit to a mathematical discounting model may be problematic because it confounds discounting rate comparison with the issue of discounting model assessment. Therefore, a model-free method to identify nonsystematic data is needed. An algorithm for identifying nonsystematic data is presented that is based on the expectation of a monotonically decreasing discounting function. This algorithm identified 13 cases out of the 161 reanalyzed data sets as nonsystematic. These nonsystematic data are presented, along with examples of data not identified as nonsystematic. This algorithm, or modifications of it, may be useful in a variety of human and nonhuman animal discounting studies (e.g., delay discounting, probability discounting) as an alternative to the R2 measure for identifying nonsystematic data. The algorithm may be used in empirical investigations to improve discounting methodology, and may be used to identify outliers to be removed from analyses.