Suboptimal choice by pigeons: an analog of human gambling behavior

Outcome expectancy and suboptimal risky choice in nonhuman primates

Animals will favor a risky option when a stimulus signaling reward bridges the choice and the outcome. The present experiments investigated signal-induced risky choices and reward-outcome expectations in rhesus and capuchin monkeys. Risky choice was assessed by preference for a large-probabilistic reward over a modest-certain reward. Outcome expectancy was assessed by providing a truncation-response to shorten the delay period. In Experiment 1 both species generally favored the risky option compared to a safe option when the outcomes were signaled and generally shortened the delays except when a signaled-loss stimulus was presented. The use of the delay-truncation response suggested that the monkeys were sensitive to the information conveyed by the stimulus. Experiments 2 and 3 were designed to investigate whether the delay-truncation response used by capuchin monkeys was strategically used reflecting explicit decision-making versus a conditioned response to reward stimuli. A perceptual judgment task was included and the selective use of the delay-truncation response on unsignaled correct trials may suggest the involvement of metacognitive processes. The capuchin monkeys generally truncated the delays except under conditions where reward would not be expected (risky-loss or incorrect-judgment). When the outcomes were unsignaled during the delay some capuchin monkeys were less likely to truncate the delay following an incorrect task response. Overall, the monkeys: (1) made more risky choices when the outcomes were signaled - consistent with gambling-like behavior. (2) selectively truncated the unsignaled delays when rewards could be anticipated (even when metacognitive-like awareness guided anticipation) - suggesting that delay truncation responses reflect explicit outcome expectancy.

Human Choice Predicted by Obtained Reinforcers, Not by Reinforcement Predictors

Macphail (1985) proposed that “intelligence” should not vary across vertebrate species when contextual variables are accounted for. Focusing on research involving choice behavior, the propensity for choosing an option that produces stimuli that predict the presence or absence of reinforcement but that also results in less food over time can be examined. This choice preference has been found multiple times in pigeons (Stagner and Zentall, 2010; Zentall and Stagner, 2011; Laude et al., 2014) and has been likened to gambling behavior demonstrated by humans (Zentall, 2014, 2016). The present experiments used a similarly structured task to examine adult human preferences for reinforcement predictors and compared findings to choice behavior demonstrated by children (Lalli et al., 2000), monkeys (Smith et al., 2017; Smith and Beran, 2020), dogs (Jackson et al., 2020), rats (Chow et al., 2017; Cunningham and Shahan, 2019; Jackson et al., 2020), and pigeons (Roper and Zentall, 1999; Stagner and Zentall, 2010). In Experiment 1, adult human participants showed no preference for reinforcement predictors. Results from Experiment 2 suggest that not only were reinforcement predictors not preferred, but that perhaps reinforcement predictors had no effect at all on choice behavior. Results from Experiments 1 and 2 were further assessed using a generalized matching equation, the findings from which support that adult human choice behavior in the present research was largely determined by reinforcement history. Overall, the present results obtained from human adult participants are different than those found from pigeons in particular, suggesting that further examination of Macphail (1985) hypothesis is warranted.

Neural circuitry of information seeking

Humans and animals navigate uncertain environments by seeking information about the future. Remarkably, we often seek information even when it has no instrumental value for aiding our decisions - as if the information is a source of value in its own right. In recent years, there has been a flourishing of research into these non-instrumental information preferences and their implementation in the brain. Individuals value information about uncertain future rewards, and do so for multiple reasons, including valuing resolution of uncertainty and overweighting desirable information. The brain motivates this information seeking by tapping into some of the same circuitry as primary rewards like food and water. However, it also employs cortex and basal ganglia circuitry that predicts and values information as distinct from primary reward. Uncovering how these circuits cooperate will be fundamental to understanding information seeking and motivated behavior as a whole, in our increasingly complex and information-rich world.

Contribution of alcohol- and cigarette-related cues to concurrent reinforcer choice in humans

Drug-related cues have been extensively studied for their contributions to decision-making processes involving drug consumption. Less studied is how drug-related stimuli may impact decisions involving outcomes not directly related to substance use. Studies using concurrent choice tasks have demonstrated that cocaine and cannabis cues result in robust and selective biases for monetary reinforcers contiguous with those cues. The purpose of this study was to provide a conceptual replication of these findings with alcohol and cigarettes. Participants recruited using crowdsourcing completed a cued concurrent choice task involving presentation of two cues (one drug and one neutral) followed by concurrent monetary offers below each image. Alcohol (Experiment 1; N = 103) and cigarette (Experiment 2; N = 256) visual cues were evaluated. Participants with hazardous alcohol use and current cigarette use showed greater-than-indifference selection for the alcohol- (p = .004) and cigarette-cued (p = .02) monetary choices, respectively. Qualitative responses indicated that the most popular rationale for responding was "image quality" despite images having no explicit impact on the reinforcer received. Low alcohol use (p = .03) and non-cigarette using controls (p < .001) showed an avoidance bias with lower-than-indifference selection for drug-cued choices. These findings replicate and extend observations that spatially contiguous drug-related cues can bias decisions involving concurrently presented non-drug reinforcers.

Suboptimal choice by pigeons is eliminated when key-pecking behavior is replaced by treadle-pressing

In the study of suboptimal choice, a reliable result is that pigeons strongly prefer an alternative that signals whether a reinforcer will be delivered or not over another alternative without that information even if the first provides a lower probability of reinforcement. In the aforementioned research, key pecking has been the operant response and illuminated keys the discriminative stimuli. In the present study we modified both of these aspects of the procedure in order to analyze the generality of suboptimal preferences of pigeons and to investigate the effect of changes in the incentive salience of the discriminative stimuli. To accomplish this, we presented pigeons a choice situation with the same parameters of reinforcement than previous research, but with treadle pressing as the choice response and ambient lights as discriminative stimuli. Under these conditions, most of the pigeons showed optimal behavior and a high degree of discrimination of the stimuli associated with the discriminative alternative. A control condition with key pecking as choice response and keylights as discriminative stimuli showed that the same pigeons turned to be suboptimal, a result that discards the possibility that the optimality found in the main condition was a consequence of a particular characteristic of our sample of subjects or of our procedure. We discuss the influence that the attribution of incentive salience to the discriminative stimuli has on suboptimal choice in both pigeons and rats.

Human and pigeon suboptimal choice

Many studies have shown that pigeons will sometimes behave suboptimally by choosing an option that provides food less frequently over one that provides food more frequently. The critical factor in driving suboptimal behavior in these procedures is that the delayed outcomes are differentially signaled on the suboptimal alternative, but not the optimal alternative. Although this procedure is frequently cited as potentially analogous to human gambling, there is little empirical data to evaluate this assertion. The present study tested both pigeon (Experiment 1) and human (Experiment 2) subjects with a suboptimal choice task. Subjects chose between a suboptimal alternative that provided a large reinforcer 20% of the time and an optimal alternative that always provided a small reinforcer. Stimuli presented during the delays signaled the outcomes on the suboptimal alternative in some conditions. When outcomes were signaled, pigeons chose the suboptimal alternative more frequently than did humans. When the outcomes were not signaled, pigeons' choices became more optimal, but humans' choices did not. Humans' suboptimal choice was unrelated to performance on a probability discounting task. Overall, these findings suggest that although both pigeons and humans can choose suboptimally, more research is needed in order to determine whether non-human performance on this task can serve as a model for human gambling.

Contribution of cannabis-related cues to concurrent reinforcer choice in humans

BACKGROUND

Drug-related cues play a critical role in the development and persistence of substance use disorder. Few human laboratory studies have evaluated how these cues contribute to decisions between concurrently presented reinforcers, and none have examined the specific role of cannabis cues. This study evaluated the contribution of cannabis-related cues to concurrent monetary reinforcer choice in humans.

METHODS

Participants with a cannabis use history (i.e., use in the past two weeks and 50 or more lifetime uses; n = 71) and controls without this history (i.e., 5 or less lifetime uses; n = 79) were recruited using Amazon Mechanical Turk. A cued concurrent choice task was used in which cannabis trials presented two cues (one cannabis and one neutral) side-by-side followed by concurrent monetary offers below each image. The primary dependent measure was choice for cannabis-cued monetary reinforcers on equal value trials. Secondary analyses evaluated individual difference variables related to choice bias.

RESULTS

Participants in the cannabis group showed a significant bias for cannabis-cued choices (mean 76.0%) whereas participants in the control group showed a significant bias against cannabis-cued choices (mean 30.3%). Reaction times on cannabis trials were faster than neutral filler trials and did not differ by group. Cannabis-cued choice was significantly associated with more frequent cannabis use (r = .44), higher cannabis demand intensity (r = .28), and lower cannabis elasticity (r = -.30).

CONCLUSIONS

These findings suggest that cannabis-related cues can influence reinforcer choice and potentially promote disadvantageous decision-making related to non-drug reinforcers.

Contribution of cocaine-related cues to concurrent monetary choice in humans

RATIONALE

Theoretical accounts highlight the importance of drug-related cues for the development and persistence of drug-taking behavior. Few studies have evaluated the ability of spatially contiguous drug cues to bias decisions between two concurrently presented non-drug reinforcers.

OBJECTIVE

Evaluate the contribution of spatially contiguous cocaine cues to choice between two concurrently presented monetary reinforcers METHODS: Participants with cocaine use disorder completed a cued concurrent choice task. Two cues (one cocaine and one control image) were presented side-by-side followed by concurrent monetary offers below each image. Concurrent choice was measured for cocaine-side advantageous, equal, and disadvantageous concurrent monetary offers. The primary dependent measure was bias for selecting cocaine-cued monetary reinforcers. Three experiments tested selectivity of cocaine-cued bias in individuals with a cocaine use history (Experiment 1), replication when including additional control trials (Experiment 2), and a potential attentional mechanism evaluated using eye-tracking technology (Experiment 3).

RESULTS

Significant and robust cocaine-cued bias at equal monetary value was observed in three experiments (mean percent choice = 65-77%) and higher Drug Abuse Screening Test (DAST) scores were associated with greater cocaine-choice bias. These experiments demonstrated that cocaine-cued bias was (1) selective to individuals with a cocaine use history, (2) specific to trials involving a cocaine cue, and (3) partially associated with attentional bias.

CONCLUSIONS

These experiments provide evidence that drug-related cues can influence choice and potentially promote maladaptive decision making during concurrent choice events. Future research evaluating prospective associations of drug-cued bias with drug-associated behaviors will help reveal the clinical relevance for substance use disorder.

Does a negative discriminative stimulus function as a punishing consequence?

The study and use of punishment in behavioral treatments has been constrained by ethical concerns. However, there remains a need to reduce harmful behavior unable to be reduced by differential-reinforcement procedures. We investigated whether response-contingent presentation of a negative discriminative stimulus previously correlated with an absence of reinforcers would punish behavior maintained by positive reinforcers. Across four conditions, pigeons were trained to discriminate between a positive discriminative stimulus (S+) signaling the presence of food, and a negative discriminative stimulus (S-) signaling the absence of food. Once learned, every five responses on average to the S+ produced S- for a duration of 1.5 s. S+ response rate decreased for a majority of pigeons when responses produced S-, compared to when they did not, or when a neutral control stimulus was presented. In Condition 5, choice between two concurrently presented S+ alternatives shifted away from the alternative producing S-, despite a 1:1 reinforcer ratio. Therefore, presenting contingent S- stimuli punishes operant behavior maintained on simple schedules and in choice situations. Development of negative discriminative stimuli as punishers of operant behavior could provide an effective approach to behavioral treatments for problem behavior and subverting suboptimal choices involved in addictions.

Reinforcement learning models of risky choice and the promotion of risk-taking by losses disguised as wins in rats

Risky decisions are inherently characterized by the potential to receive gains or incur losses, and these outcomes have distinct effects on subsequent decision-making. One important factor is that individuals engage in loss-chasing, in which the reception of a loss is followed by relatively increased risk-taking. Unfortunately, the mechanisms of loss-chasing are poorly understood, despite the potential importance for understanding pathological choice behavior. The goal of the present experiment was to illuminate the mechanisms governing individual differences in loss-chasing and risky-choice behaviors. Rats chose between a low-uncertainty outcome that always delivered a variable amount of reward and a high-uncertainty outcome that probabilistically delivered reward. Loss-processing and loss-chasing were assessed in the context of losses disguised as wins (LDWs), which are loss outcomes that are presented along with gain-related stimuli. LDWs have been suggested to interfere with adaptive decision-making in humans and thus potentially increase loss-making. Here, the rats presented with LDWs were riskier, in that they made more choices for the high-uncertainty outcome. A series of nonlinear models were fit to individual rats' data to elucidate the possible psychological mechanisms that best account for individual differences in high-uncertainty choices and loss-chasing behaviors. The models suggested that the rats presented with LDWs were more prone to showing a stay bias following high-uncertainty outcomes compared to rats not presented with LDWs. These results collectively suggest that LDWs acquire conditioned reinforcement properties that encourage continued risk-taking and increase loss-chasing following previous high-risk decisions. (PsycINFO Database Record

Pigeons play the percentages: computation of probability in a bird

The ability to compute probability, previously shown in nonverbal infants, apes, and monkeys, was examined in three experiments with pigeons. After responding to individually presented keys in an operant chamber that delivered reinforcement with varying probabilities, pigeons chose between these keys on probe trials. Pigeons strongly preferred a 75% reinforced key over a 25% reinforced key, even when the total number of reinforcers obtained on each key was equated. When both keys delivered 50% reinforcement, pigeons showed indifference between them, even though three times more reinforcers were obtained on one key than on the other. It is suggested that computation of probability may be common to many classes of animals and may be driven by the need to forage successfully for nutritional food items, mates, and areas with a low density of predators.

Melioration revisited: a systematic replication of Vaughan (1981)

Organisms that behave so as to forfeit a relatively higher overall rate of reinforcement in favor of a relatively lower rate are said to engage in suboptimal choice. Suboptimal choice has been linked with maladaptive behavior in humans. Melioration theory offers one explanatory framework for suboptimal choice. Melioration theory suggests behavior is controlled by differences in local reinforcer rates between alternatives. Vaughan (1981) arranged two experimental conditions in which maximizing the overall rate of reinforcement required behavior that was compatible, or incompatible, with melioration. Vaughan found pigeons allocated more time to a locally richer alternative even when doing so resulted in suboptimal choice. However, Vaughan did not show whether these effects could systematically reverse and did not provide within-session data to show that choice across short time spans remains under the control of differences in local reinforcer rates. The present study used pigeons to replicate and extend Vaughan's findings. We investigated shifts in overall- and within-session choice across repeated conditions, according to arranged local contingencies. Behavior systematically followed changes in local contingencies for most pigeons. Within-session data suggests that, providing differences in local reinforcer rates are discriminated, pigeons will allocate more time to a locally richer alternative, even if this leads to suboptimal choice. These findings facilitate the more confident use of similar procedures that investigate how melioration contributes to suboptimal choice.

Ultimate explanations and suboptimal choice

Researchers have unraveled multiple cases in which behavior deviates from rationality principles. We propose that such deviations are valuable tools to understand the adaptive significance of the underpinning mechanisms. To illustrate, we discuss in detail an experimental protocol in which animals systematically incur substantial foraging losses by preferring a lean but informative option over a rich but non-informative one. To understand how adaptive mechanisms may fail to maximize food intake, we review a model inspired by optimal foraging principles that reconciles sub-optimal choice with the view that current behavioral mechanisms were pruned by the optimizing action of natural selection. To move beyond retrospective speculation, we then review critical tests of the model, regarding both its assumptions and its (sometimes counterintuitive) predictions, all of which have been upheld. The overall contention is that (a) known mechanisms can be used to develop better ultimate accounts and that (b) to understand why mechanisms that generate suboptimal behavior evolved, we need to consider their adaptive value in the animal's characteristic ecology.

Incentive salience attribution is not the sole determinant of suboptimal choice in rats: Conditioned inhibition matters

Previous research has identified clear differences between pigeons and rats in the suboptimal choice procedure. Pigeons behave suboptimally, preferring an alternative with discriminative stimuli and a smaller probability of reinforcement, over another with a higher probability of reinforcement, but without discriminative stimuli. In contrast, rats behave optimally showing the opposite preference. It has been proposed that these dissimilarities are consequence of a higher sensitivity to conditioned inhibition in rats than in pigeons. Alternatively, recent research suggests that differences in optimality can be accounted for by a differential incentive salience of the stimuli employed as discriminative stimuli, and that both species are suboptimal when such stimuli have high incentive salience; specifically, rats were found to be suboptimal when levers were used as discriminative stimuli. However, in the evaluation of this hypothesis, a conditioned inhibitor was not employed. In the present report, eight rats were exposed to a choice procedure that integrated both variables discussed above: a conditioned inhibitor was associated with the discriminative alternative and the stimuli had high incentive salience. A clear preference for the optimal alternative was found, suggesting that the conditioned inhibitor had a considerable impact on rats' preference, and that species-differences remain even in procedures in which the discriminative stimuli have incentive salience.

Suboptimal choice in rats: Incentive salience attribution promotes maladaptive decision-making

Stimuli that are more predictive of subsequent reward also function as better conditioned reinforcers. Moreover, stimuli attributed with incentive salience function as more robust conditioned reinforcers. Some theories have suggested that conditioned reinforcement plays an important role in promoting suboptimal choice behavior, like gambling. The present experiments examined how different stimuli, those attributed with incentive salience versus those without, can function in tandem with stimulus-reward predictive utility to promote maladaptive decision-making in rats. One group of rats had lights associated with goal-tracking as the reward-predictive stimuli and another had levers associated with sign-tracking as the reward-predictive stimuli. All rats were first trained on a choice procedure in which the expected value across both alternatives was equivalent but differed in their stimulus-reward predictive utility. Next, the expected value across both alternatives was systematically changed so that the alternative with greater stimulus-reward predictive utility was suboptimal in regard to primary reinforcement. The results demonstrate that in order to obtain suboptimal choice behavior, incentive salience alongside strong stimulus-reward predictive utility may be necessary; thus, maladaptive decision-making can be driven more by the value attributed to stimuli imbued with incentive salience that reliably predict a reward rather than the reward itself.

Optimal behavior by rats in a choice task is associated to a persistent conditioned inhibition effect

When given a choice between an alternative with a low probability of reinforcement and discriminative stimuli, and another with a higher probability of reinforcement and non-discriminative stimuli, pigeons show a clear preference for the former but rats clearly prefer the later. It has been reported that pigeon's suboptimal choice is associated to a diminishing effect of the stimulus correlated with non-reinforcement. In the present paper, we explored the possibility that rats' optimal choice is more strongly influenced than pigeons' by the stimulus associated to non-reinforcement and that the effects of it do not dissipate during training. We trained rats to choose between an alternative with 0.50 probability of reinforcement and discriminative stimuli, and an alternative with 0.75 probability of reinforcement and non-discriminative stimuli. We replicated the strong preference for the optimal alternative. Then, after several sessions of training, we presented summation trials in which both the stimulus associated to reinforcement and the stimulus associated to non-reinforcement were simultaneously presented. The results showed that the stimulus associated to non-reinforcement exerted a strong effect on choice, and, more importantly, that it did not seem to dissipate across training. These results suggest that the strong difference found between pigeons and rats in the suboptimal choice procedure is potentially related to differences in the impact of conditioned inhibitors.

Rats are optimal in a choice task in which pigeons are not

In an extensive list of studies, it has been found that pigeons prefer an alternative associated with discriminative stimuli over another associated with non-discriminative stimuli, even when the probability of reinforcement is higher in the latter. This behavior has been named "suboptimal choice". In the present experiment, we evaluated whether rats, another widely studied species within the Experimental Analysis of Behavior, also shows this behavior. We systematically replicated the procedure employed with pigeons, and found that rats are not suboptimal, i.e., they prefer the non-discriminative alternative associated with .5 probability of reinforcement, over the discriminative alternative associated with .2 probability of reinforcement. This effect occurred even though rats discriminated the contingencies of reinforcement associated with each stimulus, suggesting that rats' optimal choice was driven by the overall probability of reinforcement of each alternative. Different procedural details are offered as possibilities for explaining this apparent inter-species difference.

Translational research into intertemporal choice: the Western scrub-jay as an animal model for future-thinking

Decisions often involve outcomes that will not materialise until later, and choices between immediate gratification and future consequences are thought to be important for human health and welfare. Combined human and animal research has identified impulsive intertemporal choice as an important factor in drug-taking and pathological gambling. In this paper, we give an overview of recent research into intertemporal choice in non-human animals, and argue that this work could offer insight into human behaviour through the development of animal models. As an example, we discuss the role of future-thinking in intertemporal choice, and review the case for the Western scrub-jay (Aphelocoma californica) as an animal model of such prospective cognition. This article is part of a Special Issue entitled: Tribute to Tom Zentall.