Suboptimal Choice by Pigeons: Evidence that the Value of the Conditioned Reinforcer Rather than its Frequency Determines Choice

Time, uncertainty, and suboptimal choice

Under certain conditions, pigeons prefer information about whether food will be forthcoming at the end of an interval to a higher chance of obtaining the food. In the typical protocol, choosing one option (Informative) is followed by one of two 10-s long terminal-link stimuli: SG always ending in food or SR never ending in food, with SG occurring only 20% of the trials. The other option (Non-informative) is also followed by one of two 10-s long terminal-link stimuli: SB or SY, both ending in food 50% of the trials. Although the Informative option yields food with a lower probability than the Non-informative (0.2 vs. 0.5), pigeons prefer it. To determine whether such preference occurs because SG and SR disambiguate the trial outcome immediately upon choice, we delayed the moment the disambiguation took place in two experiments. In Experiment 1, when the Informative option was chosen, SG always ensued for t seconds of the terminal-link, and then the standard contingencies followed. Experiment 2 was similar, except that SR always ensued for t seconds. Across conditions, t varied from 0 to 10 s. In both experiments, preference for the Informative option decreased with t, but the effect was stronger in Experiment 1. We discuss the implication of these findings for functional and mechanistic models of suboptimal choice.

Paradoxical choice and the reinforcing value of information

Signals that reduce uncertainty can be valuable because well-informed decision-makers can better align their preferences to opportunities. However, some birds and mammals display an appetite for informative signals that cannot be used to increase returns. We explore the role that reward-predictive stimuli have in fostering such preferences, aiming at distinguishing between two putative underlying mechanisms. The 'information hypothesis' proposes that reducing uncertainty is reinforcing per se, somewhat consistently with the concept of curiosity: a motivation to know in the absence of tractable extrinsic benefits. In contrast, the 'conditioned reinforcement hypothesis', an associative account, proposes asymmetries in secondarily acquired reinforcement: post-choice stimuli announcing forthcoming rewards (S⁺) reinforce responses more than stimuli signalling no rewards (S^-) inhibit responses. In three treatments, rats faced two equally profitable options delivering food probabilistically after a fixed delay. In the informative option (Info), food or no food was signalled immediately after choice, whereas in the non-informative option (NoInfo) outcomes were uncertain until the delay lapsed. Subjects preferred Info when (1) both outcomes were explicitly signalled by salient auditory cues, (2) only forthcoming food delivery was explicitly signalled, and (3) only the absence of forthcoming reward was explicitly signalled. Acquisition was slower in (3), when food was not explicitly signalled, showing that signals for positive outcomes have a greater influence on the development of preference than signals for negative ones. Our results are consistent with an elaborated conditioned reinforcement account, and with the conjecture that both uncertainty reduction and conditioned reinforcement jointly act to generate preference.

Forced-exposure trials increase suboptimal choice

The influence of single option or forced-exposure (FE) trials was studied in the suboptimal choice task. Pigeons chose between an optimal alternative that led to food half of the time and a suboptimal alternative that led to food 20% of the time. Choice of the suboptimal alternative was compared across groups of subjects that received different numbers of FE trials during training. In Experiment 1, subjects received 100% FE trials, 67% FE trials, or only choice trials. Pigeons in the two groups that had FE trials developed extreme preference for the signaled suboptimal alternative over the unsignaled optimal alternative, while pigeons that had no FE trials showed pronounced individual differences. Experiment 2 compared 10% and 90% FE trials. When neither alternative signaled trial outcomes, both groups of subjects strongly preferred the optimal alternative. When the suboptimal alternative provided differential signals, the subjects in the 90% FE group developed strong preference for the suboptimal alternative and subjects in the 10% FE group maintained preference for the optimal alternative. The results of both experiments demonstrate that FE trials can have substantial effects on the development of preference in the suboptimal choice task.

Behavioral ephemera, difficult discriminations, and behavioral stability

Every species possesses abilities for successfully interacting with its environment. These result from phylogeny. In the laboratory, one may arrange artificial conditions that thwart an organism's abilities. The result may be a "phenomenon." With sufficient training, however, the phenomenon may prove to be ephemeral, as the organism's basic abilities reassert themselves. Pigeons respond extremely well to differences and nondifferences in rate of obtaining food. This ability may be thwarted in a variety of ways, but the results tend to be ephemeral. An example is an experiment that pitted pigeons' preference for unimpeded responding against their ability to respond to food rate. In a concurrent-chains procedure, the terminal links were identical variable-interval schedules, but in one terminal link, every response produced a timeout. The duration of the timeout varied, and preference varied with it, but the relation vanished with training, in keeping with the equality of food rate across the 2 terminal links. Some other examples of "phenomena" that tend to disappear with sufficient training and sufficient variation in experimental parameters are behavioral contrast, conditioned reinforcement, resistance to extinction, and suboptimal choice. These "phenomena" depend on pigeons' failing to make difficult discriminations. They appear to be behavioral ephemera.

Human Choice Predicted by Obtained Reinforcers, Not by Reinforcement Predictors

Macphail (1985) proposed that “intelligence” should not vary across vertebrate species when contextual variables are accounted for. Focusing on research involving choice behavior, the propensity for choosing an option that produces stimuli that predict the presence or absence of reinforcement but that also results in less food over time can be examined. This choice preference has been found multiple times in pigeons (Stagner and Zentall, 2010; Zentall and Stagner, 2011; Laude et al., 2014) and has been likened to gambling behavior demonstrated by humans (Zentall, 2014, 2016). The present experiments used a similarly structured task to examine adult human preferences for reinforcement predictors and compared findings to choice behavior demonstrated by children (Lalli et al., 2000), monkeys (Smith et al., 2017; Smith and Beran, 2020), dogs (Jackson et al., 2020), rats (Chow et al., 2017; Cunningham and Shahan, 2019; Jackson et al., 2020), and pigeons (Roper and Zentall, 1999; Stagner and Zentall, 2010). In Experiment 1, adult human participants showed no preference for reinforcement predictors. Results from Experiment 2 suggest that not only were reinforcement predictors not preferred, but that perhaps reinforcement predictors had no effect at all on choice behavior. Results from Experiments 1 and 2 were further assessed using a generalized matching equation, the findings from which support that adult human choice behavior in the present research was largely determined by reinforcement history. Overall, the present results obtained from human adult participants are different than those found from pigeons in particular, suggesting that further examination of Macphail (1985) hypothesis is warranted.

The Δ-∑ hypothesis: How contrast and reinforcement rate combine to generate suboptimal choice

When given a choice between two alternatives, each offering food after the same delay with different but signaled probabilities, pigeons often prefer the low probability alternative. This preference is surprising because pigeons fail to maximize the rate of food intake; they exhibit a suboptimal preference. We advance a new explanation, the Δ-∑ hypothesis, in which the difference in probability of reinforcement within terminal links (Δ) and the overall reinforcement probability rate of each alternative (∑) are the key variables responsible for such suboptimal preference. We tested the Δ-∑ hypothesis in two experiments. In Experiment 1, we manipulated the Δs while maintaining constant all other parameters of the task, in particular the ∑s. We predicted a preference for the alternative with the larger Δ. In Experiment 2, we examined the effect of the overall reinforcement probabilities, the ∑s, while maintaining constant all other parameters of the task, in particular the Δs. We predicted a preference for the larger ∑. The results of both experiments support the Δ-∑ hypothesis.

Human and pigeon suboptimal choice

Many studies have shown that pigeons will sometimes behave suboptimally by choosing an option that provides food less frequently over one that provides food more frequently. The critical factor in driving suboptimal behavior in these procedures is that the delayed outcomes are differentially signaled on the suboptimal alternative, but not the optimal alternative. Although this procedure is frequently cited as potentially analogous to human gambling, there is little empirical data to evaluate this assertion. The present study tested both pigeon (Experiment 1) and human (Experiment 2) subjects with a suboptimal choice task. Subjects chose between a suboptimal alternative that provided a large reinforcer 20% of the time and an optimal alternative that always provided a small reinforcer. Stimuli presented during the delays signaled the outcomes on the suboptimal alternative in some conditions. When outcomes were signaled, pigeons chose the suboptimal alternative more frequently than did humans. When the outcomes were not signaled, pigeons' choices became more optimal, but humans' choices did not. Humans' suboptimal choice was unrelated to performance on a probability discounting task. Overall, these findings suggest that although both pigeons and humans can choose suboptimally, more research is needed in order to determine whether non-human performance on this task can serve as a model for human gambling.

The influence of outcome delay on suboptimal choice

Under certain conditions pigeons will choose an option that provides less probable food over one that provides more probable food. This suboptimal choice behavior occurs when the outcomes are delayed and stimuli during the delay differentially signal the upcoming outcomes on the suboptimal alternative, but not the optimal alternative. The present study assessed whether duration of the outcome delay affects pigeons' suboptimal preference. Pigeons chose between a suboptimal alternative that provided food 20% of the time and an optimal alternative that provided food 80% of the time. Stimuli presented during the delays signaled the outcomes on the suboptimal alternative, but not on the optimal alternative. The outcome delays were 5 s in some conditions and 20 s in others. The results of two experiments demonstrate that behavior is generally more suboptimal when the outcome delays are longer but tends to stay relatively suboptimal if subjects experience the long delay condition before the short delay condition. The finding that behavior is more suboptimal with longer delays to the outcomes is consistent with the view that pigeons' suboptimal choice is influenced by both conditioned and primary reinforcement and is inconsistent with the view that suboptimal choice is influenced solely by signal value.

Suboptimal choice in pigeons: Does the predictive value of the conditioned reinforcer alone determine choice?

Prior research has found that pigeons are indifferent between an option that always provides a signal for reinforcement and an alternative that provides a signal for reinforcement only 50% of the time (and a signal for the absence of reinforcement 50% of the time). This suboptimal choice suggests that the frequency of the signal for reinforcement plays virtually no role and choice depends only on the predictive value of the signal for reinforcement associated with each alternative. In the present research we tested the hypothesis that if there are two or three signals for reinforcement associated with the suboptimal alternative but each occurs only 25% or 17% of the time, respectively, pigeons would show a greater preference for the suboptimal alternative. Although we found that increasing the number of signals for reinforcement associated with the suboptimal alternative did not increase the preference for the suboptimal alternative (relative to a single signal for reinforcement) extended training on this task resulted in a significant preference for the suboptimal alternative by both groups. This result suggests that contrast between the expected outcome at the time of choice (50% reinforcement) and the value of the signal for reinforcement (100% reinforcement) is also responsible for choice of the suboptimal alternative.

The role of 'jackpot' stimuli in maladaptive decision-making: dissociable effects of D1/D2 receptor agonists and antagonists

RATIONALE

Laboratory experiments often model risk through a choice between a large, uncertain (LU) reward against a small, certain (SC) reward as an index of an individual's risk tolerance. An important factor generally lacking from these procedures are reward-associated cues that may modulate risk preferences.

OBJECTIVE

We tested whether the addition of cues signaling 'jackpot' wins to LU choices would modulate risk preferences and if these cue effects were mediated by dopaminergic signaling.

METHODS

Three groups of rats chose between LU and SC rewards for which the LU probability of reward decreased across blocks. The unsignaled group received a non-informative stimulus of trial outcome. The signaled group received a jackpot signal prior to reward delivery and blackout on losses. The signaled-light group received a similar jackpot for wins, but a salient loss signal distinct from the win signal.

RESULTS

Presenting win signals decreased the discounting of LU value for both signaled groups regardless of loss signal, while the unsignaled group showed discounting similar to previous research without cues. Pharmacological challenges with D1/D2 agonists and antagonists revealed that D1 antagonism increased and decreased sensitives to the relative probability of reward for unsignaled and signaled groups, respectively, while D2 agonists decreased sensitivities to the relative magnitude of reward.

CONCLUSION

The results highlight how signals predictive of wins can promote maladaptive risk taking in individuals, while loss signals have reduced effect. Additionally, the presence of reward-predictive cues may change the underlying neurobehavioral mechanisms mediating decision-making under risk.

Ultimate explanations and suboptimal choice

Researchers have unraveled multiple cases in which behavior deviates from rationality principles. We propose that such deviations are valuable tools to understand the adaptive significance of the underpinning mechanisms. To illustrate, we discuss in detail an experimental protocol in which animals systematically incur substantial foraging losses by preferring a lean but informative option over a rich but non-informative one. To understand how adaptive mechanisms may fail to maximize food intake, we review a model inspired by optimal foraging principles that reconciles sub-optimal choice with the view that current behavioral mechanisms were pruned by the optimizing action of natural selection. To move beyond retrospective speculation, we then review critical tests of the model, regarding both its assumptions and its (sometimes counterintuitive) predictions, all of which have been upheld. The overall contention is that (a) known mechanisms can be used to develop better ultimate accounts and that (b) to understand why mechanisms that generate suboptimal behavior evolved, we need to consider their adaptive value in the animal's characteristic ecology.

Gambling-like behavior in pigeons: 'jackpot' signals promote maladaptive risky choice

Individuals often face choices that have uncertain outcomes and have important consequences. As a model of this environment, laboratory experiments often offer a choice between an uncertain, large reward that varies in its probability of delivery against a certain but smaller reward as a measure of an individual’s risk aversion. An important factor generally lacking from these procedures are gambling related cues that may moderate risk preferences. The present experiment offered pigeons choices between unreliable and certain rewards but, for the Signaled group on winning choices, presented a ‘jackpot’ signal prior to reward delivery. The Unsignaled group received an ambiguous stimulus not informative of choice outcomes. For the Signaled group, presenting win signals effectively blocked value discounting for the large, uncertain outcome as the probability of a loss increased, whereas the Unsignaled group showed regular preference changes similar to previous research lacking gambling related cues. These maladaptive choices were further shown to be unaffected by more salient loss signals and resistant to response cost increases. The results suggest an important role of an individual’s sensitivity to outcome-correlated cues in influencing risky choices that may moderate gambling behaviors in humans, particularly in casino and other gambling-specific environments.

When good news leads to bad choices

Pigeons and other animals sometimes deviate from optimal choice behavior when given informative signals for delayed outcomes. For example, when pigeons are given a choice between an alternative that always leads to food after a delay and an alternative that leads to food only half of the time after a delay, preference changes dramatically depending on whether the stimuli during the delays are correlated with (signal) the outcomes or not. With signaled outcomes, pigeons show a much greater preference for the suboptimal alternative than with unsignaled outcomes. Key variables and research findings related to this phenomenon are reviewed, including the effects of durations of the choice and delay periods, probability of reinforcement, and gaps in the signal. We interpret the available evidence as reflecting a preference induced by signals for good news in a context of uncertainty. Other explanations are briefly summarized and compared.

When good pigeons make bad decisions: Choice with probabilistic delays and outcomes

Pigeons chose between an (optimal) alternative that sometimes provided food after a 10-s delay and other times after a 40-s delay and another (suboptimal) alternative that sometimes provided food after 10 s but other times no food after 40 s. When outcomes were not signaled during the delays, pigeons strongly preferred the optimal alternative. When outcomes were signaled, choices of the suboptimal alternative increased and most pigeons preferred the alternative that provided no food after the long delay despite the cost in terms of obtained food. The pattern of results was similar whether the short delays occurred on 25% or 50% of the trials. Shortening the 40-s delay to food sharply reduced suboptimal choices, but shortening the delay to no food had little effect. The results suggest that a signaled delay to no food does not punish responding in probabilistic choice procedures. The findings are discussed in terms of conditioned reinforcement by signals for good news.

Suboptimal Choice in Pigeons: Stimulus Value Predicts Choice over Frequencies

Pigeons have shown suboptimal gambling-like behavior when preferring a stimulus that infrequently signals reliable reinforcement over alternatives that provide greater reinforcement overall. As a mechanism for this behavior, recent research proposed that the stimulus value of alternatives with more reliable signals for reinforcement will be preferred relatively independently of their frequencies. The present study tested this hypothesis using a simplified design of a Discriminative alternative that, 50% of the time, led to either a signal for 100% reinforcement or a blackout period indicative of 0% reinforcement against a Nondiscriminative alternative that always led to a signal that predicted 50% reinforcement. Pigeons showed a strong preference for the Discriminative alternative that remained despite reducing the frequency of the signal for reinforcement in subsequent phases to 25% and then 12.5%. In Experiment 2, using the original design of Experiment 1, the stimulus following choice of the Nondiscriminative alternative was increased to 75% and then to 100%. Results showed that preference for the Discriminative alternative decreased only when the signals for reinforcement for the two alternatives predicted the same probability of reinforcement. The ability of several models to predict this behavior are discussed, but the terminal link stimulus value offers the most parsimonious account of this suboptimal behavior.