Risky choice in pigeons: preference for amount variability using a token-reinforcement system

A critical review of risk-sensitive foraging

Foraging is risk sensitive if choices depend on the variability of returns from the options as well as their mean return. Risk-sensitive foraging is important in behavioural ecology, psychology and neurophysiology. It has been explained both in terms of mechanisms and in terms of evolutionary advantage. We provide a critical review, evaluating both mechanistic and evolutionary accounts. Some derivations of risk sensitivity from mechanistic models based on psychophysics are not convincing because they depend on an inappropriate use of Jensen's inequality. Attempts have been made to link risk sensitivity to the ecology of a species, but again these are not convincing. The field of risk-sensitive foraging has provided a focus for theoretical and empirical work and has yielded important insights, but we lack a simple and empirically defendable general account of it in either mechanistic or evolutionary terms. However, empirical analysis of choice sequences under theoretically motivated experimental designs and environmental settings appears a promising avenue for mapping the scope and relative merits of existing theories. Simply put, the devil is in the sequence.

On the value of advanced information about delayed rewards

In a variety of laboratory preparations, several animal species prefer signaled over unsignaled outcomes. Here we examine whether pigeons prefer options that signal the delay to reward over options that do not and how this preference changes with the ratio of the delays. We offered pigeons repeated choices between two alternatives leading to a short or a long delay to reward. For one alternative (informative), the short and long delays were reliably signaled by different stimuli (e.g., SS for short delays, SL for long delays). For the other (non-informative), the delays were not reliably signaled by the stimuli presented (S1 and S2). Across conditions, we varied the durations of the short and long delays, hence their ratio, while keeping the average delay to reward constant. Pigeons preferred the informative over the non-informative option and this preference became stronger as the ratio of the long to the short delay increased. A modified version of the Δ-Σ hypothesis (González et al., J Exp Anal Behav 113(3):591-608. https://doi.org/10.1002/jeab.595 , 2020a) incorporating a contrast-like process between the immediacies to reward signaled by each stimulus accounted well for our findings. Functionally, we argue that a preference for signaled delays hinges on the potential instrumental advantage typically conveyed by information.

Perception of visual variance is mediated by subcortical mechanisms

Variance characterizes the structure of the environment. This statistical concept plays a critical role in evaluating the reliability of evidence for human decision-making. The present study examined the involvement of subcortical structures in the processing of visual variance. To this end, we used a stereoscope to sequentially present two circle arrays in a dichoptic or monocular fashion while participants compared the perceived variance of the two arrays. In Experiment 1, two arrays were presented monocularly to the same eye, dichopticly to different eyes, or binocularly to both eyes. The variance judgment was less accurate in different-eye condition than the other conditions. In Experiment 2, the first circle array was split into a large-variance and a small-variance set, with either the large-variance or small-variance set preceding the presentation of the second circle array in the same eye. The variance of the first array was judged larger when the second array was preceded by the large-variance set in the same eye, showing that the perception of variance was modulated by the visual variance processed in the same eye. Taken together, these findings provide evidence for monocular processing of visual variance, suggesting that subcortical structures capture the statistical structure of the visual world.

Student preference for and performance in fixed- versus mixed-duration schedules

Duration schedules of reinforcement for continuous behavior abide by several preexisting operant behavioral economic equations for reinforcer cost, otherwise known as price, and consumption. Duration schedules require behaviors to occur for a set duration of time prior to accessing reinforcement, unlike interval schedules that produce reinforcement after the first instance of a behavior after a given period. Despite extensive examples of naturally occurring duration schedules, translational research regarding duration schedules is quite limited. Further, a lack of research investigating the implementation of such reinforcement schedules, combined with concepts such as preference, represents a gap in the applied behavior analysis literature. The current study measured three elementary students' preferences for fixed- and mixed-duration schedules of reinforcement during academic work completion. Results suggest students prefer mixed-duration schedules of reinforcement that provide the opportunity to access reinforcement at a reduced price and that such arrangements could be employed to increase work completion and academic-engaged time.

Space, feature, and risk sensitivity in homing pigeons (Columba livia): Broadening the conversation on the role of the avian hippocampus in memory

David Sherry has been a pioneer in investigating the avian hippocampal formation (HF) and spatial memory. Following on his work and observations that HF is sensitive to the occurrence of reward (food), we were interested in carrying out an exploratory study to investigate possible HF involvement in the representation goal value and risk. Control sham-lesioned and hippocampal-lesioned pigeons were trained in an open field to locate one food bowl containing a constant two food pellets on all trials, and two variable bowls with one containing five pellets on 75% (High Variable) and another on 25% (Low Variable) of their respective trials (High-Variable and Low-Variable bowls were never presented together). One pairing of pigeons learned bowl locations (space); another bowl colors (feature). Trained to color, hippocampal-lesioned pigeons performed as rational agents in their bowl choices and were indistinguishable from the control pigeons, a result consistent with HF regarded as unimportant for non-spatial memory. By contrast, when trained to location, hippocampal-lesioned pigeons differed from the control pigeons. They made more first-choice errors to bowls that never contained food, consistent with a role of HF in spatial memory. Intriguingly, the hippocampal-lesioned pigeons also made fewer first choices to both variable bowls, suggesting that hippocampal lesions resulted in the pigeons becoming more risk averse. Acknowledging that the results are preliminary and further research is needed, the data nonetheless support the general hypothesis that HF-dependent memory representations of space capture properties of reward value and risk, properties that contribute to decision making when confronted with a choice.

Effects of remifentanil/histamine mixtures in rats responding under a choice procedure

Intravenous drug self-administration remains the 'gold standard' for assessing abuse liability. Failure of a drug to maintain self-administration might indicate the absence of positive reinforcing effects but might also indicate the presence of aversive effects. Sensitivity to aversive and punishing effects of drugs (as well as nondrug stimuli) might collectively determine the likelihood of use, abuse and relapse. Using a choice procedure, this study compared the effects of remifentanil (mu opioid receptor agonist; 0.001-0.01 mg/kg/infusion) and histamine (H1-4 receptor agonist; 0.32-3.2 mg/kg/infusion), alone and in mixtures, to test the hypothesis that remifentanil/histamine mixtures are less reinforcing compared with remifentanil alone and less punishing compared with histamine alone. Male Sprague-Dawley rats (n = 10) chose between an intravenous infusion + a pellet and a pellet alone. Rats were indifferent to saline, chose remifentanil + a pellet over a pellet alone, and chose a pellet alone over histamine + a pellet. The effects of remifentanil/histamine mixtures generally were different from the constituent doses of histamine alone but not from remifentanil alone. A mixture containing 3.2 mg/kg/infusion histamine and either 0.001 or 0.0032 mg/kg/infusion remifentanil was not different from saline but was different from the effects of the constituent dose, insofar as choice increased compared with 3.2 mg/kg/infusion histamine alone and decreased compared with 0.001 or 0.0032 mg/kg/infusion remifentanil alone. Reinforcing doses of remifentanil combined with punishing doses of histamine can yield mixtures that are neither preferred nor avoided, offering 'proof-of-principle' for using drug mixtures to avoid adverse effects of opioid receptor agonists.

Probing the decision-making mechanisms underlying choice between drug and nondrug rewards in rats

Delineating the decision-making mechanisms underlying choice between drug and nondrug rewards remains a challenge. This study adopts an original approach to probe these mechanisms by comparing response latencies during sampling versus choice trials. While lengthening of latencies during choice is predicted in a deliberative choice model (DCM), the race-like response competition mechanism postulated by the Sequential choice model (SCM) predicts a shortening of latencies during choice compared to sampling. Here, we tested these predictions by conducting a retrospective analysis of cocaine-versus-saccharin choice experiments conducted in our laboratory. We found that rats engage deliberative decision-making mechanisms after limited training, but adopt a SCM-like response selection mechanism after more extended training, while their behavior is presumably habitual. Thus, the DCM and SCM may not be general models of choice, as initially formulated, but could be dynamically engaged to control choice behavior across early and extended training.

The functional equivalence of two variants of the suboptimal choice task: choice proportion and response latency as measures of value

In the suboptimal-choice task, birds systematically choose the leaner but informative option (suboptimal) over the richer but non-informative option (optimal). The task has two variations. In the standard task, the optimal option includes two terminal link stimuli. In the original task, it includes a single terminal link stimulus. Two models, the temporal information account (Cunningham and Shahan, J Exp Psychol Anim Learn Cogn 44:1-22, 2018) and the ∆-∑ hypothesis (González et al., J Exp Anal Behav 113:591-608, 2020), presuppose that these procedures are equivalent, but no formal comparison is available. Here we test whether or not these procedures are functionally equivalent. One group of pigeons was trained with the standard procedure, another group with the original procedure, and a third group was trained with a hybrid of the other two (i.e., the two options were the optimal links of the standard and original procedures). Our findings indicate that the number of terminal link stimuli in the optimal option is inconsequential vis-à-vis choice. Moreover, our findings also indicate that latencies to respond are a sensitive metric of value and choice. As predicted by the Sequential Choice Model, we were able to predict simultaneous choices from the latencies of sequential choices and observed a substantial shortening of latencies during simultaneous choices.

Unpredictability as a modulator of drug self-administration: Relevance for substance-use disorders

Drug self-administration has been regarded as a gold-standard preclinical model of addiction and substance-use disorder (SUD). However, investigators are becoming increasingly aware, that certain aspects of addiction or SUDs experienced by humans are not accurately captured in our preclinical self-administration models. The current review will focus on two such aspects of current preclinical drug self-administration models: 1) Predictable vs. unpredictable drug access in terms of the time and effort put into obtaining drugs (i.e., response requirement) and drug quality (i.e., amount) and 2) rich vs. lean access to drugs. Some behavioral and neurobiological mechanisms that could contribute to excessive allocation of behavior toward drug-seeking and drug-taking at the expense of engaging in nondrug-related activities are discussed, and some directions for future research are identified. Based on the experiments reviewed, lean and unpredictable drug access could worsen drug-seeking and drug-taking behavior in individuals with SUDs. Once more fully explored, this area of research will help determine whether and how unpredictable and lean cost requirements affect drug self-administration in preclinical laboratory studies with nonhuman subjects and will help determine whether incorporating these conditions in current self-administration models will increase their predictive validity.

Choice between variable and fixed cocaine injections in male rhesus monkeys

RATIONALE

The schedule of drug availability may enhance choice of a drug. In non-human subjects, reinforcers are chosen more often when available under variable schedules of reinforcement relative to fixed schedules.

OBJECTIVE

To determine whether variable-drug access is an important determinant of cocaine choice by manipulating the schedule, drug dose, and combination of schedule + dose.

METHOD

Four male rhesus monkeys chose between cocaine doses (0.025-0.4 mg/kg/injection). In control conditions, the schedule and dose of each drug delivery were fixed. In other conditions, the reinforcement schedule (i.e., variable-ratio schedule), dose of each cocaine delivery, or both were variable on one lever while all aspects on the other lever remained fixed.

RESULTS

When cocaine dose was equal on average (0.1 mg/kg/injection), 2 of 4 subjects chose cocaine associated with the variable schedule more than the fixed schedule. All subjects chose the variable dose that was equal on average to the fixed dose, and this difference was statistically significant. Three of 4 subjects chose cocaine associated with the variable combination over the fixed option (when the dose was equal on average). During dose-response determinations (when dose on the variable and fixed options were not equal), making the schedule, dose, or both variable generally did not alter cocaine's potency as a reinforcer.

CONCLUSION

While many factors contribute to drug choice, unpredictable drug access is a feature that may be common in the natural environment and could play a key role in the allocation of behavior to drug alternatives by patients with substance-use disorders.

Children's preference for mixed- versus fixed-ratio schedules of reinforcement: A translational study of risky choice

Laboratory research has shown that when subjects are given a choice between fixed-ratio and bi-valued mixed-ratio schedules of reinforcement, preference typically emerges for the mixed-ratio schedule even with a larger ratio requirement. The current study sought to replicate and extend these findings to children's math problem completion. Using an ABCBC reversal design, four fourth-grade students were given the choice of completing addition problems reinforced on either a fixed-ratio 5 schedule or one of three mixed-ratio schedules; an equivalent mixed-ratio (1, 9) schedule, a mixed-ratio (1, 11) schedule with a 20% larger ratio requirement, and an equally lean mixed-ratio (5, 7) schedule without the small fixed-ratio 1 component. This was followed by a reversal back to the preceding phase in which preference for the mixed-ratio schedule had been observed, and a final reversal back to the mixed-ratio (5, 7) phase. Findings were consistent with previous research in that all children preferred the mixed-ratio (1, 9) schedule over the equivalent fixed-ratio 5 schedule. Preference persisted for the leaner mixed-ratio (1, 11) schedule for three of the four children. Indifference or preference for the fixed-ratio 5 alternative was observed in phases containing the mixed-ratio (5, 7) schedule. These results extend previous research on risky choice to children's math problem completion and highlight the importance of a small ratio component in the emergence of preference for bi-valued mixed-ratio schedules. Implications of these results for arranging reinforcement to increase children's academic responding are discussed.

Substitution effects in a generalized token economy with pigeons

Pigeons made repeated choices between earning and exchanging reinforcer-specific tokens (green tokens exchangeable for food, red tokens exchangeable for water) and reinforcer-general tokens (white tokens exchangeable for food or water) in a closed token economy. Food and green food tokens could be earned on one panel; water and red water tokens could be earned on a second panel; white generalized tokens could be earned on either panel. Responses on one key produced tokens according to a fixed-ratio schedule, whereas responses on a second key produced exchange periods, during which all previously earned tokens could be exchanged for the appropriate commodity. Most conditions were conducted in a closed economy, and pigeons distributed their token allocation in ways that permitted food and water consumption. When the price of all tokens was equal and low, most pigeons preferred the generalized tokens. When token-production prices were manipulated, pigeons reduced production of the tokens that increased in price while increasing production of the generalized tokens that remained at a fixed price. The latter is consistent with a substitution effect: Generalized tokens increased and were exchanged for the more expensive reinforcer. When food and water were made freely available outside the session, token production and exchange was sharply reduced but was not eliminated, even in conditions when it no longer produced tokens. The results join with other recent data in showing sustained generalized functions of token reinforcers, and demonstrate the utility of token-economic methods for assessing demand for and substitution among multiple commodities in a laboratory context.

Trends in Behavior-Analytic Gambling Research and Treatment

The purpose of the present review was to analyze research outcomes for all gambling studies reported in the behavior analysis literature. We used the search term "gambling" to identify articles that were published in behaviorally oriented journals between the years 1992 and 2012 and categorized the content of each article as empirical or conceptual. Next, we examined and categorized the empirical articles by inclusion of an experimental manipulation and treatment to alleviate at least some aspect of pathological gambling, participant population used, type of gambling task employed in the research, whether the participants in the study actually gambled, and the behavioral phenomena of interest. The results show that the rate of publication of gambling research has increased in the last 6 years, and a vast majority of articles are empirical. Of the empirical articles, examinations of treatment techniques or methods are scarce; slot machine play is the most represented form of gambling, and slightly greater than half of the research included compensation based on gambling outcomes within experiments. We discuss implications and future directions based on these observations of the published literature.

The several roles of stimuli in token reinforcement

Three experiments were conducted with pigeons to identify the stimulus functions of tokens in second-order token-reinforcement schedules. All experiments employed two-component multiple schedules with a token-reinforcement schedule in one component and a schedule with equivalent response requirements and/or reinforcer density in the other. In Experiment 1, response rates were lower under a token-reinforcement schedule than under a tandem schedule with the same response requirements, suggesting a discriminative role for the tokens. In Experiment 2, response rates varied systematically with signaling functions of the tokens in a series of conditions designed to explore other aspects of the temporal-correlative relations between tokens and food. In Experiment 3, response rates were reduced but not eliminated by presenting tokens independent of responding, yoked to their temporal occurrence in a preceding token component, suggesting both a reinforcing function and eliciting/evocative functions based on stimulus-food relations. Only when tokens were removed entirely was responding eliminated. On the whole, the results suggest that tokens, as stimuli temporally correlated with food, may serve multiple stimulus functions in token-reinforcement procedures--reinforcing, discriminative, or eliciting--depending on the precise arrangement of the contingencies in which they are embedded.

Random-ratio schedules produce greater demand for i.v. drug administration than fixed-ratio schedules in rhesus monkeys

RATIONALE

Organisms emit more responses when food is provided according to random as compared with fixed schedules of reinforcement. Similarly, many human behaviors deemed compulsive are maintained on variable schedules (e.g., gambling). If greater amounts of behavior are maintained by drugs of abuse when earned according to variably reinforced schedules, this would suggest that excessive drug-taking behavior may be due in part to the nature of drug availability.

OBJECTIVES

The aim was to determine whether random schedules of contingent intravenous drug delivery would produce more responding than similarly priced fixed schedules.

METHODS

Six rhesus macaque subjects responded to produce cocaine (0.003-0.03 mg/kg/inj), remifentanil (0.01-1.0 μg/kg/inj), or ketamine (0.01-0.1 mg/kg/inj) according to either fixed or random ratio requirements that increased systematically across sessions. Demand curves were generated with the most effective dose of each drug and compared across drug and schedule type.

RESULTS

Cocaine and remifentanil maintained higher levels and rates of responding when earned according to random-ratio schedules as compared with fixed-ratio schedules. This difference was most pronounced when drugs were available at high unit prices. Differences in responding across the schedule types generated by ketamine-a lesser-valued reinforcer-were qualitatively similar but smaller in magnitude.

CONCLUSIONS

The current study provides a systematic replication across reinforcer type demonstrating that drugs delivered after a random number of responses generate more behavior than those delivered according to a fixed schedule. The variable nature of the availability of drugs of abuse-particularly those that are scarce or expensive-may be a contributing factor to excessive drug intake by humans. This effect is most likely to be observed when more highly demanded (reinforcing) drugs are being consumed.

Generalized conditioned reinforcement with pigeons in a token economy

Six pigeons were studied in a token economy in which tokens could be produced and exchanged for food on one side of an experimental chamber and for water on the opposite side of the chamber. Responses on one key produced tokens according to a token-production fixed ratio (FR) schedule. Responses on a second key produced an exchange period during which tokens were exchanged for water or food. In Experiment 1a, food tokens could be earned and exchanged under restricted food budgets, and water tokens could be earned and exchanged under water restricted budgets. In Experiment 1b, a third (generalized) token type could be earned and exchanged for either food or water under water restricted budgets. Across Experiments 1a and 1b, the number of tokens accumulated prior to exchange increased as the exchange-production schedule was increased. In Experiment 1b, pigeons produced more generalized than specific tokens, suggesting enhanced reinforcing efficacy of generalized tokens. In Experiment 2, the FR token-production price was manipulated under water restriction and then under food restriction. Production of each token type generally declined as a function of its own price and increased as a function of the price of the alternate type, demonstrating own-price and cross-price elasticity. Production of food and water tokens often changed together, indicating complementarity. Production of specific and generalized tokens changed in opposite directions, indicating substitutability. This is the first demonstration of sustained generalized functions of tokens in nonhumans, and illustrates a promising method for exploring economic contingencies in a controlled environment.