Scaling pigeons' choice of feeds: bigger is better

Hyperbolae

Hyperbolic relations between independent and dependent variables are ubiquitous in the experimental analysis of behavior, mentioned in over 150 articles in the Journal of the Experimental Analysis of Behavior. There are two principal forms of hyperbolae: The first describes the relation between response rate and reinforcement rate on variable-interval schedules of reinforcement; it rises asymptotically toward a maximum. The second describes the relation between the current equivalent value of an incentive and its delay or (im)probability; it falls from a maximum toward an asymptote of 0. Where do these come from? What do their parameters mean? How are they related? This article answers the first two questions and addresses the last.

Parallel processing of value-related information during multi-attribute decisions

When choosing between options with multiple attributes, do we integrate all attributes into a unified measure for comparison, or does the comparison also occur at the level of each attribute, involving parallel processes that can dynamically influence each other? What happens when independent sensory features all carry information about the same decision factor, such as reward value? To investigate these questions, we asked human participants to perform a two-alternative forced choice reaching task in which the reward value of a target was indicated by two visual attributes-its brightness ("bottom-up," BU feature) and its orientation ("top-down," TD feature). If decisions always occur after the integration of both features, there should be no difference in the reaction time (RT) regardless of the attribute combinations that drove the choice. Counter to that prediction, RT distributions depended on the attribute combinations of given targets and the choices made by participants. RTs were shortest when both attributes were congruent or when the choice was based on the bottom-up feature, and longer when the attributes were in conflict (favoring opposite options). In conflict trials, nearly two-thirds of participants made faster decisions when choosing the option favored by the bottom-up feature than when choosing the top-down-favored option. We also observed mid-reach changes-of-mind in a subset of conflict trials, mostly changing from the bottom-up to the top-down-favored target. These data suggest that multi-attribute value-based decisions are better explained by a distributed process including competition among different features than by a competition based on a single, integrated estimate of value.NEW & NOTEWORTHY We show that during value-based decisions, humans do not always use all reward-related information to make their choice, but instead can "jump the gun" using partial information. In particular, when different sources of information were in conflict, early decisions were mostly based on fast bottom-up information, and sometimes followed by corrective changes-of-mind based on slower top-down information. This supports parallel decision processes among different information sources, as opposed to a single integrated "common currency."

Impulsivity and risk-seeking as Bayesian inference under dopaminergic control

Bayesian models successfully account for several of dopamine (DA)'s effects on contextual calibration in interval timing and reward estimation. In these models, tonic levels of DA control the precision of stimulus encoding, which is weighed against contextual information when making decisions. When DA levels are high, the animal relies more heavily on the (highly precise) stimulus encoding, whereas when DA levels are low, the context affects decisions more strongly. Here, we extend this idea to intertemporal choice and probability discounting tasks. In intertemporal choice tasks, agents must choose between a small reward delivered soon and a large reward delivered later, whereas in probability discounting tasks, agents must choose between a small reward that is always delivered and a large reward that may be omitted with some probability. Beginning with the principle that animals will seek to maximize their reward rates, we show that the Bayesian model predicts a number of curious empirical findings in both tasks. First, the model predicts that higher DA levels should normally promote selection of the larger/later option, which is often taken to imply that DA decreases 'impulsivity,' and promote selection of the large/risky option, often taken to imply that DA increases 'risk-seeking.' However, if the temporal precision is sufficiently decreased, higher DA levels should have the opposite effect-promoting selection of the smaller/sooner option (higher impulsivity) and the small/safe option (lower risk-seeking). Second, high enough levels of DA can result in preference reversals. Third, selectively decreasing the temporal precision, without manipulating DA, should promote selection of the larger/later and large/risky options. Fourth, when a different post-reward delay is associated with each option, animals will not learn the option-delay contingencies, but this learning can be salvaged when the post-reward delays are made more salient. Finally, the Bayesian model predicts correlations among behavioral phenotypes: Animals that are better timers will also appear less impulsive.

Bidding for delayed rewards: Accumulation as delay discounting, delay discounting as regulation, demand functions as corollary

In the accumulation paradigm animals press one manipulandum to accumulate pellets or seconds of access to food, and then press another manipulandum, or run some distance, to collect it. The accumulation may be interpreted as delay discounting, with the animals adjusting the distal amount to compensate for its distance or delay. The amount accumulated before being collected is a linear function of the distance or time that the experimental paradigm stipulates for collecting it. That linear function follows from the Unit-Price/Unit-Amount axiom. The inverse of the linear compensation functions gives a delay-discount hyperbola. The advantages of the accumulation paradigm and analytic framework for delay discounting studies are noted. Compensation functions are then derived from a behavioral regulation model, which generalizes them to contexts where the individual's budget for response cost becomes over-taxed. In turn, such compensation/regulation models lead directly to representative demand functions. In sum, regulation models provide a theoretical grounding for demand functions, compensation functions, and delay discounting hyperbolas. The parameter that links them is the unit amount k, the slope of the compensation function and of the discount function, the setpoint for consumption in a regulation model, and the ideal quantity consumed at minimal price in a demand analysis.

Systematic Analysis of Pigeons' Discrimination of Pixelated Stimuli: A Hierarchical Pattern Recognition System Is Not Identifiable

Pigeons learned to discriminate two different patterns displayed with miniature light-emitting diode arrays. They were then tested with 84 interspersed, non-reinforced degraded pattern pairs. Choices ranged between 100% and 50% for one or other of the patterns. Stimuli consisting of few pixels yielded low choice scores whereas those consisting of many pixels yielded a broad range of scores. Those patterns with a high number of pixels coinciding with those of the rewarded training stimulus were preferred and those with a high number of pixels coinciding with the non-rewarded training pattern were avoided; a discrimination index based on this correlated 0.74 with the pattern choices. Pixels common to both training patterns had a minimal influence. A pixel-by-pixel analysis revealed that eight pixels of one pattern and six pixels of the other pattern played a prominent role in the pigeons’ choices. These pixels were disposed in four and two clusters of neighbouring locations. A summary index calculated on this basis still only yielded a weak 0.73 correlation. The individual pigeons’ data furthermore showed that these clusters were a mere averaging mirage. The pigeons’ performance depends on deep learning in a midbrain-based multimillion synapse neuronal network. Pixelated visual patterns should be helpful when simulating perception of patterns with artificial networks.

Reinforcer quality matters: A test of the Mathematical Principles of Reinforcement with domestic hens (Gallus gallus domesticus)

This study evaluated the ability of Killeen's (1994) Mathematical Principles of Reinforcement to account for the effects of changes in reinforcer quality on hens' rates of responding on fixed-ratio schedules. Hens were trained to peck a key on a fixed-ratio schedule of reinforcement and then experienced an ascending series of ratio values in two separate conditions. In different conditions, the food reinforcer was either wheat or puffed wheat. Response rates initially increased with increases in ratio requirement before eventually decreasing at larger ratios. Quantitative fits of the model accounted for the data well. The fits revealed that different foods were systematically associated with changes in the specific activation parameter, a, and these were consistent with previous reports of preference for those food items.

Comparative evaluations of reward dimensions in honey bees: evidence from two-alternative forced choice proboscis-extension conditioning

A major challenge in understanding choice behaviour is determining how subjects evaluate alternatives that differ along multiple dimensions. Of particular interest is whether similar dimensions are compared to each other or whether each alternative is assigned an absolute value (utility). We assumed that choice proportions would follow Weber's effect, according to which discrimination is proportional to relative difference (difference/mean). We tested honey bees in a two-alternative forced choice (2AFC) paradigm of proboscis-extension response (PER) conditioning. Subjects were conditioned over six trials to associate each of two odours with sucrose solution rewards and then tested in a choice trial between the two alternatives. Each group of subjects was tested in one treatment, and there were four treatments in each of six experiments. Rewards differed in delay, duration of feeding, and sucrose concentration. In each treatment, the high-profitability alternative was better than the low-profitability alternative along a single dimension, but between treatments of each experiment values in another dimension monotonically increased. Proboscis-response proportions during the conditioning phase tended to be greater for the high-profitability alternative, and choice proportions for it in the choice phase ranged between 0.72 and 0.89 in the 24 treatments. We show for the first time that harnessed bees are sensitive to reward delay. Preferences did not differ statistically between the different treatments of any of the experiments. The results support comparative evaluation of alternatives and are pertinent to multi-attribute choice, with implications for context-dependent preferences. We also discuss the potential advantages of the 2AFC PER simultaneous choice assay.

Models of ADHD: Five ways smaller sooner is better

BACKGROUND

In the delay discounting paradigm choices are made between a good of small utility available soon (SS) vs. a good of greater utility available later (LL). Versions of the task may be used with human and non-human subjects, they generate characteristic data, and are of relevance the topic of this special issue, the imputed impulsivity of organisms categorized as ADHD, and animal models of ADHD.

METHOD

Mathematical models of judgment were formulated. The first model adds the utility of a good to the utility of the delay. It provides distinct information about future perspective, the marginal utility of the goods offered, and the relative weight on delay. It grounds hyperbolic discounting. It predicts slower rates of discount for more preferred goods. It can reduce to an exponential function on future perspective time. When choice is mediated by conditioning, not judgment, a model of conditioned reinforcement strength is written. For studies where only the degree of preference for the LL is reported, a Thurstonian model of discrimination provides a shell that embraces all of the earlier models, providing additional information about the precision and bias of those preferential judgments.

COMPARISON WITH OTHER MODELS

All other prominent models are nested within, or approximations to, the models described here.

RESULTS

The models are fit to exemplary data. They provide some support for theories of delay aversion, and explanations for some inconsistencies in the literature. They involve parameters of general utility and provide connection to economic models.

A temporal basis for Weber's law in value perception

Weber's law—the observation that the ability to perceive changes in magnitudes of stimuli is proportional to the magnitude—is a widely observed psychophysical phenomenon. It is also believed to underlie the perception of reward magnitudes and the passage of time. Since many ecological theories state that animals attempt to maximize reward rates, errors in the perception of reward magnitudes and delays must affect decision-making. Using an ecological theory of decision-making (TIMERR), we analyze the effect of multiple sources of noise (sensory noise, time estimation noise, and integration noise) on reward magnitude and subjective value perception. We show that the precision of reward magnitude perception is correlated with the precision of time perception and that Weber's law in time estimation can lead to Weber's law in value perception. The strength of this correlation is predicted to depend on the reward history of the animal. Subsequently, we show that sensory integration noise (either alone or in combination with time estimation noise) also leads to Weber's law in reward magnitude perception in an accumulator model, if it has balanced Poisson feedback. We then demonstrate that the noise in subjective value of a delayed reward, due to the combined effect of noise in both the perception of reward magnitude and delay, also abides by Weber's law. Thus, in our theory we prove analytically that the perception of reward magnitude, time, and subjective value change all approximately obey Weber's law.

Rats (Rattus norvegicus) and pigeons (Columbia livia) are sensitive to the distance to food, but only rats request more food when distance increases

Three experiments investigated foraging by rats and pigeons. In Experiment 1, each response on a manipulandum delivered food to a cup, with the distance between the manipulandum and the cup varying across conditions. The number of responses made before traveling to collect and eat the food increased with distance for rats, but not for pigeons. In Experiment 2, two manipulanda were placed at different distances from a fixed food source; both pigeons and rats preferentially used the manipulandum closest to the food source. Experiment 3 was a systematic replication of Experiment 1 with pigeons. In different conditions, each peck on the left key increased the upcoming hopper duration by 0.5, 1.5 or 2.5s. Completing a ratio requirement on the right key of 1, 4, 8, 16 or 32 pecks, depending on the condition, then produced the food hopper for a duration that depended on the number of prior left pecks. As the ratio requirement increased on the right key, pigeons responded more on the left key and earned more food. Overall, the results replicate previous research, underlining similarities and differences between these species. The results are discussed in terms of optimal foraging, reinforcer sensitivity and delay discounting.

Food choice in the laboratory pigeon

Although food reward plays a large role in learning and behavioral experiments, there have been few studies examining the most motivating food reward for pigeons. Brown (1969) found that pigeons had a tendency to prefer peas, while Killeen et al. (1993) found pigeons to prefer peas and popcorn in Experiment 1A. We looked to further explore these options as well as expand upon the types of foods examined beyond mainly grains and seeds. Pigeons were presented with six novel foods (granulated peanuts, popping corn, freeze-dried mealworms, bread crumbs, split peas, and sunflower hearts) allocated into two sets of three food items. Once the most consumed food from each food set was determined, they were pooled together with sorghum seeds (a familiar food) to form a third set. Sunflower hearts were the most consumed of all the food items, followed by corn and granulated peanuts. We discuss the potential factors mediating consumption choice, including nutritional profile and food particle size.

Economics, ecologics, and mechanics: The dynamics of responding under conditions of varying motivation

The mechanics of behavior developed by Killeen (1994) is extended to deal with deprivation and satiation and with recovery of arousal at the beginning of sessions. The extended theory is validated against satiation curves and within-session changes in response rates. Anomalies, such as (a) the positive correlation between magnitude of an incentive and response rates in some contexts and a negative correlation in other contexts and (b) the greater prominence of incentive effects when magnitude is varied within the session rather than between sessions, are explained in terms of the basic interplay of drive and incentive motivation. The models are applied to data from closed economies in which changes of satiation levels play a key role in determining the changes in behavior. Relaxation of various assumptions leads to closed-form models for response rates and demand functions in these contexts, ones that show reasonable accord with the data and reinforce arguments for unit price as a controlling variable. The central role of deprivation level in this treatment distinguishes it from economic models. It is argued that traditional experiments should be redesigned to reveal basic principles, that ecologic experiments should be redesigned to test the applicability of those principles in more natural contexts, and that behavioral economics should consist of the applications of these principles to economic contexts, not the adoption of economic models as alternatives to behavioral analysis.

Rapid acquisition of preference in concurrent chains when alternatives differ on multiple dimensions of reinforcement

Pigeons responded in a concurrent-chains procedure in which terminal-link reinforcer variables were changed unpredictably across sessions. In Experiment 1, the terminal-link schedules were fixed-interval (FI) 8 s and FI 16 s, and the reinforcer magnitudes were 2 s and 4 s. In Experiment 2 the probability of reinforcement (100% or 50%) was varied with immediacy and magnitude. Multiple-regression analyses showed that pigeons' initial-link response allocation was determined by current-session reinforcer variables, similar to previous studies which have varied only immediacy (Grace, Bragason, & McLean, 2003). Sensitivity coefficients were positive and statistically significant for all reinforcer variables in both experiments. Analyses of responding within individual sessions showed that final levels of preference for dominated sessions, in which all reinforcer variables favored the same terminal link, were more extreme than for tradeoff sessions in which at least one reinforcer variable favored each alternative. This result implies that response allocation was determined by multiple reinforcer variables within individual sessions, consistent with the concatenated matching law. However, in Experiment 2, there was a nonlinear (sigmoidal) relationship between response allocation and relative value, which suggests the possibility that reinforcer variables may interact during acquisition, contrary to the matching law.

Comparison of the effects of clozapine and 8-hydroxy-2-(di-n-propylamino)tetralin (8-OH-DPAT) on progressive ratio schedule performance: evidence against the involvement of 5-HT1A receptors in the behavioural effects of clozapine

RATIONALE

Performance on progressive ratio schedules has been proposed as a means of assessing the effects of drugs on the efficacy of reinforcers. A mathematical model (Killeen PR (1994) Mathematical principles of reinforcement. Behav Brain Sci 17:105-172) affords a basis for quantifying the effects of drugs on progressive ratio schedule performance. The model postulates a bitonic function relating response rate and ratio size. One parameter of the function, a, expresses the motivational effect of the reinforcer, whereas another parameter, delta, expresses the minimum time needed to execute a response, and is regarded as an index of 'motor capacity'. Previously we found that the atypical antipsychotic clozapine increased a, indicating an increase in reinforcer efficacy; a similar effect was observed with the 5-hydroxytryptamine (5-HT)(1A) receptor agonist 8-hydroxy-2-(di-n-propylamino)tetralin (8-OH-DPAT). It has been suggested that some of clozapine's behavioural effects are mediated by agonistic action at 5-HT(1A) receptors.

OBJECTIVE

This study was conducted to compare the effects of clozapine and 8-OH-DPAT on progressive ratio schedule performance.

METHODS

Rats were trained under a time-constrained progressive ratio schedule (50-min sessions). In experiment 1, they received acute doses of clozapine (4 mg kg(-1)) and 8-OH-DPAT (100 microg kg(-1)), alone and in combination with the 5-HT(1A) receptor antagonist N-[2-(4-[2-methoxyphenyl]-1-piperazinyl)ethyl]-N-2-yridinylcyclohexanecarboxamide (WAY-100635; 30 microg kg(-1)). In experiment 2, the effects of clozapine (2, 4 and 8 mg kg(-1)) and 8-OH-DPAT (25, 50 and 100 microg kg(-1)) were compared between intact rats and rats whose 5-HTergic pathways had been ablated by 5,7-dihydroxytryptamine (5,7-DHT).

RESULTS

In both experiments, clozapine and 8-OH-DPAT increased a and delta. In experiment 1, WAY-100635 abolished the effect of 8-OH-DPAT on a and delta, but did not alter clozapine's effects on these parameters. In experiment 2, the effects of clozapine and 8-OH-DPAT did not differ between sham-lesioned and 5,7-DHT-lesioned rats.

CONCLUSIONS

The results confirm previous findings on the effects of clozapine and 8-OH-DPAT on progressive ratio schedule performance. 8-OH-DPAT's effects are probably mediated by post-synaptic 5-HT(1A) receptors; clozapine's effects are mediated by a different mechanism, which does not appear to involve 5-HT(1A) receptors and which does not depend upon an intact 5-HTergic pathway.

MPR

Mathematical Principles of Reinforcement (MPR) is a theory of reinforcement schedules. This paper reviews the origin of the principles constituting MPR: arousal, association and constraint. Incentives invigorate responses, in particular those preceding and predicting the incentive. The process that generates an associative bond between stimuli, responses and incentives is called coupling. The combination of arousal and coupling constitutes reinforcement. Models of coupling play a central role in the evolution of the theory. The time required to respond constrains the maximum response rates, and generates a hyperbolic relation between rate of responding and rate of reinforcement. Models of control by ratio schedules are developed to illustrate the interaction of the principles. Correlations among parameters are incorporated into the structure of the models, and assumptions that were made in the original theory are refined in light of current data.

The principal components of response strength

As Skinner (1938) described it, response strength is the "state of the reflex with respect to all its static properties" (p. 15), which include response rate, latency, probability, and persistence. The relations of those measures to one another was analyzed by probabilistically reinforcing, satiating, and extinguishing pigeons' key pecking in a trials paradigm. Reinforcement was scheduled according to variable-interval, variable-ratio, and fixed-interval contingencies. Principal components analysis permitted description in terms of a single latent variable, strength, and this was validated with confirmatory factor analyses. Overall response rate was an excellent predictor of this state variable.

Models of ratio schedule performance

Predictions of P. R. Killeen's (1994) mathematical principles of reinforcement were tested for responding on ratio reinforcement schedules. The type of response key, the number of sessions per condition, and first vs. second half of a session had negligible effects on responding. Longer reinforcer durations and larger grain types engendered more responding, affecting primarily the parameter alpha (specific activation). Key pecking was faster than treadle pressing, affecting primarily the parameter delta (response time). Longer intertrial intervals led to higher overall response rates and shorter postreinforcement pauses and higher running rates, and ruled out some competing explanations. The treadle data required a distinction between the energetic requirements and rate-limiting properties of extended responses. The theory was extended to predict pause durations and run rates on ratio schedules.

Models of ratio schedule performance

Predictions of P. R. Killeen's (1994) mathematical principles of reinforcement were tested for responding on ratio reinforcement schedules. The type of response key, the number of sessions per condition, and first vs. second half of a session had negligible effects on responding. Longer reinforcer durations and larger grain types engendered more responding, affecting primarily the parameter alpha (specific activation). Key pecking was faster than treadle pressing, affecting primarily the parameter delta (response time). Longer intertrial intervals led to higher overall response rates and shorter postreinforcement pauses and higher running rates, and ruled out some competing explanations. The treadle data required a distinction between the energetic requirements and rate-limiting properties of extended responses. The theory was extended to predict pause durations and run rates on ratio schedules.

Effects of food-pellet size on rate, latency, and topography of autoshaped key pecks and gapes in pigeons

Four pigeons responded under autoshaping contingencies in which different conditional stimuli (red or green keylights) were associated with unconditional stimuli of different magnitudes (large or small food pellets) over successive trials within a session. Both topography (beak opening or gape) and strength (rates and latencies of key pecks and gapes) of responding during the conditional stimuli depended on the magnitude of the correlated unconditional stimulus. Key-peck and gape rates were higher and latencies were shorter in large-pellet trials than in small-pellet trials. Gape amplitudes varied directly with pellet size, although conditional and unconditional gapes were larger than either pellet. These findings were replicated when the key colors were presented either on one or two keys and after reversals of the color-size correlations. Because the unconditional stimulus was varied through pellet size, magnitude was not confounded with food-access duration or quality. These results demonstrate the effects of the magnitude of the unconditional stimulus, in that rates and latencies of both key pecks (which are directed movements toward the key) and gapes (which are independent of the bird's position and key properties) varied with pellet size. Gape measures were unique in that two dimensions (response strength and topography) of a single response class varied simultaneously with magnitude.