Uncertainty reduction, conditioned reinforcement, and observing

When good news leads to bad choices

Pigeons and other animals sometimes deviate from optimal choice behavior when given informative signals for delayed outcomes. For example, when pigeons are given a choice between an alternative that always leads to food after a delay and an alternative that leads to food only half of the time after a delay, preference changes dramatically depending on whether the stimuli during the delays are correlated with (signal) the outcomes or not. With signaled outcomes, pigeons show a much greater preference for the suboptimal alternative than with unsignaled outcomes. Key variables and research findings related to this phenomenon are reviewed, including the effects of durations of the choice and delay periods, probability of reinforcement, and gaps in the signal. We interpret the available evidence as reflecting a preference induced by signals for good news in a context of uncertainty. Other explanations are briefly summarized and compared.

Incentive theory: II. Models for choice

Incentive theory is extended to account for concurrent chained schedules of reinforcement. The basic model consists of additive contributions from the primary and secondary effects of reinforcers, which serve to direct the behavior activated by reinforcement. The activation is proportional to the rate of reinforcement and interacts multiplicatively with the directive effects. The two free parameters are q, the slope of the delay of reinforcement gradient, whose value is constant across many experiments, and b, a bias parameter. The model is shown to provide an excellent description of all results from studies that have varied the terminal-link schedules, and of many of the results from studies that have varied initial-link schedules. The model is extended to diverse modifications of the terminal links, such as varied amount of reinforcement, varied signaling of the terminal-link schedules, and segmentation of the terminal-link schedules. It is demonstrated that incentive theory provides an accurate and integrated account of many of the phenomena of choice.

Preference for multiple versus mixed schedules of reinforcement

Five pigeons were trained in a concurrent-chain procedure. In the initial links, equal nonindependent variable-interval schedules were available concurrently on two keys. Completing the schedule on either key led to exclusive presentation of one of two further variable-interval schedules for a fixed period of time. During these terminal links, as many reinforcers as were scheduled could be obtained. If the response producing this terminal link occurred on one key, differential stimuli signaled which variable-interval schedule had been produced. If the response producing the terminal link occurred on the other key, no such differential stimuli were available. Once the fixed period of time elapsed, the initial links were reinstated. In Experiment 1, the period of time for which the terminal links were available was always 10 s and the absolute duration of the initial links was varied. Subjects preferred the alternative leading to the multiple schedule when the initial-link duration was short, but preferred the alternative leading to the mixed schedule when the initial-link durations were longer. In Experiment 2, both the initial-link duration and the duration of the terminal links were varied. The effect of initial-link duration was identical to that in Experiment 1 and there was no systematic effect of varying the terminal-link duration.

Choice and multiple reinforcers

Pigeons chose between equivalent two-component mixed and multiple terminal-link schedules of reinforcement in the concurrent-chains procedure. The pigeons preferred the multiple schedule over the mixed when the components of the compound schedules were differentiated in terms of density of reinforcement, but the pigeons were indifferent when the components were differentiated in terms of number of reinforcers per cycle. Taken together, these results indicate that a local variable, the interval to the first reinforcer, but not a molar variable, the number of reinforcers, was sufficient to differentiate the components and thereby evoke preference.

Free-choice preference under uncertainty

Preferences in pigeons for free choice over forced choice under uncertain contingencies were compared with the one under certain contingencies in multiple concurrent-chain schedules of reinforcement. The uncertain condition examined the preference for two alternatives over one alternative when reinforcement probability at the end of the terminal link equalled 0.5, and with all keys in each terminal link lit green (two mixed fixed-interval extinction keys versus one mixed fixed-interval extinction key). Key and schedule arrangement in the certain condition was the same except that a peck on any terminal-link key after the FI interval always produced food. When naive pigeons were first exposed to uncertain contingencies, preference for two lit keys over one lit key was observed, and the preference was confirmed by sequential reversals of the terminal-link contingencies. However, no consistent preference was observed when uncertain condition followed the certain condition. Under certain contingencies, unlike the earlier experiments, very small and inconsistent preferences for free choice were demonstrated. A possible reason for the different preferences in the uncertain conditions was that pigeons may lessen their sensitivity to the circumstances with uncertainty by any history or carry-over effect of the prior contingency.

Sample-duration effects on pigeons' delayed matching as a function of predictability of duration

Three experiments assessed the impact of sample duration on pigeons' delayed matching as a function of whether or not the samples themselves signaled how long they would remain on. When duration was uncorrelated with the sample appearing on each matching trial, the typical effect of duration was observed: Choice accuracy was higher with long (15-s) than with short (5-s) durations. By contrast, this difference either disappeared or reversed when the 5- and 15-s durations were correlated with the sample stimuli. Sample duration itself cued comparison choice by some birds in the latter (predictable) condition when duration was also correlated with the reinforced choice alternatives. However, even when duration could not provide a cue for choice, pigeons matched predictably short-duration samples as accurately as, or more accurately than, predictably long-duration samples. Moreover, this result was observed independently of whether the contextual conditions of the retention interval were the same as, or different from, those of the intertrial interval. These results strongly support the view that conditional stimulus control by the samples is partly a function of their conditioned reinforcing properties, as determined by the relative reduction in overall delay to reinforcement that they signal.

Choice with uncertain outcomes: conditioned reinforcement effects

Pigeons responded on concurrent chains with equal initial- and terminal-link durations. In all conditions, the terminal links of one chain ended reliably in reinforcement; the terminal links on the alternative chain ended in either food or blackout. In Experiment 1, the terminal-link stimuli were correlated with (signaled) the outcome, and the durations of the initial and terminal links were varied across conditions. Preference did not vary systematically across conditions. In Experiment 2, terminal-link durations were varied under different stimulus conditions. The initial links were variable-interval 80-s schedules. Preference for the reliable alternative was generally higher in unsignaled than in signaled conditions. Preference increased with terminal-link durations only in the unsignaled conditions. There were no consistent differences between conditions with and without a common signal for reinforcement on the two chains. In the first series of conditions in Experiment 3, a single response was required in the initial links, and the stimulus conditions during 50-s terminal links were varied. Preference for the reliable outcome approached 1.0 in unsignaled conditions and was considerably lower (below .50 for 3 of 5 subjects) in signaled conditions. In a final series of signaled conditions with relatively long terminal links, preference varied with duration of the initial links. The results extend previous findings and are discussed in terms of the delay reduction signaled by terminal-link stimuli.