Effects of signaled versus unsignaled delay of reinforcement on choice

Temporal context effects on suboptimal choice

Choice can be driven both by rewards and stimuli that signal those rewards. Under certain conditions, pigeons will prefer options that lead to less probable reward when the reward is signaled. A recently quantified model, the Signal for Good News (SiGN) model, assumes that in the context of uncertainty, signals for a reduced delay to reward reinforce choice. The SiGN model provides an excellent fit to previous results from pigeons and the current studies are the first to test a priori quantitative predictions. Pigeons chose between a suboptimal alternative that led to signaled 20% food and an optimal alternative that led to 50% food. The duration of the choice period was manipulated across conditions in two experiments. Pigeons strongly preferred the suboptimal alternative at the shorter durations and strongly preferred the optimal alternative at the longer durations. The results from both experiments fit well with predictions from the SiGN model and show that altering the duration of the choice period has a dramatic effect in that it changes which of the two options pigeons prefer. More generally, these results suggest that the relative value of options is not fixed, but instead depends on the temporal context.

Outcome expectancy and suboptimal risky choice in nonhuman primates

Animals will favor a risky option when a stimulus signaling reward bridges the choice and the outcome. The present experiments investigated signal-induced risky choices and reward-outcome expectations in rhesus and capuchin monkeys. Risky choice was assessed by preference for a large-probabilistic reward over a modest-certain reward. Outcome expectancy was assessed by providing a truncation-response to shorten the delay period. In Experiment 1 both species generally favored the risky option compared to a safe option when the outcomes were signaled and generally shortened the delays except when a signaled-loss stimulus was presented. The use of the delay-truncation response suggested that the monkeys were sensitive to the information conveyed by the stimulus. Experiments 2 and 3 were designed to investigate whether the delay-truncation response used by capuchin monkeys was strategically used reflecting explicit decision-making versus a conditioned response to reward stimuli. A perceptual judgment task was included and the selective use of the delay-truncation response on unsignaled correct trials may suggest the involvement of metacognitive processes. The capuchin monkeys generally truncated the delays except under conditions where reward would not be expected (risky-loss or incorrect-judgment). When the outcomes were unsignaled during the delay some capuchin monkeys were less likely to truncate the delay following an incorrect task response. Overall, the monkeys: (1) made more risky choices when the outcomes were signaled - consistent with gambling-like behavior. (2) selectively truncated the unsignaled delays when rewards could be anticipated (even when metacognitive-like awareness guided anticipation) - suggesting that delay truncation responses reflect explicit outcome expectancy.

What reward does a child prefer for behaving well at the dentist?

Background

Paediatric dentists often report using positive reinforcement to encourage their young patients to show co-operative behaviour. For effective reinforcement to take place the reward should be salient to the individual. To date, there is little research into what reward a young patient will choose when attending the dentist.

Aim

To identify what reward children between the age of 4-8 years will choose when attending the dentist, and to determine the extent of agreement between children and caregivers in reward choice.

Method

Observational study. Fifty-two children from different age groups (4-5 years, 6-7 years and 8 years) attending a primary-care dental clinic were asked to choose between a range of different rewards. The caregiver attending with them was also asked to anticipate the child's preferred choice.

Results

There was no clear favourite reward for children from both genders and different age group. However, no child chose the 'sticker' reward that is traditionally given out at the dentist. Overall carers agreed with the child's choice of toy on 18 occasions (34.6%), but there were significant differences across the age groups with carers of older children showing less agreement.

Conclusion

To ensure that rewards are salient, children should be given a choice of rewards when attending the dental clinic. Parents ability to predict their child's preferred rewards decreases as the child ages.

Clinical relevance

A child's motivation to co-operate during dental treatment can be increased by offering a range of rewards. Asking children to choose their reward from a limited range will increase the saliency of the reward for the child.

When good news leads to bad choices

Pigeons and other animals sometimes deviate from optimal choice behavior when given informative signals for delayed outcomes. For example, when pigeons are given a choice between an alternative that always leads to food after a delay and an alternative that leads to food only half of the time after a delay, preference changes dramatically depending on whether the stimuli during the delays are correlated with (signal) the outcomes or not. With signaled outcomes, pigeons show a much greater preference for the suboptimal alternative than with unsignaled outcomes. Key variables and research findings related to this phenomenon are reviewed, including the effects of durations of the choice and delay periods, probability of reinforcement, and gaps in the signal. We interpret the available evidence as reflecting a preference induced by signals for good news in a context of uncertainty. Other explanations are briefly summarized and compared.

Local effects of delayed food

Five pigeons were trained on a procedure in which seven concurrent variable-interval schedules arranged seven different food-rate ratios in random sequence in each session. Each of these components lasted for 10 response-produced food deliveries, and components were separated by 10-s blackouts. We varied delays to food (signaled by blackout) between the two response alternatives in an experiment with three phases: In Phase 1, the delay on one alternative was 0 s, and the other was varied between 0 and 8 s; in Phase 2, both delays were equal and were varied from 0 to 4 s; in Phase 3, the two delays summed to 8 s, and each was varied from 1 to 7 s. The results showed that increasing delay affected local choice, measured by a pulse in preference, in the same way as decreasing magnitude, but we found also that increasing the delay at the other alternative increased local preference. This result casts doubt on the traditional view that a reinforcer strengthens a response depending only on the reinforcer's value discounted by any response-reinforcer delay. The results suggest that food guides, rather than strengthens, behavior.

The role of terminal-link stimuli in concurrent-chain schedules: revisited using a behavioral-history procedure

A behavioral-history procedure was used to study the function of terminal-link stimuli as conditioned reinforcers in multiple concurrent-chain schedules of reinforcement. First, three pigeons were exposed to multiple concurrent-chain schedules in which the two multiple-schedule components were correlated with a blue and a white stimulus, respectively. In each component the initial links were equal independent variable-interval (VI) 15 s schedules. A fixed-interval (FI) 10 s schedule operated on the red key in one terminal link while extinction operated on the green key in the alternative terminal link. When large preferences for the red stimulus had been established, two tests were conducted. In the terminal-link test, under new initial-link stimuli--purple and brown--an FI 10 s schedule operated for both the red and green terminal-link stimuli. In the subsequent initial-link test, the blue and white initial-link stimuli were reintroduced, and, as in the terminal-link test, FI 10s operated for both the red and the green terminal-link stimuli. In the terminal-link test, the three pigeons showed no preference for the terminal links with the red stimulus, but showed clear and consistent preferences for the red stimulus when blue and white stimuli were reintroduced as initial-link stimuli in the initial-link test. This suggests that there are multiple sources of control over initial-link response allocation in concurrent-chains, including control by both terminal- and initial-link stimuli.

Small and large fixed ratios with the same unit price

Key pressing of rats was maintained under multiple and discrete-trial choice schedules with reinforcer units of 45 mg food pellets or 3.5 s dips of sucrose solution. Both smaller and larger fixed ratio (FR) schedules were associated with the same unit price in a manner, for example, that each of eight iterations of FR120 was associated with delivery of a single reinforcer unit and one instance of FR960 was associated with eight reinforcer units. FR requirement varied between 20 and 1560 per aggregate reinforcer and unit price varied between 20 and 240 per reinforcer unit. During multiple schedules with food reinforcers, rates and patterns of responding were comparable over nearly a 50-fold range of FR requirements (20-1380) when unit price was 20; over nearly a six-fold range of FR requirements (120-720) when unit price was 120; and was only marginally maintained when unit price was 240. Demand for food pellets was comparatively inelastic at FRs between 20 and 120, during which subjects did not receive supplemental feeding outside experimental sessions, but was elastic at FRs greater than 240, when subjects sometimes did receive supplemental feeding. In a discrete-trial choice procedure with a constant unit price of 120 for sucrose solution, subjects were indifferent between smaller FRs and alternative FRs as large as 480, but began switching away from larger FRs that were 600 or greater. Because responding had been comparably maintained under both FR120 and FRs as large as 960 in the multiple schedule, results from the choice procedure indicated that choice performance was influenced by variables other than FR requirement and unit price. Because aggregate reinforcers were the same for smaller and larger FRs, the most likely reason for preferring smaller FRs was the nearness in time to some reinforcer.

Challenges to timing-based theories of operant behavior

Recent theories of behavior have proposed that associative learning principles be replaced by a theoretical framework that assumes the animal has a veridical record of the temporal relations between events. I argue here that such a theory omits critical features of learned behavior: functional differences between different types of temporal relations, the critical nature of response-reinforcer delays, and the necessity of conditioned value as a theoretical construct.