Observing and conditioned reinforcement

Negative effects of positive reinforcement

Procedures classified as positive reinforcement are generally regarded as more desirable than those classified as aversive-those that involve negative reinforcement or punishment. This is a crude test of the desirability of a procedure to change or maintain behavior. The problems can be identified on the basis of theory, experimental analysis, and consideration of practical cases. Theoretically, the distinction between positive and negative reinforcement has proven difficult (some would say the distinction is untenable). When the distinction is made purely in operational terms, experiments reveal that positive reinforcement has aversive functions. On a practical level, positive reinforcement can lead to deleterious effects, and it is implicated in a range of personal and societal problems. These issues challenge us to identify other criteria for judging behavioral procedures.

Analyzing the reinforcement process at the human level: can application and behavioristic interpretation replace laboratory research?

Critics have questioned the value of human operant conditioning experiments in the study of fundamental processes of reinforcement. Contradictory results from human and animal experiments have been attributed to the complex social and verbal history of the human subject. On these grounds, it has been contended that procedures that mimic those conventionally used with animal subjects represent a "poor analytic preparation" for the explication of reinforcement principles. In defending the use of conventional operant methods for human research, we make three points: (a) Historical variables play a critical role in research on processes of reinforcement, regardless of whether the subjects are humans or animals. (b) Techniques are available for detecting, analyzing, and counteracting such historical and extra-experimental influences; these include long-term observations, steady state designs, and, when variables are not amenable to direct control (e.g., age, gender, species), selection of subjects with common characteristics. (c) Other forms of evidence that might be used to validate conditioning principles-applied behavior analysis and behavioristic interpretation-have inherent limitations and cannot substitute for experimental analysis. We conclude that human operant conditioning experiments are essential for the analysis of the reinforcement process at the human level, but caution that their value depends on the extent to which the traditional methods of the experimental analysis of behavior are properly applied.

Stimulus control: part I

In his effort to distinguish operant from respondent conditioning, Skinner stressed the lack of an eliciting stimulus and rejected the prevailing stereotype of Pavlovian "stimulus-response" psychology. But control by antecedent stimuli, whether classified as conditional or discriminative, is ubiquitous in the natural setting. With both respondent and operant behavior, symmetrical gradients of generalization along unrelated dimensions may be obtained following differential reinforcement in the presence and the absence of the stimulus. The slopes of these gradients serve as measures of stimulus control, and they can be steepened without applying differential reinforcement to any two points along the test dimension. Increases and decreases in stimulus control occur under the same conditions as those leading to increases and decreases in observing responses, indicating that it is the increasing frequency and duration of observation (and perhaps also of attention) that produces the separation in performances during discrimination learning.

Stimulus control: part II

The second part of my tutorial stresses the systematic importance of two parameters of discrimination training: (a) the magnitude of the physical difference between the positive and the negative stimulus (disparity) and (b) the magnitude of the difference between the positive stimulus, in particular, and the background stimulation (salience). It then examines the role these variables play in such complex phenomena as blocking and overshadowing, progressive discrimination training, and the transfer of control by fading. It concludes by considering concept formation and imitation, which are important forms of application, and recent work on equivalence relations.

Choice, conditioned reinforcement, and the prius effect

Psychologists have long been intrigued with the rationales that underlie our decisions. Similarly, the concept of conditioned reinforcement has a venerable history, particularly in accounting for behavior not obviously maintained by primary reinforcers. The studies of choice and of conditioned reinforcement have often developed in lockstep. Many contemporary approaches to these fundamental topics share an emphasis on context and on relative value. We trace the evolution of thinking about the potency of conditioned reinforcers from stimuli that were thought to acquire their value from pairings with more fundamental reinforcers to stimuli that acquire their value by being differentially correlated with these more fundamental reinforcers. We discuss some seminal experiments (including several that have been underappreciated) and some ongoing data, all of which have propelled us to the conclusion that the strength of conditioned reinforcers is determined by their signaling a relative improvement in the organism's relation to reinforcement.

Lateral habenula neurons signal errors in the prediction of reward information

Humans and animals have the ability to predict future events, which they cultivate by continuously searching their environment for sources of predictive information. However, little is known about the neural systems that motivate this behavior. We hypothesized that information-seeking is assigned value by the same circuits that support reward-seeking, such that neural signals encoding reward prediction errors (RPEs) include analogous information prediction errors (IPEs). To test this, we recorded from neurons in the lateral habenula, a nucleus that encodes RPEs, while monkeys chose between cues that provided different chances to view information about upcoming rewards. We found that a subpopulation of lateral habenula neurons transmitted signals resembling IPEs, responding when reward information was unexpectedly cued, delivered or denied. These signals evaluated information sources reliably, even when the monkey's decisions did not. These neurons could provide a common instructive signal for reward-seeking and information-seeking behavior.

Sub-Optimal Choice by Pigeons: Failure to Support The Allais Paradox

Pigeons show a preference for an alternative that provides them with discriminative stimuli (sometimes a stimulus that predicts reinforcement and at other times a stimulus that predicts the absence of reinforcement) over an alternative that provides them with non discriminative stimuli, even if the non discriminative stimulus alternative is associated with 2.5 times as much reinforcement (Stagner & Zentall, 1910). In Experiment 1 we found that the delay to reinforcement associated with the non discriminative stimuli could be reduced by almost one half before the pigeons were indifferent between the two alternatives. In Experiment 2 we tested the hypothesis that the preference for the discriminative stimulus alternative resulted from the fact that, like humans, the pigeons were attracted by the stimulus that consistently predicted reinforcement (the Allais paradox). When the probability of reinforcement associated with the discriminative stimulus that predicted reinforcement was reduced from 100% to 80% the pigeons still showed a strong preference for the discriminative stimulus alternative. Thus, under these conditions, the Allais paradox cannot account for the sub-optimal choice behavior shown by pigeons. Instead we propose that sub-optimal choice results from positive contrast between the low expectation of reinforcement associated with the discriminative stimulus alternative and the much higher obtained reinforcement when the stimulus associated with reinforcement appears. We propose that similar processes can account for sub-optimal gambling behavior by humans.

Suboptimal choice behavior by pigeons

Contrary to the law of effect and optimal foraging theory, pigeons show suboptimal choice behavior by choosing an alternative that provides 20% reinforcement over another that provides 50% reinforcement. They choose the 20% reinforcement alternative--in which 20% of the time, that choice results in a stimulus that always predicts reinforcement, and 80% of the time, it results in another stimulus that predicts its absence--rather than the 50% reinforcement alternative, which results in one of two stimuli, each of which predicts reinforcement 50% of the time. This choice behavior may be related to suboptimal human monetary gambling behavior, because in both cases, the organism overemphasizes the infrequent occurrence of the winning event and underemphasizes the more frequent occurrence of the losing event.

Revisiting the role of bad news in maintaining human observing behavior

Results from studies of observing responses have suggested that stimuli maintain observing owing to their special relationship to primary reinforcement (the conditioned-reinforcement hypothesis), and not because they predict the availability and nonavailability of reinforcement (the information hypothesis). The present article first reviews a study that challenges that conclusion and then reports a series of five brief experiments that provide further support for the conditioned-reinforcement view. In Experiments 1 through 3, participants preferred occasional good news (a stimulus correlated with reinforcement) or no news (a stimulus uncorrelated with reinforcement) to occasional bad news (a stimulus negatively correlated with reinforcement). In Experiment 4 bad news was preferred to no news when the absence of stimulus change following a response to the bad-news option was reliably associated with good news. When this association was weakened in Experiment 5 the results were intermediate. The results support the conclusion that information is reinforcing only when it is positive or useful. As required by the conditioned-reinforcement hypothesis, useless information does not maintain observing.

Maladaptive "gambling" by pigeons

When humans buy a lottery ticket or gamble at a casino they are engaging in an activity that on average leads to a loss of money. Although animals are purported to engage in optimal foraging behavior, similar sub-optimal behavior can be found in pigeons. They show a preference for an alternative that is associated with a low probability of reinforcement (e.g., one that is followed by a red hue on 20% of the trials and then reinforcement or by a green hue on 80% of the trials and then the absence of reinforcement) over an alternative that is associated with a higher probability of reinforcement (e.g., blue or yellow each of which is followed by reinforcement 50% of the time). This effect appears to result from the strong conditioned reinforcement associated with the stimulus that is always followed by reinforcement. Surprisingly, although it is experienced four times as much, the stimulus that is never followed by reinforcement does not appear to result in significant conditioned inhibition (perhaps due to the absence of observing behavior). Similarly, human gamblers tend to overvalue wins and undervalue losses. Thus, this animal model may provide a useful analog to human gambling behavior, one that is free from the influence of human culture, language, social reinforcement, and other experiential biases that may influence human gambling behavior.

Maladaptive choice behaviour by pigeons: an animal analogue and possible mechanism for gambling (sub-optimal human decision-making behaviour)

Consistent with human gambling behaviour but contrary to optimal foraging theory, pigeons showed maladaptive choice behaviour in experiment 1 by choosing an alternative that provided on average two food pellets over an alternative that provided a certain three food pellets. On 20 per cent of the trials, choice of the two-pellet alternative resulted in a stimulus that always predicted ten food pellets; on the remaining 80 per cent of the trials, the two-pellet alternative resulted in a different stimulus that always predicted zero food pellets. Choice of the three-pellet alternative always resulted in three food pellets. This choice behaviour mimics human monetary gambling in which the infrequent occurrence of a stimulus signalling the winning event (10 pellets) is overemphasized and the more frequent occurrence of a stimulus signalling the losing event (zero pellets) is underemphasized, compared with the certain outcome associated with not gambling (the signal for three pellets). In experiment 2, choice of the two-pellet alternative resulted in ten pellets with a probability of 20 per cent following presentation of either stimulus. Choice of the three-pellet alternative continued to result in three food pellets. In this case, the pigeons reliably chose the alternative that provided a certain three pellets over the alternative that provided an average of two pellets. Thus, in experiment 1, the pigeons were responding to obtain the discriminative stimuli signalling reinforcement and the absence of reinforcement, rather than to obtain the variability in reinforcement.

Pavlovian-instrumental interaction in 'observing behavior'

Subjects typically choose to be presented with stimuli that predict the existence of future reinforcements. This so-called ‘observing behavior’ is evident in many species under various experimental conditions, including if the choice is expensive, or if there is nothing that subjects can do to improve their lot with the information gained. A recent study showed that the activities of putative midbrain dopamine neurons reflect this preference for observation in a way that appears to challenge the common prediction-error interpretation of these neurons. In this paper, we provide an alternative account according to which observing behavior arises from a small, possibly Pavlovian, bias associated with the operation of working memory.

The theory of Reinforcement Learning (RL) has been influential in explaining basic learning and behavior in humans and other animals, and in accounting for key features of the activity of dopamine neurons. However, perhaps due to this very success, paradigms that challenge RL are at a premium. One case concerns so-called ‘observing behavior’, in which, at least in some versions, animals elect to observe cues that are predictive of future rewarding outcomes, although the observations themselves have no direct behavioral relevance. In a recent experiment on observing, the activity of monkey dopaminergic neurons was also found to be incompatible with classic RL. However, as is often the case, this was a task that allowed for potential interactions from a secondary behavioral system in which responses are directly triggered by values. In this paper we show that a model incorporating a next order of refinement associated with such Pavlovian interactions can explain this type of observing behavior.

Observing responses and serial stimuli: searching for the reinforcing properties of the S-

The control exerted by a stimulus associated with an extinction component (S-) on observing responses was determined as a function of its temporal relation with the onset of the reinforcement component. Lever pressing by rats was reinforced on a mixed random-interval extinction schedule. Each press on a second lever produced stimuli associated with the component of the schedule in effect. In Experiment 1 a response-dependent clock procedure that incorporated different stimuli associated with an extinction component of a variable duration was used. When a single S- was presented throughout the extinction component, the rate of observing remained relatively constant across this component. In the response-dependent clock procedure, observing responses increased from the beginning to the end of the extinction component. This result was replicated in Experiment 2, using a similar clock procedure but keeping the number of stimuli per extinction component constant. We conclude that the S- can function as a conditioned reinforcer, a neutral stimulus or as an aversive stimulus, depending on its temporal location within the extinction component.

Control of behavior by an establishing stimulus

Seventeen pigeons were exposed to a three-key discrete-trial procedure in which a peck on the lit center key produced food if, and only if, the left keylight was lit. The center key was illuminated by a peck on the lit right key. Of interest was whether subjects pecked the right key before or after the response-independent onset of the left keylight. Pecks on the right key after left-keylight onset suggest control of behavior by the left keylight-an establishing stimulus. In three experiments, the strength of center-keylight onset as conditioned reinforcer for a response on the right key was manipulated by altering the size of the reduction in time to food delivery correlated with its onset. Control of pigeons' key pecks by onset of the left keylight occurred on more trials per session when the center keylight was a relatively weak conditioned reinforcer and on fewer trials per session when the center keylight was a relatively strong condtioned reinforcer. Differences across conditions in the degree of control by onset of the establishing stimulus were greatest when changes in conditioned reinforcer strength occurred relatively frequently and were signaled. The results provide evidence of the function of an establishing stimulus.

On conditioned reinforcing effects of negative discriminative stimuli

Observing responses by pigeons were studied during sessions in which a food key and an observing key were available continuously. A variable-interval schedule and extinction alternated randomly on the food key. In one condition, food-key pecking during extinction decreased reinforcement frequency during the next variable-interval component, and in the other condition such pecking did not affect reinforcement frequency. Observing responses either changed both keylight colors from white to green (S+) or to red (S-) depending on the condition on the food key, or the observing responses never produced the S+ but produced the S- when extinction was in effect on the food key. Observing responses that produced only S- were maintained only when food-key pecking during extinction decreased reinforcement frequency in the subsequent variable-interval component. The red light conformed to conventional definitions of a negative discriminative stimulus, rendering results counter to previous findings that production of S- alone does not maintain observing. Rather than offering support for an informational account of conditioned reinforcement, the results are discussed in terms of a molar analysis to account for how stimuli acquire response-maintaining properties.

Delay and number of food reinforcers: Effects on choice and latencies

Pigeons were given a choice between two identical-duration situations (terminal links of chain schedules). One terminal link of the choice pair provided two food deliveries, and the other provided five. The exact times of these food deliveries differed between the terminal links and were varied over conditions. A single response during the initial link gave immediate access to the corresponding terminal link. Forced trials, during which only one of the initial-link keys was lighted, were interspersed with choice trials during which both initial-link keys were lighted. Choice tended to favor whichever terminal link was correlated with the higher sum of the immediacies (i.e., the sum of the reciprocals of the delays to each of the reinforcers following the choice, with all delays measured from the choice). Latencies on forced trials and on choice trials also were related (negatively) to the sum of the immediacies. This correlation among response measures (choice and latencies) suggests that both measures are manifestations of the effect of conditioned reinforcement on response tendencies.

Conditioned reinforcement of human observing behavior by descriptive and arbitrary verbal stimuli

College students earned monetary reinforcers by pressing a key according to a compound schedule with variable-interval and extinction components. Pressing additional keys occasionally produced displays of either of two verbal stimuli; one was uncorrelated with the schedule components, and the other was correlated with the extinction component. In Experiments 1 and 2, the display area of the apparatus was blank unless an observing key was pressed, whereupon a descriptive message appeared. Most students preferred an uncorrelated stimulus stating that "Some of this time scores are TWICE AS LIKELY as normal, and some of this time NO SCORES can be earned" over a stimulus stating that "At this time NO SCORES can be earned." In Experiment 3, the display area indicated that "The Current Status of the Program is: NOT SHOWN." Presses on the observing keys replaced this message with stimuli that provided arbitrary labels for the schedule conditions. All of the students preferred a stimulus stating that "The Current Status of the Program is: B" over an uncorrelated stimulus stating that "The Current Status of the Program is: either A or B." Thus, under some circumstances, observing was maintained by a stimulus correlated with extinction-a finding that poses a challenge for Pavolvian accounts of conditioned reinforcement. Differences in the maintenance of observing by the descriptive and arbitrary stimuli may be attributed to differences in either the strength or nature of the instructional control exerted by the verbal stimuli.

Information on response requirements compared with information on food density as a reinforcer of observing in pigeons

On a variable-interval schedule, pecking the key to the pigeon's right (observing response) produced red or green displays relating to the delivery of grain and its dependence on pecking the key to the left (food key). During various blocks of sessions, mixed (no stimulus change) schedules including the following pairs of components were temporarily converted by the observing response to their corresponding multiple (correlated stimuli) schedules: variable-interval 60-s, extinction; variable-interval 60-s, variable-time (response-independent) 60-s; extinction, variable-time 60-s. Differences in food delivery maintained substantial rates of responding on the observing key, without regard to pecking requirements on the food key. Although stimuli correlated with differences in the response requirement on the food key maintained higher observing rates than those maintained by uncorrelated stimuli, they were much lower than those based on food. The value of predictive stimuli as reinforcers is determined by the value of the events predicted. In particular, the cost of pecking appears to be low, and this may place limitations on the applicability of energy-based and economic models of behavior.

Verbal self-reports of delayed matching to sample by humans

Undergraduates participated in two experiments to develop methods for the experimental analysis of self-reports about behavior. The target behavior was the choice response in a delayed-matching-to-sample task in which monetary reinforcement was contingent upon both speed and accuracy of the choice. In Experiment 1, the temporal portion of the contingency was manipulated within each session, and the presence and absence of feedback about reinforcement was manipulated across sessions. As the time limits became stricter, target response speeds increased, but accuracy and reinforcement rates decreased. When feedback was withheld, further reductions in speed and reinforcement occurred, but only at the strictest time limit. Thus, the procedures were successful in producing systematic variation in the speed, accuracy, and reinforcement of the target behavior. Experiment 2 was designed to assess the influence of these characteristics on self-reports. In self-report conditions, each target response was followed by a computer-generated query: "Did you earn points?" The subject reported by pressing "Yes" or "No" buttons, with the sole consequence of advancing the session. In some cases, feedback about reinforcement of the target response followed the reports; in other cases it was withheld. Self-reports were less accurate when the target responses occurred under greater time pressure. When feedback was withheld, the speed of the target response influenced reports, in that the probability of a "Yes" report increased directly with the speed of accurate target responses. In addition, imposing the self-report procedure disrupted target performance by reducing response speeds at the strictest time limit. These results allow investigation of issues in both behavioral and cognitive psychology. More important, the overall order in the data suggests promise for the experimental analysis of self-reports by human subjects.

Basic research needed for stimulating the development of behavioral technologies

The costs of disconnection between the basic and applied sectors of behavior analysis are reviewed, and some solutions to these problems are proposed. Central to these solutions are collaborations between basic and applied behavioral scientists in programmatic research that addresses the behavioral basis and solution of human behavior problems. This kind of collaboration parallels the deliberate interactions between basic and applied researchers that have proven to be so profitable in other scientific fields, such as medicine. Basic research questions of particular relevance to the development of behavioral technologies are posed in the following areas: response allocation, resistance to change, countercontrol, formation and differentiation/discrimination of stimulus and response classes, analysis of low-rate behavior, and rule-governed behavior. Three interrelated strategies to build connections between the basic and applied analysis of behavior are identified: (a) the development of nonhuman animal models of human behavior problems using operations that parallel plausible human circumstances, (b) replication of the modeled relations with human subjects in the operant laboratory, and (c) tests of the generality of the model with actual human problems in natural settings.

Escape from serial stimuli leading to food

If the functional relations governing the strength of a conditioned reinforcer correspond to those obtained with other Pavlovian procedures (e.g., Kaplan, 1984), the termination of stimuli appearing early in the interval between successive food deliveries should be reinforcing. During initial training we presented four key colors, followed by food, in a recurrent sequence to each of 6 pigeons. This established a baseline level of autoshaped pecking. In later sessions, we terminated each of these colors or only the first color for a brief period following each peck, replacing the original color with a standard substitute to avoid darkening the key. Pecking decreased in the presence of the last color in the sequence but increased in the presence of the first. In accord with contemporary models of Pavlovian conditioning, these and other data suggest that the behavioral effects of stimuli in a chain may be better understood in terms of what each stimulus predicts, as measured by relative time to the terminal reinforcer, than in the exclusively positive terms of the traditional formulation (Skinner, 1938). The same model may also account for the initial pause under fixed-interval and fixed-ratio schedules of reinforcement.

The effect of negative stimulus presentations on observing-response rates

Theories of observing differ in predicting whether or not a signal for absence of reinforcement (S-) is capable of reinforcing observing responses. Experiments in which S- was first removed from and then restored to the procedure have yielded mixed results. The present experiments suggest that failure to control for the direct effect of presenting S- may have been responsible. Pigeons and operant procedures were used. Experiment 1 showed that presentations of S-, even when not contingent on observing, can raise the rate of an observing response that was reinforced only by presentations of a signal (S+) that accompanied a schedule of food delivery. Experiment 2 showed that this effect resulted from bursts of responding that followed offsets of S-. Experiment 3 showed that, when the presence of S- was held constant, lower rates occurred when S- was dependent on, rather than independent of, observing. These results support theories that characterize S- as incapable of reinforcing observing responses.

Observing responses: maintained by good news only?

Observing responses are those that produce stimuli correlated with the availability (S+) or non-availability (S-) of reinforcement but that has no influence on the actual delivery or timing of reinforcement. Prior research has shown that observing is maintained by the occasional production of the S+ ("good news") and not by production of the equally informative S- ("bad news"). However, for both humans and rats the S- maintains observing when it is at least implicitly correlated with good news. In the present study, pigeons could obtain both good and bad news by responding during the appropriate key color. In one condition, the bad news was actually more informative about reinforcement than was the good news. Nevertheless, a preponderance of the birds' responses was made on the nominally good-news option. The present results offer further support for the central role of good news in maintaining observing responses and are entirely consistent with the traditional conditioned-reinforcement (or classical conditioning) interpretation of observing.

Conditioned reinforcement and response strength

Stimuli associated with primary reinforcers appear themselves to acquire the capacity to strengthen behavior. This paper reviews research on the strengthening effects of conditioned reinforcers within the context of contemporary quantitative choice theories and behavioral momentum theory. Based partially on the finding that variations in parameters of conditioned reinforcement appear not to affect response strength as measured by resistance to change, long-standing assertions that conditioned reinforcers do not strengthen behavior in a reinforcement-like fashion are considered. A signposts or means-to-an-end account is explored and appears to provide a plausible alternative interpretation of the effects of stimuli associated with primary reinforcers. Related suggestions that primary reinforcers also might not have their effects via a strengthening process are explored and found to be worthy of serious consideration.

Choice, optimal foraging, and the delay-reduction hypothesis

Behaving organisms are continually choosing. Recently the theoretical and empirical study of decision making by behavioral ecologists and experimental psychologists have converged in the area of foraging, particularly food acquisition. This convergence has raised the interdisciplinary question of whether principles that have emerged from the study of decision making in the operant conditioning laboratory are consistent with decision making in naturally occurring foraging. One such principle, the “parameter-free delay-reduction hypothesis, ” developed in studies of choice in the operant conditioning laboratory, states that the effectiveness of a stimulus as a reinforcer may be predicted most accurately by calculating the decrease in time to food presentation correlated with the onset of the stimulus, relative to the length of time to food presentation measured from the onset of the preceding stimulus. Since foraging involves choice, the delay-reduction hypothesis may be extended to predict aspects of foraging. We discuss the strategy of assessing parameters of foraging with operant laboratory analogues to foraging. We then compare the predictions of the delay-reduction hypothesis with those of optimal foraging theory, developed by behavioral ecologists, showing that, with two exceptions, the two positions make comparable predictions. The delay-reduction hypothesis is also compared to several contemporary pscyhological accounts of choice. Results from several of our experiments with pigeons, designed as operant conditioning simulations of foraging, have shown the following: The more time subjects spend searching for or traveling between potential food sources, the less selective they become, that is, the more likely they are to accept the less preferred outcome; increasing time spent procuring (“handling”) food increases selectivity; how often the preferred outcome is available has a greater effect on choice then how often the less preferred outcome is available; subjects maximize reinforcement whether it is the rate, amount, or probability of reinforcement that is varied; there are no significant differences between subjects performing under different types of deprivation (open vs. closed economies). These results are all consistent with the delay-reduction hypothesis. Moreover, they suggest that the technology of the operant conditioning laboratory may have fruitful application in the study of foraging, and, in doing so, they underscore the importance of an interdisciplinary approach to behavior.