The operant reserve: A computer simulation in (accelerated) real time

Theory of reinforcement schedules

The three principles of reinforcement are (1) events such as incentives and reinforcers increase the activity of an organism; (2) that activity is bounded by competition from other responses; and (3) animals approach incentives and their signs, guided by their temporal and physical conditions, together called the "contingencies of reinforcement." Mathematical models of each of these principles comprised mathematical principles of reinforcement (MPR; Killeen, 1994). Over the ensuing decades, MPR was extended to new experimental contexts. This article reviews the basic theory and its extensions to satiation, warm-up, extinction, sign tracking, pausing, and sequential control in progressive-ratio and multiple schedules. In the latter cases, a single equation balancing target and competing responses governs behavioral contrast and behavioral momentum. Momentum is intrinsic in the fundamental equations, as behavior unspools more slowly from highly aroused responses conditioned by higher rates of incitement than it does from responses from leaner contexts. Habits are responses that have accrued substantial behavioral momentum. Operant responses, being predictors of reinforcement, are approached by making them: The sight and feel of a paw on a lever is approached by placing paw on lever, as attempted for any sign of reinforcement. Behavior in concurrent schedules is governed by approach to momentarily richer patches (melioration). Applications of MPR in behavioral pharmacology and delay discounting are noted.

Longer operant lever-press duration requirements induce fewer but longer response bouts in rats

Operant behavior is organized in bouts that are particularly visible under variable-interval (VI) schedules of reinforcement. Previous research showed that increasing the work required to produce a response decreases the rate at which bouts are emitted and increases the minimum interresponse time (IRT). In the current study, the minimum effective IRT was directly manipulated by changing the minimum duration of effective lever presses reinforced on a VI 40-s schedule. Contrary to assumptions of previous models, response durations were variable. Response durations were typically 0.5 s greater than the minimum duration threshold; durations that exceeded this threshold were approximately log-normally distributed. As the required duration threshold increased, rats emitted fewer but longer bouts. This effect may reflect an effort-induced reduction in motivation and a duration-induced facilitation of a response-outcome association.

On the "Strength" of Behavior

The place of the concept of response strength in a natural science of behavior has been the subject of much debate. This article reconsiders the concept of response strength for reasons linked to the foundations of a natural science of behavior. The notion of response strength is implicit in many radical behaviorists' work. Palmer (2009) makes it explicit by applying the response strength concept to three levels: (1) overt behavior, (2) covert behavior, and (3) latent or potential behavior. We argue that the concept of response strength is superfluous in general, and an explication of the notion of giving causal status to nonobservable events like latent behavior or response strength is harmful to a scientific endeavor. Interpreting EEG recordings as indicators of changes in response strength runs the risk of reducing behavior to underlying mechanisms, regardless of whether such suggestions are accompanied by behavioral observations. Many radical behaviorists understand behavior as a discrete unit, inviting conceptual mistakes reflected in the notion of response strength. A molar view is suggested as an alternative that accounts for the temporally extended nature of behavior and avoids the perils of a response-strength based approach.

A Critical Review of the Support for Variability as an Operant Dimension

There is abundant evidence that behavioral variability is more predominant when reinforcement is contingent on it than when it is not, and the interpretation of direct reinforcement of variability suggested by Page and Neuringer, Journal of Experimental Psychology: Animal Behavior Processes, 11(3), 429-452 (1985) has been widely accepted. Even so, trying to identify the underlying mechanisms in the emergence of stochastic-like variability in a variability contingency is intricate. There are several challenges to characterizing variability as directly reinforced, most notably because reinforcement traditionally has been found to produce repetitive responding, but also because directly reinforced variability does not always relate to independent variables the same way as more commonly studied repetitive responding does. The challenging findings in variability experiments are discussed, along with alternative hypotheses on how variability contingencies may engender the high variability that they undeniably do. We suggest that the typical increase in behavioral variability that is often demonstrated when reinforcement is contingent on it may be better explained in terms of a dynamic interaction of reinforcement and extinction working on several specific responses rather than as directly reinforced.

Toward the Unification of Molecular and Molar Analyses

Three categories of behavior analysis may be called molecular, molar, and unified. Molecular analyses focus on how manual shaping segments moment-to-moment behaving into new, unified, hierarchically organized patterns. Manual shaping is largely atheoretical, qualitative, and practical. Molar analyses aggregate behaviors and then compute a numerical average for the aggregate. Typical molar analyses involve average rate of, or average time allocated to, the aggregated behaviors. Some molar analyses have no known relation to any behavior stream. Molar analyses are usually quantitative and often theoretical. Unified analyses combine automated shaping of moment-to-moment behaving and molar aggregates of the shaped patterns. Unified controlling relations suggest that molar controlling relations like matching confound shaping and strengthening effects of reinforcement. If a molecular analysis is about how reinforcement organizes individual behavior moment by moment, and a molar analysis is about how reinforcement encourages more or less of an activity aggregated over time, then a unified analysis handles both kinds of analyses. Only theories engendered by computer simulation appear to be able to unify all three categories of behavior analysis.

Moving Beyond Reinforcement and Response Strength

Behavior analysis has often simultaneously depended upon and denied an implicit, hypothetical process of reinforcement as response strengthening. I discuss what I see as problematic about the use of such an implicit, possibly inaccurate, and likely unfalsifiable theory and describe issues to consider with respect to an alternative view without response strengthening. In my take on such an approach, important events (i.e., "reinforcers") provide a means to measure learning about predictive relations in the environment by modulating (i.e., inducing) performance dependent upon what is predicted and the relevant motivational mode or behavioral system active at that time (i.e., organismic state). Important events might be phylogenetically important, or they might acquire importance by being useful as signals for guiding an organism to where, when, or how currently relevant events might be obtained (or avoided). Given the role of learning predictive relations in such an approach, it is suggested that a potentially useful first step is to work toward formal descriptions of the structure of the predictive relations embodied in common facets of operant behavior (e.g., response-reinforcer contingencies, conditioned reinforcement, and stimulus control). Ultimately, the success of such an approach will depend upon how well it integrates formal characterizations of predictive relations (and how they are learned without response strengthening) and the relevant concomitant changes in organismic state across time. I also consider how thinking about the relevant processes in such a way might improve both our basic science and our technology of behavior.

Extinction learning deficit in a rodent model of attention-deficit hyperactivity disorder

Background

Deficient operant extinction has been hypothesized to be constitutive of ADHD dysfunction. In order to elucidate the behavioral mechanisms underlying this deficit, the performance of an animal model of ADHD, the spontaneously hypertensive rat (SHR), was compared against the performance of a control strain, the Wistar-Kyoto rat (WKY) during extinction.

Method

Following extensive training of lever pressing under variable interval schedules of food reinforcement (reported previously), SHR and WKY rats were exposed to two sessions of extinction training. Extinction data was analyzed using the Dynamic Bi-Exponential Refractory Model (DBERM) of operant performance. DBERM assumes that operant responses are organized in bouts separated by pauses; during extinction, bouts may decline across multiple dimensions, including frequency and length. DBERM parameters were estimated using hierarchical Bayesian modeling.

Results

SHR responded more than WKY during the first extinction session. DBERM parameter estimates revealed that, at the onset of extinction, SHR produced more response bouts than WKY. Over the course of extinction, response bouts progressively shortened for WKY but not for SHR.

Conclusions

Based on prior findings on the sensitivity of DBERM parameters to motivational and schedule manipulations, present data suggests that (1) more frequent response bouts in SHR are likely related to greater incentive motivation, and (2) the persistent length of bouts in SHR are likely related to a slower updating of the response-outcome association. Overall, these findings suggest specific motivational and learning deficits that may explain ADHD-related impairments in operant performance.

Discrimination of variable schedules is controlled by interresponse times proximal to reinforcement

In Experiment 1, food-deprived rats responded to one of two schedules that were, with equal probability, associated with a sample lever. One schedule was always variable ratio, while the other schedule, depending on the trial within a session, was: (a) a variable-interval schedule; (b) a tandem variable-interval, differential-reinforcement-of-low-rate schedule; or (c) a tandem variable-interval, differential-reinforcement-of-high-rate schedule. Completion of a sample-lever schedule, which took approximately the same time regardless of schedule, presented two comparison levers, one associated with each sample-lever schedule. Pressing the comparison lever associated with the schedule just presented produced food, while pressing the other produced a blackout. Conditional-discrimination accuracy was related to the size of the difference in reinforced interresponse times and those that preceded it (predecessor interresponse times) between the variable-ratio and other comparison schedules. In Experiment 2, control by predecessor interresponse times was accentuated by requiring rats to discriminate between a variable-ratio schedule and a tandem schedule that required emission of a sequence of a long, then a short interresponse time in the tandem's terminal schedule. These discrimination data are compatible with the copyist model from Tanno and Silberberg (2012) in which response rates are determined by the succession of interresponse times between reinforcers weighted so that each interresponse time's role in rate determination diminishes exponentially as a function of its distance from reinforcement.

Selection dynamics in joint matching to rate and magnitude of reinforcement

Virtual organisms animated by a selectionist theory of behavior dynamics worked on concurrent random interval schedules where both the rate and magnitude of reinforcement were varied. The selectionist theory consists of a set of simple rules of selection, recombination, and mutation that act on a population of potential behaviors by means of a genetic algorithm. An extension of the power function matching equation, which expresses behavior allocation as a joint function of exponentiated reinforcement rate and reinforcer magnitude ratios, was fitted to the virtual organisms' data, and over a range of moderate mutation rates was found to provide an excellent description of their behavior without residual trends. The mean exponents in this range of mutation rates were 0.83 for the reinforcement rate ratio and 0.68 for the reinforcer magnitude ratio, which are values that are comparable to those obtained in experiments with live organisms. These findings add to the evidence supporting the selectionist theory, which asserts that the world of behavior we observe and measure is created by evolutionary dynamics.

The isolation of motivational, motoric, and schedule effects on operant performance: a modeling approach

Dissociating motoric and motivational effects of pharmacological manipulations on operant behavior is a substantial challenge. To address this problem, we applied a response-bout analysis to data from rats trained to lever press for sucrose on variable-interval (VI) schedules of reinforcement. Motoric, motivational, and schedule factors (effort requirement, deprivation level, and schedule requirements, respectively) were manipulated. Bout analysis found that interresponse times (IRTs) were described by a mixture of two exponential distributions, one characterizing IRTs within response bouts, another characterizing intervals between bouts. Increasing effort requirement lengthened the shortest IRT (the refractory period between responses). Adding a ratio requirement increased the length and density of response bouts. Both manipulations also decreased the bout-initiation rate. In contrast, food deprivation only increased the bout-initiation rate. Changes in the distribution of IRTs over time showed that responses during extinction were also emitted in bouts, and that the decrease in response rate was primarily due to progressively longer intervals between bouts. Taken together, these results suggest that changes in the refractory period indicate motoric effects, whereas selective alterations in bout initiation rate indicate incentive-motivational effects. These findings support the use of response-bout analyses to identify the influence of pharmacological manipulations on processes underlying operant performance.

Categorical counting

Pigeons pecked on three keys, responses to one of which could be reinforced after a few pecks, to a second key after a somewhat larger number of pecks, and to a third key after the maximum pecking requirement. The values of the pecking requirements and the proportion of trials ending with reinforcement were varied. Transits among the keys were an orderly function of peck number, and showed approximately proportional changes with changes in the pecking requirements, consistent with Weber's law. Standard deviations of the switch points between successive keys increased more slowly within a condition than across conditions. Changes in reinforcement probability produced changes in the location of the psychometric functions that were consistent with models of timing. Analyses of the number of pecks emitted and the duration of the pecking sequences demonstrated that peck number was the primary determinant of choice, but that passage of time also played some role. We capture the basic results with a standard model of counting, which we qualify to account for the secondary experiments.

The Journal of the Experimental Analysis of Behavior at zero, fifty, and one hundred

The experimental content areas represented in JEAB in its first volume (1958) and 50 years later in Volume 87 are in many ways similar with regard to research on schedules of reinforcement, research with human subjects, and several other topics. Experimental analysis has not been displaced by quantitative analysis. Much less research on aversive control has been published in recent than in earlier years. Wishes for progress in the next 50 years include experiments on verbal behavior, the sources of novel behavior, and observing responses based on stimuli correlated with escape or avoidance.

Quantitative behavior analysis and human values

Many scientists believe that among the virtues of quantitative science are that its facts are free from personal, social, political, economic, and other cultural influences, or at least, if they are not, they should be. Radical behaviorism suggests, however, that a science of behavior must apply to peoples' everyday professional behaviors, including those of quantitative behavior analysts. The behaviors of quantitative behavior analysts, however, like the behaviors of everyone else, depend on the cultures to which they belong. A quantitative science of behavior must therefore describe and explain the cultural and human values of quantitative behavior analysts. In this sense, a quantitative science of behavior must apply to itself. No such "reflexive behavior analysis" currently exists and its development might shed considerable light on the basic nature of behavior analysis.