Molecular analyses of the principal components of response strength

Bouts, Pauses, and Units of Operant Performance: A Primer

Operant behavior typically occurs in bouts and pauses. The microstructural analysis of bouts and pauses reveals important and separable information about the physical characteristics of the operant and the motivation behind it. An analysis of interresponse times (IRTs) often reveals a mixture of two exponential distributions. One corresponds to short IRTs within ongoing response bouts, reflecting motor properties of the operant, and the other corresponds to longer intervals between bouts, reflecting the motivation behind the response. Partitioning responses into bout initiations and within-bout responses via this two-mode framework reveals the mechanisms underlying behavior maintenance and change. This approach is used in the fields of neurotoxicology, behavioral pharmacology, and behavioral neuroscience to disentangle the contribution of motivational and motoric variables to the pattern of operant behavior. In this article, we present a primer aimed at providing essential concepts related to the analysis of response bouts and temporal dynamics of operant performance.

Hyperbolae

Hyperbolic relations between independent and dependent variables are ubiquitous in the experimental analysis of behavior, mentioned in over 150 articles in the Journal of the Experimental Analysis of Behavior. There are two principal forms of hyperbolae: The first describes the relation between response rate and reinforcement rate on variable-interval schedules of reinforcement; it rises asymptotically toward a maximum. The second describes the relation between the current equivalent value of an incentive and its delay or (im)probability; it falls from a maximum toward an asymptote of 0. Where do these come from? What do their parameters mean? How are they related? This article answers the first two questions and addresses the last.

The temporal structure of goal-directed and habitual operant behavior

Operant behavior can reflect the influence of goal-directed and habitual processes. These can be distinguished by changes to response rate following devaluation of the reinforcing outcome. Whether a response is goal directed or habitual depends on whether devaluation affects response rate. Response rate can be decomposed into frequencies of bouts and pauses by analyzing the distribution of interresponse times. This study sought to characterize goal-directed and habitual behaviors in terms of bout-initiation rate, within-bout response rate, bout length, and bout duration. Data were taken from three published studies that compared sensitivity to devaluation following brief and extended training with variable-interval schedules. Analyses focused on goal-directed and habitual responding, a comparison of a habitual response to a similarly trained response that had been converted back to goal-directed status after a surprising event, and a demonstration of contextual control of habit and goal direction in the same subjects. Across experiments and despite responses being clearly distinguished as goal directed and habitual by total response rate, analyses of bout-initiation rate, within-bout rate, bout length, and bout duration did not reveal a pattern that distinguished goal-directed from habitual responding.

How associations become behavior

The Rescorla and Wagner (1972) model is the first mathematical theory to explain associative learning in the presence of multiple stimuli. Its main theoretical construct is that of associative strength, but this is connected to behavior only loosely. We propose a model in which behavior is described by a collection of Poisson processes, each with a rate proportional to an associative strength. The model predicts that the time between behaviors follows an exponential or hypoexponential distribution. This prediction is supported by two data sets on autoshaped and instrumental behavior in rats.

Theory of reinforcement schedules

The three principles of reinforcement are (1) events such as incentives and reinforcers increase the activity of an organism; (2) that activity is bounded by competition from other responses; and (3) animals approach incentives and their signs, guided by their temporal and physical conditions, together called the "contingencies of reinforcement." Mathematical models of each of these principles comprised mathematical principles of reinforcement (MPR; Killeen, 1994). Over the ensuing decades, MPR was extended to new experimental contexts. This article reviews the basic theory and its extensions to satiation, warm-up, extinction, sign tracking, pausing, and sequential control in progressive-ratio and multiple schedules. In the latter cases, a single equation balancing target and competing responses governs behavioral contrast and behavioral momentum. Momentum is intrinsic in the fundamental equations, as behavior unspools more slowly from highly aroused responses conditioned by higher rates of incitement than it does from responses from leaner contexts. Habits are responses that have accrued substantial behavioral momentum. Operant responses, being predictors of reinforcement, are approached by making them: The sight and feel of a paw on a lever is approached by placing paw on lever, as attempted for any sign of reinforcement. Behavior in concurrent schedules is governed by approach to momentarily richer patches (melioration). Applications of MPR in behavioral pharmacology and delay discounting are noted.

The effect of brief mindfulness training on the micro-structure of human free-operant responding: Mindfulness affects stimulus-driven responding

BACKGROUND AND OBJECTIVES

The current study examines the extent to which mindfulness impacts on operant conditioning processes, and explores the suggestion that mindfulness training serves to make humans more sensitive to the current reinforcement contingencies with which they are presented. In particular, the effect of mindfulness on the micro-structure of human schedule performance was explored. It was expected that mindfulness might impact bout-initiation responding to a greater degree than within-bout responding, premised on the assumption that bout-initiation responses are habitual and not under conscious control, but within-bout responses are goal-directed and conscious.

METHODS

Nonclinical participants experienced one of three brief (15min) interventions: focused attention breathing exercise (mindfulness), an unfocused attention breathing exercises, or no intervention. They then responded on a multiple random ratio (RR) random interval (RI) schedule.

RESULTS

In the no intervention and unfocused attention groups, overall and within-bout response rates were higher on the RR than the RI schedule, but bout-initiation rates were the same on the two schedules. However, for the mindfulness groups all forms of responding were higher for the RR than the RI schedule. Previous work has noted that habitual, and/or unconscious or fringe-conscious events, are impacted by mindfulness training.

LIMITATIONS

A nonclinical sample may limit generality.

CONCLUSIONS

The current pattern of results suggests that this is also true in schedule-controlled performance, and offers an insight into the manner in which mindfulness alongside conditioning-based interventions, to bring all responses under conscious control.

Pupillary dynamics of mice performing a Pavlovian delay conditioning task reflect reward-predictive signals

Pupils can signify various internal processes and states, such as attention, arousal, and working memory. Changes in pupil size have been associated with learning speed, prediction of future events, and deviations from the prediction in human studies. However, the detailed relationships between pupil size changes and prediction are unclear. We explored pupil size dynamics in mice performing a Pavlovian delay conditioning task. A head-fixed experimental setup combined with deep-learning-based image analysis enabled us to reduce spontaneous locomotor activity and to track the precise dynamics of pupil size of behaving mice. By setting up two experimental groups, one for which mice were able to predict reward in the Pavlovian delay conditioning task and the other for which mice were not, we demonstrated that the pupil size of mice is modulated by reward prediction and consumption, as well as body movements, but not by unpredicted reward delivery. Furthermore, we clarified that pupil size is still modulated by reward prediction even after the disruption of body movements by intraperitoneal injection of haloperidol, a dopamine D2 receptor antagonist. These results suggest that changes in pupil size reflect reward prediction signals. Thus, we provide important evidence to reconsider the neuronal circuit involved in computing reward prediction error. This integrative approach of behavioral analysis, image analysis, pupillometry, and pharmacological manipulation will pave the way for understanding the psychological and neurobiological mechanisms of reward prediction and the prediction errors essential to learning and behavior.

Longer operant lever-press duration requirements induce fewer but longer response bouts in rats

Operant behavior is organized in bouts that are particularly visible under variable-interval (VI) schedules of reinforcement. Previous research showed that increasing the work required to produce a response decreases the rate at which bouts are emitted and increases the minimum interresponse time (IRT). In the current study, the minimum effective IRT was directly manipulated by changing the minimum duration of effective lever presses reinforced on a VI 40-s schedule. Contrary to assumptions of previous models, response durations were variable. Response durations were typically 0.5 s greater than the minimum duration threshold; durations that exceeded this threshold were approximately log-normally distributed. As the required duration threshold increased, rats emitted fewer but longer bouts. This effect may reflect an effort-induced reduction in motivation and a duration-induced facilitation of a response-outcome association.

Internal-Clock Models and Misguided Views of Mechanistic Explanations: A Reply to Eckard & Lattal (2020)

Eckard and Lattal's Perspectives on Behavior Science, 43(1), 5-19 (2020) critique of internal clock (IC) mechanisms is based on narrow concepts of clocks, of their internality, of their mechanistic nature, and of scientific explanations in general. This reply broadens these concepts to characterize all timekeeping objects-physical and otherwise-as clocks, all intrinsic properties of such objects as internal to them, and all simulatable explanations of such properties as mechanisms. Eckard and Lattal's critique reflects a restrictive billiard-ball view of causation, in which environmental manipulations and behavioral effects are connected by a single chain of contiguous events. In contrast, this reply offers a more inclusive stochastic view of causation, in which environmental manipulations are probabilistically connected to behavioral effects. From either view of causation, computational ICs are hypothetical and unobservable, but their heuristic value and parsimony can only be appreciated from a stochastic view of causation. Billiard-ball and stochastic views have contrasting implications for potential explanations of interval timing. As illustrated by accounts of the variability in start times in fixed-interval schedules of reinforcement, of the two views of causality examined, only the stochastic account supports falsifiable predictions beyond simple replications. It is thus not surprising that the experimental analysis of behavior has progressively adopted a stochastic view of causation, and that it has reaped its benefits. This reply invites experimental behavior analysts to continue on that trajectory.

Simulating bout-and-pause patterns with reinforcement learning

Animal responses occur according to a specific temporal structure composed of two states, where a bout is followed by a long pause until the next bout. Such a bout-and-pause pattern has three components: the bout length, the within-bout response rate, and the bout initiation rate. Previous studies have investigated how these three components are affected by experimental manipulations. However, it remains unknown what underlying mechanisms cause bout-and-pause patterns. In this article, we propose two mechanisms and examine computational models developed based on reinforcement learning. The model is characterized by two mechanisms. The first mechanism is choice-an agent makes a choice between operant and other behaviors. The second mechanism is cost-a cost is associated with the changeover of behaviors. These two mechanisms are extracted from past experimental findings. Simulation results suggested that both the choice and cost mechanisms are required to generate bout-and-pause patterns and if either of them is knocked out, the model does not generate bout-and-pause patterns. We further analyzed the proposed model and found that it reproduced the relationships between experimental manipulations and the three components that have been reported by previous studies. In addition, we showed alternative models can generate bout-and-pause patterns as long as they implement the two mechanisms.

Human free-operant performance varies with a concurrent task: Probability learning without a task, and schedule-consistent with a task

Three experiments examined human rates and patterns of responding during exposure to various schedules of reinforcement with or without a concurrent task. In the presence of the concurrent task, performances were similar to those typically noted for nonhumans. Overall response rates were higher on medium-sized ratio schedules than on smaller or larger ratio schedules (Experiment 1), on interval schedules with shorter than longer values (Experiment 2), and on ratio compared with interval schedules with the same rate of reinforcement (Experiment 3). Moreover, bout-initiation responses were more susceptible to influence by rates of reinforcement than were within-bout responses across all experiments. In contrast, in the absence of a concurrent task, human schedule performance did not always display characteristics of nonhuman performance, but tended to be related to the relationship between rates of responding and reinforcement (feedback function), irrespective of the schedule of reinforcement employed. This was also true of within-bout responding, but not bout-initiations, which were not affected by the presence of a concurrent task. These data suggest the existence of two strategies for human responding on free-operant schedules, relatively mechanistic ones that apply to bout-initiation, and relatively explicit ones, that tend to apply to within-bout responding, and dominate human performance when other demands are not made on resources.

Factors controlling the micro-structure of human free-operant behaviour: Bout-initiation and within-bout responses are effected by different aspects of the schedule

Two experiments examined factors controlling human free-operant performance in relation to predictions based on the nature of bout-initiation and within-bout responding. Overall, responding was higher for a random ratio (RR) than a random interval (RI) schedule, with equal rates of reinforcement. Bout-initiation rates were not different across the two schedules, but within-bout rates were higher on the RR schedule. Response cost reduced overall rates of responding, but tended to suppress bout-initiation responding more than within-bout responding (Experiments 1 & 2). In contrast, reinforcement magnitude increased all forms of responding (Experiment 2). One explanation consistent with these effects is that bout-initiation responses are controlled by overall rates of reinforcement through their impact on the context (i.e. are stimulus-driven), but that within-bout responses are controlled by response reinforcement (i.e. are goal-directed). These current findings are discussed in the light of these theoretical suggestions.

A computational formulation of the behavior systems account of the temporal organization of motivated behavior

The behavior systems framework suggests that motivated behavior-e.g., seeking food and mates, avoiding predators-consists of sequences of actions organized within nested behavioral states. This framework has bridged behavioral ecology and experimental psychology, providing key insights into critical behavioral processes. In particular, the behavior systems framework entails a particular organization of behavior over time. The present paper examines whether such organization emerges from a generic Markov process, where the current behavioral state determines the probability distribution of subsequent behavioral states. This proposition is developed as a systematic examination of increasingly complex Markov models, seeking a computational formulation that balances adherence to the behavior systems approach, parsimony, and conformity to data. As a result of this exercise, a nonstationary partially hidden Markov model is selected as a computational formulation of the predatory subsystem. It is noted that the temporal distribution of discrete responses may further unveil the structure and parameters of the model but, without proper mathematical modeling, these discrete responses may be misleading. Opportunities for further elaboration of the proposed computational formulation are identified, including developments in its architecture, extensions to defensive and reproductive subsystems, and methodological refinements.

Interval timing under a behavioral microscope: Dissociating motivational and timing processes in fixed-interval performance

The distribution of latencies and interresponse times (IRTs) of rats was compared between two fixed-interval (FI) schedules of food reinforcement (FI 30 s and FI 90 s), and between two levels of food deprivation. Computational modeling revealed that latencies and IRTs were well described by mixture probability distributions embodying two-state Markov chains. Analysis of these models revealed that only a subset of latencies is sensitive to the periodicity of reinforcement, and prefeeding only reduces the size of this subset. The distribution of IRTs suggests that behavior in FI schedules is organized in bouts that lengthen and ramp up in frequency with proximity to reinforcement. Prefeeding slowed down the lengthening of bouts and increased the time between bouts. When concatenated, latency and IRT models adequately reproduced sigmoidal FI response functions. These findings suggest that behavior in FI schedules fluctuates in and out of schedule control; an account of such fluctuation suggests that timing and motivation are dissociable components of FI performance. These mixture-distribution models also provide novel insights on the motivational, associative, and timing processes expressed in FI performance. These processes may be obscured, however, when performance in timing tasks is analyzed in terms of mean response rates.

Toward the Unification of Molecular and Molar Analyses

Three categories of behavior analysis may be called molecular, molar, and unified. Molecular analyses focus on how manual shaping segments moment-to-moment behaving into new, unified, hierarchically organized patterns. Manual shaping is largely atheoretical, qualitative, and practical. Molar analyses aggregate behaviors and then compute a numerical average for the aggregate. Typical molar analyses involve average rate of, or average time allocated to, the aggregated behaviors. Some molar analyses have no known relation to any behavior stream. Molar analyses are usually quantitative and often theoretical. Unified analyses combine automated shaping of moment-to-moment behaving and molar aggregates of the shaped patterns. Unified controlling relations suggest that molar controlling relations like matching confound shaping and strengthening effects of reinforcement. If a molecular analysis is about how reinforcement organizes individual behavior moment by moment, and a molar analysis is about how reinforcement encourages more or less of an activity aggregated over time, then a unified analysis handles both kinds of analyses. Only theories engendered by computer simulation appear to be able to unify all three categories of behavior analysis.

Quantitative analysis of performance on a progressive-ratio schedule: effects of reinforcer type, food deprivation and acute treatment with Δ⁹-tetrahydrocannabinol (THC)

Rats' performance on a progressive-ratio schedule maintained by sucrose (0.6M, 50 μl) and corn oil (100%, 25 μl) reinforcers was assessed using a model derived from Killeen's (1994) theory of schedule-controlled behaviour, 'Mathematical Principles of Reinforcement'. When the rats were maintained at 80% of their free-feeding body weights, the parameter expressing incentive value, a, was greater for the corn oil than for the sucrose reinforcer; the response-time parameter, δ, did not differ between the reinforcer types, but a parameter derived from the linear waiting principle (T0), indicated that the minimum post-reinforcement pause was longer for corn oil than for sucrose. When the rats were maintained under free-feeding conditions, a was reduced, indicating a reduction of incentive value, but δ was unaltered. Under the food-deprived condition, the CB1 cannabinoid receptor agonist Δ(9)-tetrahydrocannabinol (THC: 0.3, 1 and 3 mg kg(-1)) increased the value of a for sucrose but not for corn oil, suggesting a selective enhancement of the incentive value of sucrose; none of the other parameters was affected by THC. The results provide new information about the sensitivity of the model's parameters to deprivation and reinforcer quality, and suggest that THC selectively enhances the incentive value of sucrose.

Extinction under a behavioral microscope: isolating the sources of decline in operant response rate

Extinction performance is often used to assess underlying psychological processes without the interference of reinforcement. For example, in the extinction/reinstatement paradigm, motivation to seek drug is assessed by measuring responding elicited by drug-associated cues without drug reinforcement. However, extinction performance is governed by several psychological processes that involve motivation, memory, learning, and motoric functions. These processes are confounded when overall response rate is used to measure performance. Based on evidence that operant responding occurs in bouts, this paper proposes an analytic procedure that separates extinction performance into several behavioral components: (1-3) the baseline bout initiation rate, within-bout response rate, and bout length at the onset of extinction; (4-6) their rates of decay during extinction; (7) the time between extinction onset and the decline of responding; (8) the asymptotic response rate at the end of extinction; (9) the refractory period after each response. Data that illustrate the goodness of fit of this analytic model are presented. This paper also describes procedures to isolate behavioral components contributing to extinction performance and make inferences about experimental effects on these components. This microscopic behavioral analysis allows the mapping of different psychological processes to distinct behavioral components implicated in extinction performance, which may further our understanding of the psychological effects of neurobiological treatments.

The isolation of motivational, motoric, and schedule effects on operant performance: a modeling approach

Dissociating motoric and motivational effects of pharmacological manipulations on operant behavior is a substantial challenge. To address this problem, we applied a response-bout analysis to data from rats trained to lever press for sucrose on variable-interval (VI) schedules of reinforcement. Motoric, motivational, and schedule factors (effort requirement, deprivation level, and schedule requirements, respectively) were manipulated. Bout analysis found that interresponse times (IRTs) were described by a mixture of two exponential distributions, one characterizing IRTs within response bouts, another characterizing intervals between bouts. Increasing effort requirement lengthened the shortest IRT (the refractory period between responses). Adding a ratio requirement increased the length and density of response bouts. Both manipulations also decreased the bout-initiation rate. In contrast, food deprivation only increased the bout-initiation rate. Changes in the distribution of IRTs over time showed that responses during extinction were also emitted in bouts, and that the decrease in response rate was primarily due to progressively longer intervals between bouts. Taken together, these results suggest that changes in the refractory period indicate motoric effects, whereas selective alterations in bout initiation rate indicate incentive-motivational effects. These findings support the use of response-bout analyses to identify the influence of pharmacological manipulations on processes underlying operant performance.

Repeated extinction and reversal learning of an approach response supports an arousal-mediated learning model

We assessed the effects of repeated extinction and reversals of two conditional stimuli (CS+/CS-) on an appetitive conditioned approach response in rats. Three results were observed that could not be accounted for by a simple linear operator model such as the one proposed by Rescorla and Wagner (1972): (1) responding to a CS- declined faster when a CS+ was simultaneously extinguished; (2) reacquisition of pre-extinction performance recovered rapidly within one session; and (3) reversal of CS+/CS- contingencies resulted in a more rapid recovery to the current CS- (former CS+) than the current CS+, accompanied by a slower acquisition of performance to the current CS+. An arousal parameter that mediates learning was introduced to a linear operator model to account for these effects. The arousal-mediated learning model adequately fit the data and predicted data from a second experiment with different rats in which only repeated reversals of CS+/CS- were assessed. According to this arousal-mediated learning model, learning is accelerated by US-elicited arousal and it slows down in the absence of US. Because arousal varies faster than conditioning, the model accounts for the decline in responding during extinction mainly through a reduction in arousal, not a change in learning. By preserving learning during extinction, the model is able to account for relapse effects like rapid reacquisition, renewal, and reinstatement.

The dynamics of conditioning and extinction

Pigeons responded to intermittently reinforced classical conditioning trials with erratic bouts of responding to the conditioned stimulus. Responding depended on whether the prior trial contained a peck, food, or both. A linear persistence-learning model moved pigeons into and out of a response state, and a Weibull distribution for number of within-trial responses governed in-state pecking. Variations of trial and intertrial durations caused correlated changes in rate and probability of responding and in model parameters. A novel prediction--in the protracted absence of food, response rates can plateau above zero--was validated. The model predicted smooth acquisition functions when instantiated with the probability of food but a more accurate jagged learning curve when instantiated with trial-to-trial records of reinforcement. The Skinnerian parameter was dominant only when food could be accelerated or delayed by pecking. These experiments provide a framework for trial-by-trial accounts of conditioning and extinction that increases the information available from the data, permitting such accounts to comment more definitively on complex contemporary models of momentum and conditioning.

An analytic form for the interresponse time analysis of Shull, Gaynor, and Grimes with applications and extensions

Shull, Gaynor and Grimes advanced a model for interresponse time distribution using probabilistic cycling between a higher-rate and a lower-rate response process. Both response processes are assumed to be random in time with a constant rate. The cycling between the two processes is assumed to have a constant transition probability that is independent of bout length. This report develops an analytic form of the model which has a natural parametrization for a higher-rate within-bout responding and a lower-rate visit-initiation responding. The analytic form provides a convenient basis for both a nonlinear least-squares data reduction technique to estimate the model's parameters and Monte Carlo simulations of the model. In addition, the analytic formulation is extended to both a refractory period for the rats' behavior and, separately, the strongly-banded behavior seen with pigeons.

Interresponse time structures in variable-ratio and variable-interval schedules

The interresponse-time structures of pigeon key pecking were examined under variable-ratio, variable-interval, and variable-interval plus linear feedback schedules. Whereas the variable-ratio and variable-interval plus linear feedback schedules generally resulted in a distinct group of short interresponse times and a broad distribution of longer interresponse times, the variable-interval schedules generally showed a much more continuous distribution of interresponse times. The results were taken to indicate that a log survivor analysis or double exponential fit of interresponse times may not be universally applicable to the task of demonstrating that operant behavior can be dichotomized into bouts of engagement and periods of disengagement.

Evidence for impulsivity in the Spontaneously Hypertensive Rat drawn from complementary response-withholding tasks

BACKGROUND

The inability to inhibit reinforced responses is a defining feature of ADHD associated with impulsivity. The Spontaneously Hypertensive Rat (SHR) has been extolled as an animal model of ADHD, but there is no clear experimental evidence of inhibition deficits in SHR. Attempts to demonstrate these deficits may have suffered from methodological and analytical limitations.

METHODS

We provide a rationale for using two complementary response-withholding tasks to doubly dissociate impulsivity from motivational and motor processes. In the lever-holding task (LHT), continual lever depression was required for a minimum interval. Under a differential reinforcement of low rates schedule (DRL), a minimum interval was required between lever presses. Both tasks were studied using SHR and two normotensive control strains, Wistar-Kyoto (WKY) and Long Evans (LE), over an overlapping range of intervals (1 - 5 s for LHT and 5 - 60 s for DRL). Lever-holding and DRL performance was characterized as the output of a mixture of two processes, timing and iterative random responding; we call this account of response inhibition the Temporal Regulation (TR) model. In the context of TR, impulsivity was defined as a bias toward premature termination of the timed intervals.

RESULTS

The TR model provided an accurate description of LHT and DRL performance. On the basis of TR parameter estimates, SHRs were more impulsive than LE rats across tasks and target times. WKY rats produced substantially shorter timed responses in the lever-holding task than in DRL, suggesting a motivational or motor deficit. The precision of timing by SHR, as measured by the variance of their timed intervals, was excellent, flouting expectations from ADHD research.

CONCLUSION

This research validates the TR model of response inhibition and supports SHR as an animal model of ADHD-related impulsivity. It indicates, however, that SHR's impulse-control deficit is not caused by imprecise timing. The use of ad hoc impulsivity metrics and of WKY as control strain for SHR impulsivity are called into question.

Resistance to change of responding maintained by unsignaled delays to reinforcement: a response-bout analysis

Previous experiments have shown that unsignaled delayed reinforcement decreases response rates and resistance to change. However, the effects of different delays to reinforcement on underlying response structure have not been investigated in conjunction with tests of resistance to change. In the present experiment, pigeons responded on a three-component multiple variable-interval schedule for food presented immediately, following brief (0.5 s), or following long (3 s) unsignaled delays of reinforcement. Baseline response rates were lowest in the component with the longest delay; they were about equal with immediate and briefly delayed reinforcers. Resistance to disruption by presession feeding, response-independent food during the intercomponent interval, and extinction was slightly but consistently lower as delays increased. Because log survivor functions of interresponse times (IRTs) deviated from simple modes of bout initiations and within-bout responding, an IRT-cutoff method was used to examine underlying response structure. These analyses suggested that baseline rates of initiating bouts of responding decreased as scheduled delays increased, and within-bout response rates tended to be lower in the component with immediate reinforcers. The number of responses per bout was not reliably affected by reinforcer delay, but tended to be highest with brief delays when total response rates were higher in that component. Consistent with previous findings, resistance to change of overall response rate was highly correlated with resistance to change of bout-initiation rates but not with within-bout responding. These results suggest that unsignaled delays to reinforcement affect resistance to change through changes in the probability of initiating a response bout rather than through changes in the underlying response structure.

Mathematical models and the experimental analysis of behavior

The use of mathematical models in the experimental analysis of behavior has increased over the years, and they offer several advantages. Mathematical models require theorists to be precise and unambiguous, often allowing comparisons of competing theories that sound similar when stated in words. Sometimes different mathematical models may make equally accurate predictions for a large body of data. In such cases, it is important to find and investigate situations for which the competing models make different predictions because, unless two models are actually mathematically equivalent, they are based on different assumptions about the psychological processes that underlie an observed behavior. Mathematical models developed in basic behavioral research have been used to predict and control behavior in applied settings, and they have guided research in other areas of psychology. A good mathematical model can provide a common framework for understanding what might otherwise appear to be diverse and unrelated behavioral phenomena. Because psychologists vary in their quantitative skills and in their tolerance for mathematical equations, it is important for those who develop mathematical models of behavior to find ways (such as verbal analogies, pictorial representations, or concrete examples) to communicate the key premises of their models to nonspecialists.

Remembering: the role of extraneous reinforcement

In two experiments, pigeons' responding on an extraneous task was explicitly reinforced during delayed matching-to-sample trials. In Experiment 1, red or green sample stimuli were followed by retention intervals of 0.2, 1, 4, or 12 sec, during which pecks to a white center key were reinforced with 2.5-sec access to wheat according to extinction, variable-interval 30-sec, and variable-interval 15-sec schedules in different conditions. A proportion of .2, .5, .7, or .9 of subsequent red or green choice responses that matched the sample were reinforced with 3-sec access to wheat. The result was that increasing center key reinforcement, or reducing reinforcer probability, lowered overall accuracy. Initial discriminability fell, but with no change in the rate of forgetting. In Experiment 2, initial discriminability was affected by extraneous reinforcers that were contingent on center key pecking, but not by noncontingent reinforcers. A plausible conclusion is that initial discriminability decreases when reinforcers strengthen competing behaviors.

Bouts of responding on variable-interval schedules: effects of deprivation level

Rats obtained food pellets on a variable-interval schedule of reinforcement by nose poking a lighted key. After training to establish baseline performance (with the mean variable interval set at either 60, 120, or 240 s), the rats were given free access to food during the hour just before their daily session. This satiation operation reduced the rate of key poking. Analysis of the interresponse time distributions (log survivor plots) indicated that key poking occurred in bouts. Prefeeding lengthened the pauses between bouts, shortened the length of bouts (less reliably), and had a relatively small decremental effect on the response rate within bouts. That deprivation level affects mainly between-bout pauses has been reported previously with fixed-ratio schedules. Thus, when the focus is on bouts, the performances maintained by variable-interval schedules and fixed-ratio schedules are similarly affected by deprivation.

Choice in a variable environment: visit patterns in the dynamics of choice

Molar and molecular views of behavior imply different approaches to data analysis. The molecular view privileges moment-to-moment analyses, whereas the molar view supports analysis of more and less extended activities. In concurrent performance, the molar view supports study of both extended patterns of choice and more local patterns of visiting the choice alternatives. Analysis of the present data illustrated the usefulness of investigating order at various levels of extendedness. Seven different reinforcer ratios were presented within each session, without cues to identify them, and pigeons pecked at two response keys that delivered food on variable-interval schedules. Choice changed rapidly within components as reinforcers were delivered and, following each reinforcer, shifted toward the alternative that produced it. If several reinforcers were delivered consecutively by one alternative, choice favored that alternative, but shifted more slowly with each new reinforcer. A discontinuation of such a series of reinforcers by the delivery of a reinforcer by the other alternative resulted in a large shift of choice toward that alternative. These effects were illuminated by analysis of visits to the two alternatives. Changes in visit length occurred primarily in the first postreinforcer visit to the repeatedly reinforced alternative. All other visits tended to be brief and equal. Performance showed multiple signs of moving in the direction of a fix-and-sample pattern that characterized steady-state performance in earlier experiments with many sessions of maintaining each schedule pair. The analyses of extended and local patterns illustrate the flexibility of a molar view of behavior.

Bouts of responding: the relation between bout rate and the rate of variable-interval reinforcement

By nose poking a lighted key, rats obtained food pellets on either a variable-interval schedule of reinforcement or a schedule that required an average of four additional responses after the end of tile variable-interval component (a tandem variable-interval variable-ratio 4 schedule). With both schedule types, the mean variable interval was varied between blocks of sessions from 16 min to 0.25 min. Total rate of key poking increased similarly as a function of the reinforcer rate for the two schedule types, but response rate was higher with than without the four-response requirement. Analysis of log survivor plots of interresponse times showed that key poking occurred in bouts. The rate of initiating bouts increased as a function of reinforcer rate but was either unaffected or was decreased by adding the four-response requirement. Within-bout response rate was insensitive to reinforcer rate and only inconsistently affected by the four-response requirement. For both kinds of schedule, the ratio of bout time to between-bout pause time was approximately a power function of reinforcer rate, with exponents above and below 1.0.

Reaction time signatures of discriminative processes: differential effects of stimulus similarity and incentive

In three experiments with pigeons, the similarity of unreinforced test stimuli to a reinforced stimulus and the frequency of reinforcement associated with a stimulus were varied. The stimulus on each trial was a small spot that appeared in different hues or, in Experiment 3, different forms. Differential response frequency and reaction time (RT) patterns emerged: Changes in similarity affected the percentage of stimuli responded to but left the shape of RT distributions about the same, whereas changes in reinforcement shifted RT distributions but had little effect on the percentage of responses. When the similarity and reinforcement variables were applied to the same stimuli (Experiment 2), their effects were largely independent. A generalization procedure (Experiment 3) replicated the similarity effects of the initial discrimination procedure. The RT distributions were modeled by a diffusion process, and implications for a memory-instance model were suggested.

Small and large fixed ratios with the same unit price

Key pressing of rats was maintained under multiple and discrete-trial choice schedules with reinforcer units of 45 mg food pellets or 3.5 s dips of sucrose solution. Both smaller and larger fixed ratio (FR) schedules were associated with the same unit price in a manner, for example, that each of eight iterations of FR120 was associated with delivery of a single reinforcer unit and one instance of FR960 was associated with eight reinforcer units. FR requirement varied between 20 and 1560 per aggregate reinforcer and unit price varied between 20 and 240 per reinforcer unit. During multiple schedules with food reinforcers, rates and patterns of responding were comparable over nearly a 50-fold range of FR requirements (20-1380) when unit price was 20; over nearly a six-fold range of FR requirements (120-720) when unit price was 120; and was only marginally maintained when unit price was 240. Demand for food pellets was comparatively inelastic at FRs between 20 and 120, during which subjects did not receive supplemental feeding outside experimental sessions, but was elastic at FRs greater than 240, when subjects sometimes did receive supplemental feeding. In a discrete-trial choice procedure with a constant unit price of 120 for sucrose solution, subjects were indifferent between smaller FRs and alternative FRs as large as 480, but began switching away from larger FRs that were 600 or greater. Because responding had been comparably maintained under both FR120 and FRs as large as 960 in the multiple schedule, results from the choice procedure indicated that choice performance was influenced by variables other than FR requirement and unit price. Because aggregate reinforcers were the same for smaller and larger FRs, the most likely reason for preferring smaller FRs was the nearness in time to some reinforcer.

Bouts of responding from variable-interval reinforcement of lever pressing by rats

Four rats obtained food pellets by lever pressing. A variable-interval reinforcement schedule assigned reinforcers on average every 2 min during one block of 20 sessions and on average every 8 min during another block. Also, at each variable-interval duration, a block of sessions was conducted with a schedule that imposed a variable-ratio 4 response requirement after each variable interval (i.e., a tandem variable-time variable-ratio 4 schedule). The total rate of lever pressing increased as a function of the rate of reinforcement and as a result of imposing the variable-ratio requirement. Analysis of log survivor plots of interresponse times indicated that lever pressing occurred in bouts that were separated by pauses. Increasing the rate of reinforcement increased total response rate by increasing the rate of initiating bouts and, less reliably, by lengthening bouts. Imposing the variable-ratio component increased response rate mainly by lengthening bouts. This pattern of results is similar to that reported previously with key poking as the response. Also, response rates within bouts were relatively insensitive to either variable.

MPR

Mathematical Principles of Reinforcement (MPR) is a theory of reinforcement schedules. This paper reviews the origin of the principles constituting MPR: arousal, association and constraint. Incentives invigorate responses, in particular those preceding and predicting the incentive. The process that generates an associative bond between stimuli, responses and incentives is called coupling. The combination of arousal and coupling constitutes reinforcement. Models of coupling play a central role in the evolution of the theory. The time required to respond constrains the maximum response rates, and generates a hyperbolic relation between rate of responding and rate of reinforcement. Models of control by ratio schedules are developed to illustrate the interaction of the principles. Correlations among parameters are incorporated into the structure of the models, and assumptions that were made in the original theory are refined in light of current data.

Complex dynamic processes in sign tracking with an omission contingency (negative automaintenance)

Hungry pigeons received food periodically, signaled by the onset of a keylight. Key pecks aborted the feeding. Subjects responded for thousands of trials, despite the contingent nonreinforcement, with varying probability as the intertrial interval was varied. Hazard functions showed the dominant tendency to be perseveration in responding and not responding. Once perseveration was accounted for, a linear operator model of associative conditioning further improved predictions. Response rates during trials were correlated with the prior probabilities of a response. Rescaled range analyses showed that the behavioral trajectories were a kind of fractional Brownian motion.