Moving Beyond Reinforcement and Response Strength

Reconceptualizing the Interaction of Behavior and Environment

The concept of response strength and the process of strengthening by reinforcement are controversial in terms of their explanatory power. We clarify potential theoretical misconceptions following from a strength-based account such as essentialist thinking and circular reasoning. These problems also arise in the practice of latent variable modeling in psychometrics. To solve these conceptual problems, we discuss the Multilevel Model of Behavioral Selection (MLBS; Borgstede & Eggert, 2021) as an alternative theoretical framework. We use blocking from Pavlovian conditioning as an example to demonstrate how the MLBS framework prevents misconceptions arising from strength-based accounts and how it provides a more parsimonious and coherent explanation of the phenomenon. We illustrate the need for precisely defined and theoretically meaningful concepts and offer a reinterpretation of "strengthening by reinforcement." The reconceptualization in terms of the MLBS renders the concept of response strength superfluous. We conclude by highlighting the importance of theoretical reconsideration, putting aside difficulties that arise when attempting to validate strength by empirical means.

Evaluating functions of praise for children diagnosed with autism spectrum disorder

We assessed whether novel praise statements could be used to (a) maintain and increase responses with existing reinforcement histories and (b) teach a previously untaught response among children diagnosed with autism spectrum disorder across two experiments. During response-stimulus pairing, two responses resulted in preferred edibles but only one also produced a praise statement. In the absence of edibles, the response continuing to produce praise tended to persist more. Next, reversing the praise contingency tended to increase the other response. However, in no case did contingent delivery of those same praise statements result in the acquisition of untaught responses. These findings suggest that conditioning praise statements could serve different functions (antecedent or consequence) depending on the reinforcement history for particular responses.

Hyperbolae

Hyperbolic relations between independent and dependent variables are ubiquitous in the experimental analysis of behavior, mentioned in over 150 articles in the Journal of the Experimental Analysis of Behavior. There are two principal forms of hyperbolae: The first describes the relation between response rate and reinforcement rate on variable-interval schedules of reinforcement; it rises asymptotically toward a maximum. The second describes the relation between the current equivalent value of an incentive and its delay or (im)probability; it falls from a maximum toward an asymptote of 0. Where do these come from? What do their parameters mean? How are they related? This article answers the first two questions and addresses the last.

A molar view of goal direction and habit

When we treat behavior within an evolutionary framework and as temporally extended flow, two fundamental questions arise: (a) What is an organism? and (b) What is behavior? An organism is a process that stays intact by constantly exchanging energy with the environment. It takes in resources and puts out waste. The behavior of an organism consists of those process parts of the organism process that make up the exchange. These activities serve the function of reproducing, which generally depends on surviving. Surviving and reproducing depend on responding to phylogenetically important events (PIEs). A PIE induces activities that enhance or mitigate the PIE. Organisms respond not only to a PIE but also to events that covary with the PIE. Both activities and environmental features may covary with a PIE. When either type of covariance is introduced to an organism, behavior adapts over time. The early stages of adaptation constitute what researchers call "goal direction," and the later stages constitute what researchers call "habit." Behavior and environment constitute a dynamic system, and manipulations of the covariances and environmental features of the system allow many experimental interventions. This molar approach allows experiments on goal direction and habit to be understood without appeal to everyday mentalistic terms.

Theory of reinforcement schedules

The three principles of reinforcement are (1) events such as incentives and reinforcers increase the activity of an organism; (2) that activity is bounded by competition from other responses; and (3) animals approach incentives and their signs, guided by their temporal and physical conditions, together called the "contingencies of reinforcement." Mathematical models of each of these principles comprised mathematical principles of reinforcement (MPR; Killeen, 1994). Over the ensuing decades, MPR was extended to new experimental contexts. This article reviews the basic theory and its extensions to satiation, warm-up, extinction, sign tracking, pausing, and sequential control in progressive-ratio and multiple schedules. In the latter cases, a single equation balancing target and competing responses governs behavioral contrast and behavioral momentum. Momentum is intrinsic in the fundamental equations, as behavior unspools more slowly from highly aroused responses conditioned by higher rates of incitement than it does from responses from leaner contexts. Habits are responses that have accrued substantial behavioral momentum. Operant responses, being predictors of reinforcement, are approached by making them: The sight and feel of a paw on a lever is approached by placing paw on lever, as attempted for any sign of reinforcement. Behavior in concurrent schedules is governed by approach to momentarily richer patches (melioration). Applications of MPR in behavioral pharmacology and delay discounting are noted.

Is superstitious responding a matter of detectability? A replication of Killeen (1978)

Organisms may sometimes behave as if a contingency exists between behavior and consequences, even if this is not actually the case. Killeen (1978) suggested that such superstition occurs because of factors that bias subjects to behave "superstitiously" rather than because of failures of discrimination. We systematically replicated Killeen's experiment and compared contingency discrimination between different consequences. Six pigeons responded in a matching-to-sample procedure in which a response-independent or response-dependent stimulus change, food delivery, or blackout occurred. The pigeons reported whether the consequence was response dependent or response independent by choosing between two side keys. Discrimination was strongest after stimulus changes, weaker after blackouts, and weakest after food deliveries. These differences persisted even after additional training, suggesting asymmetries that may reflect differences in the disruptive effects of different consequences on remembering and/or behavioral mnemonics. Importantly, the pigeons were not biased to report response-dependent consequences unless that response was consistent with locational biases; that is, they behaved "superstitiously" when there was a reason to be biased to do so. These findings corroborate Killeen's and demonstrate that behavior may deviate from contingencies not necessarily because subjects cannot discriminate those contingencies but because they are biased to behave otherwise.

Aversive control versus stimulus control by punishment

Estes (1944) reported that adding electric shock punishment to extinction hastened response suppression but that responding increased when shock was removed. This result contributed to a view that reinforcement and punishment are asymmetrical processes because punishment has only indirect and temporary suppressive effects. Azrin and Holz (1966) suggested the result might be interpreted instead as shock serving as a discriminative stimulus for the absence of reinforcement. Here, to further examine potential stimulus control by punishment in a similar preparation, two groups of rats initially responded for food plus punishment and a third group for food alone. Reinforcement was then removed for all groups for the remaining three phases. With P and N denoting punishment and no punishment, the four phases for the three groups were: P-P-N-N, P-N-P-N, and N-P-N-N. We found some evidence for stimulus control by shock deliveries for group N-P-N-N (as suggested by Azrin and Holz), but all other changes in responding appeared due to introduction or removal of the aversive properties of shock. Although punishment may indeed have temporary effects under many circumstances, we argue that the view that this implies asymmetrical reinforcement and punishment processes was based on the flawed assumption that reinforcement has direct strengthening effects.

A Theory of the Extinction Burst

A preliminary theory of a temporary increase in the rate of an operant response with the transition to extinction (i.e., the extinction burst) is proposed. The theory assumes reinforcers are events permitting access to some valuable activity, and that such activity can compete for allocation with the target response under some conditions (e.g., very high reinforcement rates). With the transition to extinction, elimination of this competition for allocation can produce an increase in the the target response, but the increase is transient because the value of the target response decreases with exposure to extinction. The theory provides a way to understand why the extinction burst is not ubiquitous, seems more common following very small ratio schedules, occurs for a short period of time following the transition to extinction, and may be eliminated with the availability of alternative reinforcement. It appears to provide a reasonable starting point for a theory of the extinction burst that does not necessarily require inclusion of invigorating effects of frustration, and it is closely aligned with Resurgence as Choice theory. Additional research on factors modulating reinforcement-related activities and how they affect the extinction burst could help to further evaluate the theory.

The matching law provides a quantitative description of social time allocation in children with autism

Recent research has developed and evaluated assessments of sociability in which time allocation near or away from an adult who initiates social interactions is used to characterize the participant as social, indifferent, or avoidant of social interaction. Though these qualitative outcomes have been useful, no studies have evaluated methods of obtaining more quantitative measures of sociability. The matching law has been demonstrated to describe a wide range of human behavior and may also be useful in describing social time allocation. We adapted the matching law and assessment of sociability procedures with the aim of providing a more precise, quantitative measure of sociability. We fitted the matching equation to the social time allocation data of 8 children with autism spectrum disorder. The equation was effective in quantifying sociability, accounted for a large proportion of variance in participants' behavior, did so equally well for participants who were social and avoidant, and provided a more sensitive measure relative to those used in previous research. The implications of this methodology, its potential utility, and directions for future research are discussed.

Machine learning with a snapshot of data: Spiking neural network 'predicts' reinforcement histories of pigeons' choice behavior

An accumulated body of choice research has demonstrated that choice behavior can be understood within the context of its history of reinforcement by measuring response patterns. Traditionally, work on predicting choice behaviors has been based on the relationship between the history of reinforcement—the reinforcer arrangement used in training conditions—and choice behavior. We suggest an alternative method that treats the reinforcement history as unknown and focuses only on operant choices to accurately predict (more precisely, retrodict) reinforcement histories. We trained machine learning models known as artificial spiking neural networks (SNNs) on previously published pigeon datasets to detect patterns in choices with specific reinforcement histories—seven arranged concurrent variable‐interval schedules in effect for nine reinforcers. Notably, SNN extracted information from a small ‘window’ of observational data to predict reinforcer arrangements. The models' generalization ability was then tested with new choices of the same pigeons to predict the type of schedule used in training. We examined whether the amount of the data provided affected the prediction accuracy and our results demonstrated that choices made by the pigeons immediately after the delivery of reinforcers provided sufficient information for the model to determine the reinforcement history. These results support the idea that SNNs can process small sets of behavioral data for pattern detection, when the reinforcement history is unknown. This novel approach can influence our decisions to determine appropriate interventions; it can be a valuable addition to our toolbox, for both therapy design and research.

Incrementing non-matching- but not matching-to-sample is rapidly learned in an automated version of the odor span task

The odor span task (OST) is frequently used to assess memory capacity in rodents. Odor stimuli are presented in a large arena and choices of session-novel odors produce food reward. The procedure can be described as an incrementing non-matching-to-sample contingency because on each trial one new stimulus is presented along with one or more previously presented (non-reinforced) comparison odors. An automated version of this task has recently been developed in which odors are presented with an olfactometer in an operant chamber using a successive conditional discrimination procedure. The present study compared the acquisition of matching- vs. non-matching-to-sample versions of the task with six rats tested under each procedure. All six rats trained on the non-matching variation showed rapid acquisition of the discrimination with high rates of responding to odor stimuli when they were session-novel and low rates of responding to subsequent presentations of those odors. However, only three of the six rats trained on the matching variation met acquisition criteria, and two of the three that did acquire the task required extensive training to do so. These results support findings from the OST that rats can differentiate between stimuli that are session-novel and those previously encountered, but also that a matching contingency is more difficult to learn than a non-matching arrangement. These findings parallel differences observed between acquisition of simple matching- and non-matching-to-sample tasks, but accounts such as novelty preference or the oddity preference effect may not be sufficient to explain the present results.

Pigeons prefer to invest early for future reinforcers

Five pigeons were trained in a series of conditions in which food was delivered after 25 responses, but only when a different (Investing) response had been made before the 25 responses had been completed. If an Investing response was not made, the 25 responses ended in blackout. In various conditions, effective Investing responses either had to be made before the first of the 25 responses, or anywhere within the 25 responses; and effective Investing responses either resulted in a stimulus change or did not. Pigeons Invested even when the consequences were temporally and spatially distant, but Investing was most likely when it produced an immediate stimulus change. When given the choice, pigeons preferred to make Investing responses at the beginning of a trial. These findings again demonstrate that behavior may be maintained by events that are separated in time and space from the present.

Resistance to extinction versus extinction as discrimination

The hypothesis that response strength might be measured by persistence of responding in the face of extinction was discredited in the 1960s because experiments showed that responding persists longer following intermittent reinforcers than following continuous reinforcers. Instead, researchers proposed that the longer persistence following intermittent reinforcers arises because intermittent reinforcement more closely resembles extinction-a discrimination theory. Attention to resistance to extinction revived because one observation seemed to support the persistence hypothesis: Following training on a multiple schedule with unequal components, responding usually persisted longer in the formerly richer component than in the formerly lean component. This observation represents an anomaly, however, because results with single schedules and concurrent schedules contradict it. We suggest that the difference in results arises because the multiple-schedule procedure, while including extensive training on stimulus discrimination, includes no training on discrimination between food available and food unavailable, whereas comparable single- and concurrent-schedule procedures include such training with repeated extinction. In Experiment 1, we replicated the original result, and in Experiment 2 showed that when the multiple-schedule procedure includes training on food/no-food discrimination, extinction following multiple schedules contradicts behavioral momentum theory and agrees with the discrimination theory and research with single and concurrent schedules.

Associative processes in addiction relapse models: A review of their Pavlovian and instrumental mechanisms, history, and terminology

Animal models of relapse to drug-seeking have borrowed heavily from associative learning approaches. In studies of relapse-like behaviour, animals learn to self-administer drugs then receive a period of extinction during which they learn to inhibit the operant response. Several triggers can produce a recovery of responding which form the basis of a variety of models. These include the passage of time (spontaneous recovery), drug availability (rapid reacquisition), extinction of an alternative response (resurgence), context change (renewal), drug priming, stress, and cues (reinstatement). In most cases, the behavioural processes driving extinction and recovery in operant drug self-administration studies are similar to those in the Pavlovian and behavioural literature, such as context effects. However, reinstatement in addiction studies have several differences with Pavlovian reinstatement, which have emerged over several decades, in experimental procedures, associative mechanisms, and terminology. Interestingly, in cue-induced reinstatement, drug-paired cues that are present during acquisition are omitted during lever extinction. The unextinguished drug-paired cue may limit the model’s translational relevance to cue exposure therapy and renders its underlying associative mechanisms ambiguous. We review major behavioural theories that explain recovery phenomena, with a particular focus on cue-induced reinstatement because it is a widely used model in addiction. We argue that cue-induced reinstatement may be explained by a combination of behavioural processes, including reacquisition of conditioned reinforcement and Pavlovian to Instrumental Transfer. While there are important differences between addiction studies and the behavioural literature in terminology and procedures, it is clear that understanding associative learning processes is essential for studying relapse.

Punishment and its putative fallout: A reappraisal

In his book Coercion and Its Fallout Murray Sidman argued against the use of punishment based on concerns about its shortcomings and side effects. Among his concerns were the temporary nature of response suppression produced by punishment, the dangers of conditioned punishment, increases in escape and avoidance responses, punishment-induced aggression, and the development of countercontrol. This paper revisits Sidman's arguments about these putative shortcomings and side effects by examining the available data. Although Sidman's concerns are reasonable and should be considered when using any form of behavioral control, there appears to be a lack of strong empirical support for the notion that these potential problems with punishment are necessarily ubiquitous, long-lasting, or specific to punishment. We describe the need for additional research on punishment in general, and especially on its putative shortcomings and side effects. We also suggest the need for more effective formal theories of punishment that provide a principled account of how, why, and when lasting effects of punishment and its potential side effects might be expected to occur or not. In addition to being necessary for a complete account of behavior, such data and theories might contribute to improved interventions for problems of human concern.

On the "Strength" of Behavior

The place of the concept of response strength in a natural science of behavior has been the subject of much debate. This article reconsiders the concept of response strength for reasons linked to the foundations of a natural science of behavior. The notion of response strength is implicit in many radical behaviorists' work. Palmer (2009) makes it explicit by applying the response strength concept to three levels: (1) overt behavior, (2) covert behavior, and (3) latent or potential behavior. We argue that the concept of response strength is superfluous in general, and an explication of the notion of giving causal status to nonobservable events like latent behavior or response strength is harmful to a scientific endeavor. Interpreting EEG recordings as indicators of changes in response strength runs the risk of reducing behavior to underlying mechanisms, regardless of whether such suggestions are accompanied by behavioral observations. Many radical behaviorists understand behavior as a discrete unit, inviting conceptual mistakes reflected in the notion of response strength. A molar view is suggested as an alternative that accounts for the temporally extended nature of behavior and avoids the perils of a response-strength based approach.

Resurgence and downshifts in alternative reinforcement rate

Resurgence refers to an increase in a previously suppressed target behavior with a relative worsening of conditions for a more recently reinforced alternative behavior. This experiment examined the relation between resurgence and the magnitude of a reduction in the rate of reinforcement for the alternative behavior. Groups of both male and female rats initially pressed a target lever for food on a variable-interval (VI) 30-s schedule. In a second phase, responding to the target lever was extinguished for all groups and pressing an alternative lever was reinforced on a VI 10-s schedule. Next, the rate of reinforcement for alternative behavior was reduced differentially across groups by arranging extinction, VI 80-s, VI 40-s, VI 20-s, or continued VI 10-s reinforcement. Target responding increased as an exponential function of the magnitude of the reduction in alternative reinforcement rates. With the exception that males appeared to show higher rates of target responding in baseline and higher rates of alternative responding in other phases, the overall pattern of responding across phases was not meaningfully different between sexes. The pattern of both target and alternative response rates across sessions and phases was well described quantitatively by the Resurgence as Choice in Context model.

Generalizing from the Past, Choosing the Future

Behavior in the present depends critically on experience in similar environments in the past. Such past experience may be important in controlling behavior not because it determines the strength of a behavior, but because it allows the structure of the current environment to be detected and used. We explore a prospective-control approach to understanding simple behavior. Under this approach, order in the environment allows even simple organisms to use their personal past to respond according to the likely future. The predicted future controls behavior, and past experience forms the building blocks of the predicted future. We explore how generalization affects the use of past experience to predict and respond to the future. First, we consider how generalization across various dimensions of an event determines the degree to which the structure of the environment exerts control over behavior. Next, we explore generalization from the past to the present as the method of deciding when, where, and what to do. This prospective-control approach is measurable and testable; it builds predictions from events that have already occurred, and assumes no agency. Under this prospective-control approach, generalization is fundamental to understanding both adaptive and maladaptive behavior.

Resurgence as Choice in Context: Treatment duration and on/off alternative reinforcement

Resurgence as Choice (RaC) is a quantitative theory suggesting that an increase in an extinguished target behavior with subsequent extinction of an alternative behavior (i.e., resurgence) is governed by the same processes as choice more generally. We present data from an experiment with rats examining a range of treatment durations with alternative reinforcement plus extinction and demonstrate that increases in treatment duration produce small but reliable decreases in resurgence. Although RaC predicted the relation between target responding and treatment duration, the model failed in other respects. First, contrary to predictions, the present experiment also replicated previous findings that exposure to cycling on/off alternative reinforcement reduces resurgence. Second, RaC did a poor job simultaneously accounting for target and alternative behaviors across conditions. We present a revised model incorporating a role for more local signaling effects of reinforcer deliveries or their absence on response allocation. Such signaling effects are suggested to impact response allocation above and beyond the values of the target and alternative behaviors as longer-term repositories of experience. The new model provides an excellent account of the data and can be viewed as an integration of RaC and a quantitative approximation of some aspects of Context Theory.

Effects of schedule type and terminal-link duration on choice in concurrent chains

Four pigeons responded in a three-component concurrent chains procedure in which the terminal-link schedules were either both fixed-interval (FI FI), both variable-interval (VI VI), both mixed-interval (MI MI) or variable-interval fixed-interval (VI FI). Across components within sessions, overall terminal-link duration was varied while schedule types were varied across conditions. For the conditions with homogeneous schedules, the strongest preference was obtained with FI FI, intermediate with MI MI, and weakest with VI VI. Preference increased with overall terminal-link duration for all schedules, but the increase was more rapid for VI FI. The hyperbolic value-added model (HVA) and cumulative decision model (CDM) were fitted to the choice data and accounted for 83.9% and 76.1% of the variance, respectively. However, deviations from both models' predictions were systematic. A modification of the CDM, which assumed that reinforcers delivered after variable delays were more effective, improved the fit of the model and substantially reduced the systematic deviations in residuals. Comparable modifications of HVA produced only limited improvement. Results show that preference in concurrent chains with homogeneous terminal links depends on the degree of variability in reinforcer delays, and that the CDM can provide an excellent account of results across different terminal-link schedules.

Still Lost in Translation

Interdisciplinary dialogue can help progress and improve professional fields. Progress and improvement may be impeded by barriers within an interdisciplinary dialogue, two of which are false dichotomies and faulty generalizations. It is our opinion that this particular interdisciplinary dialogue will advance productively by avoiding: false dichotomies about the medical model and Disability Studies in Education (DSE) framework; false dichotomies about using a normality approach to establish goals; and faulty generalizations about practitioners within and outside the field in which we are trained. Most helping professionals care passionately about improving the quality of lives of the people with whom they work. Centering interdisciplinary conversations on this shared value can help avoid false dichotomies and faulty generalizations.

Resurgence as Choice: Implications for promoting durable behavior change

Resurgence is an increase in a previously suppressed behavior resulting from a worsening in reinforcement conditions for current behavior. Resurgence is often observed following successful treatment of problem behavior with differential reinforcement when reinforcement for an alternative behavior is subsequently omitted or reduced. The efficacy of differential reinforcement has long been conceptualized in terms of quantitative models of choice between concurrent operants (i.e., the matching law). Here, we provide an overview of a novel quantitative model of resurgence called Resurgence as Choice (RaC), which suggests that resurgence results from these same basic choice processes. We review the failures of the only other quantitative model of resurgence (i.e., Behavioral Momentum Theory) and discuss its shortcomings with respect to the limited range of circumstances about which it makes predictions in applied settings. Finally, we describe how RaC overcomes these shortcomings and discuss implications of the model for promoting durable behavior change.

Suing for Peace in the War Against Mentalism

The antimentalists' war against mentalism has not vanquished it. To examine why, we focus on two theses-mind as causal and internal-and three standard attacks against mentalism as defined by both theses: 1) mentalism implies dualism; 2) mind is unobservable, which hinders its scientific study; and 3) mentalism is impractical. These salients fail because: 1) if the mind is causal and internal, it must be material; 2) the observable/unobservable distinction is too problematic, with antimentalists equivocal about where to draw that line, with some even embracing publicly unobservable behavior as causally relevant; and 3) mentalism has not been demonstrated to be less practical than antimentalism. For the war on mentalism to succeed, stronger attacks must be devised, both scientific and philosophical. We contemplate some possibilities, while expressing doubts as to the wisdom of continuing the war. Peace may be better than war, and the resulting intellectual commerce may be good for both sides.

Does a negative discriminative stimulus function as a punishing consequence?

The study and use of punishment in behavioral treatments has been constrained by ethical concerns. However, there remains a need to reduce harmful behavior unable to be reduced by differential-reinforcement procedures. We investigated whether response-contingent presentation of a negative discriminative stimulus previously correlated with an absence of reinforcers would punish behavior maintained by positive reinforcers. Across four conditions, pigeons were trained to discriminate between a positive discriminative stimulus (S+) signaling the presence of food, and a negative discriminative stimulus (S-) signaling the absence of food. Once learned, every five responses on average to the S+ produced S- for a duration of 1.5 s. S+ response rate decreased for a majority of pigeons when responses produced S-, compared to when they did not, or when a neutral control stimulus was presented. In Condition 5, choice between two concurrently presented S+ alternatives shifted away from the alternative producing S-, despite a 1:1 reinforcer ratio. Therefore, presenting contingent S- stimuli punishes operant behavior maintained on simple schedules and in choice situations. Development of negative discriminative stimuli as punishers of operant behavior could provide an effective approach to behavioral treatments for problem behavior and subverting suboptimal choices involved in addictions.

Effects of orientation and differential reinforcement II: transitivity and transfer across five-member sets

A recent report by Amd et al. (2017) demonstrated that orienting towards successively presented stimulus-stimulus pairs yielded significantly more transitive relations then when those same pairs were differentially reinforced following training for three, 3-member stimulus sets. We build on that work in four important ways. First, transitivity yields produced by Pavlovian and instrumental procedures were compared following training for three 5-member sets (A1-B1-C1-D1-E1, A2-B2-C2-D2-E2, A3-B3-C3-D3-E3), where the 'A' stimuli were emotional faces and all remaining stimuli were nonsense words. Second, our instrumental task here required two orienting/observing responses per trial. Third, we compared differences in multi-nodal transfer following Pavlovian and instrumental relational learning procedures. Finally, we tested whether functioning as 'end terms' in a relational series can mitigate transfer following instrumental conditioning. Transitivity, as measured by sorting tests, was significantly more pronounced following Pavlovian training. Transfer, assessed before and after relational training with two visual analog scales corresponding to valence and arousal dimensions, appeared marginally more robust observed for participants exposed to the Pavlovian condition. Transfer magnitude was positively related with demonstrations of transitivity, regardless of type of conditioning.