Divided stimulus control depends on differential and nondifferential reinforcement: Testing a quantitative model

We investigated the effects of differential and nondifferential reinforcers on divided control by compound-stimulus dimensions. Six pigeons responded in a delayed matching-to-sample procedure in which a blue or yellow sample stimulus flashed on/off at a fast or slow rate, and subjects reported its color or alternation frequency. The dimension to report was unsignaled (Phase 1) or signaled (Phase 2). Correct responses were reinforced with a probability of .70, and the probability of reinforcers for errors varied across conditions. Comparison choice depended on reinforcer ratios for correct and incorrect responding; as the frequency of error reinforcers according to a dimension increased, control (measured by log d) by that dimension decreased and control by the other dimension increased. Davison and Nevin's (1999) model described data when the dimension to report was unsignaled, whereas model fits were poorer when it was signaled, perhaps due to carryover between conditions. We are the first to test this quantitative model of divided control with reinforcers for errors and when the dimension to report is signaled; hence, further research is needed to establish the model's generality. We question whether divided stimulus control is dimensional and suggest it may instead reflect joint control by compound stimuli and reinforcer ratios.

Theory of reinforcement schedules

The three principles of reinforcement are (1) events such as incentives and reinforcers increase the activity of an organism; (2) that activity is bounded by competition from other responses; and (3) animals approach incentives and their signs, guided by their temporal and physical conditions, together called the "contingencies of reinforcement." Mathematical models of each of these principles comprised mathematical principles of reinforcement (MPR; Killeen, 1994). Over the ensuing decades, MPR was extended to new experimental contexts. This article reviews the basic theory and its extensions to satiation, warm-up, extinction, sign tracking, pausing, and sequential control in progressive-ratio and multiple schedules. In the latter cases, a single equation balancing target and competing responses governs behavioral contrast and behavioral momentum. Momentum is intrinsic in the fundamental equations, as behavior unspools more slowly from highly aroused responses conditioned by higher rates of incitement than it does from responses from leaner contexts. Habits are responses that have accrued substantial behavioral momentum. Operant responses, being predictors of reinforcement, are approached by making them: The sight and feel of a paw on a lever is approached by placing paw on lever, as attempted for any sign of reinforcement. Behavior in concurrent schedules is governed by approach to momentarily richer patches (melioration). Applications of MPR in behavioral pharmacology and delay discounting are noted.

Is superstitious responding a matter of detectability? A replication of Killeen (1978)

Organisms may sometimes behave as if a contingency exists between behavior and consequences, even if this is not actually the case. Killeen (1978) suggested that such superstition occurs because of factors that bias subjects to behave "superstitiously" rather than because of failures of discrimination. We systematically replicated Killeen's experiment and compared contingency discrimination between different consequences. Six pigeons responded in a matching-to-sample procedure in which a response-independent or response-dependent stimulus change, food delivery, or blackout occurred. The pigeons reported whether the consequence was response dependent or response independent by choosing between two side keys. Discrimination was strongest after stimulus changes, weaker after blackouts, and weakest after food deliveries. These differences persisted even after additional training, suggesting asymmetries that may reflect differences in the disruptive effects of different consequences on remembering and/or behavioral mnemonics. Importantly, the pigeons were not biased to report response-dependent consequences unless that response was consistent with locational biases; that is, they behaved "superstitiously" when there was a reason to be biased to do so. These findings corroborate Killeen's and demonstrate that behavior may deviate from contingencies not necessarily because subjects cannot discriminate those contingencies but because they are biased to behave otherwise.

Primacy and recency effects in hierarchical renewal in rats

Previous studies on hierarchical resurgence produced mixed results regarding the order and magnitude of recurrence of responses trained initially (primacy effect) or more recently (recency effect). Although changes in contextual stimuli could explain such differences, in resurgence procedures contextual stimuli are not commonly presented, thus their effects on multiple operants trained sequentially remain unclear. Renewal procedures, in contrast, have been useful to determine the effects of exteroceptive contextual stimuli on response recurrence. Thus, primacy and recency effects were studied using a renewal procedure in which three contexts were presented sequentially. Lever presses by rats were reinforced on a different lever under each training context and were then exposed to extinction in a different context. Presses on a fourth lever were never reinforced. During renewal testing, the three training contexts were presented in the same or inverse order relative to training. A strong primacy effect was found in rats exposed to the original training order. Both primacy and recency effects were found when the rats were exposed to contexts in inverse order. These results suggest that the magnitude of renewal of hierarchically trained responses is affected by training order and order of presentation of contextual stimuli during testing.

Effects of brief post-sample cues signaling presence or absence of reinforcers in delayed matching-to-sample

When short-term memory is assessed in the delayed matching-to-sample (DMTS) procedure, performance is better when cues signal larger reinforcer magnitudes or higher reinforcer probabilities for correct responding. Previous studies demonstrating signaled-magnitude or signaled-probability effects presented cues for a prolonged period during the sample stimulus and/or retention interval. The present study asked whether a signaled-probability effect would occur with brief post-sample cues that signaled the presence or absence of reinforcement. Five pigeons responded in a DMTS task in which sample stimuli were sometimes followed by a 0.5-s cue signaling that reinforcers would either be available or not available in the current trial, and the retention interval varied from 0.5s to 20s. A reliable signaled-probability effect was found when reinforcers were arranged independently and for all correct responses, whereas a smaller, less systematic effect was found when reinforcers were arranged dependently and probabilistically. These findings highlight the importance of reinforcement contingencies and contingency discriminability in remembering, and add to the evidence showing that cues signaling differential reinforcement in DMTS may affect processes during the retention interval and comparison phase, rather than attention to the sample stimulus.

Machine learning with a snapshot of data: Spiking neural network 'predicts' reinforcement histories of pigeons' choice behavior

An accumulated body of choice research has demonstrated that choice behavior can be understood within the context of its history of reinforcement by measuring response patterns. Traditionally, work on predicting choice behaviors has been based on the relationship between the history of reinforcement—the reinforcer arrangement used in training conditions—and choice behavior. We suggest an alternative method that treats the reinforcement history as unknown and focuses only on operant choices to accurately predict (more precisely, retrodict) reinforcement histories. We trained machine learning models known as artificial spiking neural networks (SNNs) on previously published pigeon datasets to detect patterns in choices with specific reinforcement histories—seven arranged concurrent variable‐interval schedules in effect for nine reinforcers. Notably, SNN extracted information from a small ‘window’ of observational data to predict reinforcer arrangements. The models' generalization ability was then tested with new choices of the same pigeons to predict the type of schedule used in training. We examined whether the amount of the data provided affected the prediction accuracy and our results demonstrated that choices made by the pigeons immediately after the delivery of reinforcers provided sufficient information for the model to determine the reinforcement history. These results support the idea that SNNs can process small sets of behavioral data for pattern detection, when the reinforcement history is unknown. This novel approach can influence our decisions to determine appropriate interventions; it can be a valuable addition to our toolbox, for both therapy design and research.

Pigeons prefer to invest early for future reinforcers

Five pigeons were trained in a series of conditions in which food was delivered after 25 responses, but only when a different (Investing) response had been made before the 25 responses had been completed. If an Investing response was not made, the 25 responses ended in blackout. In various conditions, effective Investing responses either had to be made before the first of the 25 responses, or anywhere within the 25 responses; and effective Investing responses either resulted in a stimulus change or did not. Pigeons Invested even when the consequences were temporally and spatially distant, but Investing was most likely when it produced an immediate stimulus change. When given the choice, pigeons preferred to make Investing responses at the beginning of a trial. These findings again demonstrate that behavior may be maintained by events that are separated in time and space from the present.

Matching Behaviours and Rewards

Matching describes how behaviour is related to rewards. The matching law holds when the ratio of an individual's behaviours equals the ratio of the rewards obtained. From its origins in the study of pigeons working for food in the laboratory, the law has been applied to a range of species, both in the laboratory and outside it (e.g., human sporting decisions). Probability matching occurs when the probability of a behaviour equals the probability of being rewarded. Input matching predicts the distribution of individuals across habitats. We evaluate the rationality of the matching law and probability matching, expose the logic of matching in real-world cases, review how recent neuroscience findings relate to matching, and suggest future research directions.

On the "Strength" of Behavior

The place of the concept of response strength in a natural science of behavior has been the subject of much debate. This article reconsiders the concept of response strength for reasons linked to the foundations of a natural science of behavior. The notion of response strength is implicit in many radical behaviorists' work. Palmer (2009) makes it explicit by applying the response strength concept to three levels: (1) overt behavior, (2) covert behavior, and (3) latent or potential behavior. We argue that the concept of response strength is superfluous in general, and an explication of the notion of giving causal status to nonobservable events like latent behavior or response strength is harmful to a scientific endeavor. Interpreting EEG recordings as indicators of changes in response strength runs the risk of reducing behavior to underlying mechanisms, regardless of whether such suggestions are accompanied by behavioral observations. Many radical behaviorists understand behavior as a discrete unit, inviting conceptual mistakes reflected in the notion of response strength. A molar view is suggested as an alternative that accounts for the temporally extended nature of behavior and avoids the perils of a response-strength based approach.

Stimulus control depends on the subjective value of the outcome

Stimuli that provide information about likely future reinforcers tend to shift behavior, provided a reliable relation between the stimulus and the reinforcer can be discriminated. Stimuli that are apparently more reliable exert greater control over behavior. We asked how the subjective value (measured in terms of preference) of reinforcers associated with stimuli influences stimulus control. Five pigeons worked on a concurrent chains procedure in which half of all trials ended in a smaller reinforcer sooner, and the other half in a larger reinforcer later. In Signaled trials, the color and flash duration on the keys in the initial link signaled the outcome of the trial. In Conflicting probe trials, the color and the flash duration signaled conflicting information about the outcome of the trial. Choice in Signaled trials shifted toward the signaled outcome, but was never exclusive. In Conflicting probe trials, control was divided idiosyncratically between the 2 stimulus dimensions, but still favored the outcome with the higher subjective value. Thus, stimulus control depends not only on the perceived reliability of stimuli, but also on the subjective value of the outcome.

Successive incrementing non-matching-to-samples in rats: An automated version of the odor span task

The odor span task is a procedure frequently used to study remembering of multiple stimuli in rodents. A large arena is used and odor stimuli are presented using scented cups. Selection of each odor is reinforced when first presented, but not on subsequent presentations; correct selections depend on remembering which stimuli were previously presented. The use of an arena setting with manual stimulus presentation makes the odor span task labor-intensive and limits experimental control; thus, an automated version of the task would be of value. The present study used an operant chamber equipped with an olfactometer and trained rats using successive conditional discrimination procedures under an incrementing non-matching-to-samples contingency. High rates of responding developed to odor stimuli when they were session-novel with low rates of responding to subsequent presentations of that odor. Additional experiments assessed variations of the procedure to determine the role of the frequency of odor presentation and the retention interval separating sample and comparison. Discrimination was impaired with long retention intervals suggesting the importance of this variable. These findings confirmed that rats differentiate between stimuli that are session-novel and those previously encountered and support the use of an automated procedure as an alternative to the odor span task.

Effects of variable durations of food availability on interval time-place learning with pigeons Columba Livia

We assessed the effects of variable durations of food availability on an interval time-place learning task. 3 pigeons were exposed to a task in which food could be obtained for responses in one of four feeders according to an RI 25 s during 3 min, after which, food could be obtained on a different feeder according to the same schedule. The correct feeder changed following a fixed sequence that was repeated four times throughout the session. After 50 training sessions, an Open Hopper Test was conducted, after which, the second training condition ensued. This condition was like the first one with the exception that the availability period could be either 1,2,3, or 6 min long. A second test was conducted after 50 sessions of this training. Another group of 3 birds experienced these conditions in the reverse order. Data suggest that birds solved the task via interval timing under the fixed duration condition, and via ordinal timing when faced with variable durations. Birds learned the fixed sequence involved in the task under both conditions. Although the present data agree with previous research exploring variability in TPL tasks, they do not necessarily support previous claims for an asymmetrical role of spatial and temporal information in these tasks.

Generalizing from the Past, Choosing the Future

Behavior in the present depends critically on experience in similar environments in the past. Such past experience may be important in controlling behavior not because it determines the strength of a behavior, but because it allows the structure of the current environment to be detected and used. We explore a prospective-control approach to understanding simple behavior. Under this approach, order in the environment allows even simple organisms to use their personal past to respond according to the likely future. The predicted future controls behavior, and past experience forms the building blocks of the predicted future. We explore how generalization affects the use of past experience to predict and respond to the future. First, we consider how generalization across various dimensions of an event determines the degree to which the structure of the environment exerts control over behavior. Next, we explore generalization from the past to the present as the method of deciding when, where, and what to do. This prospective-control approach is measurable and testable; it builds predictions from events that have already occurred, and assumes no agency. Under this prospective-control approach, generalization is fundamental to understanding both adaptive and maladaptive behavior.

Being there on time: Reinforcer effects on timing and locating

In research on timing, reinforcers often are assumed to influence discrimination of elapsed time. We asked whether changes in choice used to measure timing arise because of joint control by elapsed time and reinforcers, rather than from the direct modification of control by elapsed time by reinforcers. Pigeons worked on a concurrent-choice task in which 1 response was 9 times more likely to produce a reinforcer, reversing between locations when 19 s had elapsed since the marker event. Across conditions, we manipulated the percentage of reinforcers arranged before the probability reversal from 5 to 95%. These changes in reinforcer percentages altered control by location-based elements of the contingency, but not by time-based elements. Choice was well described by a model that assumes that control by the contingency is weakened by generalization across the time and location of reinforcers, and that these generalizations become more likely at later times since a marker. These findings add to a growing body of research that suggests that reinforcers share the same function as other environmental events in determining how the environment controls behavior.

Out of the Lab: Shaping an Ecological and Constructional Cultural Systems Science

Contemporary societies face critical, interlocking, "wicked" challenges, including economic inequities and marginalization, personal and collective violence, ethnic and religious conflicts, degradation of "the commons," climate change, and more, and all of these issues clearly are grounded in behavior. An adequate culturo-behavior science could be positioned to advance and leverage research and interventions supporting community well-being, and contribute to overcoming urgent societal and global challenges. The current state of cultural systems science, however, is limited by theory and methodology, and by competition for attention with well-established research and practice opportunities related to individual-level challenges. In this article, the author explores those limitations, and suggests a more expansive perspective drawing on historical and contemporary ecological science and contemporary theories of complex systems. Research guided by established science within those disciplines offers opportunities to move cultural systems science out of the lab, and into a more adequate, environmentally rich stance drawing on ecological strategies, recursively integrating contextual observations, conceptual advances, and in vivo experimentation. Examples of each of those strategies and exploration of developmental programs of research grounded in such integration are explored.

Resurgence as Choice in Context: Treatment duration and on/off alternative reinforcement

Resurgence as Choice (RaC) is a quantitative theory suggesting that an increase in an extinguished target behavior with subsequent extinction of an alternative behavior (i.e., resurgence) is governed by the same processes as choice more generally. We present data from an experiment with rats examining a range of treatment durations with alternative reinforcement plus extinction and demonstrate that increases in treatment duration produce small but reliable decreases in resurgence. Although RaC predicted the relation between target responding and treatment duration, the model failed in other respects. First, contrary to predictions, the present experiment also replicated previous findings that exposure to cycling on/off alternative reinforcement reduces resurgence. Second, RaC did a poor job simultaneously accounting for target and alternative behaviors across conditions. We present a revised model incorporating a role for more local signaling effects of reinforcer deliveries or their absence on response allocation. Such signaling effects are suggested to impact response allocation above and beyond the values of the target and alternative behaviors as longer-term repositories of experience. The new model provides an excellent account of the data and can be viewed as an integration of RaC and a quantitative approximation of some aspects of Context Theory.

Discriminative control of saccade latencies

Recent studies have demonstrated that saccadic reaction times (SRTs) are influenced by the temporal regularities of dynamic environments (Vullings & Madelain, 2018). Here, we ask whether discriminative control (i.e., the possibility to use external stimuli signaling the future state of the environment) of latencies in a search task might be established using reinforcement contingencies. Eight participants made saccades within 80-750 ms toward a target displayed among distractors. We constructed two latency classes, "short" and "long," using the first and last quartiles of the individual baseline distributions. We then used a latency-contingent display paradigm in which finding the visual target among other items was made contingent upon specific SRTs. For a first group, the postsaccadic target was displayed only following short latencies with leftward saccades, and following long latencies with rightward saccades. The opposite was true for a second group. When short- and long-latency saccades were reinforced (i.e., the target was displayed) depending on the saccade direction, median latencies differed by 74 ms on average (all outside the 98% null hypothesis confidence intervals). Posttraining, in the absence of reinforcement, we still observed strong differences in latency distributions, averaging 64 ms for leftward versus rightward saccades. Our results demonstrate the discriminative control of SRTs, further supporting the effects of reinforcement learning for saccade. This study reveals that saccade triggering is finely controlled by learned temporal and spatial properties of the environment using predictive mechanisms.

Environment tracking and signal following in a reinforcer-ratio reversal procedure

Several studies suggest that the degree of control by reinforcer ratios (environment tracking) and by exteroceptive stimuli that signal future reinforcer availability (signal following) depends on environmental certainty: As reinforcers become more likely at one location, environmental contingencies exert stronger control and exteroceptive stimuli exert weaker control. This research has not yet been extended to environments in which reinforcer availability changes across time, even though such changes are present in most natural environments. Thus, in the present experiment, we examined environment tracking and signal following in a concurrent schedule in which the reinforcer ratio reversed to its reciprocal 30 s after a reinforcer delivery and keylight-color stimuli signaled the likely or definite time or location of the next reinforcer. Across conditions, we manipulated environmental certainty by varying the probability of reinforcer deliveries on the locally richer key. This made the location of future reinforcers at a particular time more or less certain, but did not change the overall reinforcer ratio. Changes in local environmental certainty had little to no effect on environment tracking and signal following; in all conditions, keylight-color stimuli strongly controlled choice and reinforcer ratios exerted weak control. The present findings suggest that the extent of environment tracking and signal following is primarily determined by global, not local, environmental certainty.

Does a negative discriminative stimulus function as a punishing consequence?

The study and use of punishment in behavioral treatments has been constrained by ethical concerns. However, there remains a need to reduce harmful behavior unable to be reduced by differential-reinforcement procedures. We investigated whether response-contingent presentation of a negative discriminative stimulus previously correlated with an absence of reinforcers would punish behavior maintained by positive reinforcers. Across four conditions, pigeons were trained to discriminate between a positive discriminative stimulus (S+) signaling the presence of food, and a negative discriminative stimulus (S-) signaling the absence of food. Once learned, every five responses on average to the S+ produced S- for a duration of 1.5 s. S+ response rate decreased for a majority of pigeons when responses produced S-, compared to when they did not, or when a neutral control stimulus was presented. In Condition 5, choice between two concurrently presented S+ alternatives shifted away from the alternative producing S-, despite a 1:1 reinforcer ratio. Therefore, presenting contingent S- stimuli punishes operant behavior maintained on simple schedules and in choice situations. Development of negative discriminative stimuli as punishers of operant behavior could provide an effective approach to behavioral treatments for problem behavior and subverting suboptimal choices involved in addictions.

Quantitative analysis of local-level resurgence

Resurgence is the recurrence of a previously reinforced and then extinguished behavior induced by the extinction of another more recently reinforced behavior. Resurgence provides insight into behavioral processes relevant to treatment relapse of a range of problem behaviors. Resurgence is typically studied across three phases: (1) reinforcement of a target response, (2) extinction of the target and concurrent reinforcement of an alternative response, and (3) extinction of the alternative response, resulting in the recurrence of target responding. Because each phase typically occurs successively and spans multiple sessions, extended time frames separate the training and resurgence of target responding. This study assessed resurgence more dynamically and throughout ongoing training in 6 pigeons. Baseline entailed 50-s trials of a free-operant psychophysical procedure, resembling Phases 1 and 2 of typical resurgence procedures. During the first 25 s, we reinforced target (left-key) responding but not alternative (right-key) responding; contingencies reversed during the second 25 s. Target and alternative responding followed the baseline reinforcement contingencies, with alternative responding replacing target responding across the 50 s. We observed resurgence of target responding during signaled and unsignaled probes that extended trial durations an additional 100 s in extinction. Furthermore, resurgence was greater and/or sooner when probes were signaled, suggesting an important role of discriminating transitions to extinction in resurgence. The data were well described by an extension of a stimulus-control model of discrimination that assumes resurgence is the result of generalization of obtained reinforcers across space and time. Therefore, the present findings introduce novel methods and quantitative analyses for assessing behavioral processes underlying resurgence.

Spatial midsession reversal learning in rats: Effects of egocentric Cue use and memory

The midsession reversal task has been used to investigate behavioral flexibility and cue use in non-human animals, with results indicating differences in the degree of control by environmental cues across species. For example, time-based control has been found in rats only when tested in a T-maze apparatus and under specific conditions in which position and orientation (i.e., egocentric) cues during the intertrial interval could not be used to aid performance. Other research in an operant setting has shown that rats often produce minimal errors around the reversal location, demonstrating response patterns similar to patterns exhibited by humans and primates in this task. The current study aimed to reduce, but not eliminate, the ability for rats to utilize egocentric cues by placing the response levers on the opposite wall of the chamber in relation to the pellet dispenser. Results showed that rats made minimal errors prior to the reversal, suggesting time-based cues were not controlling responses, and that they switched to the second correct stimulus within a few trials after the reversal event. Video recordings also revealed highly structured patterns of behavior by the majority of rats, which often differed depending on which response was reinforced. We interpret these findings as evidence that rats are adept at utilizing their own egocentric cues and that these cues, along with memory for the recent response-reinforcement contingencies, aid in maximizing reinforcement over the session.

The non-Darwinian evolution of behavers and behaviors

Many readers of this journal have been schooled in both Darwinian evolution and Skinnerian psychology, which have in common the vision of powerful control of their subjects by their sequalae. Individuals of species that generate more successful offspring come to dominate their habitat; responses of those individuals that generate more reinforcers come to dominate the repertoire of the individual in that context. This is unarguable. What is questionable is how large a role these forces of selection play in the larger landscape of existing organisms and the repertoires of their individuals. Here it is argued that non-Darwinian and non-Skinnerian selection play much larger roles in both than the reader may appreciate. The argument is based on the history of, and recent advances in, microbiology. Lessons from that history re-illuminate the three putative domains of selection by consequences: The evolution of species, response repertoires, and cultures. It is argued that before, beneath, and after the cosmically brief but crucial epoch of Darwinian evolution that shaped creatures such as ourselves, non-Darwinian forces pervade all three domains.