The unit of selection: what do reinforcers reinforce?

Coal Is Not Black, Snow Is Not White, Food Is Not a Reinforcer: The Roles of Affordances and Dispositions in the Analysis of Behavior

Reinforcers comprise sequences of actions in context. Just as the white of snow and black of coal depend on the interaction of an organism's visual system and the reflectances in its surrounds, reinforcers depend on an organism's motivational state and the affordances-possibilities for perception and action-in its surrounds. Reinforcers are not intrinsic to things but are a relation between what the thing affords, its context, the organism, and his or her history as capitulated in their current state. Reinforcers and other affordances are potentialities rather than intrinsic features. Realizing those potentialities requires motivational operations and stimulus contexts that change the state of the organism-they change its disposition to make the desired response. An expansion of the three-term contingency is suggested in order to help keep us mindful of the importance of behavioral systems, states, emotions, and dispositions in our research programs.

Goal-Directed Behavior and Instrumental Devaluation: A Neural System-Level Computational Model

Devaluation is the key experimental paradigm used to demonstrate the presence of instrumental behaviors guided by goals in mammals. We propose a neural system-level computational model to address the question of which brain mechanisms allow the current value of rewards to control instrumental actions. The model pivots on and shows the computational soundness of the hypothesis for which the internal representation of instrumental manipulanda (e.g., levers) activate the representation of rewards (or "action-outcomes", e.g., foods) while attributing to them a value which depends on the current internal state of the animal (e.g., satiation for some but not all foods). The model also proposes an initial hypothesis of the integrated system of key brain components supporting this process and allowing the recalled outcomes to bias action selection: (a) the sub-system formed by the basolateral amygdala and insular cortex acquiring the manipulanda-outcomes associations and attributing the current value to the outcomes; (b) three basal ganglia-cortical loops selecting respectively goals, associative sensory representations, and actions; (c) the cortico-cortical and striato-nigro-striatal neural pathways supporting the selection, and selection learning, of actions based on habits and goals. The model reproduces and explains the results of several devaluation experiments carried out with control rats and rats with pre- and post-training lesions of the basolateral amygdala, the nucleus accumbens core, the prelimbic cortex, and the dorso-medial striatum. The results support the soundness of the hypotheses of the model and show its capacity to integrate, at the system-level, the operations of the key brain structures underlying devaluation. Based on its hypotheses and predictions, the model also represents an operational framework to support the design and analysis of new experiments on the motivational aspects of goal-directed behavior.

Stress-induced modulation of instrumental behavior: from goal-directed to habitual control of action

Actions that are directed at achieving pleasant or avoiding unpleasant states are referred to as instrumental. The acquisition of instrumental actions can be controlled by two anatomically and functionally distinct processes: a goal-directed process that is based on the prefrontal cortex and dorsomedial striatum and encodes the causal relationship between an action and the motivational value of the outcome and a dorsolateral striatum-based habit process that learns associations between actions and antecedent stimuli. Here, we review recent research showing that stress modulates the control of instrumental action in a manner that favors habitual over goal-directed action. At the neuroendocrine level, this stress-induced shift towards habit action requires the concerted action of glucocorticoids and noradrenergic arousal and is most likely accompanied by opposite functional changes in the corticostriatal circuits underlying goal-directed and habitual actions. Although generally adaptive, these changes in the control of instrumental action under stress may promote dysfunctional behaviors and the development of psychiatric disorders such as addiction.

The dynamics of conditioning and extinction

Pigeons responded to intermittently reinforced classical conditioning trials with erratic bouts of responding to the conditioned stimulus. Responding depended on whether the prior trial contained a peck, food, or both. A linear persistence-learning model moved pigeons into and out of a response state, and a Weibull distribution for number of within-trial responses governed in-state pecking. Variations of trial and intertrial durations caused correlated changes in rate and probability of responding and in model parameters. A novel prediction--in the protracted absence of food, response rates can plateau above zero--was validated. The model predicted smooth acquisition functions when instantiated with the probability of food but a more accurate jagged learning curve when instantiated with trial-to-trial records of reinforcement. The Skinnerian parameter was dominant only when food could be accelerated or delayed by pecking. These experiments provide a framework for trial-by-trial accounts of conditioning and extinction that increases the information available from the data, permitting such accounts to comment more definitively on complex contemporary models of momentum and conditioning.

Autoshaping and automaintenance: a neural-network approach

This article presents an interpretation of autoshaping, and positive and negative automaintenance, based on a neural-network model. The model makes no distinction between operant and respondent learning mechanisms, and takes into account knowledge of hippocampal and dopaminergic systems. Four simulations were run, each one using an A-B-A design and four instances of feedfoward architectures. In A, networks received a positive contingency between inputs that simulated a conditioned stimulus (CS) and an input that simulated an unconditioned stimulus (US). Responding was simulated as an output activation that was neither elicited by nor required for the US. B was an omission-training procedure. Response directedness was defined as sensory feedback from responding, simulated as a dependence of other inputs on responding. In Simulation 1, the phenomena were simulated with a fully connected architecture and maximally intense response feedback. The other simulations used a partially connected architecture without competition between CS and response feedback. In Simulation 2, a maximally intense feedback resulted in substantial autoshaping and automaintenance. In Simulation 3, eliminating response feedback interfered substantially with autoshaping and automaintenance. In Simulation 4, intermediate autoshaping and automaintenance resulted from an intermediate response feedback. Implications for the operant-respondent distinction and the behavior-neuroscience relation are discussed.

Neural-network simulations of two context-dependence phenomena

This paper describes simulations of two context-dependence phenomena in Pavlovian conditioning, using a neural-network model that draws on knowledge from neuroscience and makes no distinction between operant and respondent learning mechanisms. One phenomenon is context specificity or the context-shift effect, the decrease of conditioned responding (CR) when the conditioned stimulus (CS) is tested in a context different from the one in which it had been paired with the unconditioned stimulus (US). The other effect is renewal, the recovery of CR in the training context after extinction in another context. For specificity (simulation 1), two neural networks were first given 200 CS-US pairings in a context. Then, the CS was tested either in the training context or a new context. Output activations in the new context were substantially lower. For renewal (simulation 2), two networks were first given 200 CS-US pairings in a context, then 100 extinction trials in either the same context or a new one, and then tested back in the training context. Output activations during the test phase were substantially higher after extinction in a new context. The results are interpreted in terms of the dynamics of activations and weights.

Motivating operations in appetite research

Appetite research frequently employs principles derived from behaviour analysis. However, it has yet to utilise the more recent theoretical advances in this field. This paper describes the concept of the motivating operation (MO)--a behaviour analytic formulation of motivation. An MO is an environmental event that (a) establishes or abolishes the reinforcing or punishing effect of another event and (b) evokes or abates behaviours associated with that event. The paper describes both unconditioned and conditioned MOs and the ways in which they may help account for a variety of eating behaviours. It then goes on to highlight the main ways in which the MO account differs from other theories of motivation employed in appetite research. These relate to (1) the ways in which they account for non-regulatory feeding, (2) the extent to which they address cognitive variables and (3) their underlying philosophical assumptions and subsequent relation to intervention.

Theoretical note: the C/T ratio in artificial neural networks

This paper describes computer simulations of the effect of the C/T ratio on acquisition rate in artificial neural networks. The networks consisted of neural processing elements that functioned according to a neurocomputational model whose learning rule is consistent with information on dopaminergic mechanisms of reinforcement. In Simulation 1, three comparisons were made: constant C and variable T, variable C and constant T, and a constant C/T with variable C and T. In the last two comparisons, C was manipulated by changing the probability of reinforcement within the intertrial interval (ITI), in the absence of the conditioned stimulus (CS). Acquisition rate tended to increase with C/T, and the invariant ratio had no effect. In Simulation 2, C was manipulated by changing the ITI, with continuous reinforcement in the presence of the CS and no reinforcements in its absence. Results were comparable to those obtained in Simulation 1. Simulation 3 further explored the effect of the invariant ratio, but with larger absolute values of C and T, which slowed acquisition significantly. The results parallel some experimental findings and theoretical implications of the Gibbon-Balsam model, showing that they can emerge from the moment-to-moment dynamics of a neural-network model. In contrast to that model, however, Simulation 3 suggests that the effect of invariant C/T ratios may be bounded.

Goal-directed instrumental action: contingency and incentive learning and their cortical substrates

Instrumental behaviour is controlled by two systems: a stimulus-response habit mechanism and a goal-directed process that involves two forms of learning. The first is learning about the instrumental contingency between the response and reward, whereas the second consists of the acquisition of incentive value by the reward. Evidence for contingency learning comes from studies of reward devaluation and from demonstrations that instrumental performance is sensitive not only the probability of contiguous reward but also to the probability of unpaired rewards. The process of incentive learning is evident in the acquisition of control over performance by primary motivational states. Preliminary lesion studies of the rat suggest that the prelimbic area of prefrontal cortex plays a role in the contingency learning, whereas the incentive learning for food rewards involves the insular cortex.