The error in total error reduction

Why is the Rescorla-Wagner model so influential?

The influence of the Rescorla-Wagner model cannot be overestimated, despite that (1) the model does not differ much computationally from its predecessors and competitors, and (2) its shortcomings are well-known in the learning community. Here we discuss the reasons behind its widespread influence in the cognitive and neural sciences, and argue that it is the constant search for general-process theories by learning scholars which eventually produced a model whose application spans many different areas of research to this day. We focus on the theoretical and empirical background of the model, the theoretical connections that it has with later developments across Marr's levels of analysis, as well as the broad variety of research that it has guided and inspired.

How common is a common error term? The rules that govern associative learning in sensory preconditioning and second-order conditioning

In standard (first-order) Pavlovian conditioning protocols, pairings of an initially neutral conditioned stimulus (CS) and a biologically significant unconditioned stimulus (US) result in the formation of a CS-US association. The strength of this association is theoretically regulated by prediction error: specifically, the difference between the total level of conditioning supported by the US and the degree to which it is predicted by all stimuli present (i.e., a common error term). In higher-order conditioning protocols (e.g., sensory preconditioning and second-order conditioning), a Pavlovian CS is used to condition responses to other stimuli with which it is paired. At present, it is unknown whether error-correction processes regulate associative learning in higher-order conditioning and, if so, whether these processes are the same as those that regulate formation of a CS-US association in first-order conditioning. Here we review studies that have provided findings relevant to this question: specifically, studies that have examined blocking and/or inhibitory learning in sensory preconditioning and second-order conditioning. These studies show that: (1) animals can form inhibitory associations between relatively neutral sensory stimuli; (2) the learning that occurs in sensory preconditioning and second-order conditioning can be blocked; and, finally, (3) a first-order CS can block conditioning to a second-order CS, and vice versa. The findings are taken to imply that a common error term regulates associative learning in higher-order conditioning, just as it regulates associative learning in first-order conditioning. They are discussed with respect to the nature of the error signal that underlies conditioning and future work that is needed to advance our understanding of the rules that govern different types of learning.

Agency rescues competition for credit assignment among predictive cues from adverse learning conditions

A fundamental assumption of learning theories is that the credit assigned to predictive cues is not simply determined by their probability of reinforcement, but by their ability to compete with other cues present during learning. This assumption has guided behavioral and neural science research for decades, and tremendous empirical and theoretical advances have been made identifying the mechanisms of cue competition. However, when learning conditions are not optimal (e.g., when training is massed), cue competition is attenuated. This failure of the learning system exposes the individual’s vulnerability to form spurious associations in the real world. Here, we uncover that cue competition in rats can be rescued when conditions are suboptimal provided that the individual has agency over the learning experience. Our findings reveal a new effect of agency over learning on credit assignment among predictive cues, and open new avenues of investigation into the underlying mechanisms.

Excitatory second-order conditioning using a backward first-order conditioned stimulus: A challenge for prediction error reduction

Prével and colleagues reported excitatory learning with a backward conditioned stimulus (CS) in a conditioned reinforcement preparation. Their results add to existing evidence of backward CSs sometimes being excitatory and were viewed as challenging the view that learning is driven by prediction error reduction, which assumes that only predictive (i.e., forward) relationships are learned. The results instead were consistent with the assumptions of both Miller’s Temporal Coding Hypothesis and Wagner’s Sometimes Opponent Processes (SOP) model. The present experiment extended the conditioned reinforcement preparation developed by Prével et al. to a backward second-order conditioning preparation, with the aim of discriminating between these two accounts. We tested whether a second-order CS can serve as an effective conditioned reinforcer, even when the first-order CS with which it was paired is a backward CS that elicits no responding. Evidence of conditioned reinforcement was found, despite no conditioned response (CR) being elicited by the first-order backward CS. The evidence of second-order conditioning in the absence of excitatory conditioning to the first-order CS is interpreted as a challenge to SOP. In contrast, the present results are consistent with the Temporal Coding Hypothesis and constitute a conceptual replication in humans of previous reports of excitatory second-order conditioning in rodents with a backward CS. The proposal is made that learning is driven by “discrepancy” with prior experience as opposed to “ prediction error.”

Methods of comparing associative models and an application to retrospective revaluation

Contemporary theories of associative learning are increasingly complex, which necessitates the use of computational methods to reveal predictions of these models. We argue that comparisons across multiple models in terms of goodness of fit to empirical data from experiments often reveal more about the actual mechanisms of learning and behavior than do simulations of only a single model. Such comparisons are best made when the values of free parameters are discovered through some optimization procedure based on the specific data being fit (e.g., hill climbing), so that the comparisons hinge on the psychological mechanisms assumed by each model rather than being biased by using parameters that differ in quality across models with respect to the data being fit. Statistics like the Bayesian information criterion facilitate comparisons among models that have different numbers of free parameters. These issues are examined using retrospective revaluation data.

Retrospective revaluation: The phenomenon and its theoretical implications

Retrospective revaluation refers to an increase (or decrease) in responding to conditioned stimulus (CS X) as a result of decreasing (or increasing) the associative strength of another CS (A) with respect to the unconditioned stimulus (i.e., A-US) that was previously trained in compound with the target CS (e.g., AX-US or just AX). We discuss the conditions under which retrospective revaluation phenomena are most apt to be observed and their implications for various models of learning that are able to account for retrospective revaluation (e.g., Dickinson and Burke, 1996; Miller and Matzel, 1988; Van Hamme and Wasserman, 1994). Although retroactive revaluation is relatively parameter specific, it is seen to be a reliable phenomenon observed across many tasks and species. As it is not anticipated by many conventional models of learning (e.g., Rescorla and Wagner, 1972), it serves as a critical benchmark for evaluating traditional and newer models.

The functions of contexts in associative learning

Although contexts play many roles during training and also during testing, over the last four decades theories of learning have predominantly focused on one or the other of the two families of functions served by contexts. In this selective review, we summarize recent data concerning these two functions and their interrelationship. The first function is similar to that of discrete cues, and allows contexts to elicit conditioned responses and compete with discrete events for behavioral control. The second function is modulatory, and similar to that of discrete occasion setters in that in this role contexts do not elicit conditioned responses by themselves, but rather modulate instrumental responding or responding to Pavlovian cues. We first present evidence for these two functions, and then suggest that the spacing of trials, amount of training, and contiguity are three determinants of the degree to which the context will play each function. We also conclude that these two functions are not mutually exclusive, and that future research would benefit from identifying the conditions under which their functions dominate behavioral control. We close by discussing some misconceptions concerning contexts. This article is part of a Special Issue entitled: SQAB 2013: Contextual Con.