Conditioned reinforcement and backward association

Early Auditory Event Related Potentials Distinguish Higher-Order From First-Order Aversive Conditioning

Stimuli in reality rarely co-occur with primary reward or punishment to allow direct associative learning of value. Instead, value is thought to be inferred through complex higher-order associations. Rodent research has demonstrated that the formation and maintenance of first-order and higher-order associations are supported by distinct neural substrates. In this study, we explored whether this pattern of findings held true for humans. Participants underwent first-order and subsequent higher-order conditioning using an aversive burst of white noise or neutral tone as the unconditioned stimuli. Four distinct tones, initially neutral, served as first-order and higher-order conditioned stimuli. Autonomic and neural responses were indexed by pupillometry and evoked response potentials (ERPs) respectively. Conditioned aversive values of first-order and higher-order stimuli led to increased autonomic responses, as indexed by pupil dilation. Distinct temporo-spatial auditory evoked response potentials were elicited by first-order and high-order conditioned stimuli. Conditioned first-order responses peaked around 260 ms and source estimation suggested a primary medial prefrontal and amygdala source. Conversely, conditioned higher-order responses peaked around 120 ms with an estimated source in the medial temporal lobe. Interestingly, pupillometry responses to first-order conditioned stimuli were diminished after higher order training, possibly signifying concomitant incidental extinction, while responses to higher-order stimuli remained. This suggests that once formed, higher order associations are at least partially independent of first order conditioned representations. This experiment demonstrates that first-order and higher-order conditioned associations have distinct neural signatures, and like rodents, the medial temporal lobe may be specifically involved with higher-order conditioning.

Higher-Order Conditioning With Simultaneous and Backward Conditioned Stimulus: Implications for Models of Pavlovian Conditioning

In a new environment, humans and animals can detect and learn that cues predict meaningful outcomes, and use this information to adapt their responses. This process is termed Pavlovian conditioning. Pavlovian conditioning is also observed for stimuli that predict outcome-associated cues; a second type of conditioning is termed higher-order Pavlovian conditioning. In this review, we will focus on higher-order conditioning studies with simultaneous and backward conditioned stimuli. We will examine how the results from these experiments pose a challenge to models of Pavlovian conditioning like the Temporal Difference (TD) models, in which learning is mainly driven by reward prediction errors. Contrasting with this view, the results suggest that humans and animals can form complex representations of the (temporal) structure of the task, and use this information to guide behavior, which seems consistent with model-based reinforcement learning. Future investigations involving these procedures could result in important new insights on the mechanisms that underlie Pavlovian conditioning.

Bed nucleus of the stria terminalis regulates fear to unpredictable threat signals

The bed nucleus of the stria terminalis (BNST) has been implicated in conditioned fear and anxiety, but the specific factors that engage the BNST in defensive behaviors are unclear. Here we examined whether the BNST mediates freezing to conditioned stimuli (CSs) that poorly predict the onset of aversive unconditioned stimuli (USs) in rats. Reversible inactivation of the BNST selectively reduced freezing to CSs that poorly signaled US onset (e.g., a backward CS that followed the US), but did not eliminate freezing to forward CSs even when they predicted USs of variable intensity. Additionally, backward (but not forward) CSs selectively increased Fos in the ventral BNST and in BNST-projecting neurons in the infralimbic region of the medial prefrontal cortex (mPFC), but not in the hippocampus or amygdala. These data reveal that BNST circuits regulate fear to unpredictable threats, which may be critical to the etiology and expression of anxiety.

Excitatory second-order conditioning using a backward first-order conditioned stimulus: A challenge for prediction error reduction

Prével and colleagues reported excitatory learning with a backward conditioned stimulus (CS) in a conditioned reinforcement preparation. Their results add to existing evidence of backward CSs sometimes being excitatory and were viewed as challenging the view that learning is driven by prediction error reduction, which assumes that only predictive (i.e., forward) relationships are learned. The results instead were consistent with the assumptions of both Miller’s Temporal Coding Hypothesis and Wagner’s Sometimes Opponent Processes (SOP) model. The present experiment extended the conditioned reinforcement preparation developed by Prével et al. to a backward second-order conditioning preparation, with the aim of discriminating between these two accounts. We tested whether a second-order CS can serve as an effective conditioned reinforcer, even when the first-order CS with which it was paired is a backward CS that elicits no responding. Evidence of conditioned reinforcement was found, despite no conditioned response (CR) being elicited by the first-order backward CS. The evidence of second-order conditioning in the absence of excitatory conditioning to the first-order CS is interpreted as a challenge to SOP. In contrast, the present results are consistent with the Temporal Coding Hypothesis and constitute a conceptual replication in humans of previous reports of excitatory second-order conditioning in rodents with a backward CS. The proposal is made that learning is driven by “discrepancy” with prior experience as opposed to “ prediction error.”

Toward a Unifying Account of Impulsivity and the Development of Self-Control

Impulsivity has traditionally been thought to involve various behavioral traits that can be measured using different laboratory protocols. Whereas some authors regard different measures of impulsivity as reflecting fundamentally distinct and unrelated behavioral tendencies (fragmentation approach), others regard those different indexes as analogue forms of the same behavioral tendency, only superficially different (unification approach). Unifying accounts range from mere intuitions to more sophisticated theoretical systems. Some of the more complete attempts at unifying are intriguing but have validity weaknesses. We propose a new unifying attempt based on theoretical points posed by other authors and supplemented by theory and research on associative learning. We then apply these assumptions to characterize the paradigms used to study impulsivity in laboratory settings and evaluate their scope as an attempt at unification. We argue that our approach possesses a good balance of parsimony and empirical and theoretical grounding, as well as a more encompassing scope, and is more suitable for experimental testing than previous theoretical frameworks. In addition, the proposed approach is capable of generating a new definition of impulsivity and outlines a hypothesis of how self-control can be developed. Finally, we examine the fragmentation approach from a different perspective, emphasizing the importance of finding similarities among seemingly different phenomena.

A conditioned reinforcer did not help to maintain an operant conditioning in the absence of a primary reinforcer in horses

The use of conditioned reinforcers is increasingly promoted in animal training. Surprisingly, the efficiency of their use remains to be demonstrated in horses. This study aimed to determine whether an auditory signal which had previously been associated with a food reward 288 times could be used as a conditioned reinforcer to replace the primary reinforcer in an unrelated operant conditioning procedure. Fourteen horses were divided into two groups of 7: No Reinforcement (NR) and Conditioned Reinforcement (CR). All horses underwent nine sessions of Pavlovian conditioning during which the word "good" was associated with food (32 associations/session). The horses then followed five sessions of operant conditioning (30 trials/session) during which they had to touch a cone signaled by an experimenter to receive a food reward. The last day, horses underwent one test session of the operant response: no reward was given, but the word "good" was said each time a CR horse touched the cone. Nothing was said in the NR group. CR horses did not achieve more correct trials than NR horses during the test. These findings again show that the conditioned reinforcement was ineffective when used instead of the primary reinforcement to maintain conditioning.