Behavior systems, associationism, and Pavlovian conditioning

Pavlovian or associative sensitization and its biological significance

Pavlovian conditioning is typically distinguished from sensitization but a Pavlovian conditional stimulus (CS) also results in sensitization. A Pavlovian CS can sensitize responding to a probe stimulus that is related to the unconditional stimulus (US) or to the US itself. Pavlovian sensitization has been studied in the defensive, sexual, and feeding systems. In Pavlovian sensitization, the focus is not on a conditional response (CR) directly elicited by the CS but on the response mode that is activated by the CS. Activation of a response mode increases the probability of particular responses and also increases reactivity to various stimuli. Pavlovian sensitization reflects this increased stimulus reactivity. Pavlovian sensitization helps uncover successful learning in situations where a conventional CR does not occur. Pavlovian sensitization also encourages broadening our conceptions of Pavlovian conditioning to include changes in afferent processes. Implications for biological fitness and for basic and translational research are discussed.

The behavioral origins of phylogenic responses and ontogenic habits

An examination of innate behavior and its possible origins suggests parallels with the formation of habitual behavior. Inflexible but adaptive responses-innate reflexive behavior, Pavlovian conditioned responses, and operant habits-may have evolved from variable behavior in phylogeny and ontogeny. This form of "plasticity-first" scientific narrative was unpopular post-Darwin but has recently gained credibility in evolutionary biology. The present article seeks to identify originating events and contingencies contributing to such inflexible but adaptive behavior at both phylogenic and ontogenic levels of selection. In ontogeny, the development of inflexible performance (i.e., habit) from variable operant behavior is reminiscent of the genetic accommodation of initially variable phylogenic traits. The effects characteristic of habit (e.g., unresponsiveness to reinforcer devaluation) are explicable as the result of a conflict between behaviors at distinct levels of selection. The present interpretation validates the practice of seeking hard analogies between evolutionary biology and operant behavior. Finding such parallels implies the validity of a claim that organismal behavior, both innate and learned, is a product of selection by consequences. A complete and coherent account of organismal behavior may ultimately focus on functional selective histories in much the same way evolutionary biology does with its subject matter.

Information and the Umwelt: A theoretical framework for the evolution of play

Play is phylogenetically widespread, and there are many proposed theories and fitness benefits of play. However, we still need a theoretical framework that unifies our understanding of the benefits that facilitated the evolution of play in so many diverse species. Starting with von Uexküll's theory of the Umwelt (i.e., the sensory-motor worlds of animals), together with the behavior systems approach, we propose that the Umwelt is an information processing system that serves basic biological functions. During development, the Umwelt undergoes a rapid expansion in the sensory and motor stimuli it processes. We argue that play is a process that converts surplus resources into information. By increasing the information content of the developing Umwelt, play confers fitness benefits. To demonstrate that play could evolve based on its information benefits, we present a model and simulation results of the evolution of a social play learning process that provides fitness-enhancing information in adult cooperative and competitive situations. Finally, we discuss this information-theoretic framework in relation to proposed hypotheses and fitness benefits of play.

Learning and organization of within-session sequences by pigeons (Columba livia)

Most animals engage in complex activities that are the combination of simpler actions expressed over a period of time. The mechanisms organizing such sequential behavior have been of long-standing biological and psychological interest. Previously, we observed pigeons' anticipatory behavior with a within-session sequence involving four choice alternatives suggestive of a potential understanding of the overall order and sequence of the items within a session. In that task, each colored alternative was correct for 24 consecutive trials as presented in a predictable sequence (i.e., A first, then B, then C, then D). To test whether these four already-trained pigeons possessed a sequential and linked representation of the ABCD items, we added a second four-item sequence involving new and distinct colored choice alternatives (i.e., E first for 24 trials, then F, then G, then H) and then alternated these ABCD and EFGH sequences over successive sessions. Over three manipulations, we tested and trained trials composed of combinations of elements from both sequences. We determined that pigeons did not learn any within-sequence associations among the elements. Despite the availability and explicit utility of such sequence cues, the data suggest instead that pigeons learned the discrimination tasks as a series of temporal associations among independent elements. This absence of any sequential linkage is consistent with the hypothesis that such representations are difficult to form in pigeons. This pattern of data suggests that for repeated sequential activities in birds, and potentially other animals including humans, there are highly effective, but underappreciated, clock-like mechanisms that control the ordering of behaviors.

Pupillary dynamics of mice performing a Pavlovian delay conditioning task reflect reward-predictive signals

Pupils can signify various internal processes and states, such as attention, arousal, and working memory. Changes in pupil size have been associated with learning speed, prediction of future events, and deviations from the prediction in human studies. However, the detailed relationships between pupil size changes and prediction are unclear. We explored pupil size dynamics in mice performing a Pavlovian delay conditioning task. A head-fixed experimental setup combined with deep-learning-based image analysis enabled us to reduce spontaneous locomotor activity and to track the precise dynamics of pupil size of behaving mice. By setting up two experimental groups, one for which mice were able to predict reward in the Pavlovian delay conditioning task and the other for which mice were not, we demonstrated that the pupil size of mice is modulated by reward prediction and consumption, as well as body movements, but not by unpredicted reward delivery. Furthermore, we clarified that pupil size is still modulated by reward prediction even after the disruption of body movements by intraperitoneal injection of haloperidol, a dopamine D2 receptor antagonist. These results suggest that changes in pupil size reflect reward prediction signals. Thus, we provide important evidence to reconsider the neuronal circuit involved in computing reward prediction error. This integrative approach of behavioral analysis, image analysis, pupillometry, and pharmacological manipulation will pave the way for understanding the psychological and neurobiological mechanisms of reward prediction and the prediction errors essential to learning and behavior.

Anxiety, fear, panic: An approach to assessing the defensive behavior system across the predatory imminence continuum

In order to effectively thwart predation, antipredator defensive behaviors must be matched to the current spatio-temporal relationship to the predator. We have proposed a model where different defensive responses are organized along a predatory imminence continuum (PIC). The PIC is a behavior system organized as a sequence of innately programmed behavioral modes, each representing a different interaction with the predator or threat. Ranging from low threat to predator contact, the PIC categorizes defense modes as pre-encounter, post-encounter, and circa-strike, corresponding to states of anxiety, fear, and panic, respectively. This experiment examined if the same significant stressor caused overexpression of all defensive responses along the PIC, including anxiety-like behavior, freezing, and panic-like responses. Female and male mice were exposed to acute stress that consisted of a series of ten pseudorandomly presented unsignaled footshocks (or no shocks). Mice were subsequently tested on a battery of tasks to assess stress effects on pre-encounter (anxiety-like), post-encounter (fear), and circa-strike (panic-like) behaviors. Results revealed that following stress, mice exhibited increased anxiety-like behavior shown through reduced average velocity within a modified open field. Furthermore, stressed mice showed increased fear following a single footshock in a new context as well as an increase in reactivity to white noise in the original stress context, with stressed mice exhibiting a more robust circa-strike-like response than controls. Therefore, significant stress exposure influenced the defensive states of anxiety, fear, and panic across the predatory imminence continuum. This research could therefore reveal how such responses become maladaptive following traumatic stress in humans.

The central nucleus of the amygdala and the construction of defensive modes across the threat-imminence continuum

In nature, animals display defensive behaviors that reflect the spatiotemporal distance of threats. Laboratory-based paradigms that elicit specific defensive responses in rodents have provided valuable insight into the brain mechanisms that mediate the construction of defensive modes with varying degrees of threat imminence. In this Review, we discuss accumulating evidence that the central nucleus of the amygdala (CeA) plays a key role in this process. Specifically, we propose that the mutually inhibitory circuits of the CeA use a winner-takes-all strategy that supports transitioning across defensive modes and the execution of specific defensive behaviors to previously formed threat associations. Our proposal provides a conceptual framework in which seemingly divergent observations regarding CeA function can be interpreted and identifies various areas of priority for future research.

More opportunities to peck for identical food availability increases foraging efficiency in pigeons

The opportunity and the information available to secure food resources drives foraging behaviour. We tested how inconsistent hole-food pairings and coverings could alter foraging performance, even when food availability is held constant. In our first experiment, we exposed pigeons (Columba livia) to a board in which each of the 60 covered holes contained one food item and to another board in which only one third of the 180 covered holes randomly contained one food item. In a second experiment, only the 60-hole board was used and the holes were not covered. The pigeons increased their body weight, gave fewer pecks per hole, revisited holes less often, and inspected fewer adjacent holes with 180 rather than 60 covered holes while eating similar amounts. However, their pecks were disproportionately higher near the edges of the board with 60 covered holes. This behaviour was not evident in the second experiment, when the food items were visible and individuals could know where food was available. Thus, the information about food location may drive foraging behaviour more directly than the information about food availability.

Neuronal metabolism in learning and memory: The anticipatory activity perspective

Current research on the molecular mechanisms of learning and memory is based on the "stimulus-response" paradigm, in which the neural circuits connecting environmental events with behavioral responses are strengthened. By contrast, cognitive and systems neuroscience emphasize the intrinsic activity of the brain that integrates information, establishes anticipatory actions, executes adaptive actions, and assesses the outcome via regulatory feedback mechanisms. We believe that the difference in the perspectives of systems and molecular studies is a major roadblock to further progress toward understanding the mechanisms of learning and memory. Here, we briefly overview the current studies in molecular mechanisms of learning and memory and propose that studying the predictive properties of neuronal metabolism will significantly advance our knowledge of how intrinsic, predictive activity of neurons shapes a new learning event. We further suggest that predictive metabolic changes in the brain may also take place in non-neuronal cells, including those of peripheral tissues. Finally, we present a path forward toward more in-depth studies of the role of cell metabolism in learning and memory.

Detailed mapping of behavior reveals the formation of prelimbic neural ensembles across operant learning

The prelimbic cortex (PrL) is involved in the organization of operant behaviors, but the relationship between longitudinal PrL neural activity and operant learning and performance is unknown. Here, we developed deep behavior mapping (DBM) to identify behavioral microstates in video recordings. We combined DBM with longitudinal calcium imaging to quantify behavioral tuning in PrL neurons as mice learned an operant task. We found that a subset of PrL neurons were strongly tuned to highly specific behavioral microstates, both task and non-task related. Overlapping neural ensembles were tiled across consecutive microstates in the response-reinforcer sequence, forming a continuous map. As mice learned the operant task, weakly tuned neurons were recruited into new ensembles, with a bias toward behaviors similar to their initial tuning. In summary, our data suggest that the PrL contains neural ensembles that jointly encode a map of behavioral states that is fine grained, is continuous, and grows during operant learning.

Beyond Single Discrete Responses: An Integrative and Multidimensional Analysis of Behavioral Dynamics Assisted by Machine Learning

Understanding behavioral systems as emergent systems comprising the environment and organism subsystems, include spatial dynamics as a primary dimension in natural settings. Nevertheless, under the standard approaches, the experimental analysis of behavior is based on the single response paradigm and the temporal distribution of discrete responses. Thus, the continuous analysis of spatial behavioral dynamics is a scarcely studied field. The technological advancements in computer vision have opened new methodological perspectives for the continuous sensing of spatial behavior. With the application of such advancements, recent studies suggest that there are multiple features embedded in the spatial dynamics of behavior, such as entropy, and that they are affected by programmed stimuli (e.g., schedules of reinforcement) at least as much as features related to discrete responses. Despite the progress, the characterization of behavioral systems is still segmented, and integrated data analysis and representations between discrete responses and continuous spatial behavior are exiguous in the experimental analysis of behavior. Machine learning advancements, such as t-distributed stochastic neighbor embedding and variable ranking, provide invaluable tools to crystallize an integrated approach for analyzing and representing multidimensional behavioral data. Under this rationale, the present work (1) proposes a multidisciplinary approach for the integrative and multilevel analysis of behavioral systems, (2) provides sensitive behavioral measures based on spatial dynamics and helpful data representations to study behavioral systems, and (3) reveals behavioral aspects usually ignored under the standard approaches in the experimental analysis of behavior. To exemplify and evaluate our approach, the spatial dynamics embedded in phenomena relevant to behavioral science, namely, water-seeking behavior and motivational operations, are examined, showing aspects of behavioral systems hidden until now.

Effort-motivated behavior resolves paradoxes in appetitive conditioning

Motivated behavior has long been studied by psychologists, ethologists, and neuroscientists. To date, many scientists agree with the view that cue and reward attraction is the product of a dopamine-dependent unconscious process called incentive salience or "wanting". This process allows the influence of multiple factors such as hunger and odors on motivational attraction. In some cases, however, the resulting motivated behavior differs from what the incentive salience hypothesis would predict. I argue that seeking behavior under reward uncertainty illustrates this situation: Organisms do not just "want" (appetite-based attraction) cues that are inconsistent or associated with reward occasionally, they "hope" that those cues will consistently predict reward procurement in the ongoing trial. Said otherwise, they become motivated to invest time and energy to find consistent cue-reward associations despite no guarantee of success (effort-based attraction). A multi-test comparison of performance between individuals trained under uncertainty and certainty reveals behavioral paradoxes suggesting that the concept of incentive salience cannot fully account for responding to inconsistent cues. A mathematical model explains how appetite-based and effort-based attractions might combine their effects.

The lesser evil: Pavlovian-instrumental transfer & aversive motivation

While our understanding of appetitive motivation includes accounts of rich cognitive phenomena, such as choice, sensory-specificity and outcome valuation, the same is not true in aversive processes. A highly sophisticated picture has emerged of Pavlovian fear conditioning and extinction, but progress in aversive motivation has been somewhat limited to these fundamental behaviors. Many differences between appetitive and aversive stimuli permit different kinds of analyses; a widely used procedure in appetitive studies that can expand the scope of aversive motivation is Pavlovian-instrumental transfer (PIT). Recently, this motivational transfer effect has been used to examine issues pertaining to sensory-specificity and the nature of defensive control in avoidance learning. Given enduring controversies and unresolved criticisms surrounding avoidance research, PIT offers a valuable, well-controlled procedure with which to similarly probe this form of motivation. Furthermore, while avoidance itself can be criticized as artificial, PIT can be an effective model for how skills learned through avoidance can be practically applied to encounters with threatening or fearful stimuli and stress. Despite sensory-related challenges presented by the limited aversive unconditioned stimuli typically used in research, transfer testing can nevertheless provide valuable information on the psychological nature of this historically controversial phenomenon.

Sign Tracking in an Enriched Environment: A Potential Ecologically Relevant Animal Model of Adaptive Behavior Change

When an object conditioned stimulus (CS) is paired with a food unconditioned stimulus (US), anticipatory goal-directed action directed at the US location (goal tracking) is accompanied by behavior directed at the object CS (sign tracking). Sign-tracking behavior appears to be compulsive and habit-like and predicts increased vulnerability to the addictive potential of drugs in animal models. A large body of the literature also suggests that environmental enrichment protects against the development of addiction-prone phenotypes. Thus, we investigated whether compulsive-like sign tracking develops in environmentally enriched rats trained directly in their enriched home environment. We demonstrate that adolescent enriched-housed male Sprague-Dawley rats readily sign track a 5% ethanol bottle CS in their home environment and at a rate higher than adolescent standard-housed rats. We also show that enriched adolescent rats sign track less than enriched adult-trained rats and that acute isolation stress affects sign- and goal-tracking performance of adolescents and adults differently. Sign tracking increased more in the adult than the adolescent rats. Whereas the younger rats showed a decrease in goal tacking after the final stressor manipulation, the adults showed increased goal tracking. Our results are consistent with recent studies, which suggest that although sign tracking performance is compulsive-like, it is not as inflexible and habit-like as previously assumed. Testing in an enriched home environment with object CSs having greater affordance than "neutral" lever CSs may provide greater ecological relevance for investigating the development and expression of adaptive and compulsive-like behaviors in translational research.

Stimulus control depends on the subjective value of the outcome

Stimuli that provide information about likely future reinforcers tend to shift behavior, provided a reliable relation between the stimulus and the reinforcer can be discriminated. Stimuli that are apparently more reliable exert greater control over behavior. We asked how the subjective value (measured in terms of preference) of reinforcers associated with stimuli influences stimulus control. Five pigeons worked on a concurrent chains procedure in which half of all trials ended in a smaller reinforcer sooner, and the other half in a larger reinforcer later. In Signaled trials, the color and flash duration on the keys in the initial link signaled the outcome of the trial. In Conflicting probe trials, the color and the flash duration signaled conflicting information about the outcome of the trial. Choice in Signaled trials shifted toward the signaled outcome, but was never exclusive. In Conflicting probe trials, control was divided idiosyncratically between the 2 stimulus dimensions, but still favored the outcome with the higher subjective value. Thus, stimulus control depends not only on the perceived reliability of stimuli, but also on the subjective value of the outcome.

A-learning: A new formulation of associative learning theory

We present a new mathematical formulation of associative learning focused on non-human animals, which we call A-learning. Building on current animal learning theory and machine learning, A-learning is composed of two learning equations, one for stimulus-response values and one for stimulus values (conditioned reinforcement). A third equation implements decision-making by mapping stimulus-response values to response probabilities. We show that A-learning can reproduce the main features of: instrumental acquisition, including the effects of signaled and unsignaled non-contingent reinforcement; Pavlovian acquisition, including higher-order conditioning, omission training, autoshaping, and differences in form between conditioned and unconditioned responses; acquisition of avoidance responses; acquisition and extinction of instrumental chains and Pavlovian higher-order conditioning; Pavlovian-to-instrumental transfer; Pavlovian and instrumental outcome revaluation effects, including insight into why these effects vary greatly with training procedures and with the proximity of a response to the reinforcer. We discuss the differences between current theory and A-learning, such as its lack of stimulus-stimulus and response-stimulus associations, and compare A-learning with other temporal-difference models from machine learning, such as Q-learning, SARSA, and the actor-critic model. We conclude that A-learning may offer a more convenient view of associative learning than current mathematical models, and point out areas that need further development.

Effects of conditioned stimulus (CS) duration, intertrial interval, and I/T ratio on appetitive Pavlovian conditioning

Pavlovian learning is influenced by at least 2 temporal variables: The time between the onset of the conditioned stimulus (CS) and presentation of the unconditioned stimulus (US), and the time between successive conditioning trials (the intertrial interval [ITI]). Wagner's Sometimes Opponent Process (SOP) model (e.g., 1981) provides a rich account of the effects of varying the absolute durations of CS and ITI. However, other theories have contrastingly emphasized the role of the relative durations of CS (T) and ITI (I). Three experiments with rats used an appetitive conditioning preparation to separate the two approaches. They manipulated absolute values of I and T over a factor of 6 and compared the effect of varying T and I/T by the same factor. Conditioning was indexed by the rate of foodcup entry during training, during common tests conducted later with different combinations of I and T, and with a reinforcers-to-criterion measure. Experiment 1 found that learning with a 10-s CS was superior to that with a 60-s CS when the I/T ratio was the same. There was little evidence of learning with the 60-s CS. Experiment 2 replicated that result and extended it to show that a 60-s CS still supported little conditioned responding when I/T was increased from 12 to 72. Experiment 3 then examined intermediate CS durations between 10-s and 60-s while I/T was controlled. The results support a role for absolute CS duration rather than the I/T ratio. Explanations based on SOP and behavior systems theory are explored. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

A computational formulation of the behavior systems account of the temporal organization of motivated behavior

The behavior systems framework suggests that motivated behavior-e.g., seeking food and mates, avoiding predators-consists of sequences of actions organized within nested behavioral states. This framework has bridged behavioral ecology and experimental psychology, providing key insights into critical behavioral processes. In particular, the behavior systems framework entails a particular organization of behavior over time. The present paper examines whether such organization emerges from a generic Markov process, where the current behavioral state determines the probability distribution of subsequent behavioral states. This proposition is developed as a systematic examination of increasingly complex Markov models, seeking a computational formulation that balances adherence to the behavior systems approach, parsimony, and conformity to data. As a result of this exercise, a nonstationary partially hidden Markov model is selected as a computational formulation of the predatory subsystem. It is noted that the temporal distribution of discrete responses may further unveil the structure and parameters of the model but, without proper mathematical modeling, these discrete responses may be misleading. Opportunities for further elaboration of the proposed computational formulation are identified, including developments in its architecture, extensions to defensive and reproductive subsystems, and methodological refinements.

Six clarifications for behaviour systems

The precursors of contemporary behaviour systems theory were hotly debated, and yet a similar critical fervour has not followed the second generation of behaviour systems research. I raise six items of potential or extant misunderstanding concerning behaviour systems perspectives, and attempt to set straight some of the assumptions and what motivated them, with attention to historical and theoretical context. The six challenges in focus are: 1) variety of conceptualisation of consummation; 2) potential misapprehensions about the role of general search; 3) ambiguity of predictions concerning response form; 4) ambiguity concerning what aspects are modelled as hierarchical; 5) assumptions of directedness; and 6) the relevance of spontaneous activity. For each of these six issues, some clarification is offered.

Timing and the transition between modes in the defensive behavior system

Antipredator defense is organized in a way that mirrors Timberlake's feeding behavior system because the goal of defense is to thwart predatory behavior. Each predatory mode has a corresponding antipredator mode. Like appetitive behavior systems, the defensive behavior system is organized around distinct modes along a spatiotemporal continuum we call the predatory imminence continuum. Behavior systems theory directs investigation toward the factors that lead to transitions between modes. In the feeding and sex systems the time between Conditional Stimulus (CS) and Unconditional Stimulus (US; e.g., CS-US interval or CS duration) is an important factor. Short CSs elicit conditional responses (CR) characteristic of more terminal modes and long CSs provoke CRs belonging to initial modes. Therefore, we asked if short CSs (10 s) would provoke CRs like the vigorous activity bursts and escape-like responses characteristic of the terminal mode of the predatory imminence continuum (Circa-Strike Behavior). Also, via analogy to appetitive systems, long CSs (3 min) were predicted to favor the intermediate mode, post-encounter behavior, which is characterized by freezing. Instead we found that both CSs produced freezing but not activity burst CRs and that freezing was actually greater with the short CS. We suggest that this difference between behavior systems flows from selection pressure that favors moving toward terminal modes in appetitive systems but away from terminal modes in the antipredator system. In addition, since appetitive reinforcers are more likely to be repeatedly experienced than predators, the learning of timing may be less relevant to defense. We also found that shock produced activity bursts and argue that when you are in the post-encounter mode (freezing) a sudden change in stimulation causes an immediate transition to circa-strike (terminal) behavior.

The behavior system for sexual learning

In this paper we review and update evidence relevant to formulating a behavior system for sexual learning. We emphasize behavioral rather than neurobiological evidence and mechanisms. Our analysis focuses on three types of responses or response modes: general search, focal search, and consummatory or copulatory behavior. We consider how these response modes are influenced by three categories of stimuli: spatially distributed contextual cues, arbitrary localized stimuli, and species-typical cues provided by the sexual partner. We characterize behavior control by these types of stimuli before and after various Pavlovian conditioning procedures in which the unconditioned stimulus is provided by copulation with a sexual partner. The results document extensive Pavlovian modifications of sexual behavior. These conditioning effects reflect new conditioned responses that come to be elicited by various categories of stimuli. In addition, the conditioning of contextual cues and localized stimuli facilitate sexual responding to species-typical cues. Thus, learning experiences enhance how the species-typical cues of a sexual partner stimulate sexual behavior. These modulatory conditioning effects not only produce significant behavioral changes but also increase rates of fertilization of eggs and numbers of offspring produced. These latter findings suggest that sexual learning can lead to differential reproductive success, which in turn can contribute to evolutionary change.

In what sense are dogs special? Canine cognition in comparative context

The great increase in the study of dog cognition in the current century has yielded insights into canine cognition in a variety of domains. In this review, we seek to place our enhanced understanding of canine cognition into context. We argue that in order to assess dog cognition, we need to regard dogs from three different perspectives: phylogenetically, as carnivoran and specifically a canid; ecologically, as social, cursorial hunters; and anthropogenically, as a domestic animal. A principled understanding of canine cognition should therefore involve comparing dogs' cognition with that of other carnivorans, other social hunters, and other domestic animals. This paper contrasts dog cognition with what is known about cognition in species that fit into these three categories, with a particular emphasis on wolves, cats, spotted hyenas, chimpanzees, dolphins, horses, and pigeons. We cover sensory cognition, physical cognition, spatial cognition, social cognition, and self-awareness. Although the comparisons are incomplete, because of the limited range of studies of some of the other relevant species, we conclude that dog cognition is influenced by the membership of all three of these groups, and taking all three groups into account, dog cognition does not look exceptional.

A common heritage of behaviour systems

Behaviour systems theory had its beginnings with Nikolaas Tinbergen’s “hierarchical systems”, an aspect of his thinking and writing that he conspicuously left out of his very memorable 1963 manifesto. This starting point has since been developed within psychology, where it has provided numerous advances. Tinbergen’s aspiration for behaviour systems had been principled integration of ethology with physiology, but the bridge among sciences it ultimately provided led to psychology. To an ethology audience, this paper attempts to reintroduce behaviour systems as a part of Tinbergen’s legacy to make accessible the theoretical developments of behaviour systems theory that have occurred outside of ethology over the last several decades. To a psychology audience, the paper serves as a reminder of the ethological origins of behaviour systems. Both sciences and their integration stand to benefit from recognising this point of common heritage.

Situating Human Sexual Conditioning

Conditioning is often thought of as a basic, automatic learning process that has limited applicability to higher-level human behavior. In addition, conditioning is seen as separable from, and even secondary to, "innate" processes. These ideas involve some misconceptions. The aim of this article is to provide a clearer, more refined sense of human sexual conditioning. After providing some background information and reviewing what is known from laboratory conditioning studies, human sexual conditioning is compared to sexual conditioning in nonhumans, to "innate" sexual responding, and to other types of human learning processes. Recommendations for moving forward in human sexual conditioning research are included.

A social insect perspective on the evolution of social learning mechanisms

The social world offers a wealth of opportunities to learn from others, and across the animal kingdom individuals capitalize on those opportunities. Here, we explore the role of natural selection in shaping the processes that underlie social information use, using a suite of experiments on social insects as case studies. We illustrate how an associative framework can encompass complex, context-specific social learning in the insect world and beyond, and based on the hypothesis that evolution acts to modify the associative process, suggest potential pathways by which social information use could evolve to become more efficient and effective. Social insects are distant relatives of vertebrate social learners, but the research we describe highlights routes by which natural selection could coopt similar cognitive raw material across the animal kingdom.

Coal Is Not Black, Snow Is Not White, Food Is Not a Reinforcer: The Roles of Affordances and Dispositions in the Analysis of Behavior

Reinforcers comprise sequences of actions in context. Just as the white of snow and black of coal depend on the interaction of an organism's visual system and the reflectances in its surrounds, reinforcers depend on an organism's motivational state and the affordances-possibilities for perception and action-in its surrounds. Reinforcers are not intrinsic to things but are a relation between what the thing affords, its context, the organism, and his or her history as capitulated in their current state. Reinforcers and other affordances are potentialities rather than intrinsic features. Realizing those potentialities requires motivational operations and stimulus contexts that change the state of the organism-they change its disposition to make the desired response. An expansion of the three-term contingency is suggested in order to help keep us mindful of the importance of behavioral systems, states, emotions, and dispositions in our research programs.

Suboptimal choice in rats: Incentive salience attribution promotes maladaptive decision-making

Stimuli that are more predictive of subsequent reward also function as better conditioned reinforcers. Moreover, stimuli attributed with incentive salience function as more robust conditioned reinforcers. Some theories have suggested that conditioned reinforcement plays an important role in promoting suboptimal choice behavior, like gambling. The present experiments examined how different stimuli, those attributed with incentive salience versus those without, can function in tandem with stimulus-reward predictive utility to promote maladaptive decision-making in rats. One group of rats had lights associated with goal-tracking as the reward-predictive stimuli and another had levers associated with sign-tracking as the reward-predictive stimuli. All rats were first trained on a choice procedure in which the expected value across both alternatives was equivalent but differed in their stimulus-reward predictive utility. Next, the expected value across both alternatives was systematically changed so that the alternative with greater stimulus-reward predictive utility was suboptimal in regard to primary reinforcement. The results demonstrate that in order to obtain suboptimal choice behavior, incentive salience alongside strong stimulus-reward predictive utility may be necessary; thus, maladaptive decision-making can be driven more by the value attributed to stimuli imbued with incentive salience that reliably predict a reward rather than the reward itself.

The sensory features of a food cue influence its ability to act as an incentive stimulus and evoke dopamine release in the nucleus accumbens core

The sensory properties of a reward-paired cue (a conditioned stimulus; CS) may impact the motivational value attributed to the cue, and in turn influence the form of the conditioned response (CR) that develops. A cue with multiple sensory qualities, such as a moving lever-CS, may activate numerous neural pathways that process auditory and visual information, resulting in CRs that vary both within and between individuals. For example, CRs include approach to the lever-CS itself (rats that “sign-track”; ST), approach to the location of reward delivery (rats that “goal-track”; GT), or an “intermediate” combination of these behaviors. We found that the multimodal sensory features of the lever-CS were important to the development and expression of sign-tracking. When the lever-CS was covered, and thus could only be heard moving, STs not only continued to approach the lever location but also started to approach the food cup during the CS period. While still predictive of reward, the auditory component of the lever-CS was a much weaker conditioned reinforcer than the visible lever-CS. This plasticity in behavioral responding observed in STs closely resembled behaviors normally seen in rats classified as “intermediates.” Furthermore, the ability of both the lever-CS and the reward-delivery to evoke dopamine release in the nucleus accumbens was also altered by covering the lever—dopamine signaling in STs resembled neurotransmission observed in rats that normally only GT. These data suggest that while the visible lever-CS was attractive, wanted, and had incentive value for STs, when presented in isolation, the auditory component of the cue was simply predictive of reward, lacking incentive salience. Therefore, the specific sensory features of cues may differentially contribute to responding and ensure behavioral flexibility.

Are Emotions Natural Kinds?

Laypeople and scientists alike believe that they know anger, or sadness, or fear, when they see it. These emotions and a few others are presumed to have specific causal mechanisms in the brain and properties that are observable (on the face, in the voice, in the body, or in experience)—that is, they are assumed to be natural kinds. If a given emotion is a natural kind and can be identified objectively, then it is possible to make discoveries about that emotion. Indeed, the scientific study of emotion is founded on this assumption. In this article, I review the accumulating empirical evidence that is inconsistent with the view that there are kinds of emotion with boundaries that are carved in nature. I then consider what moving beyond a natural-kind view might mean for the scientific understanding of emotion.

Dopamine Prediction Errors in Reward Learning and Addiction: From Theory to Neural Circuitry

Midbrain dopamine (DA) neurons are proposed to signal reward prediction error (RPE), a fundamental parameter in associative learning models. This RPE hypothesis provides a compelling theoretical framework for understanding DA function in reward learning and addiction. New studies support a causal role for DA-mediated RPE activity in promoting learning about natural reward; however, this question has not been explicitly tested in the context of drug addiction. In this review, we integrate theoretical models with experimental findings on the activity of DA systems, and on the causal role of specific neuronal projections and cell types, to provide a circuit-based framework for probing DA-RPE function in addiction. By examining error-encoding DA neurons in the neural network in which they are embedded, hypotheses regarding circuit-level adaptations that possibly contribute to pathological error signaling and addiction can be formulated and tested.

The Origins and Organization of Vertebrate Pavlovian Conditioning

Pavlovian conditioning is the process by which we learn relationships between stimuli and thus constitutes a basic building block for how the brain constructs representations of the world. We first review the major concepts of Pavlovian conditioning and point out many of the pervasive misunderstandings about just what conditioning is. This brings us to a modern redefinition of conditioning as the process whereby experience with a conditional relationship between stimuli bestows these stimuli with the ability to promote adaptive behavior patterns that did not occur before the experience. Working from this framework, we provide an in-depth analysis of two examples, fear conditioning and food-based appetitive conditioning, which include a description of the only partially overlapping neural circuitry of each. We also describe how these circuits promote the basic characteristics that define Pavlovian conditioning, such as error-correction-driven regulation of learning.

Nucleus Accumbens and Posterior Amygdala Mediate Cue-Triggered Alcohol Seeking and Suppress Behavior During the Omission of Alcohol-Predictive Cues

Neurobiological mechanisms that influence behavior in the presence of alcohol-associated stimuli involve processes that organize behavior during the presence of these cues, and separately, regulation of behavior in their absence. However, little is known about anatomical structures that might mediate this regulation. Here we examined nucleus accumbens shell (AcbSh) as a possible neural substrate mediating behavior modulation triggered by the presence and absence of alcohol-associated environmental cues and contexts. We also examined subregions of basal amygdala nuclei- rostral basolateral (BLA) and basal posterior (BAP)- as they provide a major source of glutamatergic input to the AcbSh. Animals were trained to associate an auditory conditioning stimulus with alcohol in a discriminative context and then subsequently tested for conditioned port-entries across contexts either previously associated or not associated with alcohol. We found that, on test to the alcohol cue alone, AcbSh inactivation prevented conditioned port-entries in contexts that either were associated with alcohol or were novel, while also increasing unconditioned port-entries during the intertrial intervals. When tested to alcohol-reinforced cues, AcbSh inactivation produced more cue-trial omissions and elevated unconditioned port-entries. Interestingly, BLA and BAP inactivation produced dissociable effects. BAP but not BLA increased unconditioned port-entries, while both manipulations prevented conditioned port-entries during the alcohol-cue. We conclude that AcbSh is necessary for modulating control over behavior otherwise guided by the presence of alcohol-predictive environmental stimuli and contexts. Moreover, this role may involve integration of functionally segregated inputs from rostral and posterior portions of basal amygdala nuclei.

Neurobehavioral perspectives on the distinction between fear and anxiety

In this review, we discuss the usefulness of the distinction between fear and anxiety. The clinical use of the labels is ambiguous, often defining one in terms of the other. We first consider what a useful, objective, and scientifically valid definition would entail and then evaluate several fear/anxiety distinctions that have been made in the neurobiological literature. A strong distinction should specify the difference in conditions that lead to fear versus anxiety. Additionally, fear and anxiety should generate distinct sets of behaviors. Ideally, the two states should be supported by distinguishable neuroanatomical circuits. Such a conceptualization would be consistent with the National Institute of Mental Health's Research Domain Criteria (RDoc). The majority of neurobiological approaches to the fear versus anxiety distinction fail to differentiate the two states in terms of behavior, often using the exact same behavioral measures as indicators. Of the two that do, only Predatory Imminence Theory provides a distinction both in terms of cause and effect. Indeed, that approach provides a ready distinction of anxiety, fear, and panic in terms of both antecedent conditions and response selection rules. Additionally, it appeals to distinct neural circuits to generate these modes of action.

Rats that sign-track are resistant to Pavlovian but not instrumental extinction

Individuals vary in the extent to which they attribute incentive salience to a discrete cue (conditioned stimulus; CS) that predicts reward delivery (unconditioned stimulus; US), which results in some individuals approaching and interacting with the CS (sign-trackers; STs) more than others (goal-trackers; GTs). Here we asked how periods of non-reinforcement influence conditioned responding in STs vs. GTs, in both Pavlovian and instrumental tasks. After classifying rats as STs or GTs by pairing a retractable lever (the CS) with the delivery of a food pellet (US), we introduced periods of non-reinforcement, first by simply withholding the US (i.e., extinction training; experiment 1), then by signaling alternating periods of reward (R) and non-reward (NR) within the same session (experiments 2 and 3). We also examined how alternating R and NR periods influenced instrumental responding for food (experiment 4). STs and GTs did not differ in their ability to discriminate between R and NR periods in the instrumental task. However, in Pavlovian settings STs and GTs responded to periods of non-reward very differently. Relative to STs, GTs very rapidly modified their behavior in response to periods of non-reward, showing much faster extinction and better and faster discrimination between R and NR conditions. These results highlight differences between Pavlovian and instrumental extinction learning, and suggest that if a Pavlovian CS is strongly attributed with incentive salience, as in STs, it may continue to bias attention toward it, and to facilitate persistent and relatively inflexible responding, even when it is no longer followed by reward.