Higher-order conditioning is impaired by hippocampal lesions

Early Auditory Event Related Potentials Distinguish Higher-Order From First-Order Aversive Conditioning

Stimuli in reality rarely co-occur with primary reward or punishment to allow direct associative learning of value. Instead, value is thought to be inferred through complex higher-order associations. Rodent research has demonstrated that the formation and maintenance of first-order and higher-order associations are supported by distinct neural substrates. In this study, we explored whether this pattern of findings held true for humans. Participants underwent first-order and subsequent higher-order conditioning using an aversive burst of white noise or neutral tone as the unconditioned stimuli. Four distinct tones, initially neutral, served as first-order and higher-order conditioned stimuli. Autonomic and neural responses were indexed by pupillometry and evoked response potentials (ERPs) respectively. Conditioned aversive values of first-order and higher-order stimuli led to increased autonomic responses, as indexed by pupil dilation. Distinct temporo-spatial auditory evoked response potentials were elicited by first-order and high-order conditioned stimuli. Conditioned first-order responses peaked around 260 ms and source estimation suggested a primary medial prefrontal and amygdala source. Conversely, conditioned higher-order responses peaked around 120 ms with an estimated source in the medial temporal lobe. Interestingly, pupillometry responses to first-order conditioned stimuli were diminished after higher order training, possibly signifying concomitant incidental extinction, while responses to higher-order stimuli remained. This suggests that once formed, higher order associations are at least partially independent of first order conditioned representations. This experiment demonstrates that first-order and higher-order conditioned associations have distinct neural signatures, and like rodents, the medial temporal lobe may be specifically involved with higher-order conditioning.

Higher-Order Conditioning and Dopamine: Charting a Path Forward

Higher-order conditioning involves learning causal links between multiple events, which then allows one to make novel inferences. For example, observing a correlation between two events (e.g., a neighbor wearing a particular sports jersey), later helps one make new predictions based on this knowledge (e.g., the neighbor's wife's favorite sports team). This type of learning is important because it allows one to benefit maximally from previous experiences and perform adaptively in complex environments where many things are ambiguous or uncertain. Two procedures in the lab are often used to probe this kind of learning, second-order conditioning (SOC) and sensory preconditioning (SPC). In second-order conditioning (SOC), we first teach subjects that there is a relationship between a stimulus and an outcome (e.g., a tone that predicts food). Then, an additional stimulus is taught to precede the predictive stimulus (e.g., a light leads to the food-predictive tone). In sensory preconditioning (SPC), this order of training is reversed. Specifically, the two neutral stimuli (i.e., light and tone) are first paired together and then the tone is paired separately with food. Interestingly, in both SPC and SOC, humans, rodents, and even insects, and other invertebrates will later predict that both the light and tone are likely to lead to food, even though they only experienced the tone directly paired with food. While these processes are procedurally similar, a wealth of research suggests they are associatively and neurobiologically distinct. However, midbrain dopamine, a neurotransmitter long thought to facilitate basic Pavlovian conditioning in a relatively simplistic manner, appears critical for both SOC and SPC. These findings suggest dopamine may contribute to learning in ways that transcend differences in associative and neurological structure. We discuss how research demonstrating that dopamine is critical to both SOC and SPC places it at the center of more complex forms of cognition (e.g., spatial navigation and causal reasoning). Further, we suggest that these more sophisticated learning procedures, coupled with recent advances in recording and manipulating dopamine neurons, represent a new path forward in understanding dopamine's contribution to learning and cognition.

Higher-Order Conditioning: What Is Learnt and How it Is Expressed

Pairing a neutral conditioned stimulus (CS) with a motivationally significant unconditioned stimulus (US) results in the CS coming to elicit conditioned responses (CRs). The widespread significance and translational value of Pavlovian conditioning are increased by the fact that pairing two neutral CSs (A and X) enables conditioning with X to affect behavior to A. There are two traditional informal accounts of such higher-order conditioning, which build on more formal associative analyses of Pavlovian conditioning. But, higher-order conditioning and Pavlovian conditioning have characteristics that are beyond these accounts: Notably, the two are influenced in different ways by the same experimental manipulations, and both generate conditioned responses that do not reflect the US per se. Here, we present a formal analysis that sought to address these characteristics.

Reinstatement of Cortical Outcome Representations during Higher-Order Learning

Naturalistic learning scenarios are characterized by infrequent experience of external feedback to guide behavior. Higher-order learning mechanisms like second-order conditioning (SOC) may allow stimuli that were never experienced together with reinforcement to acquire motivational value. Despite its explanatory potential for real-world learning, surprisingly little is known about the neural mechanism underlying such associative transfer of value in SOC. Here, we used multivariate cross-session, cross-modality searchlight classification on functional magnetic resonance imaging data obtained from humans during SOC. We show that visual first-order conditioned stimuli (CS) reinstate cortical patterns representing previously paired gustatory outcomes in the lateral orbitofrontal cortex (OFC). During SOC, this OFC region showed increased functional covariation with amygdala, where neural pattern similarity between second-order CS and outcomes increased from early to late stages of SOC. Our data suggest a mechanism by which motivational value is conferred to stimuli that were never paired with reinforcement.

Neuronal Computation Underlying Inferential Reasoning in Humans and Mice

Every day we make decisions critical for adaptation and survival. We repeat actions with known consequences. But we also draw on loosely related events to infer and imagine the outcome of entirely novel choices. These inferential decisions are thought to engage a number of brain regions; however, the underlying neuronal computation remains unknown. Here, we use a multi-day cross-species approach in humans and mice to report the functional anatomy and neuronal computation underlying inferential decisions. We show that during successful inference, the mammalian brain uses a hippocampal prospective code to forecast temporally structured learned associations. Moreover, during resting behavior, coactivation of hippocampal cells in sharp-wave/ripples represent inferred relationships that include reward, thereby "joining-the-dots" between events that have not been observed together but lead to profitable outcomes. Computing mnemonic links in this manner may provide an important mechanism to build a cognitive map that stretches beyond direct experience, thus supporting flexible behavior.

Animal affect and decision-making

The scientific study of animal affect (emotion) is an area of growing interest. Whilst research on mechanism and causation has predominated, the study of function is less advanced. This is not due to a lack of hypotheses; in both humans and animals, affective states are frequently proposed to play a pivotal role in coordinating adaptive responses and decisions. However, exactly how they might do this (what processes might implement this function) is often left rather vague. Here we propose a framework for integrating animal affect and decision-making that is couched in modern decision theory and employs an operational definition that aligns with dimensional concepts of core affect and renders animal affect empirically tractable. We develop a model of how core affect, including short-term (emotion-like) and longer-term (mood-like) states, influence decision-making via processes that we label affective options, affective predictions, and affective outcomes and which correspond to similar concepts in schema of the links between human emotion and decision-making. Our framework is generalisable across species and generates questions for future research.

Multiple associative structures created by reinforcement and incidental statistical learning mechanisms

Learning the structure of the world can be driven by reinforcement but also occurs incidentally through experience. Reinforcement learning theory has provided insight into how prediction errors drive updates in beliefs but less attention has been paid to the knowledge resulting from such learning. Here we contrast associative structures formed through reinforcement and experience of task statistics. BOLD neuroimaging in human volunteers demonstrates rigid representations of rewarded sequences in temporal pole and posterior orbito-frontal cortex, which are constructed backwards from reward. By contrast, medial prefrontal cortex and a hippocampal-amygdala border region carry reward-related knowledge but also flexible statistical knowledge of the currently relevant task model. Intriguingly, ventral striatum encodes prediction error responses but not the full RL- or statistically derived task knowledge. In summary, representations of task knowledge are derived via multiple learning processes operating at different time scales that are associated with partially overlapping and partially specialized anatomical regions.

Associative learning occurs through reinforcement mechanisms as well as incidentally through experience of statistical relationships. Here, the authors report that these two learning processes are associated with specialized anatomical regions that operate at different time scales.

Forming attitudes via neural activity supporting affective episodic simulations

Humans have the adaptive capacity for imagining hypothetical episodes. Such episodic simulation is based on a neural network that includes the ventromedial prefrontal cortex (vmPFC). This network draws on existing knowledge (e.g., of familiar people and places) to construct imaginary events (e.g., meeting with the person at that place). Here, we test the hypothesis that a simulation changes attitudes towards its constituent elements. In two experiments, we demonstrate how imagining meeting liked versus disliked people (unconditioned stimuli, UCS) at initially neutral places (conditioned stimuli, CS) changes the value of these places. We further provide evidence that the vmPFC codes for representations of those elements (i.e., of individual people and places). Critically, attitude changes induced by the liked UCS are based on a transfer of positive affective value between the representations (i.e., from the UCS to the CS). Thereby, we reveal how mere imaginings shape attitudes towards elements (i.e., places) from our real-life environment.

To What Extent Memory Could Contribute to Impaired Food Valuation and Choices in Obesity?

Obesity is associated with a diverse array of cognitive and affective deficits, among which impairments in food valuation and choices have received increasing attention. The neural underpinnings of such impairments, however, remain poorly understood, partly because a complete understanding of these processes under normal conditions has yet to be achieved. A rapidly growing literature on the interaction between memory and decision-making has begun to highlight the integral role of memory in decision making especially in the real world, as well as the role of the hippocampus in supporting flexible decision making. Perhaps not coincidentally, altered memory performances in obesity have been well documented, and the underlying neurobiological bases of these memory alterations have also started to be better described, involving pathologies at the biochemical, cellular, and circuit levels. Despite such correspondence, the link between memory impairments and food valuation/choice deficits in obesity has received little attention. In this article, we first summarize the growing empirical support for the relevance of memory for decision making, focusing on flexible value-based decisions. We then describe converging evidence on different forms of memory impairments accompanying obesity. Building on these findings, we formulate a general neuropsychological framework and discuss how dysfunctions in the formation and retrieval of memory may interfere with adaptive decision making for food. Finally, we stress the important practical implications of this framework, arguing that memory deficits are likely a significant contributor to suboptimal food purchase and eating behavior exhibited by obese individuals.

The hippocampus is critical for value-based decisions guided by dissociative inference

The hippocampus supports flexible decision-making through memory integration: bridging across episodes and inferring associations between stimuli that were never presented together ('associative inference'). A pre-requisite for memory integration is flexible representations of the relationships between stimuli within episodes (AB) but also of the constituent units (A,B). Here we investigated whether the hippocampus is required for parsing experienced episodes into their constituents to infer their re-combined within-episode associations ('dissociative inference'). In three experiments male rats were trained on an appetitive conditioning task using compound auditory stimuli (AB+, BA+, CD-, DC-). At test either the compound or individual stimuli were presented as well as new stimuli. Rats with hippocampal lesions acquired and retained the compound discriminations as well as controls. Single constituent stimuli (A, B, C, D) were presented for the first time at test, so the only value with which they could be associated was the one from the compound to which they belonged. Controls inferred constituent tones' corresponding values while hippocampal rats did not, treating them as merely familiar stimuli with no associated value. This finding held whether compound training occurred before or after hippocampal lesions, suggesting that hippocampus-dependent inferential processes more likely occur at retrieval. The findings extend recent discoveries about the role of the hippocampus in intrinsic value representation, demonstrating hippocampal contributions to allocating value from primary rewards to individual stimuli. Importantly, we discovered that dissociative inferences serve to restructure or reparse patterns of directly acquired associations when animals are faced with environmental changes and need to extract relevant information from a multiplex memory. The hippocampus is critical for this fundamental flexible use of associations.

Dexamethasone impairs encoding and expression of aversive conditioning promoted by pentylenetetrazole

Behavioral and neuroendocrine responses following threatening situations promote the release of corticosterone, which is known to modulate trauma-related learning and memory process. However, it remains unknown whether the aversive learning generated by interoceptive fear conditioning is affected by glucocorticoid modulation. Therefore, the present study aimed to investigate the role of dexamethasone suppression in encoding and expression of pentylenetetrazole-induced olfactory fear conditioning (OFC) and in contextual second-order conditioning promoted by the conditioned odor. Adult male Long-Evans rats were treated with dexamethasone 60 min before the encoding or the expression in both OFC and contextual second-order conditioning. Dexamethasone treatment impaired encoding and expression of the OFC, but failed to impair encoding and expression of the contextual second-order conditioning. Altogether, our results show that although OFC and thereafter contextual second-order conditioning may allow the study of traumatic memories, each order of conditioning seems to present specific features related to their pharmacological modulation. These findings highlight the importance of addressing the role of neuromodulatory systems in first-order and second-order conditioning to gain a better understanding of these phenomena and support future therapies related to traumatic memories.

Dorsal hippocampus contributes to model-based planning

Planning can be defined as action selection that leverages an internal model of the outcomes likely to follow each possible action. Its neural mechanisms remain poorly understood. Here we adapt recent advances from human research for rats, presenting for the first time an animal task that produces many trials of planned behavior per session, making multitrial rodent experimental tools available to study planning. We use part of this toolkit to address a perennially controversial issue in planning: the role of the dorsal hippocampus. Although prospective hippocampal representations have been proposed to support planning, intact planning in animals with damaged hippocampi has been repeatedly observed. Combining formal algorithmic behavioral analysis with muscimol inactivation, we provide causal evidence directly linking dorsal hippocampus with planning behavior. Our results and methods open the door to new and more detailed investigations of the neural mechanisms of planning in the hippocampus and throughout the brain.

Western diet and the weakening of the interoceptive stimulus control of appetitive behavior

In obesogenic environments food-related external cues are thought to overwhelm internal cues that normally regulate energy intake. We investigated how this shift from external to internal stimulus control might occur. Experiment 1 showed that rats could use stimuli arising from 0 and 4h food deprivation to predict sucrose delivery. Experiment 2 then examined (a) the ability of these deprivation cues to compete with external cues and (b) how consuming a Western-style diet (WD) affects that competition. Rats were trained to use both their deprivation cues and external cues as compound discriminative stimuli. Half of the rats were then placed on WD while the others remained on chow, and external cues were removed to assess learning about deprivation state cues. When tested with external cues removed, chow-fed rats continued to discriminate using only deprivation cues, while WD-fed rats did not. The WD-fed group performed similarly to control groups trained with a noncontingent relationship between deprivation cues and sucrose reinforcement. Previous studies provided evidence that discrimination based on interoceptive deprivation cues depends on the hippocampus and that WD intake could interfere with hippocampal functioning. A third experiment assessed the effects of neurotoxic hippocampal lesions on weight gain and on sensitivity to the appetite-suppressing effects of the satiety hormone cholecystokinin (CCK). Relative to controls, hippocampal-lesioned rats gained more weight and showed reduced sensitivity to a 1.0ug but not 2.0 or 4.0ug CCK doses. These findings suggest that WD intake reduces utilization of interoceptive energy state signals to regulate appetitive behavior via a mechanism that involves the hippocampus.

Higher-order conditioning and the retrosplenial cortex

The retrosplenial cortex (RSC) is known to contribute to contextual and spatial learning and memory. This is consistent with its well-established connectivity; the RSC is located at the interface of visuo-spatial association areas and the parahippocampal-hippocampal memory system. However, the RSC also contributes to learning and memory for discrete cues. For example, both permanent lesions and temporary inactivation of the RSC have been shown to impair sensory preconditioning, a form of higher-order conditioning. The purpose of the present experiment was to examine the role of the RSC in a closely related higher-order conditioning paradigm: second-order conditioning. Sham and RSC lesioned rats received first-order conditioning in which one visual stimulus (V1) was paired with footshock and one visual stimulus (V2) was not. Following first-order conditioning, one auditory stimulus (A1) was then paired with V1 and a second auditory stimulus (A2) was paired with V2. Although lesions of the RSC impaired the first-order discrimination, they had no impact on the acquisition of second-order conditioning. Thus, the RSC does not appear necessary for acquisition/expression of second-order fear conditioning. The role of the RSC in higher-order conditioning, as well as a possible dissociation from the hippocampus, is discussed.

A boost of confidence: The role of the ventromedial prefrontal cortex in memory, decision-making, and schemas

The ventromedial prefrontal cortex (vmPFC) has been implicated in a wide array of functions across multiple domains. In this review, we focus on the vmPFC's involvement in mediating strategic aspects of memory retrieval, memory-related schema functions, and decision-making. We suggest that vmPFC generates a confidence signal that informs decisions and memory-guided behaviour. Confidence is central to these seemingly diverse functions: (1) Strategic retrieval: lesions to the vmPFC impair an early, automatic, and intuitive monitoring process ("feeling of rightness"; FOR) often associated with confabulation (spontaneous reporting of erroneous memories). Critically, confabulators typically demonstrate high levels of confidence in their false memories, suggesting that faulty monitoring following vmPFC damage may lead to indiscriminate confidence signals. (2) Memory schemas: the vmPFC is critically involved in instantiating and maintaining contextually relevant schemas, broadly defined as higher level knowledge structures that encapsulate lower level representational elements. The correspondence between memory retrieval cues and these activated schemas leads to FOR monitoring. Stronger, more elaborate schemas produce stronger FOR and influence confidence in the veracity of memory candidates. (3) Finally, we review evidence on the vmPFC's role in decision-making, extending this role to decision-making during memory retrieval. During non-mnemonic and mnemonic decision-making the vmPFC automatically encodes confidence. Confidence signal in the vmPFC is revealed as a non-linear relationship between a first-order monitoring assessment and second-order action or choice. Attempting to integrate the multiple functions of the vmPFC, we propose a posterior-anterior organizational principle for this region. More posterior vmPFC regions are involved in earlier, automatic, subjective, and contextually sensitive functions, while more anterior regions are involved in controlled actions based on these earlier functions. Confidence signals reflect the non-linear relationship between first-order, posterior-mediated and second-order, anterior-mediated processes and are represented along the entire axis.

Episodic Memory and Beyond: The Hippocampus and Neocortex in Transformation

The last decade has seen dramatic technological and conceptual changes in research on episodic memory and the brain. New technologies, and increased use of more naturalistic observations, have enabled investigators to delve deeply into the structures that mediate episodic memory, particularly the hippocampus, and to track functional and structural interactions among brain regions that support it. Conceptually, episodic memory is increasingly being viewed as subject to lifelong transformations that are reflected in the neural substrates that mediate it. In keeping with this dynamic perspective, research on episodic memory (and the hippocampus) has infiltrated domains, from perception to language and from empathy to problem solving, that were once considered outside its boundaries. Using the component process model as a framework, and focusing on the hippocampus, its subfields, and specialization along its longitudinal axis, along with its interaction with other brain regions, we consider these new developments and their implications for the organization of episodic memory and its contribution to functions in other domains.

The Origins and Organization of Vertebrate Pavlovian Conditioning

Pavlovian conditioning is the process by which we learn relationships between stimuli and thus constitutes a basic building block for how the brain constructs representations of the world. We first review the major concepts of Pavlovian conditioning and point out many of the pervasive misunderstandings about just what conditioning is. This brings us to a modern redefinition of conditioning as the process whereby experience with a conditional relationship between stimuli bestows these stimuli with the ability to promote adaptive behavior patterns that did not occur before the experience. Working from this framework, we provide an in-depth analysis of two examples, fear conditioning and food-based appetitive conditioning, which include a description of the only partially overlapping neural circuitry of each. We also describe how these circuits promote the basic characteristics that define Pavlovian conditioning, such as error-correction-driven regulation of learning.