Formal learning theory dissociates brain regions with different temporal integration

Temporal dynamics of the semantic versus affective representations of valence during reversal learning

Valence, the representation of a stimulus in terms of good or bad, plays a central role in models of affect, value-based learning theories, and value-based decision-making models. Previous work used Unconditioned Stimulus (US) to support a theoretical division between two different types of valence representations for a stimulus: the semantic representation of valence, i.e., stored accumulated knowledge about the value of the stimulus, and the affective representation of valence, i.e., the valence of the affective response to this stimulus. The current work extended past research by using a neutral Conditioned Stimulus (CS) in the context of reversal learning, a type of associative learning. The impact of expected uncertainty (the variability of rewards) and unexpected uncertainty (reversal) on the evolving temporal dynamics of the two types of valence representations of the CS was tested in two experiments. Results show that in an environment presenting the two types of uncertainty, the adaptation process (learning rate) of the choices and of the semantic valence representation is slower than the adaptation of the affective valence representation. In contrast, in environments with only unexpected uncertainty (i.e., fixed rewards), there is no difference in the temporal dynamics of the two types of valence representations. Implications for models of affect, value-based learning theories, and value-based decision-making models are discussed.

Human threat circuits: Threats of pain, aggressive conspecific, and predator elicit distinct BOLD activations in the amygdala and hypothalamus

Introduction

Threat processing, enabled by threat circuits, is supported by a remarkably conserved neural architecture across mammals. Threatening stimuli relevant for most species include the threat of being attacked by a predator or an aggressive conspecific and the threat of pain. Extensive studies in rodents have associated the threats of pain, predator attack and aggressive conspecific attack with distinct neural circuits in subregions of the amygdala, the hypothalamus and the periaqueductal gray. Bearing in mind the considerable conservation of both the anatomy of these regions and defensive behaviors across mammalian species, we hypothesized that distinct brain activity corresponding to the threats of pain, predator attack and aggressive conspecific attack would also exist in human subcortical brain regions.

Methods

Forty healthy female subjects underwent fMRI scanning during aversive classical conditioning. In close analogy to rodent studies, threat stimuli consisted of painful electric shocks, a short video clip of an attacking bear and a short video clip of an attacking man. Threat processing was conceptualized as the expectation of the aversive stimulus during the presentation of the conditioned stimulus.

Results

Our results demonstrate differential brain activations in the left and right amygdala as well as in the left hypothalamus for the threats of pain, predator attack and aggressive conspecific attack, for the first time showing distinct threat-related brain activity within the human subcortical brain. Specifically, the threat of pain showed an increase of activity in the left and right amygdala and the left hypothalamus compared to the threat of conspecific attack (pain > conspecific), and increased activity in the left amygdala compared to the threat of predator attack (pain > predator). Threat of conspecific attack revealed heightened activity in the right amygdala, both in comparison to threat of pain (conspecific > pain) and threat of predator attack (conspecific > predator). Finally, for the condition threat of predator attack we found increased activity in the bilateral amygdala and the hypothalamus when compared to threat of conspecific attack (predator > conspecific). No significant clusters were found for the contrast predator attack > pain.

Conclusion

Results suggest that threat type-specific circuits identified in rodents might be conserved in the human brain.

Human inference reflects a normative balance of complexity and accuracy

We must often infer latent properties of the world from noisy and changing observations. Complex, probabilistic approaches to this challenge such as Bayesian inference are accurate but cognitively demanding, relying on extensive working memory and adaptive processing. Simple heuristics are easy to implement but may be less accurate. What is the appropriate balance between complexity and accuracy? Here we model a hierarchy of strategies of variable complexity and find a power law of diminishing returns: increasing complexity gives progressively smaller gains in accuracy. The rate of diminishing returns depends systematically on the statistical uncertainty in the world, such that complex strategies do not provide substantial benefits over simple ones when uncertainty is either too high or too low. In between, there is a complexity dividend. In two psychophysical experiments, we confirm specific model predictions about how working memory and adaptivity should be modulated by uncertainty.

Mixing memory and desire: How memory reactivation supports deliberative decision-making

Memories affect nearly every aspect of our mental life. They allow us to both resolve uncertainty in the present and to construct plans for the future. Recently, renewed interest in the role memory plays in adaptive behavior has led to new theoretical advances and empirical observations. We review key findings, with particular emphasis on how the retrieval of many kinds of memories affects deliberative action selection. These results are interpreted in a sequential inference framework, in which reinstatements from memory serve as "samples" of potential action outcomes. The resulting model suggests a central role for the dynamics of memory reactivation in determining the influence of different kinds of memory in decisions. We propose that representation-specific dynamics can implement a bottom-up "product of experts" rule that integrates multiple sets of action-outcome predictions weighted based on their uncertainty. We close by reviewing related findings and identifying areas for further research. This article is categorized under: Psychology > Reasoning and Decision Making Neuroscience > Cognition Neuroscience > Computation.

Reward and fictive prediction error signals in ventral striatum: asymmetry between factual and counterfactual processing

Reward prediction error, the difference between the expected and obtained reward, is known to act as a reinforcement learning neural signal. In the current study, we propose a model fitting approach that combines behavioral and neural data to fit computational models of reinforcement learning. Briefly, we penalized subject-specific fitted parameters that moved away too far from the group median, except when that deviation led to an improvement in the model's fit to neural responses. By means of a probabilistic monetary learning task and fMRI, we compared our approach with standard model fitting methods. Q-learning outperformed actor-critic at both behavioral and neural level, although the inclusion of neuroimaging data into model fitting improved the fit of actor-critic models. We observed both action-value and state-value prediction error signals in the striatum, while standard model fitting approaches failed to capture state-value signals. Finally, left ventral striatum correlated with reward prediction error while right ventral striatum with fictive prediction error, suggesting a functional hemispheric asymmetry regarding prediction-error driven learning.

Multiple associative structures created by reinforcement and incidental statistical learning mechanisms

Learning the structure of the world can be driven by reinforcement but also occurs incidentally through experience. Reinforcement learning theory has provided insight into how prediction errors drive updates in beliefs but less attention has been paid to the knowledge resulting from such learning. Here we contrast associative structures formed through reinforcement and experience of task statistics. BOLD neuroimaging in human volunteers demonstrates rigid representations of rewarded sequences in temporal pole and posterior orbito-frontal cortex, which are constructed backwards from reward. By contrast, medial prefrontal cortex and a hippocampal-amygdala border region carry reward-related knowledge but also flexible statistical knowledge of the currently relevant task model. Intriguingly, ventral striatum encodes prediction error responses but not the full RL- or statistically derived task knowledge. In summary, representations of task knowledge are derived via multiple learning processes operating at different time scales that are associated with partially overlapping and partially specialized anatomical regions.

Associative learning occurs through reinforcement mechanisms as well as incidentally through experience of statistical relationships. Here, the authors report that these two learning processes are associated with specialized anatomical regions that operate at different time scales.

Network Brain-Computer Interface (nBCI): An Alternative Approach for Cognitive Prosthetics

Brain computer interfaces (BCIs) have been applied to sensorimotor systems for many years. However, BCI technology has broad potential beyond sensorimotor systems. The emerging field of cognitive prosthetics, for example, promises to improve learning and memory for patients with cognitive impairment. Unfortunately, our understanding of the neural mechanisms underlying these cognitive processes remains limited in part due to the extensive individual variability in neural coding and circuit function. As a consequence, the development of methods to ascertain optimal control signals for cognitive decoding and restoration remains an active area of inquiry. To advance the field, robust tools are required to quantify time-varying and task-dependent brain states predictive of cognitive performance. Here, we suggest that network science is a natural language in which to formulate and apply such tools. In support of our argument, we offer a simple demonstration of the feasibility of a network approach to BCI control signals, which we refer to as network BCI (nBCI). Finally, in a single subject example, we show that nBCI can reliably predict online cognitive performance and is superior to certain common spectral approaches currently used in BCIs. Our review of the literature and preliminary findings support the notion that nBCI could provide a powerful approach for future applications in cognitive prosthetics.

Simultaneous representation of a spectrum of dynamically changing value estimates during decision making

Decisions are based on value expectations derived from experience. We show that dorsal anterior cingulate cortex and three other brain regions hold multiple representations of choice value based on different timescales of experience organized in terms of systematic gradients across the cortex. Some parts of each area represent value estimates based on recent reward experience while others represent value estimates based on experience over the longer term. The value estimates within these areas interact with one another according to their temporal scaling. Some aspects of the representations change dynamically as the environment changes. The spectrum of value estimates may act as a flexible selection mechanism for combining experience-derived value information with other aspects of value to allow flexible and adaptive decisions in changing environments.

The Vivid Present: Visualization Abilities Are Associated with Steep Discounting of Future Rewards

Humans and other animals discount the value of future rewards, a phenomenon known as delay discounting. Individuals vary widely in the extent to which they discount future rewards, and these tendencies have been associated with important life outcomes. Recent studies have demonstrated that imagining the future reduces subsequent discounting behavior, but no research to date has examined whether a similar principle applies at the trait level, and whether training visualization changes discounting. The current study examined if individual differences in visualization abilities are linked to individual differences in discounting and whether practicing visualization can change discounting behaviors in a lasting way. Participants (n = 48) completed the Vividness of Visual Imagery Questionnaire (VVIQ) and delay discounting task and then underwent a 4-week intervention consisting of visualization training (intervention) or relaxation training (control). Contrary to our hypotheses, participants who reported greater visualization abilities (lower scores) on the VVIQ were higher discounters. To further examine this relationship, an additional 106 participants completed the VVIQ and delay discounting task. In the total sample (n = 154), there was a significant negative correlation between VVIQ scores and discount rates, showing that individuals who are better visualizers are also higher discounters. Consistent with this relationship but again to our surprise, visualization training tended, albeit weakly, to increase discount rates, and those whose VVIQ decreased the most were those whose discount rates increased the most. These results suggest a novel association between visualization abilities and delay discounting.

The neurobiology of uncertainty: implications for statistical learning

The capacity for assessing the degree of uncertainty in the environment relies on estimating statistics of temporally unfolding inputs. This, in turn, allows calibration of predictive and bottom-up processing, and signalling changes in temporally unfolding environmental features. In the last decade, several studies have examined how the brain codes for and responds to input uncertainty. Initial neurobiological experiments implicated frontoparietal and hippocampal systems, based largely on paradigms that manipulated distributional features of visual stimuli. However, later work in the auditory domain pointed to different systems, whose activation profiles have interesting implications for computational and neurobiological models of statistical learning (SL). This review begins by briefly recapping the historical development of ideas pertaining to the sensitivity to uncertainty in temporally unfolding inputs. It then discusses several issues at the interface of studies of uncertainty and SL. Following, it presents several current treatments of the neurobiology of uncertainty and reviews recent findings that point to principles that serve as important constraints on future neurobiological theories of uncertainty, and relatedly, SL. This review suggests it may be useful to establish closer links between neurobiological research on uncertainty and SL, considering particularly mechanisms sensitive to local and global structure in inputs, the degree of input uncertainty, the complexity of the system generating the input, learning mechanisms that operate on different temporal scales and the use of learnt information for online prediction.This article is part of the themed issue 'New frontiers for statistical learning in the cognitive sciences'.

Cognitive emotion regulation modulates the balance of competing influences on ventral striatal aversive prediction error signals

Cognitive emotion regulation (CER) is a critical human ability to face aversive emotional stimuli in a flexible way, via recruitment of specific prefrontal brain circuits. Animal research reveals a central role of ventral striatum in emotional behavior, for both aversive conditioning, with striatum signaling aversive prediction errors (aPE), and for integrating competing influences of distinct striatal inputs from regions such as the prefrontal cortex (PFC), amygdala, hippocampus and ventral tegmental area (VTA). Translating these ventral striatal findings from animal research to human CER, we hypothesized that successful CER would affect the balance of competing influences of striatal afferents on striatal aPE signals, in a way favoring PFC as opposed to 'subcortical' (i.e., non-isocortical) striatal inputs. Using aversive Pavlovian conditioning with and without CER during fMRI, we found that during CER, superior regulators indeed reduced the modulatory impact of 'subcortical' striatal afferents (hippocampus, amygdala and VTA) on ventral striatal aPE signals, while keeping the PFC impact intact. In contrast, inferior regulators showed an opposite pattern. Our results demonstrate that ventral striatal aPE signals and associated competing modulatory inputs are critical mechanisms underlying successful cognitive regulation of aversive emotions in humans.

Reinforcement learning models and their neural correlates: An activation likelihood estimation meta-analysis

Reinforcement learning describes motivated behavior in terms of two abstract signals. The representation of discrepancies between expected and actual rewards/punishments-prediction error-is thought to update the expected value of actions and predictive stimuli. Electrophysiological and lesion studies have suggested that mesostriatal prediction error signals control behavior through synaptic modification of cortico-striato-thalamic networks. Signals in the ventromedial prefrontal and orbitofrontal cortex are implicated in representing expected value. To obtain unbiased maps of these representations in the human brain, we performed a meta-analysis of functional magnetic resonance imaging studies that had employed algorithmic reinforcement learning models across a variety of experimental paradigms. We found that the ventral striatum (medial and lateral) and midbrain/thalamus represented reward prediction errors, consistent with animal studies. Prediction error signals were also seen in the frontal operculum/insula, particularly for social rewards. In Pavlovian studies, striatal prediction error signals extended into the amygdala, whereas instrumental tasks engaged the caudate. Prediction error maps were sensitive to the model-fitting procedure (fixed or individually estimated) and to the extent of spatial smoothing. A correlate of expected value was found in a posterior region of the ventromedial prefrontal cortex, caudal and medial to the orbitofrontal regions identified in animal studies. These findings highlight a reproducible motif of reinforcement learning in the cortico-striatal loops and identify methodological dimensions that may influence the reproducibility of activation patterns across studies.

Cognitive emotion regulation enhances aversive prediction error activity while reducing emotional responses

Cognitive emotion regulation is a powerful way of modulating emotional responses. However, despite the vital role of emotions in learning, it is unknown whether the effect of cognitive emotion regulation also extends to the modulation of learning. Computational models indicate prediction error activity, typically observed in the striatum and ventral tegmental area, as a critical neural mechanism involved in associative learning. We used model-based fMRI during aversive conditioning with and without cognitive emotion regulation to test the hypothesis that emotion regulation would affect prediction error-related neural activity in the striatum and ventral tegmental area, reflecting an emotion regulation-related modulation of learning. Our results show that cognitive emotion regulation reduced emotion-related brain activity, but increased prediction error-related activity in a network involving ventral tegmental area, hippocampus, insula and ventral striatum. While the reduction of response activity was related to behavioral measures of emotion regulation success, the enhancement of prediction error-related neural activity was related to learning performance. Furthermore, functional connectivity between the ventral tegmental area and ventrolateral prefrontal cortex, an area involved in regulation, was specifically increased during emotion regulation and likewise related to learning performance. Our data, therefore, provide first-time evidence that beyond reducing emotional responses, cognitive emotion regulation affects learning by enhancing prediction error-related activity, potentially via tegmental dopaminergic pathways.

Enhanced striatal responses during expectancy coding in alcohol dependence

BACKGROUND

Individuals with alcohol dependence are known to make disadvantageous decisions, possibly caused by alterations in either reward or punishment sensitivity, which lead to persistent alcohol use despite its adverse consequences. Previous studies in alcohol dependence have mainly focused on reward anticipation processing and results from these studies are mixed. To clarify the nature of the motivational deficit that underlies disadvantageous choice in alcohol dependence, the current study sought to characterize the neural representation of expected value in individuals with alcohol dependence, separating expectancy-related processing of gains and losses, as a function of outcome magnitude and outcome probability.

METHOD

Functional MRI was used to examine brain responses during the expectation of gains and losses in patients with alcohol dependence (n=19) and healthy controls (n=19). The task manipulated outcome magnitude (€1 and €5) and outcome probability (30% and 70%).

RESULTS

Compared to healthy controls, patients with alcohol dependence were more responsive to the expectancy of large wins, in the caudate and putamen. This effect was driven by a higher caudate activity in the contrast comparing €5 vs. €1 trials in patients with alcohol dependence. There were no group differences in the responses to the expectancy for loss. The patient group reported lower expectancies of winning in the trial-by-trial ratings.

CONCLUSIONS

Patients with alcohol dependence showed caudate hyperactivity when expecting wins. The result contrasts with past work using the monetary incentive delay task, showing caudate hypoactivity; the passive nature of our task contrasts with an active response requirement in the MIDT studies.

Deficits in predictive coding underlie hallucinations in schizophrenia

The neural mechanisms that produce hallucinations and other psychotic symptoms remain unclear. Previous research suggests that deficits in predictive signals for learning, such as prediction error signals, may underlie psychotic symptoms, but the mechanism by which such deficits produce psychotic symptoms remains to be established. We used model-based fMRI to study sensory prediction errors in human patients with schizophrenia who report daily auditory verbal hallucinations (AVHs) and sociodemographically matched healthy control subjects. We manipulated participants' expectations for hearing speech at different periods within a speech decision-making task. Patients activated a voice-sensitive region of the auditory cortex while they experienced AVHs in the scanner and displayed a concomitant deficit in prediction error signals in a similar portion of auditory cortex. This prediction error deficit correlated strongly with increased activity during silence and with reduced volumes of the auditory cortex, two established neural phenotypes of AVHs. Furthermore, patients with more severe AVHs had more deficient prediction error signals and greater activity during silence within the region of auditory cortex where groups differed, regardless of the severity of psychotic symptoms other than AVHs. Our findings suggest that deficient predictive coding accounts for the resting hyperactivity in sensory cortex that leads to hallucinations.

Prediction errors to emotional expressions: the roles of the amygdala in social referencing

Social referencing paradigms in humans and observational learning paradigms in animals suggest that emotional expressions are important for communicating valence. It has been proposed that these expressions initiate stimulus-reinforcement learning. Relatively little is known about the role of emotional expressions in reinforcement learning, particularly in the context of social referencing. In this study, we examined object valence learning in the context of a social referencing paradigm. Participants viewed objects and faces that turned toward the objects and displayed a fearful, happy or neutral reaction to them, while judging the gender of these faces. Notably, amygdala activation was larger when the expressions following an object were less expected. Moreover, when asked, participants were both more likely to want to approach, and showed stronger amygdala responses to, objects associated with happy relative to objects associated with fearful expressions. This suggests that the amygdala plays two roles in social referencing: (i) initiating learning regarding the valence of an object as a function of prediction errors to expressions displayed toward this object and (ii) orchestrating an emotional response to the object when value judgments are being made regarding this object.

Social learning in humans and other animals

Decisions made by individuals can be influenced by what others think and do. Social learning includes a wide array of behaviors such as imitation, observational learning of novel foraging techniques, peer or parental influences on individual preferences, as well as outright teaching. These processes are believed to underlie an important part of cultural variation among human populations and may also explain intraspecific variation in behavior between geographically distinct populations of animals. Recent neurobiological studies have begun to uncover the neural basis of social learning. Here we review experimental evidence from the past few decades showing that social learning is a widespread set of skills present in multiple animal species. In mammals, the temporoparietal junction, the dorsomedial, and dorsolateral prefrontal cortex, as well as the anterior cingulate gyrus, appear to play critical roles in social learning. Birds, fish, and insects also learn from others, but the underlying neural mechanisms remain poorly understood. We discuss the evolutionary implications of these findings and highlight the importance of emerging animal models that permit precise modification of neural circuit function for elucidating the neural basis of social learning.

Categorization = decision making + generalization

We rarely, if ever, repeatedly encounter exactly the same situation. This makes generalization crucial for real world decision making. We argue that categorization, the study of generalizable representations, is a type of decision making, and that categorization learning research would benefit from approaches developed to study the neuroscience of decision making. Similarly, methods developed to examine generalization and learning within the field of categorization may enhance decision making research. We first discuss perceptual information processing and integration, with an emphasis on accumulator models. We then examine learning the value of different decision making choices via experience, emphasizing reinforcement learning modeling approaches. Next we discuss how value is combined with other factors in decision making, emphasizing the effects of uncertainty. Finally, we describe how a final decision is selected via thresholding processes implemented by the basal ganglia and related regions. We also consider how memory related functions in the hippocampus may be integrated with decision making mechanisms and contribute to categorization.

A mixture of delta-rules approximation to bayesian inference in change-point problems

Error-driven learning rules have received considerable attention because of their close relationships to both optimal theory and neurobiological mechanisms. However, basic forms of these rules are effective under only a restricted set of conditions in which the environment is stable. Recent studies have defined optimal solutions to learning problems in more general, potentially unstable, environments, but the relevance of these complex mathematical solutions to how the brain solves these problems remains unclear. Here, we show that one such Bayesian solution can be approximated by a computationally straightforward mixture of simple error-driven 'Delta' rules. This simpler model can make effective inferences in a dynamic environment and matches human performance on a predictive-inference task using a mixture of a small number of Delta rules. This model represents an important conceptual advance in our understanding of how the brain can use relatively simple computations to make nearly optimal inferences in a dynamic world.

Neural evidence for adaptive strategy selection in value-based decision-making

In everyday life, humans often encounter complex environments in which multiple sources of information can influence their decisions. We propose that in such situations, people select and apply different strategies representing different cognitive models of the decision problem. Learning advances by evaluating the success of using a strategy and eventually by switching between strategies. To test our strategy selection model, we investigated how humans solve a dynamic learning task with complex auditory and visual information, and assessed the underlying neural mechanisms with functional magnetic resonance imaging. Using the model, we were able to capture participants' choices and to successfully attribute expected values and reward prediction errors to activations in the dopaminoceptive system (e.g., ventral striatum [VS]) as well as decision conflict to signals in the anterior cingulate cortex. The model outperformed an alternative approach that did not update decision strategies, but the relevance of information itself. Activation of sensory areas depended on whether the selected strategy made use of the respective source of information. Selection of a strategy also determined how value-related information influenced effective connectivity between sensory systems and the VS. Our results suggest that humans can structure their search for and use of relevant information by adaptively selecting between decision strategies.

Dissociating hippocampal and striatal contributions to sequential prediction learning

Behavior may be generated on the basis of many different kinds of learned contingencies. For instance, responses could be guided by the direct association between a stimulus and response, or by sequential stimulus-stimulus relationships (as in model-based reinforcement learning or goal-directed actions). However, the neural architecture underlying sequential predictive learning is not well understood, in part because it is difficult to isolate its effect on choice behavior. To track such learning more directly, we examined reaction times (RTs) in a probabilistic sequential picture identification task in healthy individuals. We used computational learning models to isolate trial-by-trial effects of two distinct learning processes in behavior, and used these as signatures to analyse the separate neural substrates of each process. RTs were best explained via the combination of two delta rule learning processes with different learning rates. To examine neural manifestations of these learning processes, we used functional magnetic resonance imaging to seek correlates of time-series related to expectancy or surprise. We observed such correlates in two regions, hippocampus and striatum. By estimating the learning rates best explaining each signal, we verified that they were uniquely associated with one of the two distinct processes identified behaviorally. These differential correlates suggest that complementary anticipatory functions drive each region's effect on behavior. Our results provide novel insights as to the quantitative computational distinctions between medial temporal and basal ganglia learning networks and enable experiments that exploit trial-by-trial measurement of the unique contributions of both hippocampus and striatum to response behavior.

Separate amygdala subregions signal surprise and predictiveness during associative fear learning in humans

It has recently been suggested that learning signals in the amygdala might be best characterized by attentional theories of associative learning [such as Pearce-Hall (PH)] and more recent hybrid variants that combine Rescorla-Wagner and PH learning models. In these models, unsigned prediction errors (PEs) determine the associability of a cue, which is used in turn to control learning of outcome expectations dynamically and reflects a function of the reliability of prior outcome predictions. Here, we employed an aversive Pavlovian reversal-learning task to investigate computational signals derived from such a hybrid model. Unlike previous accounts, our paradigm allowed for the separate assessment of associability at the time of cue presentation and PEs at the time of outcome. We combined this approach with high-resolution functional magnetic resonance imaging to understand how different subregions of the human amygdala contribute to associative learning. Signal changes in the corticomedial amygdala and in the midbrain represented unsigned PEs at the time of outcome showing increased responses irrespective of whether a shock was unexpectedly administered or omitted. In contrast, activity in basolateral amygdala regions correlated negatively with associability at the time of cue presentation. Thus, whereas the corticomedial amygdala and the midbrain reflected immediate surprise, the basolateral amygdala represented predictiveness and displayed increased responses when outcome predictions became more reliable. These results extend previous findings on PH-like mechanisms in the amygdala and provide unique insights into human amygdala circuits during associative learning.

Neurobiological mechanisms underlying the blocking effect in aversive learning

Current theories of classical conditioning assume that learning depends on the predictive relationship between events, not just on their temporal contiguity. Here we employ the classic experiment substantiating this reasoning-the blocking paradigm-in combination with functional magnetic resonance imaging (fMRI) to investigate whether human amygdala responses in aversive learning conform to these assumptions. In accordance with blocking, we demonstrate that significantly stronger behavioral and amygdala responses are evoked by conditioned stimuli that are predictive of the unconditioned stimulus than by conditioned stimuli that have received the same pairing with the unconditioned stimulus, yet have no predictive value. When studying the development of this effect, we not only observed that it was related to the strength of previous conditioned responses, but also that predictive compared with nonpredictive conditioned stimuli received more overt attention, as measured by fMRI-concurrent eye tracking, and that this went along with enhanced amygdala responses. We furthermore observed that prefrontal regions play a role in the development of the blocking effect: ventromedial prefrontal cortex (subgenual anterior cingulate) only exhibited responses when conditioned stimuli had to be established as nonpredictive for an outcome, whereas dorsolateral prefrontal cortex also showed responses when conditioned stimuli had to be established as predictive. Most importantly, dorsolateral prefrontal cortex connectivity to amygdala flexibly switched between positive and negative coupling, depending on the requirements posed by predictive relationships. Together, our findings highlight the role of predictive value in explaining amygdala responses and identify mechanisms that shape these responses in human fear conditioning.

The influence of serotonin on fear learning

Learning of associations between aversive stimuli and predictive cues is the basis of Pavlovian fear conditioning and is driven by a mismatch between expectation and outcome. To investigate whether serotonin modulates the formation of such aversive cue-outcome associations, we used functional magnetic resonance imaging (fMRI) and dietary tryptophan depletion to reduce brain serotonin (5-HT) levels in healthy human subjects. In a Pavlovian fear conditioning paradigm, 5-HT depleted subjects compared to a non-depleted control group exhibited attenuated autonomic responses to cues indicating the upcoming of an aversive event. These results were closely paralleled by reduced aversive learning signals in the amygdala and the orbitofrontal cortex, two prominent structures of the neural fear circuit. In agreement with current theories of serotonin as a motivational opponent system to dopamine in fear learning, our data provide first empirical evidence for a role of serotonin in representing formally derived learning signals for aversive events.

Distorted expectancy coding in problem gambling: is the addictive in the anticipation?

BACKGROUND

Pathologic gamblers are known to have abnormal neural responses associated with experiencing monetary wins and losses. However, neural responsiveness during reward and loss expectations in pathologic gamblers has not yet been investigated.

METHODS

We used a functional magnetic resonance imaging paradigm that allowed us to investigate the dissociable reward- and loss-related expectancies with various probabilities of winning or losing different amounts of money in 15 patients with problem gambling (PRGs) and 16 healthy control subjects (HCs).

RESULTS

Compared with HCs, PRGs showed stronger activation in the bilateral ventral striatum to 5 euro than to 1 euro trials. PRGs also showed more activation of the bilateral ventral striatum and left orbitofrontal cortex associated with gain-related expected value than HCs. In addition, regression analyses indicated a highly significant negative correlation between gambling severity scores and right amygdala activation associated with gain-related expected value coding. There were no group differences in brain activation for loss-related expected value.

CONCLUSIONS

PRGs show higher activity in the reward system during reward expectation than HCs, whereas we observed no difference between PRGs and HC in the loss value system. Furthermore, the negative relation between gambling severity and amygdala activation in gain expected value coding suggests that more severe PRGs are less likely to be risk aversive during gambling. Our study provides evidence that PRGs are characterized by abnormally increased reward expectancy coding, which may render them overoptimistic with regard to gambling outcomes.

A stable sparse fear memory trace in human amygdala

Pavlovian fear conditioning is highly conserved across species, providing a powerful model of aversive learning. In rodents, fear memory is stored and reactivated under the influence of the amygdala. There is no evidence for an equivalent mechanism in primates, and an opposite mechanism is proposed whereby primate amygdala contributes only to an initial phase of aversive learning, subsequently ceding fear memory to extra-amygdalar regions. Here, we reexamine this question by exploiting human high-resolution functional magnetic resonance imaging in conjunction with multivariate methods. By assuming a sparse neural coding, we show it is possible, at an individual subject level, to discriminate responses to conditioned (CS+ and CS-) stimuli in both basolateral and centro-cortical amygdala nuclei. The strength of this discrimination increased over time and was tightly coupled to the behavioral expression of fear, consistent with an expression of a stable fear memory trace. These data highlight that the human basolateral and centro-cortical amygdala support initial learning as well more enduring fear memory storage. A sparse neuronal representation for fear, here revealed by multivariate pattern classification, resolves why an enduring memory trace has proven elusive in previous human studies.

The integration of negative affect, pain and cognitive control in the cingulate cortex

It has been argued that emotion, pain and cognitive control are functionally segregated in distinct subdivisions of the cingulate cortex. However, recent observations encourage a fundamentally different view. Imaging studies demonstrate that negative affect, pain and cognitive control activate an overlapping region of the dorsal cingulate--the anterior midcingulate cortex (aMCC). Anatomical studies reveal that the aMCC constitutes a hub where information about reinforcers can be linked to motor centres responsible for expressing affect and executing goal-directed behaviour. Computational modelling and other kinds of evidence suggest that this intimacy reflects control processes that are common to all three domains. These observations compel a reconsideration of the dorsal cingulate's contribution to negative affect and pain.

Transient neural activation in human amygdala involved in aversive conditioning of face and voice

Elucidating the neural mechanisms involved in aversive conditioning helps find effective treatments for psychiatric disorders such as anxiety disorder and phobia. Previous studies using fMRI and human subjects have reported that the amygdala plays a role in this phenomenon. However, the noxious stimuli that were used as unconditioned stimuli in previous studies (e.g., electric shock) might have been ecologically invalid because we seldom encounter such stimuli in daily life. Therefore, we investigated whether a face stimulus could be conditioned by using a voice that had negative emotional valence and was collected from a real-life environment. A skin conductance response showed that healthy subjects were conditioned by using these stimuli. In an fMRI study, there was greater amygdala activation in response to the faces that had been paired with the voice than to those that had not. The right amygdala showed transient activity in the early stage of acquisition. A psychophysiological interaction analysis indicated that the subcortical pathway from the medial geniculate body to the amygdala played a role in conditioning. Modulation of the subcortical pathway by voice stimuli preceded the transient activity in the amygdala. The finding that an ecologically valid stimulus elicited the conditioning and amygdala response suggests that our brain is automatically processing unpleasant stimuli in daily life.

Internal-time temporal difference model for neural value-based decision making

The temporal difference (TD) learning framework is a major paradigm for understanding value-based decision making and related neural activities (e.g., dopamine activity). The representation of time in neural processes modeled by a TD framework, however, is poorly understood. To address this issue, we propose a TD formulation that separates the time of the operator (neural valuation processes), which we refer to as internal time, from the time of the observer (experiment), which we refer to as conventional time. We provide the formulation and theoretical characteristics of this TD model based on internal time, called internal-time TD, and explore the possible consequences of the use of this model in neural value-based decision making. Due to the separation of the two times, internal-time TD computations, such as TD error, are expressed differently, depending on both the time frame and time unit. We examine this operator-observer problem in relation to the time representation used in previous TD models. An internal time TD value function exhibits the co-appearance of exponential and hyperbolic discounting at different delays in intertemporal choice tasks. We further examine the effects of internal time noise on TD error, the dynamic construction of internal time, and the modulation of internal time with the internal time hypothesis of serotonin function. We also relate the internal TD formulation to research on interval timing and subjective time.

Neural correlates of aversive conditioning: development of a functional imaging paradigm for the investigation of anxiety disorders

The purpose of the present study was to establish a short paradigm for the examination of classical aversive conditioning processes for application in patients with anxiety disorders. We measured behavioral, autonomic and neural correlates of the paradigm in healthy subjects, applying functional magnetic resonance imaging (fMRI) and measurement of skin conductance. Therefore, neutral visual stimuli were paired with an unpleasant white noise as unconditioned stimulus. Twenty healthy subjects performed three experimental phases of learning: familiarization, acquisition and extinction. Subjective ratings of valence and arousal after each phase of conditioning as well as skin conductance measurement indicated successful conditioning. During acquisition, fMRI results showed increased activation for the conditioned stimulus (CS+(unpaired)) when compared with the non-conditioned stimulus (CS-) in the right amygdala, the insulae, the anterior cingulate cortex and the parahippocampal gyrus, all regions known to be involved in emotional processing. In addition, a linearly decreasing activation in the right amygdala/hippocampus for the CS- across the acquisition phase was found. There were no significant differences between CS+ and CS- during extinction. In conclusion, the applicability of this paradigm for the evaluation of neural correlates in conditioning and extinction processes has been proven. Thus, we present a promising paradigm for the examination of the fear-circuit in patients with anxiety disorders and additionally effects of cognitive-behavioral interventions.

Multiple timescales of memory in lateral habenula and dopamine neurons

Midbrain dopamine neurons are thought to signal predictions about future rewards based on the memory of past rewarding experience. Little is known about the source of their reward memory and the factors that control its timescale. Here we recorded from dopamine neurons, as well as one of their sources of input, the lateral habenula, while animals predicted upcoming rewards based on the past reward history. We found that lateral habenula and dopamine neurons accessed two distinct reward memories: a short-timescale memory expressed at the start of the task and a near-optimal long-timescale memory expressed when a future reward outcome was revealed. The short- and long-timescale memories were expressed in different forms of reward-oriented eye movements. Our data show that the habenula-dopamine pathway contains multiple timescales of memory and provide evidence for their role in motivated behavior.

Dynamic causal modelling of anticipatory skin conductance responses

Anticipatory skin conductance responses [SCRs] are a widely used measure of aversive conditioning in humans. Here, we describe a dynamic causal model [DCM] of how anticipatory, evoked, and spontaneous skin conductance changes are generated by sudomotor nerve activity. Inversion of this model, using variational Bayes, provides a means of inferring the most likely sympathetic nerve activity, given observed skin conductance responses. In two fear conditioning experiments, we demonstrate the predictive validity of the DCM by showing it has greater sensitivity to the effects of conditioning, relative to alternative (conventional) response estimates. Furthermore, we establish face validity by showing that trial-by-trial estimates of anticipatory sudomotor activity are better predicted by formal learning models, relative to response estimates from peak-scoring approaches. The model furnishes a potentially powerful approach to characterising SCR that exploits knowledge about how these signals are generated.

Individual differences and the neural representations of reward expectation and reward prediction error

Reward expectation and reward prediction errors are thought to be critical for dynamic adjustments in decision-making and reward-seeking behavior, but little is known about their representation in the brain during uncertainty and risk-taking. Furthermore, little is known about what role individual differences might play in such reinforcement processes. In this study, it is shown behavioral and neural responses during a decision-making task can be characterized by a computational reinforcement learning model and that individual differences in learning parameters in the model are critical for elucidating these processes. In the fMRI experiment, subjects chose between high- and low-risk rewards. A computational reinforcement learning model computed expected values and prediction errors that each subject might experience on each trial. These outputs predicted subjects' trial-to-trial choice strategies and neural activity in several limbic and prefrontal regions during the task. Individual differences in estimated reinforcement learning parameters proved critical for characterizing these processes, because models that incorporated individual learning parameters explained significantly more variance in the fMRI data than did a model using fixed learning parameters. These findings suggest that the brain engages a reinforcement learning process during risk-taking and that individual differences play a crucial role in modeling this process.

Overlapping neural systems mediating extinction, reversal and regulation of fear

Learned fear is a process allowing quick detection of associations between cues in the environment and prediction of imminent threat. Adaptive function in a changing environment, however, requires organisms to quickly update this learning and have the ability to hinder fear responses when predictions are no longer correct. Here we focus on three strategies that can modify conditioned fear, namely extinction, reversal and regulation of fear, and review their underlying neural mechanisms. By directly comparing neuroimaging data from three separate studies that employ each strategy, we highlight overlapping brain structures that comprise a general circuitry in the human brain. This circuitry potentially enables the flexible control of fear, regardless of the particular task demands.

Using conditional maximization to determine hyperparameters in model-based fMRI

In model-based analysis of fMRI data, a neural or cognitive mathematical model of behavior is used to predict changes in fMRI activity. The model predictions are often applied as a parametric modulation of the main stimulus effect within the context of the general linear model (GLM). Using a mathematical model has become an important method for connecting fMRI signals to behavior because the model represents how stimulus processing leads to behavior, and the parametric modulation represents a specific test about the profile of stimulus-related fMRI activity (for review and discussion, see O'Doherty et al., 2007). However, in some cases the parameters of the mathematical model may be under-determined because there is a range of values that equally well account for behavior, or perhaps an exploratory analysis is desired. Thus, in order to fully gauge the applicability of some mathematical model it would be useful to understand how fMRI analysis depends on those parameters. Here, a conditional maximization procedure is developed to search for parameter values in the mathematical model as hyperparameters in the GLM. Simulations and analysis with real fMRI data show that conditional maximization is an effective and simple procedure for estimating hyperparameters. General recommendations and caveats for using hyperparameters in model-based fMRI analysis are also presented.

The optimal level of fuzz: Case studies in a methodology for psychological research

Cognitive Science research is hard to conduct, because researchers must take phenomena from the world and turn them into laboratory tasks for which a reasonable level of experimental control can be achieved. Consequently, research necessarily makes tradeoffs between internal validity (experimental control) and external validity (the degree to which a task represents behavior outside of the lab). Researchers are thus seeking the best possible tradeoff between these constraints, which we refer to as the optimal level of fuzz. We present two principles for finding the optimal level of fuzz, in research, and then illustrate these principles using research from motivation, individual differences, and cognitive neuroscience.

A dual role for prediction error in associative learning

Confronted with a rich sensory environment, the brain must learn statistical regularities across sensory domains to construct causal models of the world. Here, we used functional magnetic resonance imaging and dynamic causal modeling (DCM) to furnish neurophysiological evidence that statistical associations are learnt, even when task-irrelevant. Subjects performed an audio-visual target-detection task while being exposed to distractor stimuli. Unknown to them, auditory distractors predicted the presence or absence of subsequent visual distractors. We modeled incidental learning of these associations using a Rescorla–Wagner (RW) model. Activity in primary visual cortex and putamen reflected learning-dependent surprise: these areas responded progressively more to unpredicted, and progressively less to predicted visual stimuli. Critically, this prediction-error response was observed even when the absence of a visual stimulus was surprising. We investigated the underlying mechanism by embedding the RW model into a DCM to show that auditory to visual connectivity changed significantly over time as a function of prediction error. Thus, consistent with predictive coding models of perception, associative learning is mediated by prediction-error dependent changes in connectivity. These results posit a dual role for prediction-error in encoding surprise and driving associative plasticity.

Emotion, decision making, and the amygdala

Emotion plays a critical role in many contemporary accounts of decision making, but exactly what underlies its influence and how this is mediated in the brain remain far from clear. Here, we review behavioral studies that suggest that Pavlovian processes can exert an important influence over choice and may account for many effects that have traditionally been attributed to emotion. We illustrate how recent experiments cast light on the underlying structure of Pavlovian control and argue that generally this influence makes good computational sense. Corresponding neuroscientific data from both animals and humans implicate a central role for the amygdala through interactions with other brain areas. This yields a neurobiological account of emotion in which it may operate, often covertly, to optimize rather than corrupt economic choice.

Blockade of endogenous opioid neurotransmission enhances acquisition of conditioned fear in humans

The endogenous opioid system is involved in fear learning in rodents, as opioid agonists attenuate and opioid antagonists facilitate the acquisition of conditioned fear. It has been suggested that an opioidergic signal, which is engaged through conditioning and acts inhibitory on unconditioned stimulus input, is the source of these effects. To clarify whether blockade of endogenous opioid neurotransmission enhances acquisition of conditioned fear in humans, and to elucidate the neural underpinnings of such an effect, we used functional magnetic resonance imaging in combination with behavioral recordings and a double-blind pharmacological intervention. All subjects underwent the same classical fear-conditioning paradigm, but subjects in the experimental group received the opioid antagonist naloxone before and during the experiment, in contrast to subjects in the control group, who received saline. Blocking endogenous opioid neurotransmission with naloxone led to more sustained responses to the unconditioned stimulus across trials, evident in both behavioral and blood oxygen level-dependent responses in pain responsive cortical regions. This effect was likely caused by naloxone blocking conditioned responses in a pain-inhibitory circuit involving opioid-rich areas such as the rostral anterior cingulate cortex, amygdala, and periaqueductal gray. Most importantly, naloxone enhanced the acquisition of fear on the behavioral level and changed the activation profile of the amygdala: whereas the control group showed rapidly decaying conditioned responses across trials, the naloxone group showed sustained conditioned responses in the amygdala. Together, these results demonstrate that in humans the endogenous opioid system has an inhibitory role in the acquisition of fear.

Policy adjustment in a dynamic economic game

Making sequential decisions to harvest rewards is a notoriously difficult problem. One difficulty is that the real world is not stationary and the reward expected from a contemplated action may depend in complex ways on the history of an animal's choices. Previous functional neuroimaging work combined with principled models has detected brain responses that correlate with computations thought to guide simple learning and action choice. Those works generally employed instrumental conditioning tasks with fixed action-reward contingencies. For real-world learning problems, the history of reward-harvesting choices can change the likelihood of rewards collected by the same choices in the near-term future. We used functional MRI to probe brain and behavioral responses in a continuous decision-making task where reward contingency is a function of both a subject's immediate choice and his choice history. In these more complex tasks, we demonstrated that a simple actor-critic model can account for both the subjects' behavioral and brain responses, and identified a reward prediction error signal in ventral striatal structures active during these non-stationary decision tasks. However, a sudden introduction of new reward structures engages more complex control circuitry in the prefrontal cortex (inferior frontal gyrus and anterior insula) and is not captured by a simple actor-critic model. Taken together, these results extend our knowledge of reward-learning signals into more complex, history-dependent choice tasks. They also highlight the important interplay between striatum and prefrontal cortex as decision-makers respond to the strategic demands imposed by non-stationary reward environments more reminiscent of real-world tasks.

Dissociable systems for gain- and loss-related value predictions and errors of prediction in the human brain

Midbrain dopaminergic neurons projecting to the ventral striatum code for reward magnitude and probability during reward anticipation and then indicate the difference between actual and predicted outcome. It has been questioned whether such a common system for the prediction and evaluation of reward exists in humans. Using functional magnetic resonance imaging and a guessing task in two large cohorts, we are able to confirm ventral striatal responses coding both reward probability and magnitude during anticipation, permitting the local computation of expected value (EV). However, the ventral striatum only represented the gain-related part of EV (EV+). At reward delivery, the same area shows a reward probability and magnitude-dependent prediction error signal, best modeled as the difference between actual outcome and EV+. In contrast, loss-related expected value (EV-) and the associated prediction error was represented in the amygdala. Thus, the ventral striatum and the amygdala distinctively process the value of a prediction and subsequently compute a prediction error for gains and losses, respectively. Therefore, a homeostatic balance of both systems might be important for generating adequate expectations under uncertainty. Prevalence of either part might render expectations more positive or negative, which could contribute to the pathophysiology of mood disorders like major depression.

The role of the ventromedial prefrontal cortex in abstract state-based inference during decision making in humans

Many real-life decision-making problems incorporate higher-order structure, involving interdependencies between different stimuli, actions, and subsequent rewards. It is not known whether brain regions implicated in decision making, such as the ventromedial prefrontal cortex (vmPFC), use a stored model of the task structure to guide choice (model-based decision making) or merely learn action or state values without assuming higher-order structure as in standard reinforcement learning. To discriminate between these possibilities, we scanned human subjects with functional magnetic resonance imaging while they performed a simple decision-making task with higher-order structure, probabilistic reversal learning. We found that neural activity in a key decision-making region, the vmPFC, was more consistent with a computational model that exploits higher-order structure than with simple reinforcement learning. These results suggest that brain regions, such as the vmPFC, use an abstract model of task structure to guide behavioral choice, computations that may underlie the human capacity for complex social interactions and abstract strategizing.

Opponent appetitive-aversive neural processes underlie predictive learning of pain relief

Termination of a painful or unpleasant event can be rewarding. However, whether the brain treats relief in a similar way as it treats natural reward is unclear, and the neural processes that underlie its representation as a motivational goal remain poorly understood. We used fMRI (functional magnetic resonance imaging) to investigate how humans learn to generate expectations of pain relief. Using a pavlovian conditioning procedure, we show that subjects experiencing prolonged experimentally induced pain can be conditioned to predict pain relief. This proceeds in a manner consistent with contemporary reward-learning theory (average reward/loss reinforcement learning), reflected by neural activity in the amygdala and midbrain. Furthermore, these reward-like learning signals are mirrored by opposite aversion-like signals in lateral orbitofrontal cortex and anterior cingulate cortex. This dual coding has parallels to 'opponent process' theories in psychology and promotes a formal account of prediction and expectation during pain.