Catecholaminergic modulation of meta-learning

Noradrenergic alpha-2a receptor stimulation enhances prediction error signaling and updating of attention sets in anterior cingulate cortex and striatum

The noradrenergic system is believed to support behavioral flexibility. A possible source mediating improved flexibility are α2A adrenoceptors (α2AR) in prefrontal cortex (PFC) or the anterior cingulate cortex (ACC). We tested this hypothesis by stimulating α2ARs using Guanfacine during attentional set shifting in male nonhuman primates. We found that α2AR stimulation improved learning from errors and updating attention sets. Neural recordings in the ACC, dorsolateral PFC, and the striatum showed that α2AR stimulation selectively enhanced neural signaling of prediction errors in neurons of the ACC and the striatum, but not in dlPFC. This modulation was accompanied by enhanced encoding of attended target features and particularly apparent in putative fast-spiking interneurons, pointing to an interneuron mediated mechanism of α2AR action. These results reveal that α2A receptors are part of the causal chain of flexibly updating attention sets through an enhancement of outcomes and prediction error signaling in ACC and striatum.

Dopaminergic manipulations affect the modulation and meta-modulation of movement speed: Evidence from two pharmacological interventions

A body of research implicates dopamine in the average speed of simple movements. However, naturalistic movements span a range of different shaped trajectories and rarely proceed at a single constant speed. Instead, speed is reduced when drawing "corners" compared to "straights" (i.e., speed modulation), and the extent of this slowing down is dependent upon the global shape of the movement trajectory (i.e., speed meta-modulation) - for example whether the shape is an ellipse or a rounded square. At present, it is not known how (or whether) dopaminergic function controls continuous changes in speed during movement execution. The current paper reports effects on these kinematic features of movement following two forms of dopamine manipulation: Study One highlights movement differences in individuals with PD both ON and OFF their dopaminergic medication (N = 32); Study Two highlights movement differences in individuals from the general population on haloperidol (a dopamine receptor blocker, or "antagonist") and placebo (N = 43). Evidence is presented implicating dopamine in speed, speed modulation and speed meta-modulation, whereby low dopamine conditions are associated with reductions in these variables. These findings move beyond vigour models implicating dopamine in average movement speed, and towards a conceptualisation that involves the modulation of speed as a function of contextual information.

Methamphetamine-induced adaptation of learning rate dynamics depend on baseline performance

The ability to calibrate learning according to new information is a fundamental component of an organism's ability to adapt to changing conditions. Yet, the exact neural mechanisms guiding dynamic learning rate adjustments remain unclear. Catecholamines appear to play a critical role in adjusting the degree to which we use new information over time, but individuals vary widely in the manner in which they adjust to changes. Here, we studied the effects of a low dose of methamphetamine (MA), and individual differences in these effects, on probabilistic reversal learning dynamics in a within-subject, double-blind, randomized design. Participants first completed a reversal learning task during a drug-free baseline session to provide a measure of baseline performance. Then they completed the task during two sessions, one with MA (20 mg oral) and one with placebo (PL). First, we showed that, relative to PL, MA modulates the ability to dynamically adjust learning from prediction errors. Second, this effect was more pronounced in participants who performed poorly at baseline. These results present novel evidence for the involvement of catecholaminergic transmission on learning flexibility and highlights that baseline performance modulates the effect of the drug.

Learning environment-specific learning rates

People often have to switch back and forth between different environments that come with different problems and volatilities. While volatile environments require fast learning (i.e., high learning rates), stable environments call for lower learning rates. Previous studies have shown that people adapt their learning rates, but it remains unclear whether they can also learn about environment-specific learning rates, and instantaneously retrieve them when revisiting environments. Here, using optimality simulations and hierarchical Bayesian analyses across three experiments, we show that people can learn to use different learning rates when switching back and forth between two different environments. We even observe a signature of these environment-specific learning rates when the volatility of both environments is suddenly the same. We conclude that humans can flexibly adapt and learn to associate different learning rates to different environments, offering important insights for developing theories of meta-learning and context-specific control.

Dopamine release in human associative striatum during reversal learning

The dopaminergic system is firmly implicated in reversal learning but human measurements of dopamine release as a correlate of reversal learning success are lacking. Dopamine release and hemodynamic brain activity in response to unexpected changes in action-outcome probabilities are here explored using simultaneous dynamic [11C]Raclopride PET-fMRI and computational modelling of behavior. When participants encounter reversed reward probabilities during a card guessing game, dopamine release is observed in associative striatum. Individual differences in absolute reward prediction error and sensitivity to errors are associated with peak dopamine receptor occupancy. The fMRI response to perseverance errors at the onset of a reversal spatially overlap with the site of dopamine release. Trial-by-trial fMRI correlates of absolute prediction errors show a response in striatum and association cortices, closely overlapping with the location of dopamine release, and separable from a valence signal in ventral striatum. The results converge to implicate striatal dopamine release in associative striatum as a central component of reversal learning, possibly signifying the need for increased cognitive control when new stimuli-responses should be learned.

A boost in self-esteem after positive social evaluation predicts social and non-social learning

Fluctuations in self-esteem resulting from social acceptance and rejection could guide social behaviour by putting us in a state that is more or less open to social experiences. However, it remains unclear whether social acceptance and rejection may shape learning from social information depending on individual differences in self-esteem changes. Here we used a social feedback paradigm to manipulate social acceptance and rejection in a between-subjects design. Subsequently, we administered a behavioural task that enables the assessment of how well individuals learn on the basis of own experiences versus social information. Participants receiving positive (N = 43) versus negative (N = 44) social evaluation demonstrated an increase in subjective self-esteem. Importantly, the effect of the social evaluation on social learning was moderated by self-esteem changes. Specifically, an increase in self-esteem, as induced by positive evaluation, was associated with increased learning from social, but decreased learning from individual information. A decrease in self-esteem in response to negative evaluation was associated with decreased learning from individual information. These data suggest that increases in self-esteem in response to positive evaluation can induce a shift in the inclination to use social versus non-social information and may open one up to constructive learning from others.

Transdiagnostic computations of uncertainty: towards a new lens on intolerance of uncertainty

People radically differ in how they cope with uncertainty. Clinical researchers describe a dispositional characteristic known as "intolerance of uncertainty", a tendency to find uncertainty aversive, reported to be elevated across psychiatric and neurodevelopmental conditions. Concurrently, recent research in computational psychiatry has leveraged theoretical work to characterise individual differences in uncertainty processing. Under this framework, differences in how people estimate different forms of uncertainty can contribute to mental health difficulties. In this review, we briefly outline the concept of intolerance of uncertainty within its clinical context, and we argue that the mechanisms underlying this construct may be further elucidated through modelling how individuals make inferences about uncertainty. We will review the evidence linking psychopathology to different computationally specified forms of uncertainty and consider how these findings might suggest distinct mechanistic routes towards intolerance of uncertainty. We also discuss the implications of this computational approach for behavioural and pharmacological interventions, as well as the importance of different cognitive domains and subjective experiences in studying uncertainty processing.

Adaptive tuning of human learning and choice variability to unexpected uncertainty

Human value-based decisions are notably variable under uncertainty. This variability is known to arise from two distinct sources: variable choices aimed at exploring available options and imprecise learning of option values due to limited cognitive resources. However, whether these two sources of decision variability are tuned to their specific costs and benefits remains unclear. To address this question, we compared the effects of expected and unexpected uncertainty on decision-making in the same reinforcement learning task. Across two large behavioral datasets, we found that humans choose more variably between options but simultaneously learn less imprecisely their values in response to unexpected uncertainty. Using simulations of learning agents, we demonstrate that these opposite adjustments reflect adaptive tuning of exploration and learning precision to the structure of uncertainty. Together, these findings indicate that humans regulate not only how much they explore uncertain options but also how precisely they learn the values of these options.

Effects of DRD2/ANKK1 and COMT Val158Met polymorphisms on stabilization against and adaptation to unexpected events

Genetic variations affecting dopaminergic neuromodulation such as the DRD2/ANKK1 and the COMT Val158Met polymorphisms contribute to goal-directed behavior that requires a balance between stabilization and updating of current states and behaviors. Dopamine is also thought to be relevant for encoding of surprise signals to sensory input and adaptive learning. A link between goal-directed behavior and learning from surprise is therefore plausible. In the present fMRI study, we investigated whether DRD2 and COMT polymorphisms are related to behavioral responses and neural signals in the caudate nucleus and dlPFC during updating or stabilizing internal models of predictable digit sequences. To-be-detected switches between sequences and to-be-ignored digit omissions within a sequence varied by information-theoretic quantities of surprise and entropy. We found that A1 noncarriers and Val-carriers showed a lower response threshold along with increased caudate and dlPFC activation to surprising switches compared with A1-carriers and Met-homozygotes, whose dlPFC activity increased with decreasing switch surprise. In contrast, there were overall smaller differences in behavioral and neural modulation by drift surprise. Our results suggest that the impact of dopamine-relevant polymorphisms in the flexibility-stability trade-off may result in part from the role of dopamine in encoding the weight afforded to events requiring updating or stabilization.

Dopaminergic challenge dissociates learning from primary versus secondary sources of information

Some theories of human cultural evolution posit that humans have social-specific learning mechanisms that are adaptive specialisations moulded by natural selection to cope with the pressures of group living. However, the existence of neurochemical pathways that are specialised for learning from social information and individual experience is widely debated. Cognitive neuroscientific studies present mixed evidence for social-specific learning mechanisms: some studies find dissociable neural correlates for social and individual learning, whereas others find the same brain areas and, dopamine-mediated, computations involved in both. Here, we demonstrate that, like individual learning, social learning is modulated by the dopamine D2 receptor antagonist haloperidol when social information is the primary learning source, but not when it comprises a secondary, additional element. Two groups (total N = 43) completed a decision-making task which required primary learning, from own experience, and secondary learning from an additional source. For one group, the primary source was social, and secondary was individual; for the other group this was reversed. Haloperidol affected primary learning irrespective of social/individual nature, with no effect on learning from the secondary source. Thus, we illustrate that dopaminergic mechanisms underpinning learning can be dissociated along a primary-secondary but not a social-individual axis. These results resolve conflict in the literature and support an expanding field showing that, rather than being specialised for particular inputs, neurochemical pathways in the human brain can process both social and non-social cues and arbitrate between the two depending upon which cue is primarily relevant for the task at hand.

Meta-control: From psychology to computational neuroscience

Research in the past decades shed light on the different mechanisms that underlie our capacity for cognitive control. However, the meta-level processes that regulate cognitive control itself remain poorly understood. Following the terminology from artificial intelligence, meta-control can be defined as a collection of mechanisms that (a) monitor the progress of controlled processing and (b) regulate the underlying control parameters in the service of current task goals and in response to internal or external constraints. From a psychological perspective, meta-control is an important concept because it may help explain and predict how and when human agents select different types of behavioral strategies. From a cognitive neuroscience viewpoint, meta-control is a useful concept for understanding the complex networks in the prefrontal cortex that guide higher-level behavior as well as their interactions with neuromodulatory systems (such as the dopamine or norepinephrine system). The purpose of the special issue is to integrate hitherto segregated strands of research across three different perspectives: 1) a psychological perspective that specifies meta-control processes on a functional level and aims to operationalize them in experimental tasks; 2) a computational perspective that builds on ideas from artificial intelligence to formalize normative solutions to meta-control problems; and 3) a cognitive neuroscience perspective that identifies neural correlates of and mechanisms underlying meta-control.

Prediction and Learning: Understanding Uncertainty

We build models of the world around us to guide perception and learning in the face of uncertainty. New evidence reveals a neurocomputational mechanism that links predictive processes across cognitive domains.

The Computational, Pharmacological, and Physiological Determinants of Sensory Learning under Uncertainty

The ability to represent and respond to uncertainty is fundamental to human cognition and decision-making. Noradrenaline (NA) is hypothesized to play a key role in coordinating the sensory, learning, and physiological states necessary to adapt to a changing world, but direct evidence for this is lacking in humans. Here, we tested the effects of attenuating noradrenergic neurotransmission on learning under uncertainty. We probed the effects of the β-adrenergic receptor antagonist propranolol (40 mg) using a between-subjects, double-blind, placebo-controlled design. Participants performed a probabilistic associative learning task, and we employed a hierarchical learning model to formally quantify prediction errors about cue-outcome contingencies and changes in these associations over time (volatility). Both unexpectedness and noise slowed down reaction times, but propranolol augmented the interaction between these main effects such that behavior was influenced more by prior expectations when uncertainty was high. Computationally, this was driven by a reduction in learning rates, with people slower to update their beliefs in the face of new information. Attenuating the global effects of NA also eliminated the phasic effects of prediction error and volatility on pupil size, consistent with slower belief updating. Finally, estimates of environmental volatility were predicted by baseline cardiac measures in all participants. Our results demonstrate that NA underpins behavioral and computational responses to uncertainty. These findings have important implications for understanding the impact of uncertainty on human biology and cognition.

Effects of methylphenidate on reinforcement learning depend on working memory capacity

RATIONALE

Brain catecholamines have long been implicated in reinforcement learning, exemplified by catecholamine drug and genetic effects on probabilistic reversal learning. However, the mechanisms underlying such effects are unclear.

OBJECTIVES AND METHODS

Here we investigated effects of an acute catecholamine challenge with methylphenidate (20 mg, oral) on a novel probabilistic reversal learning paradigm in a within-subject, double-blind randomised design. The paradigm was designed to disentangle effects on punishment avoidance from effects on reward perseveration. Given the known large individual variability in methylphenidate's effects, we stratified our effects by working memory capacity and trait impulsivity, putatively modulating the effects of methylphenidate, in a large sample (n = 102) of healthy volunteers.

RESULTS

Contrary to our prediction, methylphenidate did not alter performance in the reversal phase of the task. Our key finding is that methylphenidate altered learning of choice-outcome contingencies in a manner that depended on individual variability in working memory span. Specifically, methylphenidate improved performance by adaptively reducing the effective learning rate in participants with higher working memory capacity.

CONCLUSIONS

This finding emphasises the important role of working memory in reinforcement learning, as reported in influential recent computational modelling and behavioural work, and highlights the dependence of this interplay on catecholaminergic function.