The anatomy of choice: dopamine and decision-making

Role of the first and second person perspective for control of behaviour: Understanding other people's facial expressions

Humans typically make probabilistic inferences about another person's affective state based on her/his bodily movements such as emotional facial expressions, emblematic gestures and whole body movements. Furthermore, humans deduce tentative predictions about the other person's intentions. Thus, the first person perspective of a subject is supplemented by the second person perspective involving theory of mind and empathy. Neuroimaging investigations have shown that the medial and lateral frontal cortex are critical nodes in the circuits underlying theory of mind, empathy, as well as intention of action. It is suggested that personal perspective taking in social interactions is paradigmatic for the capability of humans to generate probabilistic accounts of the outside world that underlie a person's control of behaviour.

Evidence for surprise minimization over value maximization in choice behavior

Classical economic models are predicated on the idea that the ultimate aim of choice is to maximize utility or reward. In contrast, an alternative perspective highlights the fact that adaptive behavior requires agents' to model their environment and minimize surprise about the states they frequent. We propose that choice behavior can be more accurately accounted for by surprise minimization compared to reward or utility maximization alone. Minimizing surprise makes a prediction at variance with expected utility models; namely, that in addition to attaining valuable states, agents attempt to maximize the entropy over outcomes and thus 'keep their options open'. We tested this prediction using a simple binary choice paradigm and show that human decision-making is better explained by surprise minimization compared to utility maximization. Furthermore, we replicated this entropy-seeking behavior in a control task with no explicit utilities. These findings highlight a limitation of purely economic motivations in explaining choice behavior and instead emphasize the importance of belief-based motivations.

Dopamine, reward learning, and active inference

Temporal difference learning models propose phasic dopamine signaling encodes reward prediction errors that drive learning. This is supported by studies where optogenetic stimulation of dopamine neurons can stand in lieu of actual reward. Nevertheless, a large body of data also shows that dopamine is not necessary for learning, and that dopamine depletion primarily affects task performance. We offer a resolution to this paradox based on an hypothesis that dopamine encodes the precision of beliefs about alternative actions, and thus controls the outcome-sensitivity of behavior. We extend an active inference scheme for solving Markov decision processes to include learning, and show that simulated dopamine dynamics strongly resemble those actually observed during instrumental conditioning. Furthermore, simulated dopamine depletion impairs performance but spares learning, while simulated excitation of dopamine neurons drives reward learning, through aberrant inference about outcome states. Our formal approach provides a novel and parsimonious reconciliation of apparently divergent experimental findings.

Active Inference, homeostatic regulation and adaptive behavioural control

We review a theory of homeostatic regulation and adaptive behavioural control within the Active Inference framework. Our aim is to connect two research streams that are usually considered independently; namely, Active Inference and associative learning theories of animal behaviour. The former uses a probabilistic (Bayesian) formulation of perception and action, while the latter calls on multiple (Pavlovian, habitual, goal-directed) processes for homeostatic and behavioural control. We offer a synthesis these classical processes and cast them as successive hierarchical contextualisations of sensorimotor constructs, using the generative models that underpin Active Inference. This dissolves any apparent mechanistic distinction between the optimization processes that mediate classical control or learning. Furthermore, we generalize the scope of Active Inference by emphasizing interoceptive inference and homeostatic regulation. The ensuing homeostatic (or allostatic) perspective provides an intuitive explanation for how priors act as drives or goals to enslave action, and emphasises the embodied nature of inference.

Neural signals encoding shifts in beliefs

Dopamine is implicated in a diverse range of cognitive functions including cognitive flexibility, task switching, signalling novel or unexpected stimuli as well as advance information. There is also longstanding line of thought that links dopamine with belief formation and, crucially, aberrant belief formation in psychosis. Integrating these strands of evidence would suggest that dopamine plays a central role in belief updating and more specifically in encoding of meaningful information content in observations. The precise nature of this relationship has remained unclear. To directly address this question we developed a paradigm that allowed us to decompose two distinct types of information content, information-theoretic surprise that reflects the unexpectedness of an observation, and epistemic value that induces shifts in beliefs or, more formally, Bayesian surprise. Using functional magnetic-resonance imaging in humans we show that dopamine-rich midbrain regions encode shifts in beliefs whereas surprise is encoded in prefrontal regions, including the pre-supplementary motor area and dorsal cingulate cortex. By linking putative dopaminergic activity to belief updating these data provide a link to false belief formation that characterises hyperdopaminergic states associated with idiopathic and drug induced psychosis.

Nociception, Pain, Negative Moods, and Behavior Selection

Recent neuroimaging studies suggest that the brain adapts with pain, as well as imparts risk for developing chronic pain. Within this context, we revisit the concepts for nociception, acute and chronic pain, and negative moods relative to behavior selection. We redefine nociception as the mechanism protecting the organism from injury, while acute pain as failure of avoidant behavior, and a mesolimbic threshold process that gates the transformation of nociceptive activity to conscious pain. Adaptations in this threshold process are envisioned to be critical for development of chronic pain. We deconstruct chronic pain into four distinct phases, each with specific mechanisms, and outline current state of knowledge regarding these mechanisms: the limbic brain imparting risk, and the mesolimbic learning processes reorganizing the neocortex into a chronic pain state. Moreover, pain and negative moods are envisioned as a continuum of aversive behavioral learning, which enhance survival by protecting against threats.

Linking unfounded beliefs to genetic dopamine availability

Unfounded convictions involving beliefs in the paranormal, grandiosity ideas or suspicious thoughts are endorsed at varying degrees among the general population. Here, we investigated the neurobiopsychological basis of the observed inter-individual variability in the propensity toward unfounded beliefs. One hundred two healthy individuals were genotyped for four polymorphisms in the COMT gene (rs6269, rs4633, rs4818, and rs4680, also known as val¹⁵⁸met) that define common functional haplotypes with substantial impact on synaptic dopamine degradation, completed a questionnaire measuring unfounded beliefs, and took part in a behavioral experiment assessing perceptual inference. We found that greater dopamine availability was associated with a stronger propensity toward unfounded beliefs, and that this effect was statistically mediated by an enhanced influence of expectations on perceptual inference. Our results indicate that genetic differences in dopaminergic neurotransmission account for inter-individual differences in perceptual inference linked to the formation and maintenance of unfounded beliefs. Thus, dopamine might be critically involved in the processes underlying one's interpretation of the relationship between the self and the world.

Dissociable Effects of Serotonin and Dopamine on the Valuation of Harm in Moral Decision Making

An aversion to harming others is a core component of human morality and is disturbed in antisocial behavior. Deficient harm aversion may underlie instrumental and reactive aggression, which both feature in psychopathy. Past work has highlighted monoaminergic influences on aggression, but a mechanistic account of how monoamines regulate antisocial motives remains elusive. We previously observed that most people show a greater aversion to inflicting pain on others than themselves. Here, we investigated whether this hyperaltruistic disposition is susceptible to monoaminergic control. We observed dissociable effects of the serotonin reuptake inhibitor citalopram and the dopamine precursor levodopa on decisions to inflict pain on oneself and others for financial gain. Computational models of choice behavior showed that citalopram increased harm aversion for both self and others, while levodopa reduced hyperaltruism. The effects of citalopram were stronger than those of levodopa. Crucially, neither drug influenced the physical perception of pain or other components of choice such as motor impulsivity or loss aversion, suggesting a direct and specific influence of serotonin and dopamine on the valuation of harm. We also found evidence for dose dependency of these effects. Finally, the drugs had dissociable effects on response times, with citalopram enhancing behavioral inhibition and levodopa reducing slowing related to being responsible for another's fate. These distinct roles of serotonin and dopamine in modulating moral behavior have implications for potential treatments of social dysfunction that is a common feature as well as a risk factor for many psychiatric disorders.

Active inference and epistemic value

We offer a formal treatment of choice behavior based on the premise that agents minimize the expected free energy of future outcomes. Crucially, the negative free energy or quality of a policy can be decomposed into extrinsic and epistemic (or intrinsic) value. Minimizing expected free energy is therefore equivalent to maximizing extrinsic value or expected utility (defined in terms of prior preferences or goals), while maximizing information gain or intrinsic value (or reducing uncertainty about the causes of valuable outcomes). The resulting scheme resolves the exploration-exploitation dilemma: Epistemic value is maximized until there is no further information gain, after which exploitation is assured through maximization of extrinsic value. This is formally consistent with the Infomax principle, generalizing formulations of active vision based upon salience (Bayesian surprise) and optimal decisions based on expected utility and risk-sensitive (Kullback-Leibler) control. Furthermore, as with previous active inference formulations of discrete (Markovian) problems, ad hoc softmax parameters become the expected (Bayes-optimal) precision of beliefs about, or confidence in, policies. This article focuses on the basic theory, illustrating the ideas with simulations. A key aspect of these simulations is the similarity between precision updates and dopaminergic discharges observed in conditioning paradigms.

Active inference, evidence accumulation, and the urn task

Deciding how much evidence to accumulate before making a decision is a problem we and other animals often face, but one that is not completely understood. This issue is particularly important because a tendency to sample less information (often known as reflection impulsivity) is a feature in several psychopathologies, such as psychosis. A formal understanding of information sampling may therefore clarify the computational anatomy of psychopathology. In this theoretical letter, we consider evidence accumulation in terms of active (Bayesian) inference using a generic model of Markov decision processes. Here, agents are equipped with beliefs about their own behavior--in this case, that they will make informed decisions. Normative decision making is then modeled using variational Bayes to minimize surprise about choice outcomes. Under this scheme, different facets of belief updating map naturally onto the functional anatomy of the brain (at least at a heuristic level). Of particular interest is the key role played by the expected precision of beliefs about control, which we have previously suggested may be encoded by dopaminergic neurons in the midbrain. We show that manipulating expected precision strongly affects how much information an agent characteristically samples, and thus provides a possible link between impulsivity and dopaminergic dysfunction. Our study therefore represents a step toward understanding evidence accumulation in terms of neurobiologically plausible Bayesian inference and may cast light on why this process is disordered in psychopathology.

The why, what, where, when and how of goal-directed choice: neuronal and computational principles

The central problems that goal-directed animals must solve are: 'What do I need and Why, Where and When can this be obtained, and How do I get it?' or the H4W problem. Here, we elucidate the principles underlying the neuronal solutions to H4W using a combination of neurobiological and neurorobotic approaches. First, we analyse H4W from a system-level perspective by mapping its objectives onto the Distributed Adaptive Control embodied cognitive architecture which sees the generation of adaptive action in the real world as the primary task of the brain rather than optimally solving abstract problems. We next map this functional decomposition to the architecture of the rodent brain to test its consistency. Following this approach, we propose that the mammalian brain solves the H4W problem on the basis of multiple kinds of outcome predictions, integrating central representations of needs and drives (e.g. hypothalamus), valence (e.g. amygdala), world, self and task state spaces (e.g. neocortex, hippocampus and prefrontal cortex, respectively) combined with multi-modal selection (e.g. basal ganglia). In our analysis, goal-directed behaviour results from a well-structured architecture in which goals are bootstrapped on the basis of predefined needs, valence and multiple learning, memory and planning mechanisms rather than being generated by a singular computation.