When is a loss a loss? Excitatory and inhibitory processes in loss-related decision-making

Additive Effects of Monetary Loss and Positive Emotion in the Human Brain

In many real-life scenarios, our decisions could lead to multiple outcomes that conflict with value. Hence, an appropriate neural representation of the net experienced value of conflicting outcomes, which play a crucial role in guiding future decisions, is critical for adaptive behavior. As some recent functional neuroimaging work has primarily focused on the concurrent processing of monetary gains and aversive information, very little is known regarding the integration of conflicting value signals involving monetary losses and appetitive information in the human brain. To address this critical gap, we conducted a functional MRI study involving healthy human male participants to examine the nature of integrating positive emotion and monetary losses. We employed a novel experimental design where the valence (positive or neutral) of an emotional stimulus indicated the type of outcome (loss or no loss) in a choice task. Specifically, we probed two plausible integration patterns while processing conflicting value signals involving positive emotion and monetary losses: interactive versus additive. We found overlapping main effects of positive (vs neutral) emotion and loss (vs no loss) in multiple brain regions, including the ventromedial prefrontal cortex, striatum, and amygdala, notably with a lack of evidence for interaction. Thus, our findings revealed the additive integration pattern of monetary loss and positive emotion outcomes, suggesting that the experienced value of the monetary loss was not modulated by the valence of the image signaling those outcomes. These findings contribute to our limited understanding of the nature of integrating conflicting outcomes in the healthy human brain with potential clinical relevance.

Motivational signals disrupt metacognitive signals in the human ventromedial prefrontal cortex

A growing body of evidence suggests that, during decision-making, BOLD signal in the ventromedial prefrontal cortex (VMPFC) correlates both with motivational variables – such as incentives and expected values – and metacognitive variables – such as confidence judgments – which reflect the subjective probability of being correct. At the behavioral level, we recently demonstrated that the value of monetary stakes bias confidence judgments, with gain (respectively loss) prospects increasing (respectively decreasing) confidence judgments, even for similar levels of difficulty and performance. If and how this value-confidence interaction is reflected in the VMPFC remains unknown. Here, we used an incentivized perceptual decision-making fMRI task that dissociates key decision-making variables, thereby allowing to test several hypotheses about the role of the VMPFC in the value-confidence interaction. While our initial analyses seemingly indicate that the VMPFC combines incentives and confidence to form an expected value signal, we falsified this conclusion with a meticulous dissection of qualitative activation patterns. Rather, our results show that strong VMPFC confidence signals observed in trials with gain prospects are disrupted in trials with no – or negative (loss) – monetary prospects. Deciphering how decision variables are represented and interact at finer scales seems necessary to better understand biased (meta)cognition.

The human ventromedial prefrontal cortex helps to determine value and confidence in certain decisions, but only in situations when there is a potential for a (monetary) reward.

The Effect of Subjective Loss in Financial Risk Taking and Negative Emotion

The current research examined the influence of subjective loss on financial risk-taking tendency and negative emotional experience through inducing the experience of subjective loss in auction scenarios. In Study 1, we found that the subjective loss experience (compared to no-loss experience) in an auction scenario induced greater financial risk propensity, especially in gambling, greater negative emotion, and greater decision regret. In addition, we found that the subjective loss experience induced stronger negative emotion but less risk propensity in investment than the actual loss experience did, but these two types of loss did not yield a difference in risk propensity in gambling in Study 2. These results implicate that subjective loss is a distinct experience from no-loss and actual loss experiences, which is reflected by the degree of associated emotional experience and subsequent risk-taking propensity. The current research highlights the complex psychological processes of the experience of loss in decision-making contexts.

Fear-induced increases in loss aversion are linked to increased neural negative-value coding

Human decisions are often influenced by emotions. An economically relevant example is the role of fear in generating loss aversion. Previous research implicates the amygdala as a key brain structure in the experience of fear and loss aversion. The neural mechanism behind emotional influences on loss aversion is, however, unclear. To address this, we measured brain activation with functional magnetic resonance imaging (fMRI) while participants made decisions about monetary gambles after viewing fearful or neutral faces. We observed that loss aversion following the presentation of neutral faces was mainly predicted by greater deactivations for prospective losses (relative to activations for prospective gains) in several brain regions, including the amygdala. By contrast, increases in loss aversion following the presentation of fearful faces were mainly predicted by greater activations for prospective losses. These findings suggest a fear-induced shift from positive to negative value coding that reflects a context-dependent involvement of distinct valuation processes.

A molecular insight into the dissociable regulation of associative learning and motivation by the synaptic protein neuroligin-1

Background

In a changing environment, a challenge for the brain is to flexibly guide adaptive behavior towards survival. Complex behavior and the underlying neural computations emerge from the structural components of the brain across many levels: circuits, cells, and ultimately the signaling complex of proteins at synapses. In line with this logic, dynamic modification of synaptic strength or synaptic plasticity is widely considered the cellular level implementation for adaptive behavior such as learning and memory. Predominantly expressed at excitatory synapses, the postsynaptic cell-adhesion molecule neuroligin-1 (Nlgn1) forms trans-synaptic complexes with presynaptic neurexins. Extensive evidence supports that Nlgn1 is essential for NMDA receptor transmission and long-term potentiation (LTP), both of which are putative synaptic mechanisms underlying learning and memory. Here, employing a comprehensive battery of touchscreen-based cognitive assays, we asked whether impaired NMDA receptor transmission and LTP in mice lacking Nlgn1 does in fact disrupt decision-making. To this end, we addressed two key decision problems: (i) the ability to learn and exploit the associative structure of the environment and (ii) balancing the trade-off between potential rewards and costs, or positive and negative utilities of available actions.

Results

We found that the capacity to acquire complex associative structures and adjust learned associations was intact. However, loss of Nlgn1 alters motivation leading to a reduced willingness to overcome effort cost for reward and an increased willingness to exert effort to escape an aversive situation. We suggest Nlgn1 may be important for balancing the weighting on positive and negative utilities in reward-cost trade-off.

Conclusions

Our findings update canonical views of this key synaptic molecule in behavior and suggest Nlgn1 may be essential for regulating distinct cognitive processes underlying action selection. Our data demonstrate that learning and motivational computations can be dissociated within the same animal model, from a detailed behavioral dissection. Further, these results highlight the complexities in mapping synaptic mechanisms to their behavioral consequences, and the future challenge to elucidate how complex behavior emerges through different levels of neural hardware.

Reward anticipation and punishment anticipation are instantiated in the brain via opponent mechanisms

fMRI investigations have examined the extent to which reward and punishment motivation are associated with common or opponent neural systems, but such investigations have been limited by confounding variables and methodological constraints. The present study aimed to address limitations of earlier approaches and more comprehensively evaluate the extent to which neural activation associated with reward and punishment motivation reflects opponent or shared systems. Participants completed a modified monetary incentive delay task, which involved the presentation of a cue followed by a target to which participants were required to make a speeded button press. Using a factorial design, cues indicated whether monetary reward and/or loss (i.e., cues signaled probability of reward, punishment, both, or neither) could be expected depending upon response speed. Neural analyses evaluated evidence of (a) directionally opposing effects by testing for regions of differential activation for reward and punishment anticipation, (b) mutual inhibition by testing for interactive effects of reward and punishment anticipation within a factorial design, and (c) opposing effects on shared outputs via a psychophysiological interaction analysis. Evidence supporting all three criteria for opponent systems was obtained. Collectively, present findings support conceptualizing reward and punishment motivation as opponent forces influencing brain and behavior and indicate that shared activation does not suggest the operation of a common neural mechanism instantiating reward and punishment motivation.

Ventral Pallidum Encodes Contextual Information and Controls Aversive Behaviors

Successful avoidance of aversive outcomes is crucial for the survival of animals. Although accumulating evidence indicates that an indirect pathway in the basal ganglia is involved in aversive behavior, the ventral pallidum (VP), which is an important component of this pathway, has so far been implicated primarily in appetitive behavior. In this study, we used single-cell recordings and bicuculline (GABAA antagonist) injections to elucidate the role of VP both in the encoding of aversive context and in active avoidance. We found 2 populations of neurons that were preferentially activated by appetitive and aversive conditioned stimuli (CSs). In addition, VP showed appetitive and aversive outcome anticipatory activities. These activity patterns indicate that VP is involved in encoding and maintaining CS-induced aversive contextual information. Furthermore, the disturbance of VP activity by bicuculline injection increased the number of error trials in aversive trials. In particular, the subjects released the response bar prematurely, showed no response at all, or failed to avoid the aversive outcome. Overall, these results suggest that VP plays a central role in controlling CS-induced negative motivation to produce avoidance behavior.

Predicting explorative motor learning using decision-making and motor noise

A fundamental problem faced by humans is learning to select motor actions based on noisy sensory information and incomplete knowledge of the world. Recently, a number of authors have asked whether this type of motor learning problem might be very similar to a range of higher-level decision-making problems. If so, participant behaviour on a high-level decision-making task could be predictive of their performance during a motor learning task. To investigate this question, we studied performance during an explorative motor learning task and a decision-making task which had a similar underlying structure with the exception that it was not subject to motor (execution) noise. We also collected an independent measurement of each participant’s level of motor noise. Our analysis showed that explorative motor learning and decision-making could be modelled as the (approximately) optimal solution to a Partially Observable Markov Decision Process bounded by noisy neural information processing. The model was able to predict participant performance in motor learning by using parameters estimated from the decision-making task and the separate motor noise measurement. This suggests that explorative motor learning can be formalised as a sequential decision-making process that is adjusted for motor noise, and raises interesting questions regarding the neural origin of explorative motor learning.

Until recently, motor learning was viewed as an automatic process that was independent, and even in conflict with higher-level cognitive processes such as decision-making. However, it is now thought that decision-making forms an integral part of motor learning. To further examine the relationship between decision-making and motor learning, we asked whether explorative motor learning could be considered a decision-making task that was adjusted for motor noise. We studied human performance in an explorative motor learning task and a decision-making task which had a similar underlying structure with the exception that it was not subject to motor (execution) noise. In addition, we independently measured each participant’s level of motor noise. Crucially, with a computational model, we were able to predict participant explorative motor learning by using parameters estimated from the decision-making task and the separate motor noise task. This suggests that explorative motor learning can be formalised as a sequential decision-making process that is adjusted for motor noise, and reinforces the view that the mechanisms which control decision-making and motor behaviour are highly integrated.

Improving Robot Motor Learning with Negatively Valenced Reinforcement Signals

Both nociception and punishment signals have been used in robotics. However, the potential for using these negatively valenced types of reinforcement learning signals for robot learning has not been exploited in detail yet. Nociceptive signals are primarily used as triggers of preprogrammed action sequences. Punishment signals are typically disembodied, i.e., with no or little relation to the agent-intrinsic limitations, and they are often used to impose behavioral constraints. Here, we provide an alternative approach for nociceptive signals as drivers of learning rather than simple triggers of preprogrammed behavior. Explicitly, we use nociception to expand the state space while we use punishment as a negative reinforcement learning signal. We compare the performance—in terms of task error, the amount of perceived nociception, and length of learned action sequences—of different neural networks imbued with punishment-based reinforcement signals for inverse kinematic learning. We contrast the performance of a version of the neural network that receives nociceptive inputs to that without such a process. Furthermore, we provide evidence that nociception can improve learning—making the algorithm more robust against network initializations—as well as behavioral performance by reducing the task error, perceived nociception, and length of learned action sequences. Moreover, we provide evidence that punishment, at least as typically used within reinforcement learning applications, may be detrimental in all relevant metrics.

Accumbens D2: Raters of the Loss Outcome

Zalocusky et al. (Nature 531:642-646, 2016) recently showed that activity in D2R+ cells in the nucleus accumbens is associated with loss sensitivity to prior outcomes and reduced subsequent risky choice, and that optogenetic stimulation of these cells decreased risky choices in risk-prone rats. While their findings are important for understanding trait-level risk-taking, future research should aim to examine the neuronal mechanisms of a broader range of facets of gain and loss processing with respect to different potential reference points.

The Computational Development of Reinforcement Learning during Adolescence

Adolescence is a period of life characterised by changes in learning and decision-making. Learning and decision-making do not rely on a unitary system, but instead require the coordination of different cognitive processes that can be mathematically formalised as dissociable computational modules. Here, we aimed to trace the developmental time-course of the computational modules responsible for learning from reward or punishment, and learning from counterfactual feedback. Adolescents and adults carried out a novel reinforcement learning paradigm in which participants learned the association between cues and probabilistic outcomes, where the outcomes differed in valence (reward versus punishment) and feedback was either partial or complete (either the outcome of the chosen option only, or the outcomes of both the chosen and unchosen option, were displayed). Computational strategies changed during development: whereas adolescents’ behaviour was better explained by a basic reinforcement learning algorithm, adults’ behaviour integrated increasingly complex computational features, namely a counterfactual learning module (enabling enhanced performance in the presence of complete feedback) and a value contextualisation module (enabling symmetrical reward and punishment learning). Unlike adults, adolescent performance did not benefit from counterfactual (complete) feedback. In addition, while adults learned symmetrically from both reward and punishment, adolescents learned from reward but were less likely to learn from punishment. This tendency to rely on rewards and not to consider alternative consequences of actions might contribute to our understanding of decision-making in adolescence.

We employed a novel learning task to investigate how adolescents and adults learn from reward versus punishment, and to counterfactual feedback about decisions. Computational analyses revealed that adults and adolescents did not implement the same algorithm to solve the learning task. In contrast to adults, adolescents’ performance did not take into account counterfactual information; adolescents also learned preferentially to seek rewards rather than to avoid punishments, whereas adults learned to seek and avoid both equally. Increasing our understanding of computational changes in reinforcement learning during adolescence may provide insights into adolescent value-based decision-making. Our results might also have implications for education, since they suggest that adolescents benefit more from positive feedback than from negative feedback in learning tasks.