How the Level of Reward Awareness Changes the Computational and Electrophysiological Signatures of Reinforcement Learning

Risk-Taking Is Associated with Decreased Subjective Value Signals and Increased Prediction Error Signals in the Hot Columbia Card Task

It remains a pressing concern to understand how neural computations relate to risky decisions. However, most observations of brain-behavior relationships in the risk-taking domain lack a rigorous computational basis or fail to emulate of the dynamic, sequential nature of real-life risky decision-making. Recent advances emphasize the role of neural prediction error (PE) signals. We modeled, according to prospect theory, the choices of n = 43 human participants (33 females, 10 males) performing an EEG version of the hot Columbia Card Task, featuring rounds of sequential decisions between stopping (safe option) and continuing with increasing odds of a high loss (risky option). Single-trial regression EEG analyses yielded a subjective value signal at centroparietal (300-700 ms) and frontocentral (>800 ms) electrodes and in the delta band, as well as PE signals tied to the feedback-related negativity, P3a, and P3b, and in the theta band. Higher risk preference (total number of risky choices) was linked to attenuated subjective value signals but increased PE signals. Higher P3-like activity associated with the most positive PE in each round predicted stopping in the present round but not risk-taking in the subsequent round. Our findings indicate that decreased representation of decision values and increased sensitivity to winning despite low odds (positive PE) facilitate risky choices at the subject level. Strong neural responses when gains are least expected (the most positive PE on each round) adaptively contribute to safer choices at the trial-by-trial level but do not affect risky choice at the round-by-round level.

Getting stuck in a rut as an emergent feature of a dynamic decision-making system

Human sensorimotor decision making has a tendency to get 'stuck in a rut', being biased towards selecting a previously implemented action structure (hysteresis). Existing explanations propose this is the consequence of an agent efficiently modifying an existing plan, rather than creating a new plan from scratch. Instead, we propose that hysteresis is an emergent property of a system learning from the consequences of its actions. To examine this, 152 participants moved a cursor to a target on a tablet device while avoiding an obstacle. Hysteresis was observed when the obstacle moved sequentially across the screen between trials, whereby the participant continued moving around the same side of the obstacle despite it now requiring a larger movement than the alternative. Two further experiments (n = 20) showed an attenuation when time and resource constraints were eased. We created a simple computational model capturing probabilistic estimate updating that showed the same patterns of results. This provides, to our knowledge, the first computational demonstration of how sensorimotor decision making can get 'stuck in a rut' through the updating of the probability estimates associated with actions.

Specificity and sensitivity of the fixed-point test for binary mixture distributions

When two cognitive processes contribute to a behavioral output-each process producing a specific distribution of the behavioral variable of interest-and when the mixture proportion of these two processes varies as a function of an experimental condition, a common density point should be present in the observed distributions of the data across said conditions. In principle, one can statistically test for the presence (or absence) of a fixed point in experimental data to provide evidence in favor of (or against) the presence of a mixture of processes, whose proportions are affected by an experimental manipulation. In this paper, we provide an empirical diagnostic of this test to detect a mixture of processes. We do so using resampling of real experimental data under different scenarios, which mimic variations in the experimental design suspected to affect the sensitivity and specificity of the fixed-point test (i.e., mixture proportion, time on task, and sample size). Resampling such scenarios with real data allows us to preserve important features of data which are typically observed in real experiments while maintaining tight control over the properties of the resampled scenarios. This is of particular relevance considering such stringent assumptions underlying the fixed-point test. With this paper, we ultimately aim at validating the fixed-point property of binary mixture data and at providing some performance metrics to researchers aiming at testing the fixed-point property on their experimental data.

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants-even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.

Reinforcement learning deficits exhibited by postnatal PCP-treated rats enable deep neural network classification

The ability to appropriately update the value of a given action is a critical component of flexible decision making. Several psychiatric disorders, including schizophrenia, are associated with impairments in flexible decision making that can be evaluated using the probabilistic reversal learning (PRL) task. The PRL task has been reverse-translated for use in rodents. Disrupting glutamate neurotransmission during early postnatal neurodevelopment in rodents has induced behavioral, cognitive, and neuropathophysiological abnormalities relevant to schizophrenia. Here, we tested the hypothesis that using the NMDA receptor antagonist phencyclidine (PCP) to disrupt postnatal glutamatergic transmission in rats would lead to impaired decision making in the PRL. Consistent with this hypothesis, compared to controls the postnatal PCP-treated rats completed fewer reversals and exhibited disruptions in reward and punishment sensitivity (i.e., win-stay and lose-shift responding, respectively). Moreover, computational analysis of behavior revealed that postnatal PCP-treatment resulted in a pronounced impairment in the learning rate throughout PRL testing. Finally, a deep neural network (DNN) trained on the rodent behavior could accurately predict the treatment group of subjects. These data demonstrate that disrupting early postnatal glutamatergic neurotransmission impairs flexible decision making and provides evidence that DNNs can be trained on behavioral datasets to accurately predict the treatment group of new subjects, highlighting the potential for DNNs to aid in the diagnosis of schizophrenia.

Anxiety attenuates learning advantages conferred by statistical stability and induces loss of volatility-attuning in brain activity

Anxiety can alter an individual's perception of their external sensory environment. Previous studies suggest that anxiety can increase the magnitude of neural responses to unexpected (or surprising) stimuli. Additionally, surprise responses are reported to be boosted during stable compared to volatile environments. Few studies, however, have examined how learning is impacted by both threat and volatility. To investigate these effects, we used threat-of-shock to transiently increase subjective anxiety in healthy adults while they performed an auditory oddball task under stable and volatile environments and while undergoing functional Magnetic Resonance Imaging (fMRI) scanning. We then used Bayesian Model Selection (BMS) mapping to identify the brain areas where different models of anxiety displayed the highest evidence. Behaviourally, we found that threat-of-shock eliminated the accuracy advantage conferred by environmental stability over volatility. Neurally, we found that threat-of-shock led to attenuation and loss of volatility-attuning of brain activity evoked by surprising sounds across most subcortical and limbic regions including the thalamus, basal ganglia, claustrum, insula, anterior cingulate, hippocampal gyrus and the superior temporal gyrus. Taken together, our findings suggest that threat eliminates learning advantages conferred by statistical stability compared to volatility. Thus, we propose that anxiety disrupts behavioural adaptation to environmental statistics, and that multiple subcortical and limbic regions are implicated in this process.

Persistent activity in human parietal cortex mediates perceptual choice repetition bias

Humans and other animals tend to repeat or alternate their previous choices, even when judging sensory stimuli presented in a random sequence. It is unclear if and how sensory, associative, and motor cortical circuits produce these idiosyncratic behavioral biases. Here, we combined behavioral modeling of a visual perceptual decision with magnetoencephalographic (MEG) analyses of neural dynamics, across multiple regions of the human cerebral cortex. We identified distinct history-dependent neural signals in motor and posterior parietal cortex. Gamma-band activity in parietal cortex tracked previous choices in a sustained fashion, and biased evidence accumulation toward choice repetition; sustained beta-band activity in motor cortex inversely reflected the previous motor action, and biased the accumulation starting point toward alternation. The parietal, not motor, signal mediated the impact of previous on current choice and reflected individual differences in choice repetition. In sum, parietal cortical signals seem to play a key role in shaping choice sequences.

Psychiatrically relevant signatures of domain-general decision-making and metacognition in the general population

Human behaviours are guided by how confident we feel in our abilities. When confidence does not reflect objective performance, this can impact critical adaptive functions and impair life quality. Distorted decision-making and confidence have been associated with mental health problems. Here, utilising advances in computational and transdiagnostic psychiatry, we sought to map relationships between psychopathology and both decision-making and confidence in the general population across two online studies (N's = 344 and 473, respectively). The results revealed dissociable decision-making and confidence signatures related to distinct symptom dimensions. A dimension characterised by compulsivity and intrusive thoughts was found to be associated with reduced objective accuracy but, paradoxically, increased absolute confidence, whereas a dimension characterized by anxiety and depression was associated with systematically low confidence in the absence of impairments in objective accuracy. These relationships replicated across both studies and distinct cognitive domains (perception and general knowledge), suggesting that they are reliable and domain general. Additionally, whereas Big-5 personality traits also predicted objective task performance, only symptom dimensions related to subjective confidence. Domain-general signatures of decision-making and metacognition characterise distinct psychological dispositions and psychopathology in the general population and implicate confidence as a central component of mental health.

The computational roots of positivity and confirmation biases in reinforcement learning

Humans do not integrate new information objectively: outcomes carrying a positive affective value and evidence confirming one's own prior belief are overweighed. Until recently, theoretical and empirical accounts of the positivity and confirmation biases assumed them to be specific to 'high-level' belief updates. We present evidence against this account. Learning rates in reinforcement learning (RL) tasks, estimated across different contexts and species, generally present the same characteristic asymmetry, suggesting that belief and value updating processes share key computational principles and distortions. This bias generates over-optimistic expectations about the probability of making the right choices and, consequently, generates over-optimistic reward expectations. We discuss the normative and neurobiological roots of these RL biases and their position within the greater picture of behavioral decision-making theories.

Implicit Counterfactual Effect in Partial Feedback Reinforcement Learning: Behavioral and Modeling Approach

Context remarkably affects learning behavior by adjusting option values according to the distribution of available options. Displaying counterfactual outcomes, the outcomes of the unchosen option alongside the chosen one (i.e., providing complete feedback), would increase the contextual effect by inducing participants to compare the two outcomes during learning. However, when the context only consists of the juxtaposition of several options and there is no such explicit counterfactual factor (i.e., only partial feedback is provided), it is not clear whether and how the contextual effect emerges. In this research, we employ Partial and Complete feedback paradigms in which options are associated with different reward distributions. Our modeling analysis shows that the model that uses the outcome of the chosen option for updating the values of both chosen and unchosen options in opposing directions can better account for the behavioral data. This is also in line with the diffusive effect of dopamine on the striatum. Furthermore, our data show that the contextual effect is not limited to probabilistic rewards, but also extends to magnitude rewards. These results suggest that by extending the counterfactual concept to include the effect of the chosen outcome on the unchosen option, we can better explain why there is a contextual effect in situations in which there is no extra information about the unchosen outcome.

What Is It Like to Be a Bass? Red Herrings, Fish Pain and the Study of Animal Sentience

Debates around fishes' ability to feel pain concern sentience: do reactions to tissue damage indicate evaluative consciousness (conscious affect), or mere nociception? Thanks to Braithwaite's discovery of trout nociceptors, and concerns that current practices could compromise welfare in countless fish, this issue's importance is beyond dispute. However, nociceptors are merely necessary, not sufficient, for true pain, and many measures held to indicate sentience have the same problem. The question of whether fish feel pain - or indeed anything at all - therefore stimulates sometimes polarized debate. Here, we try to bridge the divide. After reviewing key consciousness concepts, we identify "red herring" measures that should not be used to infer sentience because also present in non-sentient organisms, notably those lacking nervous systems, like plants and protozoa (P); spines disconnected from brains (S); decerebrate mammals and birds (D); and humans in unaware states (U). These "S.P.U.D. subjects" can show approach/withdrawal; react with apparent emotion; change their reactivity with food deprivation or analgesia; discriminate between stimuli; display Pavlovian learning, including some forms of trace conditioning; and even learn simple instrumental responses. Consequently, none of these responses are good indicators of sentience. Potentially more valid are aspects of working memory, operant conditioning, the self-report of state, and forms of higher order cognition. We suggest new experiments on humans to test these hypotheses, as well as modifications to tests for "mental time travel" and self-awareness (e.g., mirror self-recognition) that could allow these to now probe sentience (since currently they reflect perceptual rather than evaluative, affective aspects of consciousness). Because "bullet-proof" neurological and behavioral indicators of sentience are thus still lacking, agnosticism about fish sentience remains widespread. To end, we address how to balance such doubts with welfare protection, discussing concerns raised by key skeptics in this debate. Overall, we celebrate the rigorous evidential standards required by those unconvinced that fish are sentient; laud the compassion and ethical rigor shown by those advocating for welfare protections; and seek to show how precautionary principles still support protecting fish from physical harm.

Neurofeedback Training of Auditory Selective Attention Enhances Speech-In-Noise Perception

Selective attention enhances cortical responses to attended sensory inputs while suppressing others, which can be an effective strategy for speech-in-noise (SiN) understanding. Emerging evidence exhibits a large variance in attentional control during SiN tasks, even among normal-hearing listeners. Yet whether training can enhance the efficacy of attentional control and, if so, whether the training effects can be transferred to performance on a SiN task has not been explicitly studied. Here, we introduce a neurofeedback training paradigm designed to reinforce the attentional modulation of auditory evoked responses. Young normal-hearing adults attended one of two competing speech streams consisting of five repeating words (“up”) in a straight rhythm spoken by a female speaker and four straight words (“down”) spoken by a male speaker. Our electroencephalography-based attention decoder classified every single trial using a template-matching method based on pre-defined patterns of cortical auditory responses elicited by either an “up” or “down” stream. The result of decoding was provided on the screen as online feedback. After four sessions of this neurofeedback training over 4 weeks, the subjects exhibited improved attentional modulation of evoked responses to the training stimuli as well as enhanced cortical responses to target speech and better performance during a post-training SiN task. Such training effects were not found in the Placebo Group that underwent similar attention training except that feedback was given only based on behavioral accuracy. These results indicate that the neurofeedback training may reinforce the strength of attentional modulation, which likely improves SiN understanding. Our finding suggests a potential rehabilitation strategy for SiN deficits.

Pupil Dilation and the Slow Wave ERP Reflect Surprise about Choice Outcome Resulting from Intrinsic Variability in Decision Confidence

Central to human and animal cognition is the ability to learn from feedback in order to optimize future rewards. Such a learning signal might be encoded and broadcasted by the brain's arousal systems, including the noradrenergic locus coeruleus. Pupil responses and the positive slow wave component of event-related potentials reflect rapid changes in the arousal level of the brain. Here, we ask whether and how these variables may reflect surprise: the mismatch between one's expectation about being correct and the outcome of a decision, when expectations fluctuate due to internal factors (e.g., engagement). We show that during an elementary decision task in the face of uncertainty both physiological markers of phasic arousal reflect surprise. We further show that pupil responses and slow wave event-related potential are unrelated to each other and that prediction error computations depend on feedback awareness. These results further advance our understanding of the role of central arousal systems in decision-making under uncertainty.

Response-based outcome predictions and confidence regulate feedback processing and learning

Influential theories emphasize the importance of predictions in learning: we learn from feedback to the extent that it is surprising, and thus conveys new information. Here, we explore the hypothesis that surprise depends not only on comparing current events to past experience, but also on online evaluation of performance via internal monitoring. Specifically, we propose that people leverage insights from response-based performance monitoring - outcome predictions and confidence - to control learning from feedback. In line with predictions from a Bayesian inference model, we find that people who are better at calibrating their confidence to the precision of their outcome predictions learn more quickly. Further in line with our proposal, EEG signatures of feedback processing are sensitive to the accuracy of, and confidence in, post-response outcome predictions. Taken together, our results suggest that online predictions and confidence serve to calibrate neural error signals to improve the efficiency of learning.

Diminished Feedback Evaluation and Knowledge Updating Underlying Age-Related Differences in Choice Behavior During Feedback Learning

In our daily lives, we continuously evaluate feedback information, update our knowledge, and adapt our behavior in order to reach desired goals. This ability to learn from feedback information, however, declines with age. Previous research has indicated that certain higher-level learning processes, such as feedback evaluation, integration of feedback information, and updating of knowledge, seem to be affected by age, and recent studies have shown how the adaption of choice behavior following feedback can differ with age. The neural mechanisms underlying this age-related change in choice behavior during learning, however, remain unclear. The aim of this study is therefore to investigate the relation between learning-related neural processes and choice behavior during feedback learning in two age groups. Behavioral and fMRI data were collected, while a group of young (age 18–30) and older (age 60–75) adults performed a probabilistic learning task consisting of 10 blocks of 20 trials each. On each trial, the participants chose between a house and a face, after which they received visual feedback (loss vs. gain). In each block, either the house or the face image had a higher probability of yielding a reward (62.5 vs. 37.5%). Participants were instructed to try to maximize their gains. Our results showed that less successful learning in older adults, as indicated by a lower learning rate, corresponded with a higher tendency to switch to the other stimulus option, and with a reduced adaptation of this switch choice behavior following positive feedback. At the neural level, activation following positive and negative feedback was found to be less distinctive in the older adults, due to a smaller feedback-evaluation response to positive feedback in this group. Furthermore, whereas young adults displayed increased levels of knowledge updating prior to adapting choice behavior, we did not find this effect in older adults. Together, our results suggest that diminished learning performance with age corresponds with diminished evaluation of positive feedback and reduced knowledge updating related to changes in choice behavior, indicating how such differences in feedback processing at the trial level in older adults might lead to reduced learning performance across trials.

Information about action outcomes differentially affects learning from self-determined versus imposed choices

The valence of new information influences learning rates in humans: good news tends to receive more weight than bad news. We investigated this learning bias in four experiments, by systematically manipulating the source of required action (free versus forced choices), outcome contingencies (low versus high reward) and motor requirements (go versus no-go choices). Analysis of model-estimated learning rates showed that the confirmation bias in learning rates was specific to free choices, but was independent of outcome contingencies. The bias was also unaffected by the motor requirements, thus suggesting that it operates in the representational space of decisions, rather than motoric actions. Finally, model simulations revealed that learning rates estimated from the choice-confirmation model had the effect of maximizing performance across low- and high-reward environments. We therefore suggest that choice-confirmation bias may be adaptive for efficient learning of action-outcome contingencies, above and beyond fostering person-level dispositions such as self-esteem.

Neurocognitive Underpinnings of Aggressive Predation in Economic Contests

Competitions are part and parcel of daily life and require people to invest time and energy to gain advantage over others and to avoid (the risk of) falling behind. Whereas the behavioral mechanisms underlying competition are well documented, its neurocognitive underpinnings remain poorly understood. We addressed this using neuroimaging and computational modeling of individual investment decisions aimed at exploiting one's counterpart ("attack") or at protecting against exploitation by one's counterpart ("defense"). Analyses revealed that during attack relative to defense (i) individuals invest less and are less successful; (ii) computations of expected reward are strategically more sophisticated (reasoning level k = 4 vs. k = 3 during defense); (iii) ventral striatum activity tracks reward prediction errors; (iv) risk prediction errors were not correlated with neural activity in either ROI or whole-brain analyses; and (v) successful exploitation correlated with neural activity in the bilateral ventral striatum, left OFC, left anterior insula, left TPJ, and lateral occipital cortex. We conclude that, in economic contests, coming out ahead (vs. not falling behind) involves sophisticated strategic reasoning that engages both reward and value computation areas and areas associated with theory of mind.

Belief formation - A driving force for brain evolution

The topic of belief has been neglected in the natural sciences for a long period of time. Recent neuroscience research in non-human primates and humans, however, has shown that beliefs are the neuropsychic product of fundamental brain processes that attribute affective meaning to concrete objects and events, enabling individual goal setting, decision making and maneuvering in the environment. With regard to the involved neural processes they can be categorized as empirical, relational, and conceptual beliefs. Empirical beliefs are about objects and relational beliefs are about events as in tool use and in interactions between subjects that develop below the level of awareness and are up-dated dynamically. Conceptual beliefs are more complex being based on narratives and participation in ritual acts. As neural processes are known to require computational space in the brain, the formation of inceasingly complex beliefs demands extra neural resources. Here, we argue that the evolution of human beliefs is related to the phylogenetic enlargement of the brain including the parietal and medial frontal cortex in humans.

Decreased transfer of value to action in Tourette syndrome

OBJECTIVE

Tourette syndrome is a neurodevelopmental disorder putatively associated with a hyperdopaminergic state. Therefore, it seems plausible that excessive dopamine transmission in Tourette syndrome alters the ability to learn based on rewards and punishments. We tested whether Tourette syndrome patients exhibited altered reinforcement learning and corresponding feedback-related EEG deflections.

METHODS

We used a reinforcement learning task providing factual and counterfactual feedback in a sample of 15 Tourette syndrome patients and matched healthy controls whilst recording EEG. The paradigm presented various reward probabilities to enforce adaptive adjustments. We employed a computational model to derive estimates of the prediction error, which we used for single-trial regression analysis of the EEG data.

RESULTS

We found that Tourette syndrome patients showed increased choice stochasticity compared to controls. The feedback-related negativity represented an axiomatic prediction error for factual feedback and did not differ between groups. We observed attenuated P3a modulation specifically for factual feedback in Tourette syndrome patients, representing impaired coding of attention allocation.

CONCLUSION

Our findings indicate that cortical prediction error coding is unaffected by Tourette syndrome. Nonetheless, the transfer of learned values into choice formation is degraded, in line with a hyperdopaminergic state.

Monetary feedback modulates performance and electrophysiological indices of belief updating in reward learning

Belief updating entails the incorporation of new information about the environment into internal models of the world. Bayesian inference is the statistically optimal strategy for performing belief updating in the presence of uncertainty. An important open question is whether the use of cognitive strategies that implement Bayesian inference is dependent upon motivational state and, if so, how this is reflected in electrophysiological signatures of belief updating in the brain. Here, we recorded the EEG of participants performing a simple reward learning task with both monetary and nonmonetary instructive feedback conditions. Our aim was to distinguish the influence of the rewarding properties of feedback on belief updating from the information content of the feedback itself. A Bayesian updating model allowed us to quantify different aspects of belief updating across trials, including the size of belief updates and the uncertainty of beliefs. Faster learning rates were observed in the monetary feedback condition compared to the instructive feedback condition, while belief updates were generally larger, and belief uncertainty smaller, with monetary compared to instructive feedback. Larger amplitudes in the monetary feedback condition were found for three ERP components: the P3a, the feedback-related negativity, and the late positive potential. These findings suggest that motivational state influences inference strategies in reward learning, and this is reflected in the electrophysiological correlates of belief updating.