Neuronal coding of prediction errors

Frontal Norepinephrine Represents a Threat Prediction Error Under Uncertainty

BACKGROUND

To adapt to threats in the environment, animals must predict them and engage in defensive behavior. While the representation of a prediction error signal for reward has been linked to dopamine, a neuromodulatory prediction error for aversive learning has not been identified.

METHODS

We measured and manipulated norepinephrine release during threat learning using optogenetics and a novel fluorescent norepinephrine sensor.

RESULTS

We found that norepinephrine response to conditioned stimuli reflects aversive memory strength. When delays between auditory stimuli and footshock are introduced, norepinephrine acts as a prediction error signal. However, temporal difference prediction errors do not fully explain norepinephrine dynamics. To explain noradrenergic signaling, we used an updated reinforcement learning model with uncertainty about time and found that it explained norepinephrine dynamics across learning and variations in temporal and auditory task structure.

CONCLUSIONS

Norepinephrine thus combines cognitive and affective information into a predictive signal and links time with the anticipation of danger.

"Blocking-like" effects in attentional set-shifting: Redundant cues facilitate shifting in male rats with medial prefrontal cortex inactivation

Without a functioning prefrontal cortex, humans and other animals are impaired in measures of cognitive control and behavioral flexibility, including attentional set-shifting. However, the reason for this is unclear with evidence suggesting both impaired and enhanced attentional shifting. We inhibited the medial prefrontal cortex (mPFC) of rats while they performed a modified version of an attentional set-shifting task to explore the nature of this apparent contradiction. Twelve adult male Lister hooded rats received AAV5-CaMKIIa-hM4D(Gi)-mCherry viral vector bilaterally into mPFC to express inhibitory 'Designer Receptors Exclusively Activated by Designer Drugs' (iDREADDs). The receptors were activated by systemic clozapine N-oxide (CNO) to inhibit mPFC function. The rats were tested in the standard attentional set-shifting task four times: twice after i.p. administration and twice after oral administration of vehicle or CNO (10 mg/kg). They were then tested twice in a modified task, with or without oral CNO. The modified task had an extra stage before the extradimensional shift, in which the relevant exemplars remained relevant and new exemplars that were fully predictive but redundant replaced the previous irrelevant exemplars. These exemplars then became relevant at the subsequent ED stage. In the standard task, mPFC inactivation impaired attentional set-shifting, consistent with previous findings. However, in the modified task, mPFC inactivation abolished ED shift-costs. The results support the suggestion that the mPFC is needed for the downregulation of attention that prevents learning about redundant and irrelevant stimuli. With mPFC inactivated, the rat learns more rapidly when previously redundant exemplars become the only relevant information.

Dopamine Release Dynamics in the Nucleus Accumbens Are Modulated by the Timing of Electrical Stimulation Pulses When Applied to the Medial Forebrain Bundle and Medial Prefrontal Cortex

Electrical brain stimulation has been used in vivo and in vitro to investigate neural circuitry. Historically, stimulation parameters such as amplitude, frequency, and pulse width were varied to investigate their effects on neurotransmitter release and behavior. These experiments have traditionally employed fixed-frequency stimulation patterns, but it has previously been found that neurons are more precisely tuned to variable input. Introducing variability into the interpulse interval of stimulation pulses will inform on how dopaminergic release can be modulated by variability in pulse timing. Here, dopaminergic release in rats is monitored in the nucleus accumbens (NAc), a key dopaminergic center which plays a role in learning and motivation, by fast-scan cyclic voltammetry. Dopaminergic release in the NAc could also be modulated by stimulation region due to differences in connectivity. We targeted two regions for stimulation─the medial forebrain bundle (MFB) and the medial prefrontal cortex (mPFC)─due to their involvement in reward processing and projections to the NAc. Our goal is to investigate how variable interpulse interval stimulation patterns delivered to these regions affect the time course of dopamine release in the NAc. We found that stimulating the MFB with these variable stimulation patterns saw a highly responsive, frequency-driven dopaminergic response. In contrast, variable stimulation patterns applied to the mPFC were not as sensitive to the variable frequency changes. This work will help inform on how stimulation patterns can be tuned specifically to the stimulation region to improve the efficiency of electrical stimulation and control dopamine release.

The homogenous hippocampus: How hippocampal cells process available and potential goals

We present here a view of the firing patterns of hippocampal cells that is contrary, both functionally and anatomically, to conventional wisdom. We argue that the hippocampus responds to efference copies of goals encoded elsewhere; and that it uses these to detect and resolve conflict or interference between goals in general. While goals can involve space, hippocampal cells do not encode spatial (or other special types of) memory, as such. We also argue that the transverse circuits of the hippocampus operate in an essentially homogeneous way along its length. The apparently different functions of different parts (e.g. memory retrieval versus anxiety) result from the different (situational/motivational) inputs on which those parts perform the same fundamental computational operations. On this view, the key role of the hippocampus is the iterative adjustment, via Papez-like circuits, of synaptic weights in cell assemblies elsewhere.

Error-based Implicit Learning in Language: The Effect of Sentence Context and Constraint in a Repetition Paradigm

Prediction errors drive implicit learning in language, but the specific mechanisms underlying these effects remain debated. This issue was addressed in an EEG study manipulating the context of a repeated unpredictable word (repetition of the complete sentence or repetition of the word in a new sentence context) and sentence constraint. For the manipulation of sentence constraint, unexpected words were presented either in high-constraint (eliciting a precise prediction) or low-constraint sentences (not eliciting any specific prediction). Repetition-induced reduction of N400 amplitudes and of power in the alpha/beta frequency band was larger for words repeated with their sentence context as compared with words repeated in a new low-constraint context, suggesting that implicit learning happens not only at the level of individual items but additionally improves sentence-based predictions. These processing benefits for repeated sentences did not differ between constraint conditions, suggesting that sentence-based prediction update might be proportional to the amount of unpredicted semantic information, rather than to the precision of the prediction that was violated. In addition, the consequences of high-constraint prediction violations, as reflected in a frontal positivity and increased theta band power, were reduced with repetition. Overall, our findings suggest a powerful and specific adaptation mechanism that allows the language system to quickly adapt its predictions when unexpected semantic information is processed, irrespective of sentence constraint, and to reduce potential costs of strong predictions that were violated.

What Role Does Striatal Dopamine Play in Goal-directed Action?

Evidence suggests that dopamine activity provides a US-related prediction error for Pavlovian conditioning and the reinforcement signal supporting the acquisition of habits. However, its role in goal-directed action is less clear. There are currently few studies that have assessed dopamine release as animals acquire and perform self-paced instrumental actions. Here we briefly review the literature documenting the psychological, behavioral and neural bases of goal-directed actions in rats and mice, before turning to describe recent studies investigating the role of dopamine in instrumental learning and performance. Plasticity in dorsomedial striatum, a central node in the network supporting goal-directed action, clearly requires dopamine release, the timing of which, relative to cortical and thalamic inputs, determines the degree and form of that plasticity. Beyond this, bilateral release appears to reflect reward prediction errors as animals experience the consequences of an action. Such signals feedforward to update the value of the specific action associated with that outcome during subsequent performance, with dopamine release at the time of action reflecting the updated predicted action value. More recently, evidence has also emerged for a hemispherically lateralised signal associated with the action; dopamine release is greater in the hemisphere contralateral to the spatial target of the action. This effect emerges over the course of acquisition and appears to reflect the strength of the action-outcome association. Thus, during goal-directed action, dopamine release signals the action, the outcome and their association to shape the learning and performance processes necessary to support this form of behavioral control.

An EEG-Based Neuroplastic Approach to Predictive Coding in People With Schizophrenia or Traumatic Brain Injury

Despite different etiologies, people with schizophrenia (SCZ) or with traumatic brain injury (TBI) both show aberrant neuroplasticity. One neuroplastic mechanism that may be affected is prediction error coding. We used a roving mismatch negativity (rMMN) paradigm which uses different lengths of standard tone trains and is optimized to assess predictive coding. Twenty-five SCZ, 22 TBI (mild to moderate), and 25 healthy controls were assessed. We used a frequency-deviant rMMN in which the number of standards preceding the deviant was either 2, 6, or 36. We evaluated repetition positivity to the standard tone immediately preceding a deviant tone (repetition positivity [RP], to assess formation of the memory trace), deviant negativity to the deviant stimulus (deviant negativity [DN], which reflects signaling of a prediction error), and the difference wave between the 2 (the MMN). We found that SCZ showed reduced DN and MMN compared with healthy controls and with people with mild to moderate TBI. We did not detect impairments in any index (RP, DN, or MMN) in people with TBI compared to controls. Our findings suggest that prediction error coding assessed with rMMN is aberrant in SCZ but intact in TBI, though there is a suggestion that severity of head injury results in poorer prediction error coding.

Eye movements reflect active statistical learning

What is the link between eye movements and sensory learning? Although some theories have argued for an automatic interaction between what we know and where we look that continuously modulates human information gathering behavior during both implicit and explicit learning, there exists limited experimental evidence supporting such an ongoing interplay. To address this issue, we used a visual statistical learning paradigm combined with a gaze-contingent stimulus presentation and manipulated the explicitness of the task to explore how learning and eye movements interact. During both implicit exploration and explicit visual learning of unknown composite visual scenes, spatial eye movement patterns systematically and gradually changed in accordance with the underlying statistical structure of the scenes. Moreover, the degree of change was directly correlated with the amount and type of knowledge the observers acquired. This suggests that eye movements are potential indicators of active learning, a process where long-term knowledge, current visual stimuli and an inherent tendency to reduce uncertainty about the visual environment jointly determine where we look.

Hierarchical control over foraging behavior by anterior cingulate cortex

Foraging is a natural behavior that involves making sequential decisions to maximize rewards while minimizing the costs incurred when doing so. The prevalence of foraging across species suggests that a common brain computation underlies its implementation. Although anterior cingulate cortex is believed to contribute to foraging behavior, its specific role has been contentious, with predominant theories arguing either that it encodes environmental value or choice difficulty. Additionally, recent attempts to characterize foraging have taken place within the reinforcement learning framework, with increasingly complex models scaling with task complexity. Here we review reinforcement learning foraging models, highlighting the hierarchical structure of many foraging problems. We extend this literature by proposing that ACC guides foraging according to principles of model-based hierarchical reinforcement learning. This idea holds that ACC function is organized hierarchically along a rostral-caudal gradient, with rostral structures monitoring the status and completion of high-level task goals (like finding food), and midcingulate structures overseeing the execution of task options (subgoals, like harvesting fruit) and lower-level actions (such as grabbing an apple).

Prediction-error-dependent processing of immediate and delayed positive feedback

Learning often involves trial-and-error, i.e. repeating behaviours that lead to desired outcomes, and adjusting behaviour when outcomes do not meet our expectations and thus lead to prediction errors (PEs). PEs have been shown to be reflected in the reward positivity (RewP), an event-related potential (ERP) component between 200 and 350 ms after performance feedback which is linked to striatal processing and assessed via electroencephalography (EEG). Here we show that this is also true for delayed feedback processing, for which a critical role of the hippocampus has been suggested. We found a general reduction of the RewP for delayed feedback, but the PE was similarly reflected in the RewP and the later P300 for immediate and delayed positive feedback, while no effect was found for negative feedback. Our results suggest that, despite processing differences between immediate and delayed feedback, positive PEs drive feedback processing and learning irrespective of delay.

The lateral habenula: A hub for value-guided behavior

The habenula is an evolutionarily highly conserved diencephalic brain region divided into two major parts, medial and lateral. Over the past two decades, studies of the lateral habenula (LHb), in particular, have identified key functions in value-guided behavior in health and disease. In this review, we focus on recent insights into LHb connectivity and its functional relevance for different types of aversive and appetitive value-guided behavior. First, we give an overview of the anatomical organization of the LHb and its main cellular composition. Next, we elaborate on how distinct LHb neuronal subpopulations encode aversive and appetitive stimuli and on their involvement in more complex decision-making processes. Finally, we scrutinize the afferent and efferent connections of the LHb and discuss their functional implications for LHb-dependent behavior. A deepened understanding of distinct LHb circuit components will substantially contribute to our knowledge of value-guided behavior.

Contemporary neurocognitive models of memory: A descriptive comparative analysis

The great complexity involved in the study of memory has given rise to numerous hypotheses and models associated with various phenomena at different levels of analysis. This has allowed us to delve deeper in our knowledge about memory but has also made it difficult to synthesize and integrate data from different lines of research. In this context, this work presents a descriptive comparative analysis of contemporary models that address the structure and function of multiple memory systems. The main goal is to outline a panoramic view of the key elements that constitute these models in order to visualize both the current state of research and possible future directions. The elements that stand out from different levels of analysis are distributed neural networks, hierarchical organization, predictive coding, homeostasis, and evolutionary perspective.

Review of EEG Affective Recognition with a Neuroscience Perspective

Emotions are a series of subconscious, fleeting, and sometimes elusive manifestations of the human innate system. They play crucial roles in everyday life-influencing the way we evaluate ourselves, our surroundings, and how we interact with our world. To date, there has been an abundance of research on the domains of neuroscience and affective computing, with experimental evidence and neural network models, respectively, to elucidate the neural circuitry involved in and neural correlates for emotion recognition. Recent advances in affective computing neural network models often relate closely to evidence and perspectives gathered from neuroscience to explain the models. Specifically, there has been growing interest in the area of EEG-based emotion recognition to adopt models based on the neural underpinnings of the processing, generation, and subsequent collection of EEG data. In this respect, our review focuses on providing neuroscientific evidence and perspectives to discuss how emotions potentially come forth as the product of neural activities occurring at the level of subcortical structures within the brain's emotional circuitry and the association with current affective computing models in recognizing emotions. Furthermore, we discuss whether such biologically inspired modeling is the solution to advance the field in EEG-based emotion recognition and beyond.

Unraveling individual differences in learning potential: A dynamic framework for the case of reading development

Children show an enormous capacity to learn during development, but with large individual differences in the time course and trajectory of learning and the achieved skill level. Recent progress in developmental sciences has shown the contribution of a multitude of factors including genetic variation, brain plasticity, socio-cultural context and learning experiences to individual development. These factors interact in a complex manner, producing children's idiosyncratic and heterogeneous learning paths. Despite an increasing recognition of these intricate dynamics, current research on the development of culturally acquired skills such as reading still has a typical focus on snapshots of children's performance at discrete points in time. Here we argue that this 'static' approach is often insufficient and limits advancements in the prediction and mechanistic understanding of individual differences in learning capacity. We present a dynamic framework which highlights the importance of capturing short-term trajectories during learning across multiple stages and processes as a proxy for long-term development on the example of reading. This framework will help explain relevant variability in children's learning paths and outcomes and fosters new perspectives and approaches to study how children develop and learn.

Striatal dopamine release tracks the relationship between actions and their consequences

The acquisition and performance of goal-directed actions has long been argued to depend on the integration of glutamatergic inputs to the posterior dorsomedial striatum (pDMS) under the modulatory influence of dopamine. Nevertheless, relatively little is known about the dynamics of striatal dopamine during goal-directed actions. To investigate this, we chronically recorded dopamine release in the pDMS as rats acquired two actions for distinct outcomes as these action-outcome associations were incremented and then subsequently degraded or reversed. We found that bilateral dopamine release scaled with action value, whereas the lateralized dopamine signal, i.e., the difference in dopamine release ipsilaterally and contralaterally to the direction of the goal-directed action, reflected the strength of the action-outcome association independently of changes in movement. Our results establish, therefore, that striatal dopamine activity during goal-directed action reflects both bilateral moment-to-moment changes in action value and the long-term action-outcome association.

Observational reinforcement learning in children and young adults

Observational learning is essential for the acquisition of new behavior in educational practices and daily life and serves as an important mechanism for human cognitive and social-emotional development. However, we know little about its underlying neurocomputational mechanisms from a developmental perspective. In this study we used model-based fMRI to investigate differences in observational learning and individual learning between children and younger adults. Prediction errors (PE), the difference between experienced and predicted outcomes, related positively to striatal and ventral medial prefrontal cortex activation during individual learning and showed no age-related differences. PE-related activation during observational learning was more pronounced when outcomes were worse than predicted. Particularly, negative PE-coding in the dorsal medial prefrontal cortex was stronger in adults compared to children and was associated with improved observational learning in children and adults. The current findings pave the way to better understand observational learning challenges across development and educational settings.

Modeling circuit mechanisms of opposing cortical responses to visual flow perturbations

In an ever-changing visual world, animals' survival depends on their ability to perceive and respond to rapidly changing motion cues. The primary visual cortex (V1) is at the forefront of this sensory processing, orchestrating neural responses to perturbations in visual flow. However, the underlying neural mechanisms that lead to distinct cortical responses to such perturbations remain enigmatic. In this study, our objective was to uncover the neural dynamics that govern V1 neurons' responses to visual flow perturbations using a biologically realistic computational model. By subjecting the model to sudden changes in visual input, we observed opposing cortical responses in excitatory layer 2/3 (L2/3) neurons, namely, depolarizing and hyperpolarizing responses. We found that this segregation was primarily driven by the competition between external visual input and recurrent inhibition, particularly within L2/3 and L4. This division was not observed in excitatory L5/6 neurons, suggesting a more prominent role for inhibitory mechanisms in the visual processing of the upper cortical layers. Our findings share similarities with recent experimental studies focusing on the opposing influence of top-down and bottom-up inputs in the mouse primary visual cortex during visual flow perturbations.

The theta-gamma code in predictive processing and mnemonic updating

Predictive processing has become a leading theory about how the brain works. Yet, it remains an open question how predictive processes are realized in the brain. Here I discuss theta-gamma coupling as one potential neural mechanism for prediction and model updating. Building on Lisman and colleagues SOCRATIC model, theta-gamma coupling has been associated with phase precession and learning phenomena in medio-temporal lobe of rodents, where it completes and retains a sequence of places or items (i.e., predictive models). These sequences may be updated upon prediction errors (i.e., model updating), signaled by dopaminergic inputs from prefrontal networks. This framework, spanning the molecular to the network level, matches excitingly well with recent findings on predictive processing, mnemonic updating, and perceptual foraging for the theta-gamma code in human cognition. In sum, I use the case of theta-gamma coupling to link the predictive processing account, a very general concept of how the brain works, to specific neural processes which may implement predictive processing and model updating at the cognitive, network, cellular and molecular level.

Tailoring Classical Conditioning Behavior in TiO2 Nanowires: ZnO QDs-Based Optoelectronic Memristors for Neuromorphic Hardware

Neuromorphic hardware equipped with associative learning capabilities presents fascinating applications in the next generation of artificial intelligence. However, research into synaptic devices exhibiting complex associative learning behaviors is still nascent. Here, an optoelectronic memristor based on Ag/TiO2 Nanowires: ZnO Quantum dots/FTO was proposed and constructed to emulate the biological associative learning behaviors. Effective implementation of synaptic behaviors, including long and short-term plasticity, and learning-forgetting-relearning behaviors, were achieved in the device through the application of light and electrical stimuli. Leveraging the optoelectronic co-modulated characteristics, a simulation of neuromorphic computing was conducted, resulting in a handwriting digit recognition accuracy of 88.9%. Furthermore, a 3 × 7 memristor array was constructed, confirming its application in artificial visual memory. Most importantly, complex biological associative learning behaviors were emulated by mapping the light and electrical stimuli into conditioned and unconditioned stimuli, respectively. After training through associative pairs, reflexes could be triggered solely using light stimuli. Comprehensively, under specific optoelectronic signal applications, the four features of classical conditioning, namely acquisition, extinction, recovery, and generalization, were elegantly emulated. This work provides an optoelectronic memristor with associative behavior capabilities, offering a pathway for advancing brain-machine interfaces, autonomous robots, and machine self-learning in the future.

Evaluating the Influence of Musical and Monetary Rewards on Decision Making through Computational Modelling

A central question in behavioural neuroscience is how different rewards modulate learning. While the role of monetary rewards is well-studied in decision-making research, the influence of abstract rewards like music remains poorly understood. This study investigated the dissociable effects of these two reward types on decision making. Forty participants completed two decision-making tasks, each characterised by probabilistic associations between stimuli and rewards, with probabilities changing over time to reflect environmental volatility. In each task, choices were reinforced either by monetary outcomes (win/lose) or by the endings of musical melodies (consonant/dissonant). We applied the Hierarchical Gaussian Filter, a validated hierarchical Bayesian framework, to model learning under these two conditions. Bayesian statistics provided evidence for similar learning patterns across both reward types, suggesting individuals' similar adaptability. However, within the musical task, individual preferences for consonance over dissonance explained some aspects of learning. Specifically, correlation analyses indicated that participants more tolerant of dissonance behaved more stochastically in their belief-to-response mappings and were less likely to choose the response associated with the current prediction for a consonant ending, driven by higher volatility estimates. By contrast, participants averse to dissonance showed increased tonic volatility, leading to larger updates in reward tendency beliefs.

Behavior-relevant top-down cross-modal predictions in mouse neocortex

Animals adapt to a constantly changing world by predicting their environment and the consequences of their actions. The predictive coding hypothesis proposes that the brain generates predictions and continuously compares them with sensory inputs to guide behavior. However, how the brain reconciles conflicting top-down predictions and bottom-up sensory information remains unclear. To address this question, we simultaneously imaged neuronal populations in the mouse somatosensory barrel cortex and posterior parietal cortex during an auditory-cued texture discrimination task. In mice that had learned the task with fixed tone-texture matching, the presentation of mismatched pairing induced conflicts between tone-based texture predictions and actual texture inputs. When decisions were based on the predicted rather than the actual texture, top-down information flow was dominant and texture representations in both areas were modified, whereas dominant bottom-up information flow led to correct representations and behavioral choice. Our findings provide evidence for hierarchical predictive coding in the mouse neocortex.

State and rate-of-change encoding in parallel mesoaccumbal dopamine pathways

The nervous system uses fast- and slow-adapting sensory detectors in parallel to enable neuronal representations of external states and their temporal dynamics. It is unknown whether this dichotomy also applies to internal representations that have no direct correlation in the physical world. Here we find that two distinct dopamine (DA) neuron subtypes encode either a state or its rate-of-change. In mice performing a reward-seeking task, we found that the animal's behavioral state and rate-of-change were encoded by the sustained activity of DA neurons in medial ventral tegmental area (VTA) DA neurons and transient activity in lateral VTA DA neurons, respectively. The neural activity patterns of VTA DA cell bodies matched DA release patterns within anatomically defined mesoaccumbal pathways. Based on these results, we propose a model in which the DA system uses two parallel lines for proportional-differential encoding of a state variable and its temporal dynamics.

Resource-rational account of sequential effects in human prediction

An abundant literature reports on 'sequential effects' observed when humans make predictions on the basis of stochastic sequences of stimuli. Such sequential effects represent departures from an optimal, Bayesian process. A prominent explanation posits that humans are adapted to changing environments, and erroneously assume non-stationarity of the environment, even if the latter is static. As a result, their predictions fluctuate over time. We propose a different explanation in which sub-optimal and fluctuating predictions result from cognitive constraints (or costs), under which humans however behave rationally. We devise a framework of costly inference, in which we develop two classes of models that differ by the nature of the constraints at play: in one case the precision of beliefs comes at a cost, resulting in an exponential forgetting of past observations, while in the other beliefs with high predictive power are favored. To compare model predictions to human behavior, we carry out a prediction task that uses binary random stimuli, with probabilities ranging from 0.05 to 0.95. Although in this task the environment is static and the Bayesian belief converges, subjects' predictions fluctuate and are biased toward the recent stimulus history. Both classes of models capture this 'attractive effect', but they depart in their characterization of higher-order effects. Only the precision-cost model reproduces a 'repulsive effect', observed in the data, in which predictions are biased away from stimuli presented in more distant trials. Our experimental results reveal systematic modulations in sequential effects, which our theoretical approach accounts for in terms of rationality under cognitive constraints.

Neural correlates of prediction error in patients with schizophrenia: evidence from an fMRI meta-analysis

Abnormal processes of learning from prediction errors, i.e. the discrepancies between expectations and outcomes, are thought to underlie motivational impairments in schizophrenia. Although dopaminergic abnormalities in the mesocorticolimbic reward circuit have been found in patients with schizophrenia, the pathway through which prediction error signals are processed in schizophrenia has yet to be elucidated. To determine the neural correlates of prediction error processing in schizophrenia, we conducted a meta-analysis of whole-brain neuroimaging studies that investigated prediction error signal processing in schizophrenia patients and healthy controls. A total of 14 studies (324 schizophrenia patients and 348 healthy controls) using the reinforcement learning paradigm were included. Our meta-analysis showed that, relative to healthy controls, schizophrenia patients showed increased activity in the precentral gyrus and middle frontal gyrus and reduced activity in the mesolimbic circuit, including the striatum, thalamus, amygdala, hippocampus, anterior cingulate cortex, insula, superior temporal gyrus, and cerebellum, when processing prediction errors. We also found hyperactivity in frontal areas and hypoactivity in mesolimbic areas when encoding prediction error signals in schizophrenia patients, potentially indicating abnormal dopamine signaling of reward prediction error and suggesting failure to represent the value of alternative responses during prediction error learning and decision making.

Exploring the steps of learning: computational modeling of initiatory-actions among individuals with attention-deficit/hyperactivity disorder

Attention-deficit/hyperactivity disorder (ADHD) is characterized by difficulty in acting in a goal-directed manner. While most environments require a sequence of actions for goal attainment, ADHD was never studied in the context of value-based sequence learning. Here, we made use of current advancements in hierarchical reinforcement-learning algorithms to track the internal value and choice policy of individuals with ADHD performing a three-stage sequence learning task. Specifically, 54 participants (28 ADHD, 26 controls) completed a value-based reinforcement-learning task that allowed us to estimate internal action values for each trial and stage using computational modeling. We found attenuated sensitivity to action values in ADHD compared to controls, both in choice and reaction-time variability estimates. Remarkably, this was found only for first-stage actions (i.e., initiatory actions), while for actions performed just before outcome delivery the two groups were strikingly indistinguishable. These results suggest a difficulty in following value estimation for initiatory actions in ADHD.

Human subcortical pathways automatically detect collision trajectory without attention and awareness

Detecting imminent collisions is essential for survival. Here, we used high-resolution fMRI at 7 Tesla to investigate the role of attention and consciousness for detecting collision trajectory in human subcortical pathways. Healthy participants can precisely discriminate collision from near-miss trajectory of an approaching object, with pupil size change reflecting collision sensitivity. Subcortical pathways from the superior colliculus (SC) to the ventromedial pulvinar (vmPul) and ventral tegmental area (VTA) exhibited collision-sensitive responses even when participants were not paying attention to the looming stimuli. For hemianopic patients with unilateral lesions of the geniculostriate pathway, the ipsilesional SC and VTA showed significant activation to collision stimuli in their scotoma. Furthermore, stronger SC responses predicted better behavioral performance in collision detection even in the absence of awareness. Therefore, human tectofugal pathways could automatically detect collision trajectories without the observers' attention to and awareness of looming stimuli, supporting "blindsight" detection of impending visual threats.

BGRL: Basal Ganglia inspired Reinforcement Learning based framework for deep brain stimulators

Deep Brain Stimulation (DBS) is an implantable medical device used for electrical stimulation to treat neurological disorders. Traditional DBS devices provide fixed frequency pulses, but personalized adjustment of stimulation parameters is crucial for optimal treatment. This paper introduces a Basal Ganglia inspired Reinforcement Learning (BGRL) approach, incorporating a closed-loop feedback mechanism to suppress neural synchrony during neurological fluctuations. The BGRL approach leverages the resemblance between the Basal Ganglia region of brain by incorporating the actor-critic architecture of reinforcement learning (RL). Simulation results demonstrate that BGRL significantly reduces synchronous electrical pulses compared to other standard RL algorithms. BGRL algorithm outperforms existing RL methods in terms of suppression capability and energy consumption, validated through comparisons using ensemble oscillators. Results shown in the paper demonstrate BGRL suppressed the synchronous electrical pulses across three signaling regimes namely regular, chaotic and bursting by 40%, 146% and 40% respectively as compared to soft actor-critic model. BGRL shows promise in effectively suppressing neural synchrony in DBS therapy, providing an efficient alternative to open-loop methodologies.

Functional architecture of dopamine neurons driving fear extinction learning

The ability to extinguish fear responses to stimuli that no longer predict danger is critical for adaptive behavior and increases the likelihood of survival. During fear extinction, dopamine (DA) neurons signal the absence of the expected aversive outcome, and this extinction prediction error (EPE) signal is crucial for initiating and driving extinction learning. However, the neural circuits underlying the EPE signal have remained elusive. Here, we investigate the input-output circuitry of EPE-encoding DA neurons in male mice. By employing projection-specific fiber photometry and optogenetics, we demonstrate that these neurons project to a restricted subregion of the nucleus accumbens. Comprehensive anatomical analyses, as well as projection-specific chemogenetic manipulations combined with recordings of DA biosensors, further uncover the dorsal raphe as one key input structure critical for generating the EPE signal. Together, our results reveal for the first time the functional architecture of EPE-encoding DA neurons crucial for driving fear extinction learning.

Contexts facilitate dynamic value encoding in the mesolimbic dopamine system

Adaptive behavior in a dynamic environment often requires rapid revaluation of stimuli that deviates from well-learned associations. The divergence between stable value-encoding and appropriate behavioral output remains a critical test to theories of dopamine's function in learning, motivation, and motor control. Yet how dopamine neurons are involved in the revaluation of cues when the world changes to alter our behavior remains unclear. Here we make use of pharmacology, in vivo electrophysiology, fiber photometry, and optogenetics to resolve the contributions of the mesolimbic dopamine system to the dynamic reorganization of reward-seeking. Male and female rats were trained to discriminate when a conditioned stimulus would be followed by sucrose reward by exploiting the prior, non-overlapping presentation of a separate discrete cue - an occasion setter. Only when the occasion setter's presentation preceded the conditioned stimulus did the conditioned stimulus predict sucrose delivery. As a result, in this task we were able to dissociate the average value of the conditioned stimulus from its immediate expected value on a trial-to-trial basis. Both the activity of ventral tegmental area dopamine neurons and dopamine signaling in the nucleus accumbens were essential for rats to successfully update behavioral responding in response to the occasion setter. Moreover, dopamine release in the nucleus accumbens following the conditioned stimulus only occurred when the occasion setter indicated it would predict reward. Downstream of dopamine release, we found that single neurons in the nucleus accumbens dynamically tracked the value of the conditioned stimulus. Together these results reveal a novel mechanism within the mesolimbic dopamine system for the rapid revaluation of motivation.

Lateral preoptic area glutamate neurons relay nociceptive information to the ventral tegmental area

The ventral tegmental area (VTA) has been proposed to play a role in pain, but the brain structures modulating VTA activity in response to nociceptive stimuli remain unclear. Here, we demonstrate that the lateral preoptic area (LPO) glutamate neurons relay nociceptive information to the VTA. These LPO glutamatergic neurons synapsing on VTA neurons respond to nociceptive stimulation and conditioned stimuli predicting nociceptive stimulation and also mediate aversion. In contrast, LPO GABA neurons synapsing in the VTA mediate reward. By ultrastructural quantitative synaptic analysis, ex vivo electrophysiology, and functional neuroanatomy we identify a complex circuitry between LPO glutamatergic and GABAergic neurons and VTA dopaminergic, GABAergic, and glutamatergic neurons. We conclude that LPO glutamatergic neurons play a causal role in the processing of nociceptive stimuli and in relaying information about nociceptive stimuli. The pathway from LPO glutamatergic neurons to the VTA represents an unpredicted interface between peripheral nociceptive information and the limbic system.

The Basolateral Amygdala: The Core of a Network for Threat Conditioning, Extinction, and Second-Order Threat Conditioning

Threat conditioning, extinction, and second-order threat conditioning studied in animal models provide insight into the brain-based mechanisms of fear- and anxiety-related disorders and their treatment. Much attention has been paid to the role of the basolateral amygdala (BLA) in such processes, an overview of which is presented in this review. More recent evidence suggests that the BLA serves as the core of a greater network of structures in these forms of learning, including associative and sensory cortices. The BLA is importantly regulated by hippocampal and prefrontal inputs, as well as by the catecholaminergic neuromodulators, norepinephrine and dopamine, that may provide important prediction-error or learning signals for these forms of learning. The sensory cortices may be required for the long-term storage of threat memories. As such, future research may further investigate the potential of the sensory cortices for the long-term storage of extinction and second-order conditioning memories.

The Rescorla-Wagner model, prediction error, and fear learning

The Rescorla-Wagner model remains one of the most important and influential theoretical accounts of the conditions under which Pavlovian learning occurs. Moreover, the experimental approaches that inspired the model continue to provide powerful behavioral tools to advance mechanistic understanding of how we and other animals learn to fear and learn to reduce fear. Here we consider key features of the Rescorla-Wagner model as applied to study of fear learning. We review evidence for key insights of the model. First, learning to fear and learning to reduce fear are governed by a common, signed prediction error. Second, this error drives variations in effectiveness of the shock US that are causal to whether and how much fear is learned or lost during a conditioning trial. We also consider behavioral and neural findings inconsistent with the model and which will be essential to understand and advance understanding of fear learning.

Reward prediction error in learning-related behaviors

Learning is a complex process, during which our opinions and decisions are easily changed due to unexpected information. But the neural mechanism underlying revision and correction during the learning process remains unclear. For decades, prediction error has been regarded as the core of changes to perception in learning, even driving the learning progress. In this article, we reviewed the concept of reward prediction error, and the encoding mechanism of dopaminergic neurons and the related neural circuities. We also discussed the relationship between reward prediction error and learning-related behaviors, including reversal learning. We then demonstrated the evidence of reward prediction error signals in several neurological diseases, including Parkinson's disease and addiction. These observations may help to better understand the regulatory mechanism of reward prediction error in learning-related behaviors.

A cell-type-specific error-correction signal in the posterior parietal cortex

Neurons in the posterior parietal cortex contribute to the execution of goal-directed navigation1 and other decision-making tasks2-4. Although molecular studies have catalogued more than 50 cortical cell types5, it remains unclear what distinct functions they have in this area. Here we identified a molecularly defined subset of somatostatin (Sst) inhibitory neurons that, in the mouse posterior parietal cortex, carry a cell-type-specific error-correction signal for navigation. We obtained repeatable experimental access to these cells using an adeno-associated virus in which gene expression is driven by an enhancer that functions specifically in a subset of Sst cells6. We found that during goal-directed navigation in a virtual environment, this subset of Sst neurons activates in a synchronous pattern that is distinct from the activity of surrounding neurons, including other Sst neurons. Using in vivo two-photon photostimulation and ex vivo paired patch-clamp recordings, we show that nearby cells of this Sst subtype excite each other through gap junctions, revealing a self-excitation circuit motif that contributes to the synchronous activity of this cell type. These cells selectively activate as mice execute course corrections for deviations in their virtual heading during navigation towards a reward location, for both self-induced and experimentally induced deviations. We propose that this subtype of Sst neurons provides a self-reinforcing and cell-type-specific error-correction signal in the posterior parietal cortex that may help with the execution and learning of accurate goal-directed navigation trajectories.

The neurophysiological Behavioral Perspective Model of consumer choice and its contribution to the intentional behaviorist research programme

Cognitive explanations raise epistemological problems not faced by accounts confined to observable variables. Many explanatory components of cognitive models are unobservable: beliefs, attitudes, and intentions, for instance, must be made empirically available to the researcher in the form of measures of observable behavior from which the latent variables are inferred. The explanatory variables are abstract and theoretical and rely, if they are to enter investigations and explanations, on reasoned agreement on how they can be captured by proxy variables derived from what people say and how they behave. Psychometrics must be founded upon a firm, intersubjective agreement among researchers and users of research on the relationship of behavioral measures to the intentional constructs to which they point and the latent variables they seek to operationalize. Only if these considerations are adequately addressed can we arrive at consistent interpretations of the data. This problem provides the substance of the intentional behaviorist research programme which seeks to provide a rationale for the cognitive explanation. Within this programme, two versions of the Behavioral Perspective Model (BPM), an extensional portrayal of socioeconomic behavior and a corresponding intentional approach, address the task of identifying where intentional explanation becomes necessary and the form it should take. This study explores a third version, based on neurophysiological substrates of consumer choice as a contributor to this task. The nature of "value" is closely related to the rationale for a neurophysiological model of consumer choice. The variables involved are operationally specified and measured with high intersubjective agreement. The intentional model (BPM-I), depicting consumer action in terms of mental processes such as perception, deliberation, and choice, extends the purview of the BPM to new situations and areas of explanation.

Twisted memories: Addiction-related engrams are strengthened by desire thinking

Associative learning plays a central role in addiction by reinforcing associations between environmental cues and addiction-related information. Unsupervised learning models posit that memories are adjusted based on how strongly these representations are coactivated during the retrieval process. From a different perspective, clinical models of addiction posit that the escalation and persistence of craving may depend on desire thinking, a thinking style orienting to prefigure information about positive addiction-related experiences. In the present work, we tested the main hypothesis that desire thinking is a key factor in the strengthening of addiction-related associations. A group of adult smoking volunteers (N = 26) engaged in a period of desire thinking before performing an associative learning task in which neutral words (cues) were shown along with images (smoking-related vs. neutral context) at different frequencies. Two retrieval tests were administered, one immediately after encoding and the other after 24 h, to test how the recall of associations changed as a function of retention interval. Two control groups, smokers (N = 21) and non-smokers (N = 22), performed a similar procedure, with a neutral imagination task replacing desire thinking. Participants who engaged in desire thinking increased their performance from the first to the second retrieval test only for the most frequent smoking-related associations. Crucially, this selective effect was not observed in the two control groups. These results provide behavioral evidence in support of the idea that desire thinking plays a role in strengthening addiction-related associations. Thus, this thinking process may be considered a target for reconsolidation-based conceptualizations of, and treatments for, addiction.

How we learn social norms: a three-stage model for social norm learning

As social animals, humans are unique to make the world function well by developing, maintaining, and enforcing social norms. As a prerequisite among these norm-related processes, learning social norms can act as a basis that helps us quickly coordinate with others, which is beneficial to social inclusion when people enter into a new environment or experience certain sociocultural changes. Given the positive effects of learning social norms on social order and sociocultural adaptability in daily life, there is an urgent need to understand the underlying mechanisms of social norm learning. In this article, we review a set of works regarding social norms and highlight the specificity of social norm learning. We then propose an integrated model of social norm learning containing three stages, i.e., pre-learning, reinforcement learning, and internalization, map a potential brain network in processing social norm learning, and further discuss the potential influencing factors that modulate social norm learning. Finally, we outline a couple of future directions along this line, including theoretical (i.e., societal and individual differences in social norm learning), methodological (i.e., longitudinal research, experimental methods, neuroimaging studies), and practical issues.

A generic neural factor linking resting-state neural dynamics and the brain's response to unexpectedness in multilevel cognition

The brain's response to change is fundamental to learning and adaptation; this implies the presence of a universal neural mechanism under various contexts. We hypothesized that this mechanism manifests in neural activity patterns across low and high levels of cognition during task processing as well as in resting-state neural dynamics, because both these elements are different facets of the same dynamical system. We tested our hypothesis by (i) characterizing (a) the neural response to changes in low-level continuous information stream and unexpectedness at different cognitive levels and (b) the spontaneous neural dynamics in resting state, and (ii) examining the associations among the dynamics according to cross-individual variability (n = 200). Our results showed that the brain's response magnitude was monotonically correlated with the magnitude of information fluctuation in a low-level task, forming a simple psychophysical function; moreover, this effect was found to be associated with the brain's response to unexpectedness in high-level cognitive tasks (including language processing). These coherent multilevel neural effects in task processing were also shown to be strongly associated with resting-state neural dynamics characterized by the waxing and waning of Alpha oscillation. Taken together, our results revealed large-scale consistency between the neural dynamic system and multilevel cognition.

A multivariate brain signature for reward

The processing of reinforcers and punishers is crucial to adapt to an ever changing environment and its dysregulation is prevalent in mental health and substance use disorders. While many human brain measures related to reward have been based on activity in individual brain regions, recent studies indicate that many affective and motivational processes are encoded in distributed systems that span multiple regions. Consequently, decoding these processes using individual regions yields small effect sizes and limited reliability, whereas predictive models based on distributed patterns yield larger effect sizes and excellent reliability. To create such a predictive model for the processes of rewards and losses, termed the Brain Reward Signature (BRS), we trained a model to predict the signed magnitude of monetary rewards on the Monetary Incentive Delay task (MID; N = 39) and achieved a highly significant decoding performance (92% for decoding rewards versus losses). We subsequently demonstrate the generalizability of our signature on another version of the MID in a different sample (92% decoding accuracy; N = 12) and on a gambling task from a large sample (73% decoding accuracy, N = 1084). We further provided preliminary data to characterize the specificity of the signature by illustrating that the signature map generates estimates that significantly differ between rewarding and negative feedback (92% decoding accuracy) but do not differ for conditions that differ in disgust rather than reward in a novel Disgust-Delay Task (N = 39). Finally, we show that passively viewing positive and negatively valenced facial expressions loads positively on our signature, in line with previous studies on morbid curiosity. We thus created a BRS that can accurately predict brain responses to rewards and losses in active decision making tasks, and that possibly relates to information seeking in passive observational tasks.

Reward disturbances in antipsychotic-naïve patients with first-episode psychosis and their association to glutamate levels

BACKGROUND

Aberrant anticipation of motivational salient events and processing of outcome evaluation in striatal and prefrontal regions have been suggested to underlie psychosis. Altered glutamate levels have likewise been linked to schizophrenia. Glutamatergic abnormalities may affect the processing of motivational salience and outcome evaluation. It remains unresolved, whether glutamatergic dysfunction is associated with the coding of motivational salience and outcome evaluation in antipsychotic-naïve patients with first-episode psychosis.

METHODS

Fifty-one antipsychotic-naïve patients with first-episode psychosis (22 ± 5.2 years, female/male: 31/20) and 52 healthy controls (HC) matched on age, sex, and parental education underwent functional magnetic resonance imaging and magnetic resonance spectroscopy (3T) in one session. Brain responses to motivational salience and negative outcome evaluation (NOE) were examined using a monetary incentive delay task. Glutamate levels were estimated in the left thalamus and anterior cingulate cortex using LCModel.

RESULTS

Patients displayed a positive signal change to NOE in the caudate (p = 0.001) and dorsolateral prefrontal cortex (DLPFC; p = 0.003) compared to HC. No group difference was observed in motivational salience or in levels of glutamate. There was a different association between NOE signal in the caudate and DLPFC and thalamic glutamate levels in patients and HC due to a negative correlation in patients (caudate: p = 0.004, DLPFC: p = 0.005) that was not seen in HC.

CONCLUSIONS

Our findings confirm prior findings of abnormal outcome evaluation as a part of the pathophysiology of schizophrenia. The results also suggest a possible link between thalamic glutamate and NOE signaling in patients with first-episode psychosis.

Investigating Intra-Individual Networks of Response Inhibition and Interference Resolution using 7T MRI

Response inhibition and interference resolution are often considered subcomponents of an overarching inhibition system that utilizes the so-called cortico-basal-ganglia loop. Up until now, most previous functional magnetic resonance imaging (fMRI) literature has compared the two using between-subject designs, pooling data in the form of a meta-analysis or comparing different groups. Here, we investigate the overlap of activation patterns underlying response inhibition and interference resolution on a within-subject level, using ultra-high field MRI. In this model-based study, we furthered the functional analysis with cognitive modelling techniques to provide a more in-depth understanding of behaviour. We applied the stop-signal task and multi-source interference task to measure response inhibition and interference resolution, respectively. Our results lead us to conclude that these constructs are rooted in anatomically distinct brain areas and provide little evidence for spatial overlap. Across the two tasks, common BOLD responses were observed in the inferior frontal gyrus and anterior insula. Interference resolution relied more heavily on subcortical components, specifically nodes of the commonly referred to indirect and hyperdirect pathways, as well as the anterior cingulate cortex, and pre-supplementary motor area. Our data indicated that orbitofrontal cortex activation is specific to response inhibition. Our model-based approach provided evidence for the dissimilarity in behavioural dynamics between the two tasks. The current work exemplifies the importance of reducing inter-individual variance when comparing network patterns and the value of UHF-MRI for high resolution functional mapping.

Inhibition of noradrenergic signalling in rodent orbitofrontal cortex impairs the updating of goal-directed actions

In a constantly changing environment, organisms must track the current relationship between actions and their specific consequences and use this information to guide decision-making. Such goal-directed behaviour relies on circuits involving cortical and subcortical structures. Notably, a functional heterogeneity exists within the medial prefrontal, insular, and orbitofrontal cortices (OFC) in rodents. The role of the latter in goal-directed behaviour has been debated, but recent data indicate that the ventral and lateral subregions of the OFC are needed to integrate changes in the relationships between actions and their outcomes. Neuromodulatory agents are also crucial components of prefrontal functions and behavioural flexibility might depend upon the noradrenergic modulation of the prefrontal cortex. Therefore, we assessed whether noradrenergic innervation of the OFC plays a role in updating action-outcome relationships in male rats. We used an identity-based reversal task and found that depletion or chemogenetic silencing of noradrenergic inputs within the OFC rendered rats unable to associate new outcomes with previously acquired actions. Silencing of noradrenergic inputs in the prelimbic cortex or depletion of dopaminergic inputs in the OFC did not reproduce this deficit. Together, our results suggest that noradrenergic projections to the OFC are required to update goal-directed actions.

Does Motor Memory Reactivation through Practice and Post-Learning Sleep Modulate Consolidation?

Retrieving previously stored information makes memory traces labile again and can trigger restabilization in a strengthened or weakened form depending on the reactivation condition. Available evidence for long-term performance changes upon reactivation of motor memories and the effect of post-learning sleep on their consolidation remains scarce, and so does the data on the ways in which subsequent reactivation of motor memories interacts with sleep-related consolidation. Eighty young volunteers learned (Day 1) a 12-element Serial Reaction Time Task (SRTT) before a post-training Regular Sleep (RS) or Sleep Deprivation (SD) night, either followed (Day 2) by morning motor reactivation through a short SRTT testing or no motor activity. Consolidation was assessed after three recovery nights (Day 5). A 2 × 2 ANOVA carried on proportional offline gains did not evidence significant Reactivation (Morning Reactivation/No Morning Reactivation; p = 0.098), post-training Sleep (RS/SD; p = 0.301) or Sleep*Reactivation interaction (p = 0.257) effect. Our results are in line with prior studies suggesting a lack of supplementary performance gains upon reactivation, and other studies that failed to disclose post-learning sleep-related effects on performance improvement. However, lack of overt behavioural effects does not detract from the possibility of sleep- or reconsolidation-related covert neurophysiological changes underlying similar behavioural performance levels.

The power of the unexpected: Prediction errors enhance stereotype-based learning

Stereotyping is a ubiquitous feature of social cognition, yet surprisingly little is known about how group-related beliefs influence the acquisition of person knowledge. Accordingly, in combination with computational modeling (i.e., Reinforcement Learning Drift Diffusion Model analysis), here we used a probabilistic selection task to explore the extent to which gender stereotypes impact instrumental learning. Several theoretically interesting effects were observed. First, reflecting the impact of cultural socialization on person construal, an expectancy-based preference for stereotype-consistent (vs. stereotype-inconsistent) responses was observed. Second, underscoring the potency of unexpected information, learning rates were faster for counter-stereotypic compared to stereotypic individuals, both for negative and positive prediction errors. Collectively, these findings are consistent with predictive accounts of social perception and have implications for the conditions under which stereotyping can potentially be reduced.

Multi-timescale analysis of midbrain dopamine neuronal firing activities

Midbrain dopamine (DA) neurons exhibit spiking and bursting patterns under physiological conditions. Based on the data on electrophysiological recordings, Yu et al. developed a 13-dimensional mathematical model to capture the detailed characteristics of the DA neuronal firing activities. We use the fitting method to simplify the original model into a 4-dimensional model. Then, the spiking-to-bursting transition is detected from a simple and robust mathematical condition. Physiologically, this condition is a balance of the restorative and the regenerative ion channels at resting potential. Geometrically, this condition imposes a transcritical bifurcation. Moreover, we combine singularity theory and singular perturbation methods to capture the geometry of three-timescale firing attractors in a universal unfolding of a cusp singularity. In particular, the planar description of the corresponding firing patterns can generate the corresponding firing attractors. This analysis provides a new idea for understanding the firing activities of the DA neuron and the specific mechanisms for the switching and dynamic regulation among different patterns.

Adaptive control of synaptic plasticity integrates micro- and macroscopic network function

Synaptic plasticity configures interactions between neurons and is therefore likely to be a primary driver of behavioral learning and development. How this microscopic-macroscopic interaction occurs is poorly understood, as researchers frequently examine models within particular ranges of abstraction and scale. Computational neuroscience and machine learning models offer theoretically powerful analyses of plasticity in neural networks, but results are often siloed and only coarsely linked to biology. In this review, we examine connections between these areas, asking how network computations change as a function of diverse features of plasticity and vice versa. We review how plasticity can be controlled at synapses by calcium dynamics and neuromodulatory signals, the manifestation of these changes in networks, and their impacts in specialized circuits. We conclude that metaplasticity-defined broadly as the adaptive control of plasticity-forges connections across scales by governing what groups of synapses can and can't learn about, when, and to what ends. The metaplasticity we discuss acts by co-opting Hebbian mechanisms, shifting network properties, and routing activity within and across brain systems. Asking how these operations can go awry should also be useful for understanding pathology, which we address in the context of autism, schizophrenia and Parkinson's disease.

Pain, from perception to action: A computational perspective

Pain is driven by sensation and emotion, and in turn, it motivates decisions and actions. To fully appreciate the multidimensional nature of pain, we formulate the study of pain within a closed-loop framework of sensory-motor prediction. In this closed-loop cycle, prediction plays an important role, as the interaction between prediction and actual sensory experience shapes pain perception and subsequently, action. In this Perspective, we describe the roles of two prominent computational theories-Bayesian inference and reinforcement learning-in modeling adaptive pain behaviors. We show that prediction serves as a common theme between these two theories, and that each of these theories can explain unique aspects of the pain perception-action cycle. We discuss how these computational theories and models can improve our mechanistic understandings of pain-centered processes such as anticipation, attention, placebo hypoalgesia, and pain chronification.

The value of confidence: Confidence prediction errors drive value-based learning in the absence of external feedback

Reinforcement learning algorithms have a long-standing success story in explaining the dynamics of instrumental conditioning in humans and other species. While normative reinforcement learning models are critically dependent on external feedback, recent findings in the field of perceptual learning point to a crucial role of internally generated reinforcement signals based on subjective confidence, when external feedback is not available. Here, we investigated the existence of such confidence-based learning signals in a key domain of reinforcement-based learning: instrumental conditioning. We conducted a value-based decision making experiment which included phases with and without external feedback and in which participants reported their confidence in addition to choices. Behaviorally, we found signatures of self-reinforcement in phases without feedback, reflected in an increase of subjective confidence and choice consistency. To clarify the mechanistic role of confidence in value-based learning, we compared a family of confidence-based learning models with more standard models predicting either no change in value estimates or a devaluation over time when no external reward is provided. We found that confidence-based models indeed outperformed these reference models, whereby the learning signal of the winning model was based on the prediction error between current confidence and a stimulus-unspecific average of previous confidence levels. Interestingly, individuals with more volatile reward-based value updates in the presence of feedback also showed more volatile confidence-based value updates when feedback was not available. Together, our results provide evidence that confidence-based learning signals affect instrumentally learned subjective values in the absence of external feedback.

Reinforcement learning models successfully simulate value-based learning processes (e.g., “How worthwhile is it to choose the same option again?”) when external reward feedback is provided (e.g., drops of sweet liquids or money). But does learning stagnate if such feedback is no longer provided? Recently, a number of studies have shown that subjective confidence can likewise act as an internal reward signal, when external feedback is not available. These results are in line with the intuitive experience that being confident about choices and actions comes with a satisfying feeling of accomplishment. To better understand the role of confidence in value-based learning, we designed a study in which participants had to learn the value of choice options in phases with and without external feedback. Behaviorally, we found signatures of self-reinforcement, such as increased confidence and choice consistency, in phases without feedback. To examine the underlying mechanisms, we compared computational models, in which learning was guided by confidence signals, with more standard reinforcement learning models. A statistical comparison of these models showed that a confidence-based model in which generic confidence prediction errors (e.g., “Am I as confident as expected?”) guide learning indeed outperformed the standard models.