Learning and selective attention

Tea With Milk? A Hierarchical Generative Framework of Sequential Event Comprehension

To make sense of the world around us, we must be able to segment a continual stream of sensory inputs into discrete events. In this review, I propose that in order to comprehend events, we engage hierarchical generative models that "reverse engineer" the intentions of other agents as they produce sequential action in real time. By generating probabilistic predictions for upcoming events, generative models ensure that we are able to keep up with the rapid pace at which perceptual inputs unfold. By tracking our certainty about other agents' goals and the magnitude of prediction errors at multiple temporal scales, generative models enable us to detect event boundaries by inferring when a goal has changed. Moreover, by adapting flexibly to the broader dynamics of the environment and our own comprehension goals, generative models allow us to optimally allocate limited resources. Finally, I argue that we use generative models not only to comprehend events but also to produce events (carry out goal-relevant sequential action) and to continually learn about new events from our surroundings. Taken together, this hierarchical generative framework provides new insights into how the human brain processes events so effortlessly while highlighting the fundamental links between event comprehension, production, and learning.

Signed Reward Prediction Errors in the Ventral Striatum Drive Episodic Memory

Recent behavioral evidence implicates reward prediction errors (RPEs) as a key factor in the acquisition of episodic memory. Yet, important neural predictions related to the role of RPEs in episodic memory acquisition remain to be tested. Humans (both sexes) performed a novel variable-choice task where we experimentally manipulated RPEs and found support for key neural predictions with fMRI. Our results show that in line with previous behavioral observations, episodic memory accuracy increases with the magnitude of signed (i.e., better/worse-than-expected) RPEs (SRPEs). Neurally, we observe that SRPEs are encoded in the ventral striatum (VS). Crucially, we demonstrate through mediation analysis that activation in the VS mediates the experimental manipulation of SRPEs on episodic memory accuracy. In particular, SRPE-based responses in the VS (during learning) predict the strength of subsequent episodic memory (during recollection). Furthermore, functional connectivity between task-relevant processing areas (i.e., face-selective areas) and hippocampus and ventral striatum increased as a function of RPE value (during learning), suggesting a central role of these areas in episodic memory formation. Our results consolidate reinforcement learning theory and striatal RPEs as key factors subtending the formation of episodic memory.SIGNIFICANCE STATEMENT Recent behavioral research has shown that reward prediction errors (RPEs), a key concept of reinforcement learning theory, are crucial to the formation of episodic memories. In this study, we reveal the neural underpinnings of this process. Using fMRI, we show that signed RPEs (SRPEs) are encoded in the ventral striatum (VS), and crucially, that SRPE VS activity is responsible for the subsequent recollection accuracy of one-shot learned episodic memory associations.

Pain Control by Co-adaptive Learning in a Brain-Machine Interface

Innovation in the field of brain-machine interfacing offers a new approach to managing human pain. In principle, it should be possible to use brain activity to directly control a therapeutic intervention in an interactive, closed-loop manner. But this raises the question as to whether the brain activity changes as a function of this interaction. Here, we used real-time decoded functional MRI responses from the insula cortex as input into a closed-loop control system aimed at reducing pain and looked for co-adaptive neural and behavioral changes. As subjects engaged in active cognitive strategies orientated toward the control system, such as trying to enhance their brain activity, pain encoding in the insula was paradoxically degraded. From a mechanistic perspective, we found that cognitive engagement was accompanied by activation of the endogenous pain modulation system, manifested by the attentional modulation of pain ratings and enhanced pain responses in pregenual anterior cingulate cortex and periaqueductal gray. Further behavioral evidence of endogenous modulation was confirmed in a second experiment using an EEG-based closed-loop system. Overall, the results show that implementing brain-machine control systems for pain induces a parallel set of co-adaptive changes in the brain, and this can interfere with the brain signals and behavior under control. More generally, this illustrates a fundamental challenge of brain decoding applications-that the brain inherently adapts to being decoded, especially as a result of cognitive processes related to learning and cooperation. Understanding the nature of these co-adaptive processes informs strategies to mitigate or exploit them.

Individual learning phenotypes drive collective behavior

Individual differences in learning can influence how animals respond to and communicate about their environment, which may nonlinearly shape how a social group accomplishes a collective task. There are few empirical examples of how differences in collective dynamics emerge from variation among individuals in cognition. Here, we use a naturally variable and heritable learning behavior called latent inhibition (LI) to show that interactions among individuals that differ in this cognitive ability drive collective foraging behavior in honey bee colonies. We artificially selected two distinct phenotypes: high-LI bees that ignore previously familiar stimuli in favor of novel ones and low-LI bees that learn familiar and novel stimuli equally well. We then provided colonies differentially composed of different ratios of these phenotypes with a choice between familiar and novel feeders. Colonies of predominantly high-LI individuals preferred to visit familiar food locations, while low-LI colonies visited novel and familiar food locations equally. Interestingly, in colonies of mixed learning phenotypes, the low-LI individuals showed a preference to visiting familiar feeders, which contrasts with their behavior when in a uniform low-LI group. We show that the shift in feeder preference of low-LI bees is driven by foragers of the high-LI phenotype dancing more intensely and attracting more followers. Our results reveal that cognitive abilities of individuals and their social interactions, which we argue relate to differences in attention, drive emergent collective outcomes.

A simple model for learning in volatile environments

Sound principles of statistical inference dictate that uncertainty shapes learning. In this work, we revisit the question of learning in volatile environments, in which both the first and second-order statistics of observations dynamically evolve over time. We propose a new model, the volatile Kalman filter (VKF), which is based on a tractable state-space model of uncertainty and extends the Kalman filter algorithm to volatile environments. The proposed model is algorithmically simple and encompasses the Kalman filter as a special case. Specifically, in addition to the error-correcting rule of Kalman filter for learning observations, the VKF learns volatility according to a second error-correcting rule. These dual updates echo and contextualize classical psychological models of learning, in particular hybrid accounts of Pearce-Hall and Rescorla-Wagner. At the computational level, compared with existing models, the VKF gives up some flexibility in the generative model to enable a more faithful approximation to exact inference. When fit to empirical data, the VKF is better behaved than alternatives and better captures human choice data in two independent datasets of probabilistic learning tasks. The proposed model provides a coherent account of learning in stable or volatile environments and has implications for decision neuroscience research.

Brain dynamics for confidence-weighted learning

Learning in a changing, uncertain environment is a difficult problem. A popular solution is to predict future observations and then use surprising outcomes to update those predictions. However, humans also have a sense of confidence that characterizes the precision of their predictions. Bayesian models use a confidence-weighting principle to regulate learning: for a given surprise, the update is smaller when the confidence about the prediction was higher. Prior behavioral evidence indicates that human learning adheres to this confidence-weighting principle. Here, we explored the human brain dynamics sub-tending the confidence-weighting of learning using magneto-encephalography (MEG). During our volatile probability learning task, subjects’ confidence reports conformed with Bayesian inference. MEG revealed several stimulus-evoked brain responses whose amplitude reflected surprise, and some of them were further shaped by confidence: surprise amplified the stimulus-evoked response whereas confidence dampened it. Confidence about predictions also modulated several aspects of the brain state: pupil-linked arousal and beta-range (15–30 Hz) oscillations. The brain state in turn modulated specific stimulus-evoked surprise responses following the confidence-weighting principle. Our results thus indicate that there exist, in the human brain, signals reflecting surprise that are dampened by confidence in a way that is appropriate for learning according to Bayesian inference. They also suggest a mechanism for confidence-weighted learning: confidence about predictions would modulate intrinsic properties of the brain state to amplify or dampen surprise responses evoked by discrepant observations.

Learning in a changing and uncertain world is difficult. In this context, facing a discrepancy between my current belief and new observations may reflect random fluctuations (e.g. my commute train is unexpectedly late, but it happens sometimes), if so, I should ignore this discrepancy and not change erratically my belief. However, this discrepancy could also denote a profound change (e.g. the train company changed and is less reliable), in this case, I should promptly revise my current belief. Human learning is adaptive: we change how much we learn from new observations, in particular, we promote flexibility when facing profound changes. A mathematical analysis of the problem shows that we should increase flexibility when the confidence about our current belief is low, which occurs when a change is suspected. Here, I show that human learners entertain rational confidence levels during the learning of changing probabilities. This confidence modulates intrinsic properties of the brain state (oscillatory activity and neuromodulation) which in turn amplifies or reduces, depending on whether confidence is low or high, the neural responses to discrepant observations. This confidence-weighting mechanism could underpin adaptive learning.

Variability in Action Selection Relates to Striatal Dopamine 2/3 Receptor Availability in Humans: A PET Neuroimaging Study Using Reinforcement Learning and Active Inference Models

Choosing actions that result in advantageous outcomes is a fundamental function of nervous systems. All computational decision-making models contain a mechanism that controls the variability of (or confidence in) action selection, but its neural implementation is unclear-especially in humans. We investigated this mechanism using two influential decision-making frameworks: active inference (AI) and reinforcement learning (RL). In AI, the precision (inverse variance) of beliefs about policies controls action selection variability-similar to decision 'noise' parameters in RL-and is thought to be encoded by striatal dopamine signaling. We tested this hypothesis by administering a 'go/no-go' task to 75 healthy participants, and measuring striatal dopamine 2/3 receptor (D2/3R) availability in a subset (n = 25) using [11C]-(+)-PHNO positron emission tomography. In behavioral model comparison, RL performed best across the whole group but AI performed best in participants performing above chance levels. Limbic striatal D2/3R availability had linear relationships with AI policy precision (P = 0.029) as well as with RL irreducible decision 'noise' (P = 0.020), and this relationship with D2/3R availability was confirmed with a 'decision stochasticity' factor that aggregated across both models (P = 0.0006). These findings are consistent with occupancy of inhibitory striatal D2/3Rs decreasing the variability of action selection in humans.

A Critical Period for Prefrontal Network Configurations Underlying Psychiatric Disorders and Addiction

The medial prefrontal cortex (mPFC) has been classically defined as the brain region responsible for higher cognitive functions, including the decision-making process. Ample information has been gathered during the last 40 years in an attempt to understand how it works. We now know extensively about the connectivity of this region and its relationship with neuromodulatory ascending projection areas, such as the dorsal raphe nucleus (DRN) or the ventral tegmental area (VTA). Both areas are well-known regulators of the reward-based decision-making process and hence likely to be involved in processes like evidence integration, impulsivity or addiction biology, but also in helping us to predict the valence of our future actions: i.e., what is “good” and what is “bad.” Here we propose a hypothesis of a critical period, during which the inputs of the mPFC compete for target innervation, establishing specific prefrontal network configurations in the adult brain. We discuss how these different prefrontal configurations are linked to brain diseases such as addiction or neuropsychiatric disorders, and especially how drug abuse and other events during early life stages might lead to the formation of more vulnerable prefrontal network configurations. Finally, we show different promising pharmacological approaches that, when combined with the appropriate stimuli, will be able to re-establish these functional prefrontocortical configurations during adulthood.

Prior cocaine self-administration impairs attention signals in anterior cingulate cortex

Although maladaptive decision-making is a defining feature of drug abuse and addiction, we have yet to ascertain how cocaine self-administration disrupts neural signals in anterior cingulate cortex (ACC), a brain region thought to contribute to attentional control. To address this issue, rats were trained on a reward-guided decision-making task; reward value was manipulated by independently varying the size of or the delay to reward over several trial blocks. Subsequently, rats self-administered either a cocaine (experimental group) or sucrose (control) during 12 consecutive days, after which they underwent a 1-month withdrawal period. Upon completion of this period, rats performed the previously learned reward-guided decision-making task while we recorded from single neurons in ACC. We demonstrate that prior cocaine self-administration attenuates attention and attention-related ACC signals in an intake-dependent manner, and that changes in attention are decoupled from ACC firing. These effects likely contribute to the impaired decision-making-typified by chronic substance abuse and relapse-observed after drug use.

Reward Prediction Error and Declarative Memory

Learning based on reward prediction error (RPE) was originally proposed in the context of nondeclarative memory. We postulate that RPE may support declarative memory as well. Indeed, recent years have witnessed a number of independent empirical studies reporting effects of RPE on declarative memory. We provide a brief overview of these studies, identify emerging patterns, and discuss open issues such as the role of signed versus unsigned RPEs in declarative learning.

Dynamic resource allocation during reinforcement learning accounts for ramping and phasic dopamine activity

For an animal to learn about its environment with limited motor and cognitive resources, it should focus its resources on potentially important stimuli. However, too narrow focus is disadvantageous for adaptation to environmental changes. Midbrain dopamine neurons are excited by potentially important stimuli, such as reward-predicting or novel stimuli, and allocate resources to these stimuli by modulating how an animal approaches, exploits, explores, and attends. The current study examined the theoretical possibility that dopamine activity reflects the dynamic allocation of resources for learning. Dopamine activity may transition between two patterns: (1) phasic responses to cues and rewards, and (2) ramping activity arising as the agent approaches the reward. Phasic excitation has been explained by prediction errors generated by experimentally inserted cues. However, when and why dopamine activity transitions between the two patterns remain unknown. By parsimoniously modifying a standard temporal difference (TD) learning model to accommodate a mixed presentation of both experimental and environmental stimuli, we simulated dopamine transitions and compared them with experimental data from four different studies. The results suggested that dopamine transitions from ramping to phasic patterns as the agent focuses its resources on a small number of reward-predicting stimuli, thus leading to task dimensionality reduction. The opposite occurs when the agent re-distributes its resources to adapt to environmental changes, resulting in task dimensionality expansion. This research elucidates the role of dopamine in a broader context, providing a potential explanation for the diverse repertoire of dopamine activity that cannot be explained solely by prediction error.

Adaptive learning under expected and unexpected uncertainty

The outcome of a decision is often uncertain, and outcomes can vary over repeated decisions. Whether decision outcomes should substantially affect behaviour and learning depends on whether they are representative of a typically experienced range of outcomes or signal a change in the reward environment. Successful learning and decision-making therefore require the ability to estimate expected uncertainty (related to the variability of outcomes) and unexpected uncertainty (related to the variability of the environment). Understanding the bases and effects of these two types of uncertainty and the interactions between them - at the computational and the neural level - is crucial for understanding adaptive learning. Here, we examine computational models and experimental findings to distil computational principles and neural mechanisms for adaptive learning under uncertainty.

Uncertainty in learning, choice, and visual fixation

Uncertainty plays a critical role in reinforcement learning and decision making. However, exactly how it influences behavior remains unclear. Multiarmed-bandit tasks offer an ideal test bed, since computational tools such as approximate Kalman filters can closely characterize the interplay between trial-by-trial values, uncertainty, learning, and choice. To gain additional insight into learning and choice processes, we obtained data from subjects' overt allocation of gaze. The estimated value and estimation uncertainty of options influenced what subjects looked at before choosing; these same quantities also influenced choice, as additionally did fixation itself. A momentary measure of uncertainty in the form of absolute prediction errors determined how long participants looked at the obtained outcomes. These findings affirm the importance of uncertainty in multiple facets of behavior and help delineate its effects on decision making.

Functional disconnection of associative cortical areas predicts performance during BCI training

Brain-computer interfaces (BCIs) have been largely developed to allow communication, control, and neurofeedback in human beings. Despite their great potential, BCIs perform inconsistently across individuals and the neural processes that enable humans to achieve good control remain poorly understood. To address this question, we performed simultaneous high-density electroencephalographic (EEG) and magnetoencephalographic (MEG) recordings in a motor imagery-based BCI training involving a group of healthy subjects. After reconstructing the signals at the cortical level, we showed that the reinforcement of motor-related activity during the BCI skill acquisition is paralleled by a progressive disconnection of associative areas which were not directly targeted during the experiments. Notably, these network connectivity changes reflected growing automaticity associated with BCI performance and predicted future learning rate. Altogether, our findings provide new insights into the large-scale cortical organizational mechanisms underlying BCI learning, which have implications for the improvement of this technology in a broad range of real-life applications.

Predictability Changes What We Remember in Familiar Temporal Contexts

The human brain constantly anticipates the future based on memories of the past. Encountering a familiar situation reactivates memory of previous encounters, which can trigger a prediction of what comes next to facilitate responsiveness. However, a prediction error can lead to pruning of the offending memory, a process that weakens its representation in the brain and leads to forgetting. Our goal in this study was to evaluate whether memories are spared from such pruning in situations that allow for accurate predictions at the categorical level, despite prediction errors at the item level. Participants viewed a sequence of objects, some of which reappeared multiple times ("cues"), followed always by novel items. Half of the cues were followed by new items from different (unpredictable) categories, while others were followed by new items from a single (predictable) category. Pattern classification of fMRI data was used to identify category-specific predictions after each cue. Pruning was observed only in unpredictable contexts, while encoding of new items was less robust in predictable contexts. These findings demonstrate that how associative memories are updated is influenced by the reliability of abstract-level predictions in familiar contexts.

Learning optimal decisions with confidence

Diffusion decision models (DDMs) are immensely successful models for decision making under uncertainty and time pressure. In the context of perceptual decision making, these models typically start with two input units, organized in a neuron-antineuron pair. In contrast, in the brain, sensory inputs are encoded through the activity of large neuronal populations. Moreover, while DDMs are wired by hand, the nervous system must learn the weights of the network through trial and error. There is currently no normative theory of learning in DDMs and therefore no theory of how decision makers could learn to make optimal decisions in this context. Here, we derive such a rule for learning a near-optimal linear combination of DDM inputs based on trial-by-trial feedback. The rule is Bayesian in the sense that it learns not only the mean of the weights but also the uncertainty around this mean in the form of a covariance matrix. In this rule, the rate of learning is proportional (respectively, inversely proportional) to confidence for incorrect (respectively, correct) decisions. Furthermore, we show that, in volatile environments, the rule predicts a bias toward repeating the same choice after correct decisions, with a bias strength that is modulated by the previous choice's difficulty. Finally, we extend our learning rule to cases for which one of the choices is more likely a priori, which provides insights into how such biases modulate the mechanisms leading to optimal decisions in diffusion models.

A computational account of threat-related attentional bias

Visual selective attention acts as a filter on perceptual information, facilitating learning and inference about important events in an agent's environment. A role for visual attention in reward-based decisions has previously been demonstrated, but it remains unclear how visual attention is recruited during aversive learning, particularly when learning about multiple stimuli concurrently. This question is of particular importance in psychopathology, where enhanced attention to threat is a putative feature of pathological anxiety. Using an aversive reversal learning task that required subjects to learn, and exploit, predictions about multiple stimuli, we show that the allocation of visual attention is influenced significantly by aversive value but not by uncertainty. Moreover, this relationship is bidirectional in that attention biases value updates for attended stimuli, resulting in heightened value estimates. Our findings have implications for understanding biased attention in psychopathology and support a role for learning in the expression of threat-related attentional biases in anxiety.

Parylene Based Memristive Devices with Multilevel Resistive Switching for Neuromorphic Applications

In this paper, the resistive switching and neuromorphic behaviour of memristive devices based on parylene, a polymer both low-cost and safe for the human body, is comprehensively studied. The Metal/Parylene/ITO sandwich structures were prepared by means of the standard gas phase surface polymerization method with different top active metal electrodes (Ag, Al, Cu or Ti of ~500 nm thickness). These organic memristive devices exhibit excellent performance: low switching voltage (down to 1 V), large OFF/ON resistance ratio (up to 10⁴), retention (≥10⁴ s) and high multilevel resistance switching (at least 16 stable resistive states in the case of Cu electrodes). We have experimentally shown that parylene-based memristive elements can be trained by a biologically inspired spike-timing-dependent plasticity (STDP) mechanism. The obtained results have been used to implement a simple neuromorphic network model of classical conditioning. The described advantages allow considering parylene-based organic memristors as prospective devices for hardware realization of spiking artificial neuron networks capable of supervised and unsupervised learning and suitable for biomedical applications.

Parietal neurons encode information sampling based on decision uncertainty

In natural behavior animals actively gather information that is relevant for learning or actions, but the mechanisms of active sampling are rarely investigated. We tested parietal neurons involved in oculomotor control in a task in which monkeys made saccades to gather visual information before reporting a decision based on the information. We show that the neurons encode, before the saccade, the information gains (reduction in decision uncertainty) that the saccade was expected to bring, correlating with the monkeys’ efficiency in processing the information in the post-saccadic fixation. Informational sensitivity is independent of the neurons’ reward sensitivity, which is unreliable across task contexts, inconsistent with the view that the cells encode economic utility. Instead, we suggest that parietal cells are involved in implementing active sampling policies, showing uncertainty-dependent boosts of neural gain that facilitate the selection of relevant cues and the efficient use of the information delivered by these cues.

Selective sustained attention: a developmental foundation for cognition

Higher-order cognition, particularly in real-life settings, often requires that parts of the sensory input be processed at the exclusion of others over a period of time. Consequently, this review focuses on the development of attention that is both selective (which entails processing parts of the sensory input at the exclusion of others) and sustained (which entails maintaining sensitivity to incoming stimuli for a period of time). Recent findings from four distinct areas of research reviewed here suggest that: (1) the underlying neural circuitry of selective sustained attention involves multiple cortical and subcortical brain regions; (2) selective sustained attention in infancy provides a developmental foundation for the emergence of executive function later in life; (3) suppression-based mechanisms of attentional selection that begin to emerge during the first year of life are important for memory and learning; and (4) selective sustained attention appears to be malleable through pre-natal and post-natal nutritional supplementation and interactions with mature social partners.

Holistic Reinforcement Learning: The Role of Structure and Attention

Compact representations of the environment allow humans to behave efficiently in a complex world. Reinforcement learning models capture many behavioral and neural effects but do not explain recent findings showing that structure in the environment influences learning. In parallel, Bayesian cognitive models predict how humans learn structured knowledge but do not have a clear neurobiological implementation. We propose an integration of these two model classes in which structured knowledge learned via approximate Bayesian inference acts as a source of selective attention. In turn, selective attention biases reinforcement learning towards relevant dimensions of the environment. An understanding of structure learning will help to resolve the fundamental challenge in decision science: explaining why people make the decisions they do.

Habits without values

Habits form a crucial component of behavior. In recent years, key computational models have conceptualized habits as arising from model-free reinforcement learning mechanisms, which typically select between available actions based on the future value expected to result from each. Traditionally, however, habits have been understood as behaviors that can be triggered directly by a stimulus, without requiring the animal to evaluate expected outcomes. Here, we develop a computational model instantiating this traditional view, in which habits develop through the direct strengthening of recently taken actions rather than through the encoding of outcomes. We demonstrate that this model accounts for key behavioral manifestations of habits, including insensitivity to outcome devaluation and contingency degradation, as well as the effects of reinforcement schedule on the rate of habit formation. The model also explains the prevalent observation of perseveration in repeated-choice tasks as an additional behavioral manifestation of the habit system. We suggest that mapping habitual behaviors onto value-free mechanisms provides a parsimonious account of existing behavioral and neural data. This mapping may provide a new foundation for building robust and comprehensive models of the interaction of habits with other, more goal-directed types of behaviors and help to better guide research into the neural mechanisms underlying control of instrumental behavior more generally. (PsycINFO Database Record (c) 2019 APA, all rights reserved).

Feature-specific prediction errors and surprise across macaque fronto-striatal circuits

To adjust expectations efficiently, prediction errors need to be associated with the precise features that gave rise to the unexpected outcome, but this credit assignment may be problematic if stimuli differ on multiple dimensions and it is ambiguous which feature dimension caused the outcome. Here, we report a potential solution: neurons in four recorded areas of the anterior fronto-striatal networks encode prediction errors that are specific to feature values of different dimensions of attended multidimensional stimuli. The most ubiquitous prediction error occurred for the reward-relevant dimension. Feature-specific prediction error signals a) emerge on average shortly after non-specific prediction error signals, b) arise earliest in the anterior cingulate cortex and later in dorsolateral prefrontal cortex, caudate and ventral striatum, and c) contribute to feature-based stimulus selection after learning. Thus, a widely-distributed feature-specific eligibility trace may be used to update synaptic weights for improved feature-based attention.

In order to adjust expectations efficiently, prediction errors need to be associated with the features that gave rise to the unexpected outcome. Here, the authors show that neurons in anterior fronto-striatal networks encode prediction errors that are specific to feature values of different stimulus dimensions.

Cortical neural modulation by previous trial outcome differentiates fast- from slow-learning rats on a visuomotor directional choice task

To better understand the neural cortical underpinnings that explain behavioral differences in learning rate, we recorded single-unit activity in primary motor (M1) and secondary motor (M2) areas while rats learned to perform a directional (left or right) operant visuomotor association task. Analysis of neural activity during the early portion of the cue period showed that neural modulation in the motor cortex was most strongly associated with two task factors: the previous trial outcome (success or error) and the current trial's directional choice (left or right). Furthermore, the fast learners, defined as those who had steeper learning curves and required fewer learning sessions to reach criterion performance, encoded the previous trial outcome factor more strongly than the directional choice factor. Conversely, the slow learners encoded directional choice more strongly than previous trial outcome. These differences in task factor encoding were observed in both the percentage of neurons and the neural modulation depth. These results suggest that fast learning is accompanied by a stronger component of previous trial outcome in the modulation representation present in motor cortex, which therefore may be a contributing factor to behavioral differences in learning rate. NEW & NOTEWORTHY We chronically recorded neural activity as rats learned a visuomotor directional choice task from a naive state. Learning rates varied. Single-unit neural modulation of two motor areas revealed that the fast learners encoded previous trial outcome more strongly than directional choice, whereas the reverse was true for slow learners. This finding provides novel evidence that rat learning rate is strongly correlated with the strength of neural modulation by previous trial outcome in motor cortex.

The principle of consistency and the cause and function of behaviour

At all levels of information processing in the brain, neural and cognitive structures tend towards a state of consistency. When two or more simultaneously active cognitive structures are logically inconsistent, arousal is increased, which activates processes with the expected consequence of increasing consistency and decreasing arousal. Increased arousal is experienced as aversive, while the expected or actual decrease in arousal as a result of increased consistency is experienced as rewarding. Modes of resolution of inconsistency can be divided into purely cognitive solutions, such as changing an attitude or an associated motor plan, and behavioural solutions, such as exploration, aggression, fear, and feeding. Models and theories consistent with the principle of consistency are numerous, have a long and continuing history, and come from many different scientific fields, such as social psychology, perception, neurocognition, learning, motor control, system control, ethology, and stress. The present paper presents a brief overview of relevant information from these fields of research, while focusing mainly on the implications of the principle of consistency for the understanding of the cause and function of behaviour. Based on this overview, it is proposed that all behaviour involving cognitive processing is caused by the activation of inconsistent cognitions and functions to increase perceived consistency.

Selective maintenance of value information helps resolve the exploration/exploitation dilemma

In natural environments with many options of uncertain value, one faces a difficult tradeoff between exploiting familiar, valuable options or searching for better alternatives. Reinforcement learning models of this exploration/exploitation dilemma typically modulate the rate of exploratory choices or preferentially sample uncertain options. The extent to which such models capture human behavior remains unclear, in part because they do not consider the constraints on remembering what is learned. Using reinforcement-based timing as a motivating example, we show that selectively maintaining high-value actions compresses the amount of information to be tracked in learning, as quantified by Shannon's entropy. In turn, the information content of the value representation controls the balance between exploration (high entropy) and exploitation (low entropy). Selectively maintaining preferred action values while allowing others to decay renders the choices increasingly exploitative across learning episodes. To adjudicate among alternative maintenance and sampling strategies, we developed a new reinforcement learning model, StrategiC ExPloration/ExPloitation of Temporal Instrumental Contingencies (SCEPTIC). In computational studies, a resource-rational selective maintenance approach was as successful as more resource-intensive strategies. Furthermore, human behavior was consistent with selective maintenance; information compression was most pronounced in subjects with superior performance and non-verbal intelligence, and in learnable vs. unlearnable contingencies. Cognitively demanding uncertainty-directed exploration recovered a more accurate representation in simulations with no foraging advantage and was strongly unsupported in our human study.

Training on an Appetitive (Delay)-Conditioning Task Enhances Oscillatory Waves During Sleep in the Cortical and Amygdalar Network

Oscillating waves during sleep play an essential role in memory consolidation. The cortical slow wave activity (SWA) and sigma waves during NREM sleep and theta waves during REM sleep increase after a variety of memory tasks including declarative, procedural and associative learning tasks. These oscillatory waves during sleep help to promote neural dialog between circuitries, which possibly plays a causal role in memory consolidation. However, the role of sleep-associated oscillating waves in a complex appetitive-conditioning paradigm is not clear. The parietal cortex and amygdala are involved in the cognitive evaluation of the environmental stimuli, and appetitive conditioning. Here, we have studied the changes in sleep architecture and oscillatory waves during NREM and REM sleep in the parietal cortices and amygdalar-local field potential (A-LFP) after appetitive-conditioning in the rat. We observed that REM sleep increased significantly after appetitive conditioning, which significantly positively correlated with performance on the appetitive-conditioning task. Further, the cortical SWA (0.1-4.5 Hz), and sigma (12-14.25 Hz) waves during NREM sleep, theta (6-9 Hz) waves during REM sleep, the amygdalar SWA (0.1-3.75 Hz) during NREM sleep and theta (6-8.25 Hz) waves during REM sleep significantly increased after appetitive conditioning. Interestingly, the augmented oscillatory waves significantly positively correlated with the performances on the appetitive-conditioning task. Our results suggest that the augmented REM sleep after conditioning may be required for the consolidation of appetitive-conditioned memory. Further, a significant correlation between augmented power in oscillatory waves during sleep and performance suggesting that these waves may be playing a crucial role in the consolidation of appetitive-conditioned memory.

Can Monetary Reward Modulate Social Attention?

Selective visual attention is a primary cognitive function, which allows the selection of the most relevant stimuli in the environment by prioritizing their processing. Several studies showed that this process can be influenced by both social signals, such as gaze direction (i.e., the Gaze Cueing Effect, GCE) and by the motivational valence of gratifying stimuli, such as monetary rewards. The aim of this study was to explore whether GCE could be modulated by a monetary reward. To this end, we created an experiment in which participants performed a gaze cuing task before and after an implicit learning task aiming to induce an association between gaze direction and monetary reward (experimental condition), or after a perceptual task (control condition). Statistical analyses were conducted following both a frequentist and a Bayesian approach. Results supported previous findings showing the presence of the GCE, i.e., faster responses in congruent trials when the target appeared in the gazed-at location. Interestingly, our results did not reveal significant differences among the conditions. Therefore, contrary to what was reported by previous attentional orienting studies with non-social stimuli, monetary reward does not seem to be able to modulate (or interfere with) the orienting of attention mediated by gaze direction as measured by the GCE. Taken together our results suggest that social signals such as gaze direction have a greater impact than monetary reward in orienting selective attention.

Abnormal reward prediction-error signalling in antipsychotic naive individuals with first-episode psychosis or clinical risk for psychosis

Ongoing research suggests preliminary, though not entirely consistent, evidence of neural abnormalities in signalling prediction errors in schizophrenia. Supporting theories suggest mechanistic links between the disruption of these processes and the generation of psychotic symptoms. However, it is unknown at what stage in the pathogenesis of psychosis these impairments in prediction-error signalling develop. One major confound in prior studies is the use of medicated patients with strongly varying disease durations. Our study aims to investigate the involvement of the meso-cortico-striatal circuitry during reward prediction-error signalling in earliest stages of psychosis. We studied patients with first-episode psychosis (FEP) and help-seeking individuals at-risk for psychosis due to sub-threshold prodromal psychotic symptoms. Patients with either FEP (n = 14), or at-risk for developing psychosis (n = 30), and healthy volunteers (n = 39) performed a reinforcement learning task during fMRI scanning. ANOVA revealed significant (p < 0.05 family-wise error corrected) prediction-error signalling differences between groups in the dopaminergic midbrain and right middle frontal gyrus (dorsolateral prefrontal cortex, DLPFC). FEP patients showed disrupted reward prediction-error signalling compared to controls in both regions. At-risk patients showed intermediate activation in the midbrain that significantly differed from controls and from FEP patients, but DLPFC activation that did not differ from controls. Our study confirms that FEP patients have abnormal meso-cortical signalling of reward-prediction errors, whereas reward-prediction-error dysfunction in the at-risk patients appears to show a more nuanced pattern of activation with a degree of midbrain impairment but preserved cortical function.

An effect of serotonergic stimulation on learning rates for rewards apparent after long intertrial intervals

Serotonin has widespread, but computationally obscure, modulatory effects on learning and cognition. Here, we studied the impact of optogenetic stimulation of dorsal raphe serotonin neurons in mice performing a non-stationary, reward-driven decision-making task. Animals showed two distinct choice strategies. Choices after short inter-trial-intervals (ITIs) depended only on the last trial outcome and followed a win-stay-lose-switch pattern. In contrast, choices after long ITIs reflected outcome history over multiple trials, as described by reinforcement learning models. We found that optogenetic stimulation during a trial significantly boosted the rate of learning that occurred due to the outcome of that trial, but these effects were only exhibited on choices after long ITIs. This suggests that serotonin neurons modulate reinforcement learning rates, and that this influence is masked by alternate, unaffected, decision mechanisms. These results provide insight into the role of serotonin in treating psychiatric disorders, particularly its modulation of neural plasticity and learning.

Serotonin (5-HT) plays many important roles in reward, punishment, patience and beyond, and optogenetic stimulation of 5-HT neurons has not crisply parsed them. The authors report a novel analysis of a reward-based decision-making experiment, and show that 5-HT stimulation increases the learning rate, but only on a select subset of choices.

Learning what matters: A neural explanation for the sparsity bias

The visual environment is filled with complex, multi-dimensional objects that vary in their value to an observer's current goals. When faced with multi-dimensional stimuli, humans may rely on biases to learn to select those objects that are most valuable to the task at hand. Here, we show that decision making in a complex task is guided by the sparsity bias: the focusing of attention on a subset of available features. Participants completed a gambling task in which they selected complex stimuli that varied randomly along three dimensions: shape, color, and texture. Each dimension comprised three features (e.g., color: red, green, yellow). Only one dimension was relevant in each block (e.g., color), and a randomly-chosen value ranking determined outcome probabilities (e.g., green > yellow > red). Participants were faster to respond to infrequent probe stimuli that appeared unexpectedly within stimuli that possessed a more valuable feature than to probes appearing within stimuli possessing a less valuable feature. Event-related brain potentials recorded during the task provided a neurophysiological explanation for sparsity as a learning-dependent increase in optimal attentional performance (as measured by the N2pc component of the human event-related potential) and a concomitant learning-dependent decrease in prediction errors (as measured by the feedback-elicited reward positivity). Together, our results suggest that the sparsity bias guides human reinforcement learning in complex environments.

The control of tonic pain by active relief learning

Tonic pain after injury characterises a behavioural state that prioritises recovery. Although generally suppressing cognition and attention, tonic pain needs to allow effective relief learning to reduce the cause of the pain. Here, we describe a central learning circuit that supports learning of relief and concurrently suppresses the level of ongoing pain. We used computational modelling of behavioural, physiological and neuroimaging data in two experiments in which subjects learned to terminate tonic pain in static and dynamic escape-learning paradigms. In both studies, we show that active relief-seeking involves a reinforcement learning process manifest by error signals observed in the dorsal putamen. Critically, this system uses an uncertainty ('associability') signal detected in pregenual anterior cingulate cortex that both controls the relief learning rate, and endogenously and parametrically modulates the level of tonic pain. The results define a self-organising learning circuit that reduces ongoing pain when learning about potential relief.

Influence of multiple action-outcome associations on the transition dynamics toward an optimal choice in rats

When faced with familiar versus novel options, animals may exploit the acquired action-outcome associations or attempt to form new associations. Little is known about which factors determine the strategy of choice behavior in partially comprehended environments. Here we examine the influence of multiple action-outcome associations on choice behavior in the context of rewarding outcomes (food) and aversive outcomes (electric foot-shock). We used a nose-poke paradigm with rats, incorporating a dilemma between a familiar option and a novel, higher-value option. In Experiment 1, two groups of rats were trained with different outcome schedules: either a single action-outcome association ("Reward-Only") or dual action-outcome associations ("Reward-Shock"; with the added opportunity to avoid an electric foot-shock). In Experiment 2, we employed the same paradigm with two groups of rats performing the task under dual action-outcome associations, with different levels of threat (a low- or high-amplitude electric foot-shock). The choice behavior was clearly influenced by the action-outcome associations, with more efficient transition dynamics to the optimal choice with dual rather than single action-outcome associations. The level of threat did not affect the transition dynamics. Taken together, the data suggested that the strategy of choice behavior was modulated by the information complexity of the environment.

Associability-modulated loss learning is increased in posttraumatic stress disorder

Disproportionate reactions to unexpected stimuli in the environment are a cardinal symptom of posttraumatic stress disorder (PTSD). Here, we test whether these heightened responses are associated with disruptions in distinct components of reinforcement learning. Specifically, using functional neuroimaging, a loss-learning task, and a computational model-based approach, we assessed the mechanistic hypothesis that overreactions to stimuli in PTSD arise from anomalous gating of attention during learning (i.e., associability). Behavioral choices of combat-deployed veterans with and without PTSD were fit to a reinforcement learning model, generating trial-by-trial prediction errors (signaling unexpected outcomes) and associability values (signaling attention allocation to the unexpected outcomes). Neural substrates of associability value and behavioral parameter estimates of associability updating, but not prediction error, increased with PTSD during loss learning. Moreover, the interaction of PTSD severity with neural markers of associability value predicted behavioral choices. These results indicate that increased attention-based learning may underlie aspects of PTSD and suggest potential neuromechanistic treatment targets.

Sensory hypersensitivity predicts enhanced attention capture by faces in the early development of ASD

Sensory sensitivity is prevalent among young children with ASD, but its relation to social communication impairment is unclear. Recently, increased sensory hypersensitivity has been linked to greater activity of the neural salience network (Green et al., 2016). Increased neural sensitivity to stimuli, especially social stimuli, could provide greater opportunity for social learning and improved outcomes. Consistent with this framework, in Experiment 1 we found that parent report of greater sensory hypersensitivity at 2 years in toddlers with ASD (N=27) was predictive of increased neural responsiveness to social stimuli (larger amplitude event-related potential/ERP responses to faces at P1, P400 and Nc) at 4 years, and this in turn was related to parent report of increased social approach at 4 years. In Experiment 2, parent report of increased perceptual sensitivity at 6 months in infants at low and high familial risk for ASD (N=35) predicted larger ERP P1 amplitude to faces at 18 months. Increased sensory hypersensitivity in early development thus predicted greater attention capture by faces in later development, and this related to more optimal social behavioral development. Sensory hypersensitivity may index a child's ability to benefit from supportive environments during development. Early sensory symptoms may not always be developmentally problematic for individuals with ASD.

Dopamine, Inference, and Uncertainty

The hypothesis that the phasic dopamine response reports a reward prediction error has become deeply entrenched. However, dopamine neurons exhibit several notable deviations from this hypothesis. A coherent explanation for these deviations can be obtained by analyzing the dopamine response in terms of Bayesian reinforcement learning. The key idea is that prediction errors are modulated by probabilistic beliefs about the relationship between cues and outcomes, updated through Bayesian inference. This account can explain dopamine responses to inferred value in sensory preconditioning, the effects of cue preexposure (latent inhibition), and adaptive coding of prediction errors when rewards vary across orders of magnitude. We further postulate that orbitofrontal cortex transforms the stimulus representation through recurrent dynamics, such that a simple error-driven learning rule operating on the transformed representation can implement the Bayesian reinforcement learning update.

Learning Where to Look for High Value Improves Decision Making Asymmetrically

Decision making in any brain is imperfect and costly in terms of time and energy. Operating under such constraints, an organism could be in a position to improve performance if an opportunity arose to exploit informative patterns in the environment being searched. Such an improvement of performance could entail both faster and more accurate (i.e., reward-maximizing) decisions. The present study investigated the extent to which human participants could learn to take advantage of immediate patterns in the spatial arrangement of serially presented foods such that a region of space would consistently be associated with greater subjective value. Eye movements leading up to choices demonstrated rapidly induced biases in the selective allocation of visual fixation and attention that were accompanied by both faster and more accurate choices of desired goods as implicit learning occurred. However, for the control condition with its spatially balanced reward environment, these subjects exhibited preexisting lateralized biases for eye and hand movements (i.e., leftward and rightward, respectively) that could act in opposition not only to each other but also to the orienting biases elicited by the experimental manipulation, producing an asymmetry between the left and right hemifields with respect to performance. Potentially owing at least in part to learned cultural conventions (e.g., reading from left to right), the findings herein particularly revealed an intrinsic leftward bias underlying initial saccades in the midst of more immediate feedback-directed processes for which spatial biases can be learned flexibly to optimize oculomotor and manual control in value-based decision making. The present study thus replicates general findings of learned attentional biases in a novel context with inherently rewarding stimuli and goes on to further elucidate the interactions between endogenous and exogenous biases.

Changes of Attention during Value-Based Reversal Learning Are Tracked by N2pc and Feedback-Related Negativity

Previously learned reward values can have a pronounced impact, behaviorally and neurophysiologically, on the allocation of selective attention. All else constant, stimuli previously associated with a high value gain stronger attentional prioritization than stimuli previously associated with a low value. The N2pc, an ERP component indicative of attentional target selection, has been shown to reflect aspects of this prioritization, by changes of mean amplitudes closely corresponding to selective enhancement of high value target processing and suppression of high value distractor processing. What has remained unclear so far is whether the N2pc also reflects the flexible and repeated behavioral adjustments needed in a volatile task environment, in which the values of stimuli are reversed often and unannounced. Using a value-based reversal learning task, we found evidence that the N2pc amplitude flexibly and reversibly tracks value-based choices during the learning of reward associated stimulus colors. Specifically, successful learning of current value-contingencies was associated with reduced N2pc amplitudes, and this effect was more apparent for distractor processing, compared with target processing. In addition, following a value reversal the feedback related negativity(FRN), an ERP component that reflects feedback processing, was amplified and co-occurred with increased N2pc amplitudes in trials following low-value feedback. Importantly, participants that showed the greatest adjustment in N2pc amplitudes based on feedback were also the most efficient learners. These results allow further insight into how changes in attentional prioritization in an uncertain and volatile environment support flexible adjustments of behavior.

Stimulus preexposure speeds or slows subsequent acquisition of associative learning depending on learning test procedures and response measure

Prior exposure to a conditioned stimulus (CS) typically results in latent inhibition-slower acquisition of associative learning about that stimulus in subsequent training. Here, we found that CS preexposure had different effects on the appetitive conditioning of rats with a sucrose unconditioned stimulus (US) depending on training test procedures, the similarity of preexposure and training procedures, and the choice of response measure. Preexposure to a visual or an auditory stimulus produced facilitation of acquisition of food-cup-directed responding when both of those cues were (separately) paired with sucrose delivery in the training test (Experiments 1 and 3). By contrast, the same preexposure procedure resulted in latent inhibition of food-cup learning if the second stimulus in the test phase was of the same modality as the preexposed stimulus (Experiment 2). In Experiment 3, latent inhibition was enhanced if both phases included a single CS or both phases included both auditory and visual CSs, compared to treatments in which only one CS was presented in one phase but two CSs were presented in the other phase. In Experiment 4, preexposure of an auditory cue slowed subsequent learning about it if the context was salient but enhanced learning if the context was of weaker salience. Finally, a measure of general activity revealed latent inhibition after preexposure in all conditions in all 4 experiments. We discuss the results within several classes of latent inhibition theories, none of which provides a comprehensive account.

Estimating Dynamic Signals From Trial Data With Censored Values

Censored data occur commonly in trial-structured behavioral experiments and many other forms of longitudinal data. They can lead to severe bias and reduction of statistical power in subsequent analyses. Principled approaches for dealing with censored data, such as data imputation and methods based on the complete data's likelihood, work well for estimating fixed features of statistical models but have not been extended to dynamic measures, such as serial estimates of an underlying latent variable over time. Here we propose an approach to the censored-data problem for dynamic behavioral signals. We developed a state-space modeling framework with a censored observation process at the trial timescale. We then developed a filter algorithm to compute the posterior distribution of the state process using the available data. We showed that special cases of this framework can incorporate the three most common approaches to censored observations: ignoring trials with censored data, imputing the censored data values, or using the full information available in the data likelihood. Finally, we derived a computationally efficient approximate Gaussian filter that is similar in structure to a Kalman filter, but that efficiently accounts for censored data. We compared the performances of these methods in a simulation study and provide recommendations of approaches to use, based on the expected amount of censored data in an experiment. These new techniques can broadly be applied in many research domains in which censored data interfere with estimation, including survival analysis and other clinical trial applications.

A Predictive Coding Account of Psychotic Symptoms in Autism Spectrum Disorder

The co-occurrence of psychotic and autism spectrum disorder (ASD) symptoms represents an important clinical challenge. Here we consider this problem in the context of a computational psychiatry approach that has been applied to both conditions-predictive coding. Some symptoms of schizophrenia have been explained in terms of a failure of top-down predictions or an enhanced weighting of bottom-up prediction errors. Likewise, autism has been explained in terms of similar perturbations. We suggest that this theoretical overlap may explain overlapping symptomatology. Experimental evidence highlights meaningful distinctions and consistencies between these disorders. We hypothesize individuals with ASD may experience some degree of delusions without the presence of any additional impairment, but that hallucinations are likely indicative of a distinct process.