An effect of serotonergic stimulation on learning rates for rewards apparent after long intertrial intervals

Dopamine transients encode reward prediction errors independent of learning rates

Biological accounts of reinforcement learning posit that dopamine encodes reward prediction errors (RPEs), which are multiplied by a learning rate to update state or action values. These values are thought to be represented by corticostriatal synaptic weights, which are updated by dopamine-dependent plasticity. This suggests that dopamine release reflects the product of the learning rate and RPE. Here, we characterize dopamine encoding of learning rates in the nucleus accumbens core (NAcc) in a volatile environment. Using a task with semi-observable states offering different rewards, we find that rats adjust how quickly they initiate trials across states using RPEs. Computational modeling and behavioral analyses show that learning rates are higher following state transitions and scale with trial-by-trial changes in beliefs about hidden states, approximating normative Bayesian strategies. Notably, dopamine release in the NAcc encodes RPEs independent of learning rates, suggesting that dopamine-independent mechanisms instantiate dynamic learning rates.

Investigating Transfer Learning in Noisy Environments: A Study of Predecessor and Successor Features in Spatial Learning Using a T-Maze

In this study, we investigate the adaptability of artificial agents within a noisy T-maze that use Markov decision processes (MDPs) and successor feature (SF) and predecessor feature (PF) learning algorithms. Our focus is on quantifying how varying the hyperparameters, specifically the reward learning rate (αr) and the eligibility trace decay rate (λ), can enhance their adaptability. Adaptation is evaluated by analyzing the hyperparameters of cumulative reward, step length, adaptation rate, and adaptation step length and the relationships between them using Spearman's correlation tests and linear regression. Our findings reveal that an αr of 0.9 consistently yields superior adaptation across all metrics at a noise level of 0.05. However, the optimal setting for λ varies by metric and context. In discussing these results, we emphasize the critical role of hyperparameter optimization in refining the performance and transfer learning efficacy of learning algorithms. This research advances our understanding of the functionality of PF and SF algorithms, particularly in navigating the inherent uncertainty of transfer learning tasks. By offering insights into the optimal hyperparameter configurations, this study contributes to the development of more adaptive and robust learning algorithms, paving the way for future explorations in artificial intelligence and neuroscience.

Monoaminergic Modulation of Learning and Cognitive Function in the Prefrontal Cortex

Extensive research has shed light on the cellular and functional underpinnings of higher cognition as influenced by the prefrontal cortex. Neurotransmitters act as key regulatory molecules within the PFC to assist with synchronizing cognitive state and arousal levels. The monoamine family of neurotransmitters, including dopamine, serotonin, and norepinephrine, play multifaceted roles in the cognitive processes behind learning and memory. The present review explores the organization and signaling patterns of monoamines within the PFC, as well as elucidates the numerous roles played by monoamines in learning and higher cognitive function.

Using machine learning to predict judgments on Western visual art along content-representational and formal-perceptual attributes

Art research has long aimed to unravel the complex associations between specific attributes, such as color, complexity, and emotional expressiveness, and art judgments, including beauty, creativity, and liking. However, the fundamental distinction between attributes as inherent characteristics or features of the artwork and judgments as subjective evaluations remains an exciting topic. This paper reviews the literature of the last half century, to identify key attributes, and employs machine learning, specifically Gradient Boosted Decision Trees (GBDT), to predict 13 art judgments along 17 attributes. Ratings from 78 art novice participants were collected for 54 Western artworks. Our GBDT models successfully predicted 13 judgments significantly. Notably, judged creativity and disturbing/irritating judgments showed the highest predictability, with the models explaining 31% and 32% of the variance, respectively. The attributes emotional expressiveness, valence, symbolism, as well as complexity emerged as consistent and significant contributors to the models' performance. Content-representational attributes played a more prominent role than formal-perceptual attributes. Moreover, we found in some cases non-linear relationships between attributes and judgments with sudden inclines or declines around medium levels of the rating scales. By uncovering these underlying patterns and dynamics in art judgment behavior, our research provides valuable insights to advance the understanding of aesthetic experiences considering visual art, inform cultural practices, and inspire future research in the field of art appreciation.

Behavioral plasticity

Behavioral plasticity allows animals to modulate their behavior based on experience and environmental conditions. Caenorhabditis elegans exhibits experience-dependent changes in its behavioral responses to various modalities of sensory cues, including odorants, salts, temperature, and mechanical stimulations. Most of these forms of behavioral plasticity, such as adaptation, habituation, associative learning, and imprinting, are shared with other animals. The C. elegans nervous system is considerably tractable for experimental studies-its function can be characterized and manipulated with molecular genetic methods, its activity can be visualized and analyzed with imaging approaches, and the connectivity of its relatively small number of neurons are well described. Therefore, C. elegans provides an opportunity to study molecular, neuronal, and circuit mechanisms underlying behavioral plasticity that are either conserved in other animals or unique to this species. These findings reveal insights into how the nervous system interacts with the environmental cues to generate behavioral changes with adaptive values.

HMM for discovering decision-making dynamics using reinforcement learning experiments

Major depressive disorder (MDD), a leading cause of years of life lived with disability, presents challenges in diagnosis and treatment due to its complex and heterogeneous nature. Emerging evidence indicates that reward processing abnormalities may serve as a behavioral marker for MDD. To measure reward processing, patients perform computer-based behavioral tasks that involve making choices or responding to stimulants that are associated with different outcomes, such as gains or losses in the laboratory. Reinforcement learning (RL) models are fitted to extract parameters that measure various aspects of reward processing (e.g. reward sensitivity) to characterize how patients make decisions in behavioral tasks. Recent findings suggest the inadequacy of characterizing reward learning solely based on a single RL model; instead, there may be a switching of decision-making processes between multiple strategies. An important scientific question is how the dynamics of strategies in decision-making affect the reward learning ability of individuals with MDD. Motivated by the probabilistic reward task within the Establishing Moderators and Biosignatures of Antidepressant Response in Clinical Care (EMBARC) study, we propose a novel RL-HMM (hidden Markov model) framework for analyzing reward-based decision-making. Our model accommodates decision-making strategy switching between two distinct approaches under an HMM: subjects making decisions based on the RL model or opting for random choices. We account for continuous RL state space and allow time-varying transition probabilities in the HMM. We introduce a computationally efficient Expectation-maximization (EM) algorithm for parameter estimation and use a nonparametric bootstrap for inference. Extensive simulation studies validate the finite-sample performance of our method. We apply our approach to the EMBARC study to show that MDD patients are less engaged in RL compared to the healthy controls, and engagement is associated with brain activities in the negative affect circuitry during an emotional conflict task.

5-HT 2A and 5-HT 2C receptor antagonism differentially modulate reinforcement learning and cognitive flexibility: behavioural and computational evidence

RATIONALE

Cognitive flexibility, the ability to adapt behaviour in response to a changing environment, is disrupted in several neuropsychiatric disorders, including obsessive-compulsive disorder and major depressive disorder. Evidence suggests that flexibility, which can be operationalised using reversal learning tasks, is modulated by serotonergic transmission. However, how exactly flexible behaviour and associated reinforcement learning (RL) processes are modulated by 5-HT action on specific receptors is unknown.

OBJECTIVES

We investigated the effects of 5-HT2A receptor (5-HT2AR) and 5-HT2C receptor (5-HT2CR) antagonism on flexibility and underlying RL mechanisms.

METHODS

Thirty-six male Lister hooded rats were trained on a touchscreen visual discrimination and reversal task. We evaluated the effects of systemic treatments with the 5-HT2AR and 5-HT2CR antagonists M100907 and SB-242084, respectively, on reversal learning and performance on probe trials where correct and incorrect stimuli were presented with a third, probabilistically rewarded, stimulus. Computational models were fitted to task choice data to extract RL parameters, including a novel model designed specifically for this task.

RESULTS

5-HT2AR antagonism impaired reversal learning only after an initial perseverative phase, during a period of random choice and then new learning. 5-HT2CR antagonism, on the other hand, impaired learning from positive feedback. RL models further differentiated these effects. 5-HT2AR antagonism decreased punishment learning rate (i.e. negative feedback) at high and low doses. The low dose also decreased reinforcement sensitivity (beta) and increased stimulus and side stickiness (i.e., the tendency to repeat a choice regardless of outcome). 5-HT2CR antagonism also decreased beta, but reduced side stickiness.

CONCLUSIONS

These data indicate that 5-HT2A and 5-HT2CRs both modulate different aspects of flexibility, with 5-HT2ARs modulating learning from negative feedback as measured using RL parameters and 5-HT2CRs for learning from positive feedback assessed through conventional measures.

Regulation of reinforcement learning parameters captures long-term changes in rat behaviour

In uncertain environments in which resources fluctuate continuously, animals must permanently decide whether to stabilise learning and exploit what they currently believe to be their best option, or instead explore potential alternatives and learn fast from new observations. While such a trade-off has been extensively studied in pretrained animals facing non-stationary decision-making tasks, it is yet unknown how they progressively tune it while learning the task structure during pretraining. Here, we compared the ability of different computational models to account for long-term changes in the behaviour of 24 rats while they learned to choose a rewarded lever in a three-armed bandit task across 24 days of pretraining. We found that the day-by-day evolution of rat performance and win-shift tendency revealed a progressive stabilisation of the way they regulated reinforcement learning parameters. We successfully captured these behavioural adaptations using a meta-learning model in which either the learning rate or the inverse temperature was controlled by the average reward rate.

Methamphetamine-induced adaptation of learning rate dynamics depend on baseline performance

The ability to calibrate learning according to new information is a fundamental component of an organism's ability to adapt to changing conditions. Yet, the exact neural mechanisms guiding dynamic learning rate adjustments remain unclear. Catecholamines appear to play a critical role in adjusting the degree to which we use new information over time, but individuals vary widely in the manner in which they adjust to changes. Here, we studied the effects of a low dose of methamphetamine (MA), and individual differences in these effects, on probabilistic reversal learning dynamics in a within-subject, double-blind, randomized design. Participants first completed a reversal learning task during a drug-free baseline session to provide a measure of baseline performance. Then they completed the task during two sessions, one with MA (20 mg oral) and one with placebo (PL). First, we showed that, relative to PL, MA modulates the ability to dynamically adjust learning from prediction errors. Second, this effect was more pronounced in participants who performed poorly at baseline. These results present novel evidence for the involvement of catecholaminergic transmission on learning flexibility and highlights that baseline performance modulates the effect of the drug.

Dorsal raphe neurons integrate the values of reward amount, delay, and uncertainty in multi-attribute decision-making

The dorsal raphe nucleus (DRN) is implicated in psychiatric disorders that feature impaired sensitivity to reward amount, impulsivity when facing reward delays, and risk-seeking when confronting reward uncertainty. However, it has been unclear whether and how DRN neurons signal reward amount, reward delay, and reward uncertainty during multi-attribute value-based decision-making, where subjects consider these attributes to make a choice. We recorded DRN neurons as monkeys chose between offers whose attributes, namely expected reward amount, reward delay, and reward uncertainty, varied independently. Many DRN neurons signaled offer attributes, and this population tended to integrate the attributes in a manner that reflected monkeys' preferences for amount, delay, and uncertainty. After decision-making, in response to post-decision feedback, these same neurons signaled signed reward prediction errors, suggesting a broader role in tracking value across task epochs and behavioral contexts. Our data illustrate how the DRN participates in value computations, guiding theories about the role of the DRN in decision-making and psychiatric disease.

Sleep apnoea, gut dysbiosis and cognitive dysfunction

Sleep disorders are becoming increasingly common, and their distinct effects on physical and mental health require elaborate investigation. Gut dysbiosis (GD) has been reported in sleep-related disorders, but sleep apnoea is of particular significance because of its higher prevalence and chronicity. Cumulative evidence has suggested a link between sleep apnoea and GD. This review highlights the gut-brain communication axis that is mediated via commensal microbes and various microbiota-derived metabolites (e.g. short-chain fatty acids, lipopolysaccharide and trimethyl amine N-oxide), neurotransmitters (e.g. γ-aminobutyric acid, serotonin, glutamate and dopamine), immune cells and inflammatory mediators, as well as the vagus nerve and hypothalamic-pituitary-adrenal axis. This review also discusses the pathological role underpinning GD and altered gut bacterial populations in sleep apnoea and its related comorbid conditions, particularly cognitive dysfunction. In addition, the review examines the preclinical and clinical evidence, which suggests that prebiotics and probiotics may potentially be beneficial in sleep apnoea and its comorbidities through restoration of eubiosis or gut microbial homeostasis that regulates neural, metabolic and immune responses, as well as physiological barrier integrity via the gut-brain axis.

Optogenetic activation of dorsal raphe serotonin neurons induces brain-wide activation

Serotonin is a neuromodulator that affects multiple behavioral and cognitive functions. Nonetheless, how serotonin causes such a variety of effects via brain-wide projections and various receptors remains unclear. Here we measured brain-wide responses to optogenetic stimulation of serotonin neurons in the dorsal raphe nucleus (DRN) of the male mouse brain using functional MRI with an 11.7 T scanner and a cryoprobe. Transient activation of DRN serotonin neurons caused brain-wide activation, including the medial prefrontal cortex, the striatum, and the ventral tegmental area. The same stimulation under anesthesia with isoflurane decreased brain-wide activation, including the hippocampal complex. These brain-wide response patterns can be explained by DRN serotonergic projection topography and serotonin receptor expression profiles, with enhanced weights on 5-HT1 receptors. Together, these results provide insight into the DR serotonergic system, which is consistent with recent discoveries of its functions in adaptive behaviors.

Active reinforcement learning versus action bias and hysteresis: control with a mixture of experts and nonexperts

Active reinforcement learning enables dynamic prediction and control, where one should not only maximize rewards but also minimize costs such as of inference, decisions, actions, and time. For an embodied agent such as a human, decisions are also shaped by physical aspects of actions. Beyond the effects of reward outcomes on learning processes, to what extent can modeling of behavior in a reinforcement-learning task be complicated by other sources of variance in sequential action choices? What of the effects of action bias (for actions per se) and action hysteresis determined by the history of actions chosen previously? The present study addressed these questions with incremental assembly of models for the sequential choice data from a task with hierarchical structure for additional complexity in learning. With systematic comparison and falsification of computational models, human choices were tested for signatures of parallel modules representing not only an enhanced form of generalized reinforcement learning but also action bias and hysteresis. We found evidence for substantial differences in bias and hysteresis across participants-even comparable in magnitude to the individual differences in learning. Individuals who did not learn well revealed the greatest biases, but those who did learn accurately were also significantly biased. The direction of hysteresis varied among individuals as repetition or, more commonly, alternation biases persisting from multiple previous actions. Considering that these actions were button presses with trivial motor demands, the idiosyncratic forces biasing sequences of action choices were robust enough to suggest ubiquity across individuals and across tasks requiring various actions. In light of how bias and hysteresis function as a heuristic for efficient control that adapts to uncertainty or low motivation by minimizing the cost of effort, these phenomena broaden the consilient theory of a mixture of experts to encompass a mixture of expert and nonexpert controllers of behavior.

Comparable roles for serotonin in rats and humans for computations underlying flexible decision-making

Serotonin is critical for adapting behavior flexibly to meet changing environmental demands. Cognitive flexibility is important for successful attainment of goals, as well as for social interactions, and is frequently impaired in neuropsychiatric disorders, including obsessive-compulsive disorder. However, a unifying mechanistic framework accounting for the role of serotonin in behavioral flexibility has remained elusive. Here, we demonstrate common effects of manipulating serotonin function across two species (rats and humans) on latent processes supporting choice behavior during probabilistic reversal learning, using computational modelling. The findings support a role of serotonin in behavioral flexibility and plasticity, indicated, respectively, by increases or decreases in choice repetition ('stickiness') or reinforcement learning rates following manipulations intended to increase or decrease serotonin function. More specifically, the rate at which expected value increased following reward and decreased following punishment (reward and punishment 'learning rates') was greatest after sub-chronic administration of the selective serotonin reuptake inhibitor (SSRI) citalopram (5 mg/kg for 7 days followed by 10 mg/kg twice a day for 5 days) in rats. Conversely, humans given a single dose of an SSRI (20 mg escitalopram), which can decrease post-synaptic serotonin signalling, and rats that received the neurotoxin 5,7-dihydroxytryptamine (5,7-DHT), which destroys forebrain serotonergic neurons, exhibited decreased reward learning rates. A basic perseverative tendency ('stickiness'), or choice repetition irrespective of the outcome produced, was likewise increased in rats after the 12-day SSRI regimen and decreased after single dose SSRI in humans and 5,7-DHT in rats. These common effects of serotonergic manipulations on rats and humans-identified via computational modelling-suggest an evolutionarily conserved role for serotonin in plasticity and behavioral flexibility and have clinical relevance transdiagnostically for neuropsychiatric disorders.

From reinforcement learning to agency: Frameworks for understanding basal cognition

Organisms play, explore, and mimic those around them. Is there a purpose to this behavior? Are organisms just behaving, or are they trying to achieve goals? We believe this is a false dichotomy. To that end, to understand organisms, we attempt to unify two approaches for understanding complex agents, whether evolved or engineered. We argue that formalisms describing multiscale competencies and goal-directedness in biology (e.g., TAME), and reinforcement learning (RL), can be combined in a symbiotic framework. While RL has been largely focused on higher-level organisms and robots of high complexity, TAME is naturally capable of describing lower-level organisms and minimal agents as well. We propose several novel questions that come from using RL/TAME to understand biology as well as ones that come from using biology to formulate new theory in AI. We hope that the research programs proposed in this piece shape future efforts to understand biological organisms and also future efforts to build artificial agents.

Distinct value computations support rapid sequential decisions

The value of the environment determines animals' motivational states and sets expectations for error-based learning1-3. How are values computed? Reinforcement learning systems can store or cache values of states or actions that are learned from experience, or they can compute values using a model of the environment to simulate possible futures3. These value computations have distinct trade-offs, and a central question is how neural systems decide which computations to use or whether/how to combine them4-8. Here we show that rats use distinct value computations for sequential decisions within single trials. We used high-throughput training to collect statistically powerful datasets from 291 rats performing a temporal wagering task with hidden reward states. Rats adjusted how quickly they initiated trials and how long they waited for rewards across states, balancing effort and time costs against expected rewards. Statistical modeling revealed that animals computed the value of the environment differently when initiating trials versus when deciding how long to wait for rewards, even though these decisions were only seconds apart. Moreover, value estimates interacted via a dynamic learning rate. Our results reveal how distinct value computations interact on rapid timescales, and demonstrate the power of using high-throughput training to understand rich, cognitive behaviors.

Effect of lysergic acid diethylamide (LSD) on reinforcement learning in humans

BACKGROUND

The non-selective serotonin 2A (5-HT2A) receptor agonist lysergic acid diethylamide (LSD) holds promise as a treatment for some psychiatric disorders. Psychedelic drugs such as LSD have been suggested to have therapeutic actions through their effects on learning. The behavioural effects of LSD in humans, however, remain incompletely understood. Here we examined how LSD affects probabilistic reversal learning (PRL) in healthy humans.

METHODS

Healthy volunteers received intravenous LSD (75 μg in 10 mL saline) or placebo (10 mL saline) in a within-subjects design and completed a PRL task. Participants had to learn through trial and error which of three stimuli was rewarded most of the time, and these contingencies switched in a reversal phase. Computational models of reinforcement learning (RL) were fitted to the behavioural data to assess how LSD affected the updating ('learning rates') and deployment of value representations ('reinforcement sensitivity') during choice, as well as 'stimulus stickiness' (choice repetition irrespective of reinforcement history).

RESULTS

Raw data measures assessing sensitivity to immediate feedback ('win-stay' and 'lose-shift' probabilities) were unaffected, whereas LSD increased the impact of the strength of initial learning on perseveration. Computational modelling revealed that the most pronounced effect of LSD was the enhancement of the reward learning rate. The punishment learning rate was also elevated. Stimulus stickiness was decreased by LSD, reflecting heightened exploration. Reinforcement sensitivity differed by phase.

CONCLUSIONS

Increased RL rates suggest LSD induced a state of heightened plasticity. These results indicate a potential mechanism through which revision of maladaptive associations could occur in the clinical application of LSD.

Dorsal raphe neurons signal integrated value during multi-attribute decision-making

The dorsal raphe nucleus (DRN) is implicated in psychiatric disorders that feature impaired sensitivity to reward amount, impulsivity when facing reward delays, and risk-seeking when grappling with reward uncertainty. However, whether and how DRN neurons signal reward amount, reward delay, and reward uncertainty during multi-attribute value-based decision-making, where subjects consider all these attributes to make a choice, is unclear. We recorded DRN neurons as monkeys chose between offers whose attributes, namely expected reward amount, reward delay, and reward uncertainty, varied independently. Many DRN neurons signaled offer attributes. Remarkably, these neurons commonly integrated offer attributes in a manner that reflected monkeys' overall preferences for amount, delay, and uncertainty. After decision-making, in response to post-decision feedback, these same neurons signaled signed reward prediction errors, suggesting a broader role in tracking value across task epochs and behavioral contexts. Our data illustrate how DRN participates in integrated value computations, guiding theories of DRN in decision-making and psychiatric disease.

Machine learning revealed symbolism, emotionality, and imaginativeness as primary predictors of creativity evaluations of western art paintings

Creativity is a compelling yet elusive phenomenon, especially when manifested in visual art, where its evaluation is often a subjective and complex process. Understanding how individuals judge creativity in visual art is a particularly intriguing question. Conventional linear approaches often fail to capture the intricate nature of human behavior underlying such judgments. Therefore, in this study, we employed interpretable machine learning to probe complex associations between 17 subjective art-attributes and creativity judgments across a diverse range of artworks. A cohort of 78 non-art expert participants assessed 54 artworks varying in styles and motifs. The applied Random Forests regressor models accounted for 30% of the variability in creativity judgments given our set of art-attributes. Our analyses revealed symbolism, emotionality, and imaginativeness as the primary attributes influencing creativity judgments. Abstractness, valence, and complexity also had an impact, albeit to a lesser degree. Notably, we observed non-linearity in the relationship between art-attribute scores and creativity judgments, indicating that changes in art-attributes did not consistently correspond to changes in creativity judgments. Employing statistical learning, this investigation presents the first attribute-integrating quantitative model of factors that contribute to creativity judgments in visual art among novice raters. Our research represents a significant stride forward building the groundwork for first causal models for future investigations in art and creativity research and offering implications for diverse practical applications. Beyond enhancing comprehension of the intricate interplay and specificity of attributes used in evaluating creativity, this work introduces machine learning as an innovative approach in the field of subjective judgment.

Distinct value computations support rapid sequential decisions

The value of the environment determines animals' motivational states and sets expectations for error-based learning1-3. How are values computed? Reinforcement learning systems can store or "cache" values of states or actions that are learned from experience, or they can compute values using a model of the environment to simulate possible futures3. These value computations have distinct trade-offs, and a central question is how neural systems decide which computations to use or whether/how to combine them4-8. Here we show that rats use distinct value computations for sequential decisions within single trials. We used high-throughput training to collect statistically powerful datasets from 291 rats performing a temporal wagering task with hidden reward states. Rats adjusted how quickly they initiated trials and how long they waited for rewards across states, balancing effort and time costs against expected rewards. Statistical modeling revealed that animals computed the value of the environment differently when initiating trials versus when deciding how long to wait for rewards, even though these decisions were only seconds apart. Moreover, value estimates interacted via a dynamic learning rate. Our results reveal how distinct value computations interact on rapid timescales, and demonstrate the power of using high-throughput training to understand rich, cognitive behaviors.

Canalization and plasticity in psychopathology

This theoretical article revives a classical bridging construct, canalization, to describe a new model of a general factor of psychopathology. To achieve this, we have distinguished between two types of plasticity, an early one that we call 'TEMP' for 'Temperature or Entropy Mediated Plasticity', and another, we call 'canalization', which is close to Hebbian plasticity. These two forms of plasticity can be most easily distinguished by their relationship to 'precision' or inverse variance; TEMP relates to increased model variance or decreased precision, whereas the opposite is true for canalization. TEMP also subsumes increased learning rate, (Ising) temperature and entropy. Dictionary definitions of 'plasticity' describe it as the property of being easily shaped or molded; TEMP is the better match for this. Importantly, we propose that 'pathological' phenotypes develop via mechanisms of canalization or increased model precision, as a defensive response to adversity and associated distress or dysphoria. Our model states that canalization entrenches in psychopathology, narrowing the phenotypic state-space as the agent develops expertise in their pathology. We suggest that TEMP - combined with gently guiding psychological support - can counter canalization. We address questions of whether and when canalization is adaptive versus maladaptive, furnish our model with references to basic and human neuroscience, and offer concrete experiments and measures to test its main hypotheses and implications. This article is part of the Special Issue on "National Institutes of Health Psilocybin Research Speaker Series".

Temporal derivative computation in the dorsal raphe network revealed by an experimentally driven augmented integrate-and-fire modeling framework

By means of an expansive innervation, the serotonin (5-HT) neurons of the dorsal raphe nucleus (DRN) are positioned to enact coordinated modulation of circuits distributed across the entire brain in order to adaptively regulate behavior. Yet the network computations that emerge from the excitability and connectivity features of the DRN are still poorly understood. To gain insight into these computations, we began by carrying out a detailed electrophysiological characterization of genetically identified mouse 5-HT and somatostatin (SOM) neurons. We next developed a single-neuron modeling framework that combines the realism of Hodgkin-Huxley models with the simplicity and predictive power of generalized integrate-and-fire models. We found that feedforward inhibition of 5-HT neurons by heterogeneous SOM neurons implemented divisive inhibition, while endocannabinoid-mediated modulation of excitatory drive to the DRN increased the gain of 5-HT output. Our most striking finding was that the output of the DRN encodes a mixture of the intensity and temporal derivative of its input, and that the temporal derivative component dominates this mixture precisely when the input is increasing rapidly. This network computation primarily emerged from prominent adaptation mechanisms found in 5-HT neurons, including a previously undescribed dynamic threshold. By applying a bottom-up neural network modeling approach, our results suggest that the DRN is particularly apt to encode input changes over short timescales, reflecting one of the salient emerging computations that dominate its output to regulate behavior.

Regulation of social hierarchy learning by serotonin transporter availability

Learning one's status in a group is a fundamental process in building social hierarchies. Although animal studies suggest that serotonin (5-HT) signaling modulates learning social hierarchies, direct evidence in humans is lacking. Here we determined the relationship between serotonin transporter (SERT) availability and brain systems engaged in learning social ranks combining computational approaches with simultaneous PET-fMRI acquisition in healthy males. We also investigated the link between SERT availability and brain activity in a non-social control condition involving learning the payoffs of slot machines. Learning social ranks was modulated by the dorsal raphe nucleus (DRN) 5-HT function. BOLD ventral striatal response, tracking the rank of opponents, decreased with DRN SERT levels. Moreover, this link was specific to the social learning task. These findings demonstrate that 5-HT plays an influence on the computations required to learn social ranks.

Region-Specific Characteristics of Astrocytes and Microglia: A Possible Involvement in Aging and Diseases

Although different regions of the brain are dedicated to specific functions, the intra- and inter-regional heterogeneity of astrocytes and microglia in these regions has not yet been fully understood. Recently, an advancement in various technologies, such as single-cell RNA sequencing, has allowed for the discovery of astrocytes and microglia with distinct molecular fingerprints and varying functions in the brain. In addition, the regional heterogeneity of astrocytes and microglia exhibits different functions in several situations, such as aging and neurodegenerative diseases. Therefore, investigating the region-specific astrocytes and microglia is important in understanding the overall function of the brain. In this review, we summarize up-to-date research on various intra- and inter-regional heterogeneities of astrocytes and microglia, and provide information on how they can be applied to aging and neurodegenerative diseases.

Information normally considered task-irrelevant drives decision-making and affects premotor circuit recruitment

Decision-making is a continuous and dynamic process with prior experience reflected in and used by the brain to guide adaptive behavior. However, most neurobiological studies constrain behavior and/or analyses to task-related variables, not accounting for the continuous internal and temporal space in which they occur. We show mice rely on information learned through recent and longer-term experience beyond just prior actions and reward - including checking behavior and the passage of time - to guide self-initiated, self-paced, and self-generated actions. These experiences are represented in secondary motor cortex (M2) activity and its projections into dorsal medial striatum (DMS). M2 integrates this information to bias strategy-level decision-making, and DMS projections reflect specific aspects of this recent experience to guide actions. This suggests diverse aspects of experience drive decision-making and its neural representation, and shows premotor corticostriatal circuits are crucial for using selective aspects of experiential information to guide adaptive behavior.

Neuromodulation and Neurophysiology on the Timescale of Learning and Decision-Making

Nervous systems evolved to effectively navigate the dynamics of the environment to achieve their goals. One framework used to study this fundamental problem arose in the study of learning and decision-making. In this framework, the demands of effective behavior require slow dynamics-on the scale of seconds to minutes-of networks of neurons. Here, we review the phenomena and mechanisms involved. Using vignettes from a few species and areas of the nervous system, we view neuromodulators as key substrates for temporal scaling of neuronal dynamics. Expected final online publication date for the Annual Review of Neuroscience, Volume 45 is July 2022. Please see http://www.annualreviews.org/page/journal/pubdates for revised estimates.

Forgetting generates a novel state that is reactivatable

Forgetting is defined as a time-dependent decline of a memory. However, it is not clear whether forgetting reverses the learning process to return the brain to the naive state. Here, using the aversive olfactory learning of pathogenic bacteria in C. elegans, we show that forgetting generates a novel state of the nervous system that is distinct from the naive state or the learned state. A transient exposure to the training condition or training odorants reactivates this novel state to elicit the previously learned behavior. An AMPA receptor and a type II serotonin receptor act in the central neuron of the learning circuit to decrease and increase the speed to reach this novel state, respectively. Together, our study systematically characterizes forgetting and uncovers conserved mechanisms underlying the rate of forgetting.

Complementary roles of serotonergic and cholinergic systems in decisions about when to act

Decision-making not only involves deciding about which action to choose but when and whether to initiate an action in the first place. Macaque monkeys tracked number of dots on a screen and could choose when to make a response. The longer the animals waited before responding, the more dots appeared on the screen and the higher the probability of reward. Monkeys waited longer before making a response when a trial’s value was less than the environment’s average value. Recordings of brain activity with fMRI revealed that activity in dorsal raphe nucleus (DRN)—a key source of serotonin (5-HT)—tracked average value of the environment. By contrast, activity in the basal forebrain (BF)—an important source of acetylcholine (ACh)—was related to decision time to act as a function of immediate and recent past context. Interactions between DRN and BF and the anterior cingulate cortex (ACC), another region with action initiation-related activity, occurred as a function of the decision time to act. Next, we performed two psychopharmacological studies. Manipulating systemic 5-HT by citalopram prolonged the time macaques waited to respond for a given opportunity. This effect was more evident during blocks with long inter-trial intervals (ITIs) where good opportunities were sparse. Manipulating systemic acetylcholine (ACh) by rivastigmine reduced the time macaques waited to respond given the immediate and recent past context, a pattern opposite to the effect observed with 5-HT. These findings suggest complementary roles for serotonin/DRN and acetylcholine/BF in decisions about when to initiate an action.

•

Both immediate context and wider environment influence decisions about when to act

•

DRN and 5-HT mediate the influence of wider environment

•

BF and ACh mediate the influence of immediate context

Serotonin neurons modulate learning rate through uncertainty

Regulating how fast to learn is critical for flexible behavior. Learning about the consequences of actions should be slow in stable environments, but accelerate when that environment changes. Recognizing stability and detecting change are difficult in environments with noisy relationships between actions and outcomes. Under these conditions, theories propose that uncertainty can be used to modulate learning rates ("meta-learning"). We show that mice behaving in a dynamic foraging task exhibit choice behavior that varied as a function of two forms of uncertainty estimated from a meta-learning model. The activity of dorsal raphe serotonin neurons tracked both types of uncertainty in the foraging task as well as in a dynamic Pavlovian task. Reversible inhibition of serotonin neurons in the foraging task reproduced changes in learning predicted by a simulated lesion of meta-learning in the model. We thus provide a quantitative link between serotonin neuron activity, learning, and decision making.

Mice alternate between discrete strategies during perceptual decision-making

Classical models of perceptual decision-making assume that subjects use a single, consistent strategy to form decisions, or that decision-making strategies evolve slowly over time. Here we present new analyses suggesting that this common view is incorrect. We analyzed data from mouse and human decision-making experiments and found that choice behavior relies on an interplay among multiple interleaved strategies. These strategies, characterized by states in a hidden Markov model, persist for tens to hundreds of trials before switching, and often switch multiple times within a session. The identified decision-making strategies were highly consistent across mice and comprised a single 'engaged' state, in which decisions relied heavily on the sensory stimulus, and several biased states in which errors frequently occurred. These results provide a powerful alternate explanation for 'lapses' often observed in rodent behavioral experiments, and suggest that standard measures of performance mask the presence of major changes in strategy across trials.

Neurocomputational mechanisms underlying the subjective value of information

Humans have a striking desire to actively seek new information, even when it is devoid of any instrumental utility. However, the mechanisms that drive individuals' subjective preference for information remain unclear. Here, we used fMRI to examine the processing of subjective information value, by having participants decide how much effort they were willing to trade-off for non-instrumental information. We showed that choices were best described by a model that accounted for: (1) the variability in individuals' estimates of uncertainty, (2) their desire to reduce that uncertainty, and (3) their subjective preference for positively valenced information. Model-based analyses revealed the anterior cingulate as a key node that encodes the subjective value of information across multiple stages of decision-making - including when information was prospectively valued, and when the outcome was definitively delivered. These findings emphasise the multidimensionality of information value, and reveal the neurocomputational mechanisms underlying the variability in individuals' desire to physically pursue informative outcomes.

Relative salience signaling within a thalamo-orbitofrontal circuit governs learning rate

Learning to predict rewards is essential for the sustained fitness of animals. Contemporary views suggest that such learning is driven by a reward prediction error (RPE)-the difference between received and predicted rewards. The magnitude of learning induced by an RPE is proportional to the product of the RPE and a learning rate. Here we demonstrate using two-photon calcium imaging and optogenetics in mice that certain functionally distinct subpopulations of ventral/medial orbitofrontal cortex (vmOFC) neurons signal learning rate control. Consistent with learning rate control, trial-by-trial fluctuations in vmOFC activity positively correlate with behavioral updating when the RPE is positive, and negatively correlates with behavioral updating when the RPE is negative. Learning rate is affected by many variables including the salience of a reward. We found that the average reward response of these neurons signals the relative salience of a reward, because it decreases after reward prediction learning or the introduction of another highly salient aversive stimulus. The relative salience signaling in vmOFC is sculpted by medial thalamic inputs. These results support emerging theoretical views that prefrontal cortex encodes and controls learning parameters.

Dynamics of history-dependent perceptual judgment

Identical physical inputs do not always evoke identical percepts. To investigate the role of stimulus history in tactile perception, we designed a task in which rats had to judge each vibrissal vibration, in a long series, as strong or weak depending on its mean speed. After a low-speed stimulus (trial n - 1), rats were more likely to report the next stimulus (trial n) as strong, and after a high-speed stimulus, they were more likely to report the next stimulus as weak, a repulsive effect that did not depend on choice or reward on trial n - 1. This effect could be tracked over several preceding trials (i.e., n - 2 and earlier) and was characterized by an exponential decay function, reflecting a trial-by-trial incorporation of sensory history. Surprisingly, the influence of trial n - 1 strengthened as the time interval between n - 1 and n grew. Human subjects receiving fingertip vibrations showed these same key findings. We are able to account for the repulsive stimulus history effect, and its detailed time scale, through a single-parameter model, wherein each new stimulus gradually updates the subject's decision criterion. This model points to mechanisms underlying how the past affects the ongoing subjective experience.

Knockout of the Serotonin Transporter in the Rat Mildly Modulates Decisional Anhedonia

Serotonin transporter gene variance has long been considered an essential factor contributing to depression. However, meta-analyses yielded inconsistent findings recently, asking for further understanding of the link between the gene and depression-related symptoms. One key feature of depression is anhedonia. While data exist on the effect of serotonin transporter gene knockout (5-HTT^-/-) in rodents on consummatory and anticipatory anhedonia, with mixed outcomes, the effect on decisional anhedonia has not been investigated thus far. Here, we tested whether 5-HTT^-/- contributes to decisional anhedonia. To this end, we established a novel touchscreen-based go/go task of visual decision-making. During the learning of stimulus discrimination, 5-HTT^+/+ rats performed more optimal decision-making compared to 5-HTT^-/- rats at the beginning, but this difference did not persist throughout the learning period. During stimulus generalization, the generalization curves were similar between both genotypes and did not alter as the learning progress. Interestingly, the response time in 5-HTT^+/+ rats increased as the session increased in general, while 5-HTT^-/- rats tended to decrease. The response time difference might indicate that 5-HTT^-/- rats altered willingness to exert cognitive effort to the categorization of generalization stimuli. These results suggest that the effect of 5-HTT ablation on decisional anhedonia is mild and interacts with learning, explaining the discrepant findings on the link between 5-HTT gene and depression.

Pivotal mental states

This paper introduces a new construct, the 'pivotal mental state', which is defined as a hyper-plastic state aiding rapid and deep learning that can mediate psychological transformation. We believe this new construct bears relevance to a broad range of psychological and psychiatric phenomena. We argue that pivotal mental states serve an important evolutionary function, that is, to aid psychological transformation when actual or perceived environmental pressures demand this. We cite evidence that chronic stress and neurotic traits are primers for a pivotal mental state, whereas acute stress can be a trigger. Inspired by research with serotonin 2A receptor agonist psychedelics, we highlight how activity at this particular receptor can robustly and reliably induce pivotal mental states, but we argue that the capacity for pivotal mental states is an inherent property of the human brain itself. Moreover, we hypothesize that serotonergic psychedelics hijack a system that has evolved to mediate rapid and deep learning when its need is sensed. We cite a breadth of evidences linking stress via a variety of inducers, with an upregulated serotonin 2A receptor system (e.g. upregulated availability of and/or binding to the receptor) and acute stress with 5-HT release, which we argue can activate this primed system to induce a pivotal mental state. The pivotal mental state model is multi-level, linking a specific molecular gateway (increased serotonin 2A receptor signaling) with the inception of a hyper-plastic brain and mind state, enhanced rate of associative learning and the potential mediation of a psychological transformation.

The Roles of Serotonin in Neuropsychiatric Disorders

The serotonergic system extends throughout the central nervous system (CNS) and the gastrointestinal (GI) tract. In the CNS, serotonin (5-HT, 5-hydroxytryptamine) modulates a broad spectrum of functions, including mood, cognition, anxiety, learning, memory, reward processing, and sleep. These processes are mediated through 5-HT binding to 5-HT receptors (5-HTRs), are classified into seven distinct groups. Deficits in the serotonergic system can result in various pathological conditions, particularly depression, schizophrenia, mood disorders, and autism. In this review, we outlined the complexity of serotonergic modulation of physiologic and pathologic processes. Moreover, we provided experimental and clinical evidence of 5-HT's involvement in neuropsychiatric disorders and discussed the molecular mechanisms that underlie these illnesses and contribute to the new therapies.

Time elapsed between choices in a probabilistic task correlates with repeating the same decision

Reinforcement learning causes an action that yields a positive outcome more likely to be taken in the future. Here, we investigate how the time elapsed from an action affects subsequent decisions. Groups of C57BL6/J mice were housed in IntelliCages with access to water and chow ad libitum; they also had access to bottles with a reward: saccharin solution, alcohol, or a mixture of the two. The probability of receiving a reward in two of the cage corners changed between 0.9 and 0.3 every 48 hr over a period of ~33 days. As expected, in most animals, the odds of repeating a corner choice were increased if that choice was previously rewarded. Interestingly, the time elapsed from the previous choice also influenced the probability of repeating the choice, and this effect was independent of previous outcome. Behavioral data were fitted to a series of reinforcement learning models. Best fits were achieved when the reward prediction update was coupled with separate learning rates from positive and negative outcomes and additionally a “fictitious” update of the expected value of the nonselected choice. Additional inclusion of a time‐dependent decay of the expected values improved the fit marginally in some cases.

Dynamic decision making and value computations in medial frontal cortex

Dynamic decision making requires an intact medial frontal cortex. Recent work has combined theory and single-neuron measurements in frontal cortex to advance models of decision making. We review behavioral tasks that have been used to study dynamic decision making and algorithmic models of these tasks using reinforcement learning theory. We discuss studies linking neurophysiology and quantitative decision variables. We conclude with hypotheses about the role of other cortical and subcortical structures in dynamic decision making, including ascending neuromodulatory systems.

Knockout serotonin transporter in rats moderates outcome and stimulus generalization

Understanding the common dimension of mental disorders (such as anxiety, depression, and drug addiction) might contribute to the construction of biological frameworks (Research Domain Criteria, RDoC) for novel ways of treatment. One common dimension at the behavioral level observed across these disorders is a generalization. Testing generalization in serotonin transporter (5-HTT) knockout (KO) rats, an animal model showing depression/anxiety-like behaviors and drug addiction-like behaviors, could therefore provide more insights into this framework. We tested the outcome and stimulus generalization in wild-type (WT) and 5-HTT KO rats. Using a newly established touchscreen-based task, subjects directly responded to visual stimuli (Gabor patch images). We measured the response time and outcome in a precise manner. We found that 5-HTT KO rats processed visual information faster than WT rats during outcome generalization. Interestingly, during stimulus generalization, WT rats gradually responded faster to the stimuli as the sessions progressed, while 5-HTT KO rats responded faster than WT in the initial sessions and did not change significantly as the sessions progressed. This observation suggests that KO rats, compared to WT rats, may be less able to update changes in information. Taken together, KO 5-HTT modulates information processing when the environment changes.

The Role of Tryptophan and Tyrosine in Executive Function and Reward Processing

The serotonergic precursor tryptophan and the dopaminergic precursor tyrosine have been shown to be important modulators of mood, behaviour and cognition. Specifically, research on the function of tryptophan has characterised this molecule as particularly relevant in the context of pathological disorders such as depression. Moreover, a large body of evidence has now been accumulated to suggest that tryptophan may also be involved in executive function and reward processing. Despite some clear differentiation with tryptophan, the data reviewed in this paper illustrates that tyrosine shares similar functions with tryptophan in the regulation of executive function and reward, and that these processes in turn, rather than acting in isolation, causally influence each other.

Serotonergic Innervations of the Orbitofrontal and Medial-prefrontal Cortices are Differentially Involved in Visual Discrimination and Reversal Learning in Rats

Cross-species studies have identified an evolutionarily conserved role for serotonin in flexible behavior including reversal learning. The aim of the current study was to investigate the contribution of serotonin within the orbitofrontal cortex (OFC) and medial prefrontal cortex (mPFC) to visual discrimination and reversal learning. Male Lister Hooded rats were trained to discriminate between a rewarded (A+) and a nonrewarded (B−) visual stimulus to receive sucrose rewards in touchscreen operant chambers. Serotonin was depleted using surgical infusions of 5,7-dihydroxytryptamine (5,7-DHT), either globally by intracebroventricular (i.c.v.) infusions or locally by microinfusions into the OFC or mPFC. Rats that received i.c.v. infusions of 5,7-DHT before initial training were significantly impaired during both visual discrimination and subsequent reversal learning during which the stimulus–reward contingencies were changed (A− vs. B+). Local serotonin depletion from the OFC impaired reversal learning without affecting initial discrimination. After mPFC depletion, rats were unimpaired during reversal learning but slower to respond at the stimuli during all the stages; the mPFC group was also slower to learn during discrimination than the OFC group. These findings extend our understanding of serotonin in cognitive flexibility by revealing differential effects within two subregions of the prefrontal cortex in visual discrimination and reversal learning.

Sub-second Dopamine and Serotonin Signaling in Human Striatum during Perceptual Decision-Making

Recent animal research indicates that dopamine and serotonin, neuromodulators traditionally linked to appetitive and aversive processes, are also involved in sensory inference and decisions based on such inference. We tested this hypothesis in humans by monitoring sub-second striatal dopamine and serotonin signaling during a visual motion discrimination task that separates sensory uncertainty from decision difficulty in a factorial design. Caudate nucleus recordings (n = 4) revealed multi-scale encoding: in three participants, serotonin tracked sensory uncertainty, and, in one participant, both dopamine and serotonin tracked deviations from expected trial transitions within our factorial design. Putamen recordings (n = 1) supported a cognition-action separation between caudate nucleus and putamen—a striatal sub-division unique to primates—with both dopamine and serotonin tracking decision times. These first-of-their-kind observations in the human brain reveal a role for sub-second dopamine and serotonin signaling in non-reward-based aspects of cognition and action.

•

Dopamine and serotonin are measured in human striatum during awake decision-making

•

Serotonin tracks sensory uncertainty in caudate nucleus

•

Dopamine and serotonin track sensory statistics in caudate nucleus

•

Dopamine and serotonin track decision times in putamen

Better Safe Than Sorry: A Common Signature of General Vulnerability for Psychopathology

Several labels, such as neuroticism, negative emotionality, and dispositional negativity, indicate a broad dimension of psychopathology. However, largely separate, often disorder-specific research lines have developed that focus on different cognitive and affective characteristics that are associated with this dimension, such as perseverative cognition (worry, rumination), reduced autobiographical memory specificity, compromised fear learning, and enhanced somatic-symptom reporting. In this article, we present a theoretical perspective within a predictive-processing framework in which we trace these phenotypically different characteristics back to a common underlying “better-safe-than-sorry” processing strategy. This implies information processing that tends to be low in sensory-perceptual detail, which allows threat-related categorical priors to dominate conscious experience and for chronic uncertainty/surprise because of a stagnated error-reduction process. This common information-processing strategy has beneficial effects in the short term but important costs in the long term. From this perspective, we suggest that the phenomenally distinct cognitive and affective psychopathological characteristics mentioned above represent the same basic processing heuristic of the brain and are only different in relation to the particular type of information involved (e.g., in working memory, in autobiographical memory, in the external and internal world). Clinical implications of this view are discussed.

Psychological mechanisms and functions of 5-HT and SSRIs in potential therapeutic change: Lessons from the serotonergic modulation of action selection, learning, affect, and social cognition

Uncertainty regarding which psychological mechanisms are fundamental in mediating SSRI treatment outcomes and wide-ranging variability in their efficacy has raised more questions than it has solved. Since subjective mood states are an abstract scientific construct, only available through self-report in humans, and likely involving input from multiple top-down and bottom-up signals, it has been difficult to model at what level SSRIs interact with this process. Converging translational evidence indicates a role for serotonin in modulating context-dependent parameters of action selection, affect, and social cognition; and concurrently supporting learning mechanisms, which promote adaptability and behavioural flexibility. We examine the theoretical basis, ecological validity, and interaction of these constructs and how they may or may not exert a clinical benefit. Specifically, we bridge crucial gaps between disparate lines of research, particularly findings from animal models and human clinical trials, which often seem to present irreconcilable differences. In determining how SSRIs exert their effects, our approach examines the endogenous functions of 5-HT neurons, how 5-HT manipulations affect behaviour in different contexts, and how their therapeutic effects may be exerted in humans - which may illuminate issues of translational models, hierarchical mechanisms, idiographic variables, and social cognition.

Beyond dichotomies in reinforcement learning

Reinforcement learning (RL) is a framework of particular importance to psychology, neuroscience and machine learning. Interactions between these fields, as promoted through the common hub of RL, has facilitated paradigm shifts that relate multiple levels of analysis in a singular framework (for example, relating dopamine function to a computationally defined RL signal). Recently, more sophisticated RL algorithms have been proposed to better account for human learning, and in particular its oft-documented reliance on two separable systems: a model-based (MB) system and a model-free (MF) system. However, along with many benefits, this dichotomous lens can distort questions, and may contribute to an unnecessarily narrow perspective on learning and decision-making. Here, we outline some of the consequences that come from overconfidently mapping algorithms, such as MB versus MF RL, with putative cognitive processes. We argue that the field is well positioned to move beyond simplistic dichotomies, and we propose a means of refocusing research questions towards the rich and complex components that comprise learning and decision-making.

Global reward state affects learning and activity in raphe nucleus and anterior insula in monkeys

People and other animals learn the values of choices by observing the contingencies between them and their outcomes. However, decisions are not guided by choice-linked reward associations alone; macaques also maintain a memory of the general, average reward rate - the global reward state - in an environment. Remarkably, global reward state affects the way that each choice outcome is valued and influences future decisions so that the impact of both choice success and failure is different in rich and poor environments. Successful choices are more likely to be repeated but this is especially the case in rich environments. Unsuccessful choices are more likely to be abandoned but this is especially likely in poor environments. Functional magnetic resonance imaging (fMRI) revealed two distinct patterns of activity, one in anterior insula and one in the dorsal raphe nucleus, that track global reward state as well as specific outcome events.