Acetylcholine-modulated plasticity in reward-driven navigation: a computational study

Brain-inspired learning rules for spiking neural network-based control: a tutorial

Robotic systems rely on spatio-temporal information to solve control tasks. With advancements in deep neural networks, reinforcement learning has significantly enhanced the performance of control tasks by leveraging deep learning techniques. However, as deep neural networks grow in complexity, they consume more energy and introduce greater latency. This complexity hampers their application in robotic systems that require real-time data processing. To address this issue, spiking neural networks, which emulate the biological brain by transmitting spatio-temporal information through spikes, have been developed alongside neuromorphic hardware that supports their operation. This paper reviews brain-inspired learning rules and examines the application of spiking neural networks in control tasks. We begin by exploring the features and implementations of biologically plausible spike-timing-dependent plasticity. Subsequently, we investigate the integration of a global third factor with spike-timing-dependent plasticity and its utilization and enhancements in both theoretical and applied research. We also discuss a method for locally applying a third factor that sophisticatedly modifies each synaptic weight through weight-based backpropagation. Finally, we review studies utilizing these learning rules to solve control tasks using spiking neural networks.

Adverse event profile of memantine and donepezil combination therapy: a real-world pharmacovigilance analysis based on FDA adverse event reporting system (FAERS) data from 2004 to 2023

Background

Donepezil in combination with memantine is a widely used clinical therapy for moderate to severe dementia. However, real-world population data on the long-term safety of donepezil in combination with memantine are incomplete and variable. Therefore, the aim of this study was to analyze the adverse events (AEs) of donepezil in combination with memantine according to US Food and Drug Administration Adverse Event Reporting System (FAERS) data to provide evidence for the safety monitoring of this therapy.

Methods

We retrospectively analyzed reports of AEs associated with the combination of donepezil and memantine from 2004 to 2023 extracted from the FAERS database. Whether there was a significant association between donepezil and memantine combination therapy and AEs was assessed using four disproportionality analysis methods, namely, the reporting odds ratio, proportional reporting ratio, Bayesian confidence propagation neural network, and multi-item gamma Poisson shrinker methods. To further investigate potential safety issues, we also analyzed differences and similarities in the time of onset and incidence of AEs stratified by sex and differences and similarities in the incidence of AEs stratified by age.

Results

Of the 2,400 adverse drug reaction (ADR) reports in which the combination of donepezil and memantine was the primary suspected drug, most of the affected patients were female (54.96%) and older than 65 years of age (79.08%). We identified 22 different system organ classes covering 100 AEs, including some common AEs such as dizziness and electrocardiogram PR prolongation; fall, pleurothotonus and myoclonus were AEs that were not listed on the drug label. Moreover, we obtained 88 reports of AEs in men and 100 reports of AEs in women; somnolence was a common AE in both men and women and was more common in women, whereas pleurothotonus was a more common AE in men. In addition, we analyzed 12 AEs in patients younger than 18 years, 16 in patients between 18 and 65 years, and 113 in patients older than 65 years. The three age groups had distinctive AEs, but lethargy was the common AE among all age groups. Finally, the median time to AE onset was 19 days in all cases. In both men and women, most AEs occurred within a month of starting donepezil plus memantine, but some continued after a year of treatment.

Conclusion

Our study identified potential and new AEs of donepezil in combination with memantine; some of these AEs were the same as in the specification, and some of the AE signals were not shown in the specification. In addition, there were sex and age differences in some of the AEs. Therefore, our findings may provide valuable insights for further studies on the safety of donepezil and memantine combination therapy, which are expected to contribute to the safe use of this therapy in clinical practice.

Cell-type-specific cholinergic control of granular retrosplenial cortex with implications for angular velocity coding across brain states

Cholinergic receptor activation enables the persistent firing of cortical pyramidal neurons, providing a key cellular basis for theories of spatial navigation involving working memory, path integration, and head direction encoding. The granular retrosplenial cortex (RSG) is important for spatially-guided behaviors, but how acetylcholine impacts RSG neurons is unknown. Here, we show that a transcriptomically, morphologically, and biophysically distinct RSG cell-type - the low-rheobase (LR) neuron - has a very distinct expression profile of cholinergic muscarinic receptors compared to all other neighboring excitatory neuronal subtypes. LR neurons do not fire persistently in response to cholinergic agonists, in stark contrast to all other principal neuronal subtypes examined within the RSG and across midline cortex. This lack of persistence allows LR neuron models to rapidly compute angular head velocity (AHV), independent of cholinergic changes seen during navigation. Thus, LR neurons can consistently compute AHV across brain states, highlighting the specialized RSG neural codes supporting navigation.

The study of plasticity has always been about gradients

The experimental study of learning and plasticity has always been driven by an implicit question: how can physiological changes be adaptive and improve performance? For example, in Hebbian plasticity only synapses from presynaptic neurons that were active are changed, avoiding useless changes. Similarly, in dopamine-gated learning synapse changes depend on reward or lack thereof and do not change when everything is predictable. Within machine learning we can make the question of which changes are adaptive concrete: performance improves when changes correlate with the gradient of an objective function quantifying performance. This result is general for any system that improves through small changes. As such, physiology has always implicitly been seeking mechanisms that allow the brain to approximate gradients. Coming from this perspective we review the existing literature on plasticity-related mechanisms, and we show how these mechanisms relate to gradient estimation. We argue that gradients are a unifying idea to explain the many facets of neuronal plasticity.

Meta-SpikePropamine: learning to learn with synaptic plasticity in spiking neural networks

We propose that in order to harness our understanding of neuroscience toward machine learning, we must first have powerful tools for training brain-like models of learning. Although substantial progress has been made toward understanding the dynamics of learning in the brain, neuroscience-derived models of learning have yet to demonstrate the same performance capabilities as methods in deep learning such as gradient descent. Inspired by the successes of machine learning using gradient descent, we introduce a bi-level optimization framework that seeks to both solve online learning tasks and improve the ability to learn online using models of plasticity from neuroscience. We demonstrate that models of three-factor learning with synaptic plasticity taken from the neuroscience literature can be trained in Spiking Neural Networks (SNNs) with gradient descent via a framework of learning-to-learn to address challenging online learning problems. This framework opens a new path toward developing neuroscience inspired online learning algorithms.

Brain inspired path planning algorithms for drones

Introduction

With the development of artificial intelligence and brain science, brain-inspired navigation and path planning has attracted widespread attention.

Methods

In this paper, we present a place cell based path planning algorithm that utilizes spiking neural network (SNN) to create efficient routes for drones. First, place cells are characterized by the leaky integrate-and-fire (LIF) neuron model. Then, the connection weights between neurons are trained by spike-timing-dependent plasticity (STDP) learning rules. Afterwards, a synaptic vector field is created to avoid obstacles and to find the shortest path.

Results

Finally, simulation experiments both in a Python simulation environment and in an Unreal Engine environment are conducted to evaluate the validity of the algorithms.

Discussion

Experiment results demonstrate the validity, its robustness and the computational speed of the proposed model.

Cholinergic activity reflects reward expectations and predicts behavioral responses

Basal forebrain cholinergic neurons (BFCNs) play an important role in associative learning, suggesting that BFCNs may participate in processing stimuli that predict future outcomes. However, the impact of outcome probabilities on BFCN activity remained elusive. Therefore, we performed bulk calcium imaging and recorded spiking of identified cholinergic neurons from the basal forebrain of mice performing a probabilistic Pavlovian cued outcome task. BFCNs responded more to sensory cues that were often paired with reward. Reward delivery also activated BFCNs, with surprising rewards eliciting a stronger response, whereas punishments evoked uniform positive-going responses. We propose that BFCNs differentially weigh predictions of positive and negative reinforcement, reflecting divergent relative salience of forecasting appetitive and aversive outcomes, partially explained by a simple reinforcement learning model of a valence-weighed unsigned prediction error. Finally, the extent of cue-driven cholinergic activation predicted subsequent decision speed, suggesting that the expectation-gated cholinergic firing is instructive to reward-seeking behaviors.

Dopamine and serotonin interplay for valence-based spatial learning

Dopamine (DA) and serotonin (5-HT) are important neuromodulators of synaptic plasticity that have been linked to learning from positive or negative outcomes or valence-based learning. In the hippocampus, both affect long-term plasticity but play different roles in encoding uncertainty or predicted reward. DA has been related to positive valence, from reward consumption or avoidance behavior, and 5-HT to aversive encoding. We propose DA produces overall LTP while 5-HT elicits LTD. Here, we compare two reward-modulated spike timing-dependent plasticity (R-STDP) rules to describe the action of these neuromodulators. We examined their role in cognitive performance and flexibility for computational models of the Morris water maze task and reversal learning. Our results show that the interplay of DA and 5-HT improves learning performance and can explain experimental evidence. This study reinforces the importance of neuromodulation in determining the direction of plasticity.

A Nonlinear Hidden Layer Enables Actor-Critic Agents to Learn Multiple Paired Association Navigation

Navigation to multiple cued reward locations has been increasingly used to study rodent learning. Though deep reinforcement learning agents have been shown to be able to learn the task, they are not biologically plausible. Biologically plausible classic actor-critic agents have been shown to learn to navigate to single reward locations, but which biologically plausible agents are able to learn multiple cue-reward location tasks has remained unclear. In this computational study, we show versions of classic agents that learn to navigate to a single reward location, and adapt to reward location displacement, but are not able to learn multiple paired association navigation. The limitation is overcome by an agent in which place cell and cue information are first processed by a feedforward nonlinear hidden layer with synapses to the actor and critic subject to temporal difference error-modulated plasticity. Faster learning is obtained when the feedforward layer is replaced by a recurrent reservoir network.

The functional role of sequentially neuromodulated synaptic plasticity in behavioural learning

To survive, animals have to quickly modify their behaviour when the reward changes. The internal representations responsible for this are updated through synaptic weight changes, mediated by certain neuromodulators conveying feedback from the environment. In previous experiments, we discovered a form of hippocampal Spike-Timing-Dependent-Plasticity (STDP) that is sequentially modulated by acetylcholine and dopamine. Acetylcholine facilitates synaptic depression, while dopamine retroactively converts the depression into potentiation. When these experimental findings were implemented as a learning rule in a computational model, our simulations showed that cholinergic-facilitated depression is important for reversal learning. In the present study, we tested the model's prediction by optogenetically inactivating cholinergic neurons in mice during a hippocampus-dependent spatial learning task with changing rewards. We found that reversal learning, but not initial place learning, was impaired, verifying our computational prediction that acetylcholine-modulated plasticity promotes the unlearning of old reward locations. Further, differences in neuromodulator concentrations in the model captured mouse-by-mouse performance variability in the optogenetic experiments. Our line of work sheds light on how neuromodulators enable the learning of new contingencies.

Cannabidiol attenuated the maintenance and reinstatement of extinguished methylphenidate-induced conditioned place preference in rats

Methylphenidate (MPH) is a mild CNS stimulant that has been used in hyperactive children, and patients with neurodegenerative and major depressive disorders. Exposure to MPH-associated cues enhances craving and arousal in drug users. On the other hand, cannabidiol (CBD) has antipsychotic potential that might be useful in alleviating symptoms of drug addiction. The aim of this study was to investigate the effect of CBD administration on extinction and reinstatement of MPH-induced conditioning place preference (CPP) in rats. Male rats received MPH (1, 2.5 or 5 mg/kg, i.p) or morphine (5 or 10 mg/kg, s.c.) during the conditioning phase. Following the establishment of CPP, during extinction training, 60 min prior to every CPP session, animals were given daily ICV CBD (10 or 50 μg/5 μL), vehicle alone (DMSO) 10 % or were treatment-naïve. On the reinstatement day animals after receiving the initial dose of MPH, 0.5 mg/kg, and were placed into the CPP box to evaluate the CPP scoring for 10-min. Our findings indicated that morphine (5 and 10 mg/kg; s.c.) and MPH (1 and 2.5 mg/kg; i.p.) induced a CPP. The ICV administration of both doses of CBD (10 and 50 μg/5 μL) prevented the reinstatement of MPH-induced CPP, which displayed shorter extinction latency compared to treatment-naïve or DMSO 10 % groups. Therefore, CBD's site of action is a potential target for reducing the risk of MPH relapse; however, more investigation is required.

Muscarinic Regulation of Spike Timing Dependent Synaptic Plasticity in the Hippocampus

Long-term changes in synaptic transmission between neurons in the brain are considered the cellular basis of learning and memory. Over the last few decades, many studies have revealed that the precise order and timing of activity between pre- and post-synaptic cells ("spike-timing-dependent plasticity; STDP") is crucial for the sign and magnitude of long-term changes at many central synapses. Acetylcholine (ACh) via the recruitment of diverse muscarinic receptors is known to influence STDP in a variety of ways, enabling flexibility and adaptability in brain network activity during complex behaviors. In this review, we will summarize and discuss different mechanistic aspects of muscarinic modulation of timing-dependent plasticity at both excitatory and inhibitory synapses in the hippocampus to shape learning and memory.

Dopamine agonist treatment increases sensitivity to gamble outcomes in the hippocampus in de novo Parkinson's disease

BACKGROUND

Parkinson's disease is associated with severe nigro-striatal dopamine depletion, leading to motor dysfunction and altered reward processing. We previously showed that drug-naïve patients with Parkinson's disease had a consistent attenuation of reward signalling in the mesolimbic and mesocortical system. Here, we address the neurobiological effects of dopaminergic therapy on reward sensitivity in the mesolimbic circuitry, and how this may contribute to neuropsychiatric symptoms.

OBJECTIVES

We tested the hypothesis that (1) dopaminergic treatment would restore the attenuated, mesolimbic and mesocortical responses to reward; and (2) restoration of reward responsivity by dopaminergic treatment would predict motor performance and the emergence of impulse control symptoms.

METHODS

In 11 drug-naïve Parkinson patients, we prospectively assessed treatment-induced changes in reward processing before, and eight weeks after initiation of monotherapy with dopamine agonists. They were compared to 10 non-medicated healthy controls who were also measured longitudinally. We used whole-brain functional magnetic resonance imaging at 3 Tesla to assess the reward responsivity of the brain to monetary gains and losses, while participants performed a simple consequential gambling task.

RESULTS

In patients, dopaminergic treatment improved clinical motor symptoms without significantly changing task performance. Dopamine agonist therapy induced a stronger reward responsivity in the right hippocampus with higher doses being less effective. None of the patients developed impulse control disorders in the follow-up period of four years.

CONCLUSIONS

Short-term treatment with first-ever dopaminergic medication partially restores deficient reward-related processing in the hippocampus in de novo Parkinson's disease.