Dopamine-signaled reward predictions generated by competitive excitation and inhibition in a spiking neural network model

Learning to Produce Syllabic Speech Sounds via Reward-Modulated Neural Plasticity

At around 7 months of age, human infants begin to reliably produce well-formed syllables containing both consonants and vowels, a behavior called canonical babbling. Over subsequent months, the frequency of canonical babbling continues to increase. How the infant's nervous system supports the acquisition of this ability is unknown. Here we present a computational model that combines a spiking neural network, reinforcement-modulated spike-timing-dependent plasticity, and a human-like vocal tract to simulate the acquisition of canonical babbling. Like human infants, the model's frequency of canonical babbling gradually increases. The model is rewarded when it produces a sound that is more auditorily salient than sounds it has previously produced. This is consistent with data from human infants indicating that contingent adult responses shape infant behavior and with data from deaf and tracheostomized infants indicating that hearing, including hearing one's own vocalizations, is critical for canonical babbling development. Reward receipt increases the level of dopamine in the neural network. The neural network contains a reservoir with recurrent connections and two motor neuron groups, one agonist and one antagonist, which control the masseter and orbicularis oris muscles, promoting or inhibiting mouth closure. The model learns to increase the number of salient, syllabic sounds it produces by adjusting the base level of muscle activation and increasing their range of activity. Our results support the possibility that through dopamine-modulated spike-timing-dependent plasticity, the motor cortex learns to harness its natural oscillations in activity in order to produce syllabic sounds. It thus suggests that learning to produce rhythmic mouth movements for speech production may be supported by general cortical learning mechanisms. The model makes several testable predictions and has implications for our understanding not only of how syllabic vocalizations develop in infancy but also for our understanding of how they may have evolved.

Learning touch preferences with a tactile robot using dopamine modulated STDP in a model of insular cortex

Neurorobots enable researchers to study how behaviors are produced by neural mechanisms in an uncertain, noisy, real-world environment. To investigate how the somatosensory system processes noisy, real-world touch inputs, we introduce a neurorobot called CARL-SJR, which has a full-body tactile sensory area. The design of CARL-SJR is such that it encourages people to communicate with it through gentle touch. CARL-SJR provides feedback to users by displaying bright colors on its surface. In the present study, we show that CARL-SJR is capable of learning associations between conditioned stimuli (CS; a color pattern on its surface) and unconditioned stimuli (US; a preferred touch pattern) by applying a spiking neural network (SNN) with neurobiologically inspired plasticity. Specifically, we modeled the primary somatosensory cortex, prefrontal cortex, striatum, and the insular cortex, which is important for hedonic touch, to process noisy data generated directly from CARL-SJR's tactile sensory area. To facilitate learning, we applied dopamine-modulated Spike Timing Dependent Plasticity (STDP) to our simulated prefrontal cortex, striatum, and insular cortex. To cope with noisy, varying inputs, the SNN was tuned to produce traveling waves of activity that carried spatiotemporal information. Despite the noisy tactile sensors, spike trains, and variations in subject hand swipes, the learning was quite robust. Further, insular cortex activities in the incremental pathway of dopaminergic reward system allowed us to control CARL-SJR's preference for touch direction without heavily pre-processed inputs. The emerged behaviors we found in this model match animal's behaviors wherein they prefer touch in particular areas and directions. Thus, the results in this paper could serve as an explanation on the underlying neural mechanisms for developing tactile preferences and hedonic touch.

Improper activation of D1 and D2 receptors leads to excess noise in prefrontal cortex

The dopaminergic system has been shown to control the amount of noise in the prefrontal cortex (PFC) and likely plays an important role in working memory and the pathophysiology of schizophrenia. We developed a model that takes into account the known receptor distributions of D1 and D2 receptors, the changes these receptors have on neuron response properties, as well as identified circuitry involved in working memory. Our model suggests that D1 receptor under-stimulation in supragranular layers gates internal noise into the PFC leading to cognitive symptoms as has been proposed in attention disorders, while D2 over-stimulation gates noise into the PFC by over-activation of cortico-striatal projecting neurons in infragranular layers. We apply this model in the context of a memory-guided saccade paradigm and show deficits similar to those observed in schizophrenic patients. We also show set-shifting impairments similar to those observed in rodents with D1 and D2 receptor manipulations. We discuss how the introduction of noise through changes in D1 and D2 receptor activation may account for many of the symptoms of schizophrenia depending on where this dysfunction occurs in the PFC.

Implications of polychronous neuronal groups for the continuity of mind

Is conceptual space continuous? The answer to this question depends on how concepts are represented in the brain. Vector space representations, which ground conceptual states in the instantaneous firing rates of neurons, have successfully captured cognitive dynamics in a broad range of domains. There is a growing body of evidence, however, that conceptual information is encoded in spatiotemporal patterns of neural spikes, sometimes called polychronous neuronal groups (PNGs). The use of PNGs to represent conceptual states, rather than employing a continuous vector space, introduces new challenges, including issues of temporally extended representations, meaning through symbol grounding, compositionality, and representational similarity. In this article, we explore how PNGs support discontinuous transitions between concepts. While the continuous dynamics of vector space approaches require such transitions to activate intermediate and blended concepts, PNGs offer the means to change the activation of concepts discretely, introducing a form of conceptual dynamics unavailable to vector space models.

Timing and expectation of reward: a neuro-computational model of the afferents to the ventral tegmental area

Neural activity in dopaminergic areas such as the ventral tegmental area is influenced by timing processes, in particular by the temporal expectation of rewards during Pavlovian conditioning. Receipt of a reward at the expected time allows to compute reward-prediction errors which can drive learning in motor or cognitive structures. Reciprocally, dopamine plays an important role in the timing of external events. Several models of the dopaminergic system exist, but the substrate of temporal learning is rather unclear. In this article, we propose a neuro-computational model of the afferent network to the ventral tegmental area, including the lateral hypothalamus, the pedunculopontine nucleus, the amygdala, the ventromedial prefrontal cortex, the ventral basal ganglia (including the nucleus accumbens and the ventral pallidum), as well as the lateral habenula and the rostromedial tegmental nucleus. Based on a plausible connectivity and realistic learning rules, this neuro-computational model reproduces several experimental observations, such as the progressive cancelation of dopaminergic bursts at reward delivery, the appearance of bursts at the onset of reward-predicting cues or the influence of reward magnitude on activity in the amygdala and ventral tegmental area. While associative learning occurs primarily in the amygdala, learning of the temporal relationship between the cue and the associated reward is implemented as a dopamine-modulated coincidence detection mechanism in the nucleus accumbens.

Coping with unpredictability: dopaminergic and neurotrophic responses to omission of expected reward in Atlantic salmon (Salmo salar L.)

Comparative studies are imperative for understanding the evolution of adaptive neurobiological processes such as neural plasticity, cognition, and emotion. Previously we have reported that prolonged omission of expected rewards (OER, or 'frustrative nonreward') causes increased aggression in Atlantic salmon (Salmo salar). Here we report changes in brain monoaminergic activity and relative abundance of brain derived neurotrophic factor (BDNF) and dopamine receptor mRNA transcripts in the same paradigm. Groups of fish were initially conditioned to associate a flashing light with feeding. Subsequently, the expected food reward was delayed for 30 minutes during two out of three meals per day in the OER treatment, while the previously established routine was maintained in control groups. After 8 days there was no effect of OER on baseline brain stem serotonin (5-HT) or dopamine (DA) activity. Subsequent exposure to acute confinement stress led to increased plasma cortisol and elevated turnover of brain stem DA and 5-HT in all animals. The DA response was potentiated and DA receptor 1 (D1) mRNA abundance was reduced in the OER-exposed fish, indicating a sensitization of the DA system. In addition OER suppressed abundance of BDNF in the telencephalon of non-stressed fish. Regardless of OER treatment, a strong positive correlation between BDNF and D1 mRNA abundance was seen in non-stressed fish. This correlation was disrupted by acute stress, and replaced by a negative correlation between BDNF abundance and plasma cortisol concentration. These observations indicate a conserved link between DA, neurotrophin regulation, and corticosteroid-signaling pathways. The results also emphasize how fish models can be important tools in the study of neural plasticity and responsiveness to environmental unpredictability.

A dynamic, embodied paradigm to investigate the role of serotonin in decision-making

Serotonin (5-HT) is a neuromodulator that has been attributed to cost assessment and harm aversion. In this review, we look at the role 5-HT plays in making decisions when subjects are faced with potential harmful or costly outcomes. We review approaches for examining the serotonergic system in decision-making. We introduce our group’s paradigm used to investigate how 5-HT affects decision-making. In particular, our paradigm combines techniques from computational neuroscience, socioeconomic game theory, human–robot interaction, and Bayesian statistics. We will highlight key findings from our previous studies utilizing this paradigm, which helped expand our understanding of 5-HT’s effect on decision-making in relation to cost assessment. Lastly, we propose a cyclic multidisciplinary approach that may aid in addressing the complexity of exploring 5-HT and decision-making by iteratively updating our assumptions and models of the serotonergic system through exhaustive experimentation.

Neuromodulation of STDP through short-term changes in firing causality

Spike timing dependent plasticity (STDP) likely plays an important role in forming and changing connectivity patterns between neurons in our brain. In a unidirectional synaptic connection between two neurons, it uses the causal relation between spiking activity of a presynaptic input neuron and a postsynaptic output neuron to change the strength of this connection. While the nature of STDP benefits unsupervised learning of correlated inputs, any incorporation of value into the learning process needs some form of reinforcement. Chemical neuromodulators such as Dopamine or Acetylcholine are thought to signal changes between external reward and internal expectation to many brain regions, including the basal ganglia. This effect is often modelled through a direct inclusion of the level of Dopamine as a third factor into the STDP rule. While this gives the benefit of direct control over synaptic modification, it does not account for observed instantaneous effects in neuronal activity on application of Dopamine agonists. Specifically, an instant facilitation of neuronal excitability in the striatum can not be explained by the only indirect effect that dopamine-modulated STDP has on a neuron's firing pattern. We therefore propose a model for synaptic transmission where the level of neuromodulator does not directly influence synaptic plasticity, but instead alters the relative firing causality between pre- and postsynaptic neurons. Through the direct effect on postsynaptic activity, our rule allows indirect modulation of the learning outcome even with unmodulated, two-factor STDP. However, it also does not prohibit joint operation together with three-factor STDP rules.

An interoceptive predictive coding model of conscious presence

We describe a theoretical model of the neurocognitive mechanisms underlying conscious presence and its disturbances. The model is based on interoceptive prediction error and is informed by predictive models of agency, general models of hierarchical predictive coding and dopaminergic signaling in cortex, the role of the anterior insular cortex (AIC) in interoception and emotion, and cognitive neuroscience evidence from studies of virtual reality and of psychiatric disorders of presence, specifically depersonalization/derealization disorder. The model associates presence with successful suppression by top-down predictions of informative interoceptive signals evoked by autonomic control signals and, indirectly, by visceral responses to afferent sensory signals. The model connects presence to agency by allowing that predicted interoceptive signals will depend on whether afferent sensory signals are determined, by a parallel predictive-coding mechanism, to be self-generated or externally caused. Anatomically, we identify the AIC as the likely locus of key neural comparator mechanisms. Our model integrates a broad range of previously disparate evidence, makes predictions for conjoint manipulations of agency and presence, offers a new view of emotion as interoceptive inference, and represents a step toward a mechanistic account of a fundamental phenomenological property of consciousness.

A reinforcement learning framework for spiking networks with dynamic synapses

An integration of both the Hebbian-based and reinforcement learning (RL) rules is presented for dynamic synapses. The proposed framework permits the Hebbian rule to update the hidden synaptic model parameters regulating the synaptic response rather than the synaptic weights. This is performed using both the value and the sign of the temporal difference in the reward signal after each trial. Applying this framework, a spiking network with spike-timing-dependent synapses is tested to learn the exclusive-OR computation on a temporally coded basis. Reward values are calculated with the distance between the output spike train of the network and a reference target one. Results show that the network is able to capture the required dynamics and that the proposed framework can reveal indeed an integrated version of Hebbian and RL. The proposed framework is tractable and less computationally expensive. The framework is applicable to a wide class of synaptic models and is not restricted to the used neural representation. This generality, along with the reported results, supports adopting the introduced approach to benefit from the biologically plausible synaptic models in a wide range of intuitive signal processing.