Neuroethology and theoretical neurobiology

A neural network model of adaptively timed reinforcement learning and hippocampal dynamics

A neural model is described of how adaptively timed reinforcement learning occurs. The adaptive timing circuit is suggested to exist in the hippocampus, and to involve convergence of dentate granule cells on CA3 pyramidal cells, and N-methyl-D-aspartate (NMDA) receptors. This circuit forms part of a model neural system for the coordinated control of recognition learning, reinforcement learning, and motor learning, whose properties clarify how an animal can learn to acquire a delayed reward. Behavioral and neural data are summarized in support of each processing stage of the system. The relevant anatomical sites are in thalamus, neocortex, hippocampus, hypothalamus, amygdala and cerebellum. Cerebellar influences on motor learning are distinguished from hippocampal influences on adaptive timing of reinforcement learning. The model simulates how damage to the hippocampal formation disrupts adaptive timing, eliminates attentional blocking and causes symptoms of medial temporal amnesia. Properties of learned expectations, attentional focussing, memory search and orienting reactions to novel events are used to analyze the blocking and amnesia data. The model also suggests how normal acquisition of subcortical emotional conditioning can occur after cortical ablation, even though extinction of emotional conditioning is retarded by cortical ablation. The model simulates how increasing the duration of an unconditioned stimulus increases the amplitude of emotional conditioning, but does not change adaptive timing; and how an increase in the intensity of a conditioned stimulus 'speeds up the clock', but an increase in the intensity of an unconditioned stimulus does not. Computer simulations of the model fit parametric conditioning data, including a Weber law property and an inverted U property. Both primary and secondary adaptively timed conditionings are simulated, as are data concerning conditioning using multiple interstimulus intervals (ISIs), gradually or abruptly changing ISIs, partial reinforcement and multiple stimuli that lead to time-averaging of responses. Neurobiologically testable predictions are made to facilitate further tests of the model.

Neural dynamics of attentionally modulated Pavlovian conditioning: blocking, interstimulus interval, and secondary reinforcement

Selective information processing in neural networks is studied through computer simulations of Pavlovian conditioning data. The model reproduces properties of blocking, inverted-U in learning as a function of interstimulus interval, anticipatory conditioned responses, secondary reinforcement, attentional focusing by conditioned motivational feedback, and limited capacity short-term memory processing. Conditioning occurs from sensory to drive representations (conditioned reinforcer learning), from drive to sensory representations (incentive motivational learning), and from sensory to motor representations (habit learning).The conditionable pathwas contain long-term memory traces that obey a non-Hebbian associative law. The neural model embodies a solution to two key design problems of conditioning, the synchronization and persistence problems. This model of vertebrate learning is compared with data and models of invertebrate learning. Predictions derived from models of vertebrate learning are compared with data about invertebrate learning, including data from Aplysia about facilitator neurons and data from Hermissenda about voltage-dependent Ca(2+) currents. A prediction is stated about classical conditioning in all species, called the secondary conditioning alternative, and if confirmed would constitute an evolutionary invariant of learning.