Simulation of anticipatory responses in classical conditioning by a neuron-like adaptive element

Goal-directed learning in adolescence: neurocognitive development and contextual influences

Adolescence is a time during which we transition to independence, explore new activities and begin pursuit of major life goals. Goal-directed learning, in which we learn to perform actions that enable us to obtain desired outcomes, is central to many of these processes. Currently, our understanding of goal-directed learning in adolescence is itself in a state of transition, with the scientific community grappling with inconsistent results. When we examine metrics of goal-directed learning through the second decade of life, we find that many studies agree there are steady gains in performance in the teenage years, but others report that adolescent goal-directed learning is already adult-like, and some find adolescents can outperform adults. To explain the current variability in results, sophisticated experimental designs are being applied to test learning in different contexts. There is also increasing recognition that individuals of different ages and in different states will draw on different neurocognitive systems to support goal-directed learning. Through adoption of more nuanced approaches, we can be better prepared to recognize and harness adolescent strengths and to decipher the purpose (or goals) of adolescence itself.

Correlation of cue-locked FRN and feedback-locked FRN in the auditory monetary incentive delay task

Reflecting the discrepancy between received and predicted outcomes, the reward prediction error (RPE) plays an important role in learning in a dynamic environment. A number of studies suggested that the feedback-related negativity (FRN) component of an event-related potential, known to be associated with unexpected outcomes, encodes RPEs. While FRN was clearly shown to be sensitive to the probability of outcomes, the effect of outcome magnitude on FRN remains to be further clarified. In studies on the neural underpinnings of reward anticipation and outcome evaluation, a monetary incentive delay (MID) task proved to be particularly useful. We investigated whether feedback-locked FRN and cue-locked dN200 responses recorded during an auditory MID task were sensitive to the probability and magnitude of outcomes. The cue-locked dN200 is associated with the update of information about the magnitude of prospective outcomes. Overall, we showed that feedback-locked FRN was modulated by both the magnitude and the probability of outcomes during an auditory version of MID task, whereas no such effect was found for cue-locked dN200. Furthermore, the cue-locked dN200, which is associated with the update of information about the magnitude of prospective outcomes, correlated with the standard feedback-locked FRN, which is associated with a negative RPE. These results further expand our knowledge on the interplay between the processing of predictive cues that forecast future outcomes and the subsequent revision of these predictions during outcome delivery.

Anticipatory reward processing in addicted populations: a focus on the monetary incentive delay task

Advances in brain imaging techniques have allowed neurobiological research to temporally analyze signals coding for the anticipation of reward. In addicted populations, both hyporesponsiveness and hyperresponsiveness of brain regions (e.g., ventral striatum) implicated in drug effects and reward system processing have been reported during anticipation of generalized reward. We discuss the current state of knowledge of reward processing in addictive disorders from a widely used and validated task: the monetary incentive delay task. Only studies applying the monetary incentive delay task in addicted and at-risk adult populations are reviewed, with a focus on anticipatory processing and striatal regions activated during task performance as well as the relationship of these regions with individual difference (e.g., impulsivity) and treatment outcome variables. We further review drug influences in challenge studies as a means to examine acute influences on reward processing in abstinent, recreationally using, and addicted populations. Generalized reward processing in addicted and at-risk populations is often characterized by divergent anticipatory signaling in the ventral striatum. Although methodologic and task variations may underlie some discrepant findings, anticipatory signaling in the ventral striatum may also be influenced by smoking status, drug metabolites, and treatment status in addicted populations. Divergent results across abstinent, recreationally using, and addicted populations demonstrate complexities in interpreting findings. Future studies would benefit from focusing on characterizing how impulsivity and other addiction-related features relate to anticipatory striatal signaling over time. Additionally, identifying how anticipatory signals recover or adjust after protracted abstinence will be important in understanding recovery processes.

SSCC TD: a serial and simultaneous configural-cue compound stimuli representation for temporal difference learning

This paper presents a novel representational framework for the Temporal Difference (TD) model of learning, which allows the computation of configural stimuli--cumulative compounds of stimuli that generate perceptual emergents known as configural cues. This Simultaneous and Serial Configural-cue Compound Stimuli Temporal Difference model (SSCC TD) can model both simultaneous and serial stimulus compounds, as well as compounds including the experimental context. This modification significantly broadens the range of phenomena which the TD paradigm can explain, and allows it to predict phenomena which traditional TD solutions cannot, particularly effects that depend on compound stimuli functioning as a whole, such as pattern learning and serial structural discriminations, and context-related effects.

Behavioral flexibility is increased by optogenetic inhibition of neurons in the nucleus accumbens shell during specific time segments

Behavioral flexibility is vital for survival in an environment of changing contingencies. The nucleus accumbens may play an important role in behavioral flexibility, representing learned stimulus–reward associations in neural activity during response selection and learning from results. To investigate the role of nucleus accumbens neural activity in behavioral flexibility, we used light-activated halorhodopsin to inhibit nucleus accumbens shell neurons during specific time segments of a bar-pressing task requiring a win–stay/lose–shift strategy. We found that optogenetic inhibition during action selection in the time segment preceding a lever press had no effect on performance. However, inhibition occurring in the time segment during feedback of results—whether rewards or nonrewards—reduced the errors that occurred after a change in contingency. Our results demonstrate critical time segments during which nucleus accumbens shell neurons integrate feedback into subsequent responses. Inhibiting nucleus accumbens shell neurons in these time segments, during reinforced performance or after a change in contingencies, increases lose–shift behavior. We propose that the activity of nucleus shell accumbens shell neurons in these time segments plays a key role in integrating knowledge of results into subsequent behavior, as well as in modulating lose–shift behavior when contingencies change.

Correlates of reward-predictive value in learning-related hippocampal neural activity

Temporal difference learning (TD) is a popular algorithm in machine learning. Two learning signals that are derived from this algorithm, the predictive value and the prediction error, have been shown to explain changes in neural activity and behavior during learning across species. Here, the predictive value signal is used to explain the time course of learning-related changes in the activity of hippocampal neurons in monkeys performing an associative learning task. The TD algorithm serves as the centerpiece of a joint probability model for the learning-related neural activity and the behavioral responses recorded during the task. The neural component of the model consists of spiking neurons that compete and learn the reward-predictive value of task-relevant input signals. The predictive-value signaled by these neurons influences the behavioral response generated by a stochastic decision stage, which constitutes the behavioral component of the model. It is shown that the time course of the changes in neural activity and behavioral performance generated by the model exhibits key features of the experimental data. The results suggest that information about correct associations may be expressed in the hippocampus before it is detected in the behavior of a subject. In this way, the hippocampus may be among the earliest brain areas to express learning and drive the behavioral changes associated with learning. Correlates of reward-predictive value may be expressed in the hippocampus through rate remapping within spatial memory representations, they may represent reward-related aspects of a declarative or explicit relational memory representation of task contingencies, or they may correspond to reward-related components of episodic memory representations. These potential functions are discussed in connection with hippocampal cell assembly sequences and their reverse reactivation during the awake state. The results provide further support for the proposal that neural processes underlying learning may be implementing a temporal difference-like algorithm.

A head-neck-eye system that learns fault-tolerant saccades to 3-D targets using a self-organizing neural model

This paper describes a head-neck-eye camera system that is capable of learning to saccade to 3-D targets in a self-organized fashion. The self-organized learning process is based on action perception cycles where the camera system performs micro saccades about a given head-neck-eye camera position and learns to map these micro saccades to changes in position of a 3-D target currently in view of the stereo camera. This motor babbling phase provides self-generated movement commands that activate correlated visual, spatial and motor information that are used to learn an internal coordinate transformation between vision and motor systems. The learned transform is used by resulting head-neck-eye camera system to accurately saccade to 3-D targets using many different combinations of head, neck, and eye positions. The interesting aspect of the learned transform is that it is robust to a wide variety of disturbances including reduced degrees of freedom of movement for the head, neck, one eye, or any combination of two of the three, movement of head and neck as a function of eye movements, changes in the stereo camera separation distance and changes in focal lengths of the cameras. These disturbances were not encountered during motor babbling phase. This feature points to general nature of the learned transform in its ability to control autonomous systems with redundant degrees of freedom in a very robust and fault-tolerant fashion.

Determination of optimum operating conditions for industrial dye wastewater treatment using adaptive heuristic criticism pH control

For a pilot-scale application, pH control in the treatment of highly contaminated dye industrial wastewater containing metallic compounds as the main pollutants has been investigated with a method using adaptive heuristic criticism control (AHCC). Subsequent experimentation on between 12 and 18 l of the wastewater was carried out using statistical experimental design methodology to evaluate the effects of three critical factors: slaked lime (calcium hydroxide, Ca(OH)(2)) concentration, iron chloride (FeCl(3)) concentration and wastewater volume. With these critical factors, the wastewater treatment process is modeled as an appropriate quadratic cost function of the turbidity of the clarified water. The model is optimized with Rosenbrock's method. Response surface topology of the wastewater treatment is given in terms of optimal concentrations of lime water and FeCl(3) and optimal wastewater volume at pH 11.

Application of adaptive heuristic criticism control (AHCC) to dye wastewater

This paper presents an experimental application of AHCC to study the coagulation process of wastewater treatment in a dye plant. Also this study includes a series of tests in which an AHCC control was used for pH control. The performance results of the AHCC controller are compared with the results obtained by using a conventional proportional-integral-derivative (PID) algorithm. It is useful to compare PID with AHCC to illustrate the extreme range of the nonlinearity of the dye wastewater treatment process. Although the removal of pollutants from wastewater is similar with AHCC and PID, our results show excellent AHCC performance in the region where conventional PID control fails.

Neurophysiological theory of Kamin blocking in fear conditioning

Kamin blocking in fear conditioning is thought to reflect diminished processing of the unconditional stimulus (US) in the presence of a conditional stimulus (CS-super(+)) that was previously paired with this US. According to Fanselow's (1998) hypothesis, the CS-super(+) drives output from the amygdala that ultimately produces analgesia by causing opiate release onto afferent pain circuits. This hypothesis was explored quantitatively through neurophysiological simulations. The results suggest that opiate-mediated, negative-feedback control of US processing is too slow for efficient blocking of cue conditioning. The reason is that conditioning-produced synaptic modifications can be induced before the opiate-mediated inhibition has any substantial effect on US processing. The results suggest the existence of an additional, faster-acting, inhibitory neurotransmitter in the blocking circuit.

Dopamine cells respond to predicted events during classical conditioning: evidence for eligibility traces in the reward-learning network

Behavioral conditioning of cue-reward pairing results in a shift of midbrain dopamine (DA) cell activity from responding to the reward to responding to the predictive cue. However, the precise time course and mechanism underlying this shift remain unclear. Here, we report a combined single-unit recording and temporal difference (TD) modeling approach to this question. The data from recordings in conscious rats showed that DA cells retain responses to predicted reward after responses to conditioned cues have developed, at least early in training. This contrasts with previous TD models that predict a gradual stepwise shift in latency with responses to rewards lost before responses develop to the conditioned cue. By exploring the TD parameter space, we demonstrate that the persistent reward responses of DA cells during conditioning are only accurately replicated by a TD model with long-lasting eligibility traces (nonzero values for the parameter lambda) and low learning rate (alpha). These physiological constraints for TD parameters suggest that eligibility traces and low per-trial rates of plastic modification may be essential features of neural circuits for reward learning in the brain. Such properties enable rapid but stable initiation of learning when the number of stimulus-reward pairings is limited, conferring significant adaptive advantages in real-world environments.

Isotropic sequence order learning using a novel linear algorithm in a closed loop behavioural system

In this article, we present an isotropic algorithm for sequence order learning. Its central goal is to learn the causal relation between two (or more) inputs in order to react to the earliest incoming signal after successful learning (like in typical classical conditioning situations). We implement this algorithm in a behaving system (a robot) thereby creating a closed loop situation where the learner's actions influence its own sensor inputs to the end of creating an autonomous agent. Autonomous behaviour implies that learning goals are internally defined within the organism's capabilities. Standard learning models for sequence learning (e.g. temporal difference (TD)-learning) need an externally defined reward. This, however, is in conflict with the requirement of an implicitly defined internal goal in autonomous behaviour. Therefore, in this study we present a system in which the external reward is replaced by a reflex loop. This loop explicitly includes the environment. Every reflex loop has the inherent disadvantage, which is that its re-actions occur each time just after a reflex-eliciting sensor event and thus 'too late'. However, a reflex can serve as the internal reference for sequence order learning, which has the task of eliminating this disadvantage by creating earlier anticipatory actions. In our system learning is achieved by modifying synaptic weights of a linear neuron with a correlation based learning rule which involves the derivative of the neuron's output. All input lines are entirely isotropic. The synaptic weight change curve of this rule is strongly related to the temporal Hebb learning rule, which was found in spike timing experiments. We find that after learning the reflex loop is replaced in functional terms with an earlier anticipatory action (and pathway). In addition, we observed that the synaptic weights stabilise as soon as the reflex remains silent.

The cerebellum in action: a simulation and robotics study

The control or prediction of the precise timing of events are central aspects of the many tasks assigned to the cerebellum. Despite much detailed knowledge of its physiology and anatomy, it remains unclear how the cerebellar circuitry can achieve such an adaptive timing function. We present a computational model pursuing this question for one extensively studied type of cerebellar-mediated learning: the classical conditioning of discrete motor responses. This model combines multiple current assumptions on the function of the cerebellar circuitry and was used to investigate whether plasticity in the cerebellar cortex alone can mediate adaptive conditioned response timing. In particular, we studied the effect of changes in the strength of the synapses formed between parallel fibres and Purkinje cells under the control of a negative feedback loop formed between inferior olive, cerebellar cortex and cerebellar deep nuclei. The learning performance of the model was evaluated at the circuit level in simulated conditioning experiments as well as at the behavioural level using a mobile robot. We demonstrate that the model supports adaptively timed responses under real-world conditions. Thus, in contrast to many other models that have focused on cerebellar-mediated conditioning, we investigated whether and how the suggested underlying mechanisms could give rise to behavioural phenomena.

Extending models of hippocampal function in animal conditioning to human amnesia

Although most analyses of amnesia have focused on the loss of explicit declarative and episodic memories following hippocampal-region damage, considerable insights into amnesia can also be realised by studying hippocampal function in simple procedural, or habit-based, associative learning tasks. Although many simple forms of associative learning are unimpaired by hippocampal damage, more complex tasks which require sensitivity to unreinforced stimuli, configurations of multiple stimuli, or contextual information are impaired by hippocampal damage. In several recent papers we have developed a computational theory of hippocampal function which argues that this brain region plays a critical role in the formation of new stimulus representations during learning (Gluck & Myers, 1993, 1995; Myers & Gluck, 1996; Myers, Gluck, & Granger, 1995). We have applied this theory to a broad range of empirical data from studies of classical conditioning in both intact and hippocampal-lesioned animals, and the model correctly accounts for these data. The classical conditioning paradigm can be adapted for use in humans, and similar results for acquisition are obtained in both normal and hippocampal-damaged humans. More recently, we have begun to address an important set of category learning studies in both normals and hippocampal-damaged amnesics. This work integrates experimental studies of amnesic category learning (Knowlton, Squire, & Gluck, 1994) with theoretical accounts of associative learning, and builds on previously established behavioural correspondences between animal conditioning and human category learning (Gluck & Bower, 1988a). Our work to date illustrates some initial progress towards a more integrative understanding of hippocampal function in both animal and human learning, which may be useful in guiding further empirical and theoretical research in human memory and amnesia.

Conditioned stimulus duration in classical trace conditioning: test of a real-time neural network model

In classical trace conditioning, the interstimulus interval (ISI) is equal to the conditioned stimulus (CS) duration plus the trace interval (TI), the interval between CS offset and unconditioned stimulus (US) onset. The Sutton-Barto-Desmond neural-network model of classical conditioning predicts that, with a sufficiently long TI, conditioning will be faster with a CS of relatively long duration than with one of shorter duration. This prediction is illustrated with simulations and tested with the rabbit nictitating membrane response. Animals were trained with a tone CS of 350- or 700-ms duration. The TI was fixed at 300 ms, so that the ISI for the two durations was 650 or 1000 ms, respectively. Another factor in the experimental design was tone intensity (63 or 83 dB). Consistent with the model's prediction, conditioning was faster with the longer ISI, but only with the louder tone. The results have implications for computational models of classical conditioning.

Experiments in artificial psychology: conditioning of asynchronous neural network models

An asynchronous model for the dynamics of neural networks admits learning behaviors characteristic of classical and operant conditioning provided that appropriate plasticity algorithms are chosen. Stimulus generalization and discrimination can also be observed. Studies of such psychological phenomena are carried out by computer simulation of networks with designated sensory, association, and motor neurons, and the results are compared to those for live subjects. Various prescriptions for plasticity are investigated, including those corresponding to reward, punishment, and unlearning routines. These are characterized by their effect on network stability as quantified by a newly proposed stability measure.

Associative neural network model for the generation of temporal patterns. Theory and application to central pattern generators

Cyclic patterns of motor neuron activity are involved in the production of many rhythmic movements, such as walking, swimming, and scratching. These movements are controlled by neural circuits referred to as central pattern generators (CPGs). Some of these circuits function in the absence of both internal pacemakers and external feedback. We describe an associative neural network model whose dynamic behavior is similar to that of CPGs. The theory predicts the strength of all possible connections between pairs of neurons on the basis of the outputs of the CPG. It also allows the mean operating levels of the neurons to be deduced from the measured synaptic strengths between the pairs of neurons. We apply our theory to the CPG controlling escape swimming in the mollusk Tritonia diomedea. The basic rhythmic behavior is shown to be consistent with a simplified model that approximates neurons as threshold units and slow synaptic responses as elementary time delays. The model we describe may have relevance to other fixed action behaviors, as well as to the learning, recall, and recognition of temporally ordered information.

Neural dynamics of attentionally modulated Pavlovian conditioning: blocking, interstimulus interval, and secondary reinforcement

Selective information processing in neural networks is studied through computer simulations of Pavlovian conditioning data. The model reproduces properties of blocking, inverted-U in learning as a function of interstimulus interval, anticipatory conditioned responses, secondary reinforcement, attentional focusing by conditioned motivational feedback, and limited capacity short-term memory processing. Conditioning occurs from sensory to drive representations (conditioned reinforcer learning), from drive to sensory representations (incentive motivational learning), and from sensory to motor representations (habit learning).The conditionable pathwas contain long-term memory traces that obey a non-Hebbian associative law. The neural model embodies a solution to two key design problems of conditioning, the synchronization and persistence problems. This model of vertebrate learning is compared with data and models of invertebrate learning. Predictions derived from models of vertebrate learning are compared with data about invertebrate learning, including data from Aplysia about facilitator neurons and data from Hermissenda about voltage-dependent Ca(2+) currents. A prediction is stated about classical conditioning in all species, called the secondary conditioning alternative, and if confirmed would constitute an evolutionary invariant of learning.

Simulation of the classically conditioned nictitating membrane response by a neuron-like adaptive element: response topography, neuronal firing, and interstimulus intervals

A neuron-like adaptive element with computational features suitable for classical conditioning, the Sutton-Barto (S-B) model, was extended to simulate real-time aspects of the conditioned nictitating membrane (NM) response. The aspects of concern were response topography, CR-related neuronal firing, and interstimulus interval (ISI) effects for forward-delay and trace conditioning paradigms. The topography of the NM CR has the following features: response latency after CS onset decreases over trials; response amplitude increases gradually within the ISI and attains its maximum coincidentally with the UR. A similar pattern characterizes the firing of some (but not all) neurons in brain regions demonstrated experimentally to be important for NM conditioning. The variant of the S-B model described in this paper consists of a set of parameters and implementation rules based on 10-ms computational time steps. It differs from the original S-B model in a number of ways. The main difference is the assumption that CS inputs to the adaptive element are not instantaneous but are instead shaped by unspecified coding processes so as to produce outputs that conform with the real-time properties of NM conditioning. The model successfully simulates the aforementioned features of NM response topography. It is also capable of simulating appropriate ISI functions, i.e. with maximum conditioning strength with ISIs of 250 ms, for forward-delay and trace paradigms. The original model's successful treatment of multiple-CS phenomena, such as blocking, conditioned inhibition, and higher-order conditioning, are retained by the present model.