Analysis and modeling of neural processes underlying sensory preconditioning

Critical roles of nicotinic acetylcholine receptors in olfactory memory formation and retrieval in crickets

Acetylcholine (ACh) is a major excitatory neurotransmitter in the insect central nervous system, and insect neurons express several types of ACh receptors (AChRs). AChRs are classified into two subgroups, muscarinic AChRs and nicotinic AChRs (nAChRs). nAChRs are also divided into two subgroups by sensitivity to α-bungarotoxin (α-BGT). The cricket Gryllus bimaculatus is one of the useful insects for studying the molecular mechanisms in olfactory learning and memory. However, the roles of nAChRs in olfactory learning and memory of the cricket are still unknown. In the present study, to investigate whether nAChRs are involved in cricket olfactory learning and memory, we tested the effects of two different AChR antagonists on long-term memory (LTM) formation and retrieval in a behavioral assay. The two AChR antagonists that we used are mecamylamine (MEC), an α-BGT-insensitive nAChR antagonist, and methyllycaconitine (MLA), an α-BGT-sensitive nAChR antagonist. In crickets, multiple-trial olfactory conditioning induced 1-day memory (LTM), whereas single-trial olfactory conditioning induced 1-h memory (mid-term memory, MTM) but not 1-day memory. Crickets injected with MEC 20 min before the retention test at 1 day after the multiple-trial conditioning exhibited no memory retrieval. This indicates that α-BGT-insensitive nAChRs participate in memory retrieval. In addition, crickets injected with MLA before the multiple-trial conditioning exhibited MTM but not LTM, indicating that α-BGT-sensitive nAChRs participate in the formation of LTM. Moreover, injection of nicotine (an nAChR agonist) before the single-trial conditioning induced LTM. Finally, the nitric oxide (NO)-cGMP signaling pathway is known to participate in the formation of LTM in crickets, and we conducted co-injection experiments with an agonist or inhibitor of the nAChR and an activator or inhibitor of the NO-cGMP signaling pathway. The results suggest that nAChR works upstream of the NO-cGMP signaling system in the LTM formation process.

Higher-Order Conditioning in the Spatial Domain

Spatial learning and memory, the processes through which a wide range of living organisms encode, compute, and retrieve information from their environment to perform goal-directed navigation, has been systematically investigated since the early twentieth century to unravel behavioral and neural mechanisms of learning and memory. Early theories about learning to navigate space considered that animals learn through trial and error and develop responses to stimuli that guide them to a goal place. According to a trial-and error learning view, organisms can learn a sequence of motor actions that lead to a goal place, a strategy referred to as response learning, which contrasts with place learning where animals learn locations with respect to an allocentric framework. Place learning has been proposed to produce a mental representation of the environment and the cartesian relations between stimuli within it-which Tolman coined the cognitive map. We propose to revisit some of the best empirical evidence of spatial inference in animals, and then discuss recent attempts to account for spatial inferences within an associative framework as opposed to the traditional cognitive map framework. We will first show how higher-order conditioning can successfully account for inferential goal-directed navigation in a variety of situations and then how vectors derived from path integration can be integrated via higher-order conditioning, resulting in the generation of higher-order vectors that explain novel route taking. Finally, implications to cognitive map theories will be discussed.

What Is Learned in Pavlovian Conditioning in Crickets? Revisiting the S-S and S-R Learning Theories

In Pavlovian conditioning in mammals, two theories have been proposed for associations underlying conditioned responses (CRs). One theory, called S-S theory, assumes an association between a conditioned stimulus (CS) and internal representation of an unconditioned stimulus (US), allowing the animal to adjust the CR depending on the current value of the US. The other theory, called S-R theory, assumes an association or connection between the CS center and the CR center, allowing the CS to elicit the CR. Whether these theories account for Pavlovian conditioning in invertebrates has remained unclear. In this article, results of our studies in the cricket Gryllus bimaculatus are reviewed. We showed that after a standard amount of Pavlovian training, crickets exhibited no response to odor CS when water US was devalued by providing it until satiation, whereas after extended training, they exhibited a CR after US devaluation. An increase of behavioral automaticity by extended training has not been reported in Pavlovian conditioning in any other animals, but it has been documented in instrumental conditioning in mammals. Our pharmacological analysis suggested that octopamine neurons mediate US (water) value signals and control execution of the CR after standard training. The control, however, diminishes with extension of training and hence the CR becomes insensitive to the US value. We also found that the nature of the habitual response after extended Pavlovian training in crickets is not the same as that after extended instrumental training in mammals concerning the context specificity. Adaptive significance and evolutionary implications for our findings are discussed.

Cortico-Hippocampal Computational Modeling Using Quantum Neural Networks to Simulate Classical Conditioning Paradigms

Most existing cortico-hippocampal computational models use different artificial neural network topologies. These conventional approaches, which simulate various biological paradigms, can get slow training and inadequate conditioned responses for two reasons: increases in the number of conditioned stimuli and in the complexity of the simulated biological paradigms in different phases. In this paper, a cortico-hippocampal computational quantum (CHCQ) model is proposed for modeling intact and lesioned systems. The CHCQ model is the first computational model that uses the quantum neural networks for simulating the biological paradigms. The model consists of two entangled quantum neural networks: an adaptive single-layer feedforward quantum neural network and an autoencoder quantum neural network. The CHCQ model adaptively updates all the weights of its quantum neural networks using quantum instar, outstar, and Widrow–Hoff learning algorithms. Our model successfully simulated several biological processes and maintained the output-conditioned responses quickly and efficiently. Moreover, the results were consistent with prior biological studies.

Green model to adapt classical conditioning learning in the hippocampus

Compared with the biological paradigms of classical conditioning, non-adaptive computational models are not capable of realistically simulating the biological behavioural functions of the hippocampal regions, because of their implausible requirement for a large number of learning trials, which can be on the order of hundreds. Additionally, these models did not attain a unified, final stable state even after hundreds of learning trials. Conversely, the output response has a different threshold for similar tasks in various models with prolonged transient response of unspecified status via the training or even testing phases. Accordingly, a green model is a combination of adaptive neuro-computational hippocampal and cortical models that is proposed by adaptively updating the whole weights in all layers for both intact networks and lesion networks using instar and outstar learning rules with adaptive resonance theory (ART). The green model sustains and expands the classical conditioning biological paradigms of the non-adaptive models. The model also overcomes the irregular output response behaviour by using the proposed feature of adaptivity. Further, the model successfully simulates the hippocampal regions without passing the final output response back to the whole network, which is considered to be biologically implausible. The results of the Green model showed a significant improvement confirmed by empirical studies of different tasks. In addition, the results indicated that the model outperforms the previously published models. All the obtained results successfully and quickly attained a stable, desired final state (with a unified concluding state of either "1" or "0") with a significantly shorter transient duration.

Application of a Prediction Error Theory to Pavlovian Conditioning in an Insect

Elucidation of the conditions in which associative learning occurs is a critical issue in neuroscience and comparative psychology. In Pavlovian conditioning in mammals, it is thought that the discrepancy, or error, between the actual reward and the predicted reward determines whether learning occurs. This theory stems from the finding of Kamin’s blocking effect, in which after pairing of a stimulus with an unconditioned stimulus (US), conditioning of a second stimulus is blocked when the two stimuli are presented in compound and paired with the same US. Whether this theory is applicable to any species of invertebrates, however, has remained unknown. We first showed blocking and one-trial blocking of Pavlovian conditioning in the cricket Gryllus bimaculatus, which supported the Rescorla–Wagner model but not attentional theories, the major competitive error-correction learning theories to account for blocking. To match the prediction error theory, a neural circuit model was proposed, and prediction from the model was tested: the results were consistent with the Rescorla–Wagner model but not with the retrieval theory, another competitive theory to account for blocking. The findings suggest that the Rescorla–Wagner model best accounts for Pavlovian conditioning in crickets and that the basic computation rule underlying Pavlovian conditioning in crickets is the same to those suggested in mammals. Moreover, results of pharmacological studies in crickets suggested that octopamine and dopamine mediate prediction error signals in appetitive and aversive conditioning, respectively. This was in contrast to the notion that dopamine mediates appetitive prediction error signals in mammals. The functional significance and evolutionary implications of these findings are discussed.

Signaling Pathways for Long-Term Memory Formation in the Cricket

Unraveling the molecular mechanisms underlying memory formation in insects and a comparison with those of mammals will contribute to a further understanding of the evolution of higher-brain functions. As it is for mammals, insect memory can be divided into at least two distinct phases: protein-independent short-term memory and protein-dependent long-term memory (LTM). We have been investigating the signaling pathway of LTM formation by behavioral-pharmacological experiments using the cricket Gryllus bimaculatus, whose olfactory learning and memory abilities are among the highest in insect species. Our studies revealed that the NO-cGMP signaling pathway, CaMKII and PKA play crucial roles in LTM formation in crickets. These LTM formation signaling pathways in crickets share a number of attributes with those of mammals, and thus we conclude that insects, with relatively simple brain structures and neural circuitry, will also be beneficial in exploratory experiments to predict the molecular mechanisms underlying memory formation in mammals.

Roles of Octopamine and Dopamine Neurons for Mediating Appetitive and Aversive Signals in Pavlovian Conditioning in Crickets

Revealing neural systems that mediate appetite and aversive signals in associative learning is critical for understanding the brain mechanisms controlling adaptive behavior in animals. In mammals, it has been shown that some classes of dopamine neurons in the midbrain mediate prediction error signals that govern the learning process, whereas other classes of dopamine neurons control execution of learned actions. In this review, based on the results of our studies on Pavlovian conditioning in the cricket Gryllus bimaculatus and by referring to the findings in honey bees and fruit-flies, we argue that comparable aminergic systems exist in the insect brain. We found that administrations of octopamine (the invertebrate counterpart of noradrenaline) and dopamine receptor antagonists impair conditioning to associate an olfactory or visual conditioned stimulus (CS) with water or sodium chloride solution (appetitive or aversive unconditioned stimulus, US), respectively, suggesting that specific octopamine and dopamine neurons mediate appetitive and aversive signals, respectively, in conditioning in crickets. These findings differ from findings in fruit-flies. In fruit-flies, appetitive and aversive signals are mediated by different dopamine neuron subsets, suggesting diversity in neurotransmitters mediating appetitive signals in insects. We also found evidences of “blocking” and “auto-blocking” phenomena, which suggested that the prediction error, the discrepancy between actual US and predicted US, governs the conditioning in crickets and that octopamine neurons mediate prediction error signals for appetitive US. Our studies also showed that activations of octopamine and dopamine neurons are needed for the execution of an appetitive conditioned response (CR) and an aversive CR, respectively, and we, thus, proposed that these neurons mediate US prediction signals that drive appetitive and aversive CRs. Our findings suggest that the basic principles of functioning of aminergic systems in associative learning, i.e., to transmit prediction error signals for conditioning and to convey US prediction signals for execution of CR, are conserved among insects and mammals, on account of the fact that the organization of the insect brain is much simpler than that of the mammalian brain. Further investigation of aminergic systems that govern associative learning in insects should lead to a better understanding of commonalities and diversities of computational rules underlying associative learning in animals.

Roles of dopamine neurons in mediating the prediction error in aversive learning in insects

In associative learning in mammals, it is widely accepted that the discrepancy, or error, between actual and predicted reward determines whether learning occurs. The prediction error theory has been proposed to account for the finding of a blocking phenomenon, in which pairing of a stimulus X with an unconditioned stimulus (US) could block subsequent association of a second stimulus Y to the US when the two stimuli were paired in compound with the same US. Evidence for this theory, however, has been imperfect since blocking can also be accounted for by competitive theories. We recently reported blocking in classical conditioning of an odor with water reward in crickets. We also reported an "auto-blocking" phenomenon in appetitive learning, which supported the prediction error theory and rejected alternative theories. The presence of auto-blocking also suggested that octopamine neurons mediate reward prediction error signals. Here we show that blocking and auto-blocking occur in aversive learning to associate an odor with salt water (US) in crickets, and our results suggest that dopamine neurons mediate aversive prediction error signals. We conclude that the prediction error theory is applicable to both appetitive learning and aversive learning in insects.

Over the river, through the woods: cognitive maps in the hippocampus and orbitofrontal cortex

The hippocampus and the orbitofrontal cortex (OFC) both have important roles in cognitive processes such as learning, memory and decision making. Nevertheless, research on the OFC and hippocampus has proceeded largely independently, and little consideration has been given to the importance of interactions between these structures. Here, evidence is reviewed that the hippocampus and OFC encode parallel, but interactive, cognitive 'maps' that capture complex relationships between cues, actions, outcomes and other features of the environment. A better understanding of the interactions between the OFC and hippocampus is important for understanding the neural bases of flexible, goal-directed decision making.

Roles of OA1 octopamine receptor and Dop1 dopamine receptor in mediating appetitive and aversive reinforcement revealed by RNAi studies

Revealing reinforcing mechanisms in associative learning is important for elucidation of brain mechanisms of behavior. In mammals, dopamine neurons are thought to mediate both appetitive and aversive reinforcement signals. Studies using transgenic fruit-flies suggested that dopamine neurons mediate both appetitive and aversive reinforcements, through the Dop1 dopamine receptor, but our studies using octopamine and dopamine receptor antagonists and using Dop1 knockout crickets suggested that octopamine neurons mediate appetitive reinforcement and dopamine neurons mediate aversive reinforcement in associative learning in crickets. To fully resolve this issue, we examined the effects of silencing of expression of genes that code the OA1 octopamine receptor and Dop1 and Dop2 dopamine receptors by RNAi in crickets. OA1-silenced crickets exhibited impairment in appetitive learning with water but not in aversive learning with sodium chloride solution, while Dop1-silenced crickets exhibited impairment in aversive learning but not in appetitive learning. Dop2-silenced crickets showed normal scores in both appetitive learning and aversive learning. The results indicate that octopamine neurons mediate appetitive reinforcement via OA1 and that dopamine neurons mediate aversive reinforcement via Dop1 in crickets, providing decisive evidence that neurotransmitters and receptors that mediate appetitive reinforcement indeed differ among different species of insects.

Roles of octopamine and dopamine in appetitive and aversive memory acquisition studied in olfactory conditioning of maxillary palpi extension response in crickets

Elucidation of reinforcing mechanisms for associative learning is an important subject in neuroscience. Based on results of our previous pharmacological studies in crickets, we suggested that octopamine and dopamine mediate reward and punishment signals, respectively, in associative learning. In fruit-flies, however, it was concluded that dopamine mediates both appetitive and aversive reinforcement, which differs from our suggestion in crickets. In our previous studies, the effect of conditioning was tested at 30 min after training or later, due to limitations of our experimental procedures, and thus the possibility that octopamine and dopamine were not needed for initial acquisition of learning was not ruled out. In this study we first established a conditioning procedure to enable us to evaluate acquisition performance in crickets. Crickets extended their maxillary palpi and vigorously swung them when they perceived some odors, and we found that crickets that received pairing of an odor with water reward or sodium chloride punishment exhibited an increase or decrease in percentages of maxillary palpi extension responses to the odor. Using this procedure, we found that octopamine and dopamine receptor antagonists impair acquisition of appetitive and aversive learning, respectively. This finding suggests that neurotransmitters mediating appetitive reinforcement differ in crickets and fruit-flies.

Critical evidence for the prediction error theory in associative learning

In associative learning in mammals, it is widely accepted that the discrepancy, or error, between actual and predicted reward determines whether learning occurs. Complete evidence for the prediction error theory, however, has not been obtained in any learning systems: Prediction error theory stems from the finding of a blocking phenomenon, but blocking can also be accounted for by other theories, such as the attentional theory. We demonstrated blocking in classical conditioning in crickets and obtained evidence to reject the attentional theory. To obtain further evidence supporting the prediction error theory and rejecting alternative theories, we constructed a neural model to match the prediction error theory, by modifying our previous model of learning in crickets, and we tested a prediction from the model: the model predicts that pharmacological intervention of octopaminergic transmission during appetitive conditioning impairs learning but not formation of reward prediction itself, and it thus predicts no learning in subsequent training. We observed such an "auto-blocking", which could be accounted for by the prediction error theory but not by other competitive theories to account for blocking. This study unambiguously demonstrates validity of the prediction error theory in associative learning.

Toward elucidating diversity of neural mechanisms underlying insect learning

Insects are widely used as models to study neural mechanisms of learning and memory. Our recent studies on crickets, together with reports on other insect species, suggest that some fundamental differences exist in neural and molecular mechanisms of learning and memory among different species of insects, particularly between crickets and fruit flies. First, we suggested that in crickets octopamine (OA) and dopamine (DA) neurons convey reward and punishment signals, respectively, in associated learning. On the other hand, it has been reported that in fruit flies different sets of DA neurons convey reward or punishment signals. Secondly, we have suggested that in crickets OA and DA neurons participate in the retrieval of appetitive and aversive memories, respectively, while this is not the case in fruit flies. Thirdly, cyclic AMP signaling is critical for short-term memory formation in fruit flies, but not in crickets. Finally, nitric oxide-cyclic GMP signaling and calcium-calmodulin signaling are critical for long-term memory (LTM) formation in crickets, but such roles have not been reported in fruit flies. Not all of these differences can be ascribed to different experimental methods used in studies. We thus suggest that there are unexpected diversities in basic mechanisms of learning and memory among different insect species, especially between crickets and fruit flies. Studies on a larger number of insect species will help clarify the diversity of learning and memory mechanisms in relation to functional adaptation to the environment and evolutionary history.

Parallel pathways for cross-modal memory retrieval in Drosophila

Memory-retrieval processing of cross-modal sensory preconditioning is vital for understanding the plasticity underlying the interactions between modalities. As part of the sensory preconditioning paradigm, it has been hypothesized that the conditioned response to an unreinforced cue depends on the memory of the reinforced cue via a sensory link between the two cues. To test this hypothesis, we studied cross-modal memory-retrieval processing in a genetically tractable model organism, Drosophila melanogaster. By expressing the dominant temperature-sensitive shibire(ts1) (shi(ts1)) transgene, which blocks synaptic vesicle recycling of specific neural subsets with the Gal4/UAS system at the restrictive temperature, we specifically blocked visual and olfactory memory retrieval, either alone or in combination; memory acquisition remained intact for these modalities. Blocking the memory retrieval of the reinforced olfactory cues did not impair the conditioned response to the unreinforced visual cues or vice versa, in contrast to the canonical memory-retrieval processing of sensory preconditioning. In addition, these conditioned responses can be abolished by blocking the memory retrieval of the two modalities simultaneously. In sum, our results indicated that a conditioned response to an unreinforced cue in cross-modal sensory preconditioning can be recalled through parallel pathways.